• Hadoop集群访问客户端
  • X window
  • Jet brain Idea和客户端开发环境

以上环境我们将在WSL的Ubuntu docker容器中实现，其中对X和Idea的访问，通过Windows Xserver实现。

1 拉取Ubuntu镜像，运行ubuntu容器。

wslu@LAPTOP-ERJ3P24M:~$ sudo docker pull ubuntu:latest
wslu@LAPTOP-ERJ3P24M:~$ sudo docker run -it --name hadoop-client -p 22222:22 -v /home/wslu/data:/root/data ubuntu:latest/bin/bash

修改ubuntu软件源为阿里云，将archive.ubuntu.com和security.ubuntu.com替换为mirrors.aliyun.com。

root@19bafff6decc:/# cd /etc/apt
root@19bafff6decc:/etc/apt# ls
apt.conf.d  auth.conf.d  preferences.d  sources.list  sources.list.d  trusted.gpg.d
root@19bafff6decc:/etc/apt# cp sources.list sources.list__
root@19bafff6decc:/etc/apt# vi sources.list
root@19bafff6decc:~# apt updateGet:1 http://mirrors.aliyun.com/ubuntu focal InRelease [265 kB]
Get:2 http://mirrors.aliyun.com/ubuntu focal-updates InRelease [114 kB]
Get:3 http://mirrors.aliyun.com/ubuntu focal-backports InRelease [108 kB]
Get:4 http://mirrors.aliyun.com/ubuntu focal-security InRelease [114 kB]
Get:5 http://mirrors.aliyun.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:6 http://mirrors.aliyun.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:7 http://mirrors.aliyun.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:8 http://mirrors.aliyun.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:9 http://mirrors.aliyun.com/ubuntu focal-updates/main amd64 Packages [2146 kB]
Get:10 http://mirrors.aliyun.com/ubuntu focal-updates/multiverse amd64 Packages [30.3 kB]
Get:11 http://mirrors.aliyun.com/ubuntu focal-updates/universe amd64 Packages [1153 kB]
Get:12 http://mirrors.aliyun.com/ubuntu focal-updates/restricted amd64 Packages [1178 kB]
Get:13 http://mirrors.aliyun.com/ubuntu focal-backports/main amd64 Packages [51.2 kB]
Get:14 http://mirrors.aliyun.com/ubuntu focal-backports/universe amd64 Packages [26.0 kB]
Get:15 http://mirrors.aliyun.com/ubuntu focal-security/restricted amd64 Packages [1104 kB]
Get:16 http://mirrors.aliyun.com/ubuntu focal-security/universe amd64 Packages [867 kB]
Get:17 http://mirrors.aliyun.com/ubuntu focal-security/multiverse amd64 Packages [25.8 kB]
Get:18 http://mirrors.aliyun.com/ubuntu focal-security/main amd64 Packages [1729 kB]
Fetched 21.7 MB in 2min 31s (144 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.

2 安装openssh。

root@19bafff6decc:/# apt install openssh-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:ca-certificates dbus distro-info-data dmsetup file gir1.2-glib-2.0 krb5-locales libapparmor1 libargon2-1 libbsd0
...
Created symlink /etc/systemd/system/multi-user.target.wants/networkd-dispatcher.service → /lib/systemd/system/networkd-dispatcher.service.
Setting up ssh-import-id (5.10-0ubuntu1) ...
Attempting to convert /etc/ssh/ssh_import_id
Processing triggers for systemd (245.4-4ubuntu3.15) ...
Processing triggers for libc-bin (2.31-0ubuntu9.7) ...
Processing triggers for ca-certificates (20210119~20.04.2) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.Your identification has been saved in /etc/ssh/ssh_host_dsa_key
Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub
The key fingerprint is:
SHA256:/1V/EvWFsBZpsZqsptvqZqlUUknDlQOi/c4ZGtwyXnA root@19bafff6decc
The key's randomart image is:
+---[DSA 1024]----+
|    ..+o..  +o   |
|   o o.oo   o= . |
|  . o E  . .+ . o|
|   . *   . +   .o|
|    * * S +   . o|
|   . @ o o     o.|
|    + +.o .   o o|
|   .  ++   . . ..|
|    .==o.   .    |
+----[SHA256]-----+
root@19bafff6decc:/# ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
Generating public/private rsa key pair.
/etc/ssh/ssh_host_rsa_key already exists.
Overwrite (y/n)? nroot@19bafff6decc:/# passwd
New password:
Retype new password:
passwd: password updated successfully

为了进行一些基本的编辑工作，我们安装vim。编辑/etc/ssh/sshd_config，将PermitRootLogin 改为 yes，将 UsePAM  改为 no。

root@19bafff6decc:/etc/ssh# apt install vim
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:alsa-topology-conf alsa-ucm-conf libasound2 libasound2-data libcanberra0 libgpm2 libltdl7 libogg0 libpython3.8libtdb1 libvorbis0a libvorbisfile3 sound-theme-freedesktop vim-common vim-runtime xxd
update-alternatives: warning: skip creation of /usr/share/man/ja/man1/ex.1.gz because associated file /usr/share/man/ja/man1/vim.1.gz (of link group ex) doesn't exist
...
/ru/man1/vim.1.gz (of link group editor) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/editor.1.gz because associated file /usr/share/man/man1/vim.1.gz (of link group editor) doesn't exist
Processing triggers for libc-bin (2.31-0ubuntu9.7) ...
Processing triggers for mime-support (3.64ubuntu1) ...root@19bafff6decc:~# cd /etc/ssh
root@19bafff6decc:/etc/ssh# ls
moduli        ssh_host_dsa_key      ssh_host_ecdsa_key.pub    ssh_host_rsa_key      sshd_config
ssh_config    ssh_host_dsa_key.pub  ssh_host_ed25519_key      ssh_host_rsa_key.pub  sshd_config.d
ssh_config.d  ssh_host_ecdsa_key    ssh_host_ed25519_key.pub  ssh_import_id
root@19bafff6decc:/etc/ssh# vi sshd_config

启动ssh。

root@19bafff6decc:~# /etc/init.d/ssh start* Starting OpenBSD Secure Shell server sshd                                                                     [ OK ]
root@19bafff6decc:~#

在WSL通过端口22222进行测试，可以连接上。

wslu@LAPTOP-ERJ3P24M:~$ ssh root@localhost -p 22222
The authenticity of host '[localhost]:22222 ([127.0.0.1]:22222)' can't be established.
ECDSA key fingerprint is SHA256:jjViz0wRMzeihyAwVsXCAL//cvcV5KJaBVHee3bmWHY.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[localhost]:22222' (ECDSA) to the list of known hosts.
root@localhost's password:
root@19bafff6decc:~#

3 安装JDK

root@19bafff6decc:~# apt install openjdk-8-jdk
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:adwaita-icon-theme at-spi2-core ca-certificates-java fontconfig fontconfig-config fonts-dejavu-core
...
Running hooks in /etc/ca-certificates/update.d...done.
done.
Processing triggers for mime-support (3.64ubuntu1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.7) ...
Processing triggers for libgdk-pixbuf2.0-0:amd64 (2.40.0+dfsg-3ubuntu0.2) ...root@19bafff6decc:~# java -version
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-8u312-b07-0ubuntu1~20.04-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)
root@19bafff6decc:~#

4 下载安装MobaXterm。

从MobaXterm free Xserver and tabbed SSH client for Windows (mobatek.net)下载安装windows下SSH和Xterm客户端。

设置SSH X11 forward，参见(203条消息) docker 容器如何开启 ssh 远程连接和 X11_帐下幕僚的博客-CSDN博客_docker x11。

root@19bafff6decc:~# apt install xorg
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
...
aspell-autobuildhash: processing: en [en_US-wo_accents-only].
Processing triggers for libc-bin (2.31-0ubuntu9.7) ...
Processing triggers for dbus (1.12.16-2ubuntu2.1) ...
Processing triggers for systemd (245.4-4ubuntu3.15) ...
Processing triggers for rygel (0.38.3-1ubuntu1) ...
Processing triggers for sgml-base (1.29.1) ...
root@19bafff6decc:~# root@19bafff6decc:~# touch .Xauthority
root@19bafff6decc:~# xauth add localhost/unix:10 MIT-MAGIC-COOKIE-1 $(mcookie)

修改 /etc/ssh/sshd_config。

# AddressFamily any
AddressFamily inetAllowTcpForwarding yes
X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost yes

重启 docker容器hadoop-client。打开MobaXterm，建新Session，建立一个新的SSH连接。

，开启SSH，输入root账号，进入。注意X1-forwaring显现绿色的对勾。

5 安装Jet brain Idea。

解压ideaIC-2021.3.3.tar.gz，移动到/root下，在MobaXterm hadoop-client容器SSH窗口下运行/root/idea-IC-213.7172.25/bin/idea.sh。

root@19bafff6decc:/# cd
root@19bafff6decc:~# cd dataroot@19bafff6decc:~/data# tar xzvf ideaIC-2021.3.3.tar.gz
idea-IC-213.7172.25/LICENSE.txt
idea-IC-213.7172.25/NOTICE.txt
idea-IC-213.7172.25/bin/appletviewer.policy
idea-IC-213.7172.25/bin/idea.svg
idea-IC-213.7172.25/bin/log.xml
idea-IC-213.7172.25/lib/3rd-party-rt.jar
idea-IC-213.7172.25/lib/3rd-party.jar
idea-IC-213.7172.25/lib/annotations.jar
...
idea-IC-213.7172.25/jbr/bin/serialver
idea-IC-213.7172.25/jbr/bin/unpack200
idea-IC-213.7172.25/jbr/lib/chrome-sandbox
idea-IC-213.7172.25/jbr/lib/jcef_helper
idea-IC-213.7172.25/jbr/lib/jexec
idea-IC-213.7172.25/jbr/lib/jspawnhelperroot@19bafff6decc:~/data# mv idea-IC-213.7172.25 /root/idea-IC-213.7172.25

6 拷贝Hadoop集群环境

将Windows PC上创建大数据职业技能竞赛实验环境之三建立的Hadoop集群环境通过scp拷贝到hadoop-client容器。确保hadoop-spark容器还在运行中，否则通过docker desktop启动，或新启一个容器。

WSL docker inspect查看hadoop-spark容器ip地址。

            "IPv6Gateway": "","MacAddress": "02:42:ac:11:00:03","Networks": {"bridge": {"IPAMConfig": null,"Links": null,"Aliases": null,"NetworkID": "3888d6f9be2b19d623c388d493ab3b878cbfae09a499a44764c0489140300ecd","EndpointID": "5e8caf8b42d26094f1c830113416586b20bed560aab519bb72310c495700c014","Gateway": "172.17.0.1","IPAddress": "172.17.0.3",

确保hadoop-spark容器SSH服务启动。

sh-4.2# service sshd status
Redirecting to /bin/systemctl status sshd.service
sshd.service - OpenSSH server daemonLoaded: loaded (/usr/lib/systemd/system/sshd.service, enabled)Active: inactive (dead)
sh-4.2# service sshd start
Redirecting to /bin/systemctl start sshd.service
sh-4.2# service sshd status
Redirecting to /bin/systemctl status sshd.service
sshd.service - OpenSSH server daemonLoaded: loaded (/usr/lib/systemd/system/sshd.service, enabled)Active: active (running)sh-4.2# passwd
Changing password for user root.
New password:
BAD PASSWORD: The password fails the dictionary check - it is too simplistic/systematic
Retype new password:
passwd: all authentication tokens updated successfully.

进入hadoop-client容器，scp拷贝hadoop目录。

scp -r root@172.17.0.3:/root/hadoop /root/hadoop

修改环境变量~/.bashrc。

JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$JAVA_HOME/bin
HADOOP_HOME=/root/hadoop/hadoop-2.7.7
PATH=$PATH:$HADOOP_HOME/bin
SPARK_HOME=/root/hadoop/spark-2.1.1-bin-hadoop2.7
PATH=$PATH:$SPARK_HOME/bin
HIVE_HOME=/root/hadoop/apache-hive-2.3.4-bin
PATH=$PATH:$HIVE_HOME/bin
FLUME_HOME=/root/hadoop/apache-flume-1.7.0-bin
FLUME_CONF_DIR=$FLUME_HOME/conf
PATH=$PATH:$FLUME_HOME/bin
ZOOKEEPER_HOME=/root/hadoop/apache-zookeeper-2.6.3-bin
PATH=$PATH:$ZOOKEEPER_HOME/bin
KAFKA_HOME=/root/hadoop/kafka_2.11-2.0.0
PATH=$PATH:$KAFKA_HOME/bin
FLINK_HOME=/root/hadoop/flink-1.10.2
PATH=$PATH:$FLINK_HOME/binexport JAVA_HOME JRE_HOME CLASSPATH PATH
export HADOOP_HOME SPARK_HOME HIVE_HOME ZOOKEEPER_HOME
export FLUME_HOME FLUME_CONF_DIR
export KAFKA_HOME FLINK_HOME

修改hadoop下有关JAVA_HOME的设置。

~/hadoop/hadoop-2.7.7/etc/hadoop/hadoop-env.sh和~/hadoop/apache-flume-1.7.0-bin/conf/flume-env.sh。

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

这样hadoop集群的客户端就搭建好了。

7 安装ping

Ubuntu Docker镜像缺少ping，安装。

root@client1:/# apt install iputils-ping
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:iputils-ping
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 40.1 kB of archives.
After this operation, 108 kB of additional disk space will be used.
Get:1 http://mirrors.aliyun.com/ubuntu focal/main amd64 iputils-ping amd64 3:20190709-3 [40.1 kB]
Fetched 40.1 kB in 0s (118 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package iputils-ping.
(Reading database ... 49064 files and directories currently installed.)
Preparing to unpack .../iputils-ping_3%3a20190709-3_amd64.deb ...
Unpacking iputils-ping (3:20190709-3) ...
Setting up iputils-ping (3:20190709-3) ...
Processing triggers for man-db (2.9.1-1) ...
wslu@LAPTOP-ERJ3P24M:~$ sudo docker commit -m "hadoop client" hadoop-client centos/hadoop-client:v1

Windows PC上创建大数据职业技能竞赛实验环境之四--客户端环境的搭建相关推荐

  1. Windows PC上创建大数据职业技能竞赛实验环境之一--基本环境的搭建

    1 大数据职业竞赛环境的基础要求 软件环境 设备类型 软件类别 软件名称.版本号 竞赛服务器 竞赛环境大数据集群操作系统 CentOS 7 大数据平台组件 Hadoop 2.7.7 Hive 2.3. ...

  2. Windows PC上创建大数据职业技能竞赛实验环境之五--hadoop、hive和spark编程

    前述,我们在Windows WSL上,通过Docker容器技术实现了Hadoop集群环境,现在看看利用现有集群环境进行编程开发. 1 设置容器开启时自运行SSH服务 参照docker中安装Ubuntu ...

  3. Windows PC上创建大数据职业技能竞赛实验环境之七--疑难问题

    1 Ubuntu Docker中文显示问题 查看locale设置,我们可以看到Ubuntu缺省local是POSIX,需要进行修改. root@client1:~# locale LANG= LANG ...

  4. Windows PC上创建大数据职业技能竞赛实验环境之六--Flume、Kafka和Flink编程

    1 Flume 参看日志采集工具Flume的安装与使用方法_厦大数据库实验室博客 (xmu.edu.cn). 查看Flume安装 root@client1:~# flume-ng version Fl ...

  5. 在Kaggle上赢得大数据竞赛的技巧和窍门

    在Kaggle上赢得大数据竞赛的技巧和窍门 解决方案 平台 数据 应用 方法 阅读1906  原文:The tips and tricks I used to succeed on Kaggle  作 ...

  6. 就业培训 | 2020第一期重庆高校毕业生大数据职业技能线上特训营开课啦

    "大数据特训营开课啦!""芝诺数据的老师太'可'了~"这几天,同学们的朋友圈被刷屏了,原来,是2020第一期重庆高校毕业生大数据职业技能线上特训营开讲了! &q ...

  7. 全国大学生大数据技能竞赛（Hadoop集群搭建）

    系列文章 全国大学生大数据技能竞赛(数仓部署) 全国大学生大数据技能竞赛(Spark on Yarn安装) 文章目录 系列文章 前言 资料链接 用VMware练习配置前准备三台虚拟机并修改网络为桥接 ...

  8. 高校大数据专业竞赛建设方案

    第一章 建设背景 1.1 政策分析  2017年1月 工业和信息化部正式发布了<大数据产业发展规划(2016-2020年)>,明确了"十三五"时期大数据产业的发展思路 ...

  9. oracle移花接木,移植苹果，Windows PC上安装MacOS

    苹果系统作为一款优秀的操作系统,仅仅体验外观我们当然无法真正感受到苹果系统的魅力.虽然大部分朋友没有苹果电脑,不过现在我们可以在Windows PC上安装实际苹果系统.由于苹果电脑和Windows差异 ...

最新文章

  1. 反向代理post参数怎么传输_都9102年了，GET和POST的区别掌握了没有？
  2. 华为手机权限开启方法8
  3. 冬日稻城亚丁，美得一塌糊涂！
  4. C++ public、protected、private区别
  5. 【渝粤教育】国家开放大学2018年秋季 1320T关系营销 参考试题
  6. Elastic Stack 安装
  7. 在 OpenShift 4 上部署 Ansible Tower 环境
  8. TCP/IP笔记 三.运输层(3)——TCP超时重传算法
  9. 【转】Add a user/Administrator to Windows Server 2008
  10. 医学相关excel表格计算机作业,求计算机作业电子表格演示文稿。
  11. 新版Excel和Word全屏打印预览的设定方法
  12. 利用excel快速制作标准正态分布表
  13. 微积分应用 计算机,微积分及其应用 P.D.Lax等著；林开亮
  14. rhadoop之mapreduce函数
  15. openproj centos安装及其输入中文变方块乱码解决
  16. 2022年3月18到5月18的思考
  17. ​CS客户端渗透测试详解(三)
  18. 腐烂的橘子（广度优先搜索）（考虑同时搜索）
  19. Java210道常见的面试
  20. 王者荣耀签到系统策划案

热门文章

  1. python程序设计实验指导书_《Python程序设计》实验指导书.pdf
  2. McAfee：黑客借＂极光漏洞＂获取企业源代码
  3. excel文件下载下来损坏 js_使用exceljs和nodejs下载的xlsx文件正在libreoffice中打开,但在excel 2007中显示已损坏...
  4. 基于机器学习的古代汉语自动分词标注算法及语料库研究
  5. Qt动画之仿Win8 Metro主界面
  6. WebService技术实现方案（转）
  7. 写给那些想做程序员和不想做程序员的人
  8. 【Linux】TIME_WAIT的作用、影响、解决方法
  9. php购物车面试题,php 购物车 细节
  10. qq炫舞摘经验秒级计算机,QQ炫舞8周年回归秒升30级活动