Docker搭建Hadoop集群

网友投稿 258 2022-10-22

Docker搭建Hadoop集群

​目录​

​​一、配置Docker网络​​

​​二、docker安装centos​​

​​三、配置ssh免密输入​​

​​四、安装Java环境​​

​​五、安装zookeeper​​

​​六、安装hadoop3.0.0​​

​​七、初始化并启动集群​​

​​八、打包镜像推送到阿里云仓库​​

一、配置Docker网络

​ 1、win10配置路由表（win10版docker配置）​

win10当前存在4个网络分别是本机网络、docker网络、1WSL网络和linux子系统网络

而docker的子网是192.168.65.0，这里在win10中添加一个路由新建网络172.18.0.0到docker子网

route add 172.18.0.0/16 mask 255.255.240.0 192.168.65.0

​

​2、win10配置路由表（Linux版docker配置）​

ifconfig

172.18.0.0.0是下一步docker需要create的network

255.255.240.0是eth0网卡子网掩码

172.19.79.255是eth0网卡的广播地址

route add 172.18.0.0/16 mask 255.255.240.0 172.19.79.255

​ 3、创建docker网络​

docker network create --subnet=172.18.0.0/16 bigdata hadoop01

docker network ls

docker network inspect bigdata

​​windows宿主机访问docker容器ip无法ping通 - 系统运维 - 亿速云​​

二、docker安装centos

​1、安装centos7​

docker pull centos:7

Docker安装在Linux子系统则执行以下命令

docker run -d -it -p 201:22 \ -v /home/cluster:/home/volume \ --name=hadoop01 \ --hostname=hadoop01 \ --network=bigdata \ --privileged=true \ centos:7 /sbin/init

Docker安装在Win10使用以下命令

docker run -d -it -p 201:22 \ -v G:/Hadoop/Docker/Cluster:/home/win10 \ --name=hadoop01 \ --hostname=hadoop01 \ --network=bigdata \ --privileged=true \ centos:7 /sbin/init

​2、查看容器ip​

docker inspect --format='{{.NetworkSettings.IPAddress}}' hadoop01

docker inspect hadoop01 | grep '"IPAddress":'

​3、进入centos​

docker exec -it hadoop01 /bin/bash

cat /etc/hosts

​4、yum安装插件​

yum -y install wget net-tools vim sudo lrzsz

​5、更新yum源​

​​centos镜像-centos下载地址-centos安装教程-阿里巴巴开源镜像站​​

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.init

wget -O /etc/yum.repos.d/CentOS-Base.repo makecache

yum -y update

​6、DNS域名解析配置​

echo '## Generated by NetworkManagernameserver 8.8.8.8nameserver 114.114.114.114' >> /etc/resolv.conf

cat /etc/resolv.conf

​7、查看系统信息​

cat /proc/version

uname -a

cat /etc/redhat-release

rpm -q centos-release

三、配置ssh免密输入

​1、安装service命令 ​

yum -y install initscripts

​ 2、修改root密码​

passwd root

​3、安装sshd服务​

yum -y install openssh-server openssh-clients

​4、修改hostname（非必须）​

hostnamectl set-hostname hadoop01

hostnamectl set-hostname hadoop02

hostnamectl set-hostname hadoop03

hostname

reboot

​5、修改hosts​

vim /etc/hosts

172.18.0.2 hadoop01172.18.0.3 hadoop02172.18.0.4 hadoop03

cat /etc/hosts

​6、sshd服务开机自启​

systemctl enable sshd.service

​7、启动sshd服务​

systemctl start sshd.service

systemctl status sshd.service

​8、修改root密码​

passwd root

​9、xshell直接连接docker容器​

因为容器ssh端口22已经映射给主机201端口了，所以ssh连接是连接主机ip:201端口

​10、配置ssh免密登录​

执行 ssh-keygen 敲三次回车生成公钥和密钥，然后再发送给自己敲两次root密码即可

ssh-keygen

ssh-copy-id -i ~/.ssh/id_rsa.pub root@127.0.0.1

​11、Linux批量命令和批量分发脚本制作​

​​Linux shell脚本实现命令批处理和文件批量分发_qq262593421的博客​​

四、安装Java环境

​1、下载jdk安装包​

​​oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm​​

​ 2、查看系统是否已安装java​

rpm -aq | grep java

​3、rpm命令安装jdk​

rpm -Uvh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

​4、创建软链接​

ln -s /usr/java/jdk1.8.0_181-cloudera /usr/java/jdk1.8

​5、配置环境变量​

cp /etc/profile /etc/profile.init

echo '## java configexport JAVA_HOME=/usr/java/jdk1.8export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport PATH=${JAVA_HOME}/bin:${PATH}:${CLASSPATH}' >> /etc/profile

tail -n 10 /etc/profile

source /etc/profile

java -version

which javawhereis java

五、安装zookeeper

​​大数据高可用技术之zookeeper3.4.5安装配置_qq262593421的博客​​

​1、创建安装目录​

mkdir /usr/local/hadoop

​2、解压文件​

tar zxpf zookeeper-3.4.5.tar.gz -C /usr/local/hadoop

​3、创建软链接​

ln -s /usr/local/hadoop/zookeeper-3.4.5 /usr/local/hadoop/zookeeper

​4、配置zookeeper​

mv /usr/local/hadoop/zookeeper/conf/zoo.cfg /usr/local/hadoop/zookeeper/conf/zoo.cfg.init

vim /usr/local/hadoop/zookeeper/conf/zoo.cfg

# The number of milliseconds of each ticktickTime=2000# The number of ticks that the initial # synchronization phase can takeinitLimit=10# The number of ticks that can pass between # sending a request and getting an acknowledgementsyncLimit=5# the directory where the snapshot is stored.# do not use /tmp for storage, /tmp here is just # example sakes.dataDir=/usr/local/hadoop/zookeeper/datadataLogDir=/usr/local/hadoop/zookeeper/logs# the port at which the clients will connectclientPort=2181## Be sure to read the maintenance section of the # administrator guide before turning on autopurge.## The number of snapshots to retain in dataDir#autopurge.snapRetainCount=3# Purge task interval in hours# Set to "0" to disable auto purge feature#autopurge.purgeInterval=1 server.1=hadoop01:2888:3888server.2=hadoop02:2888:3888server.3=hadoop03:2888:3888

​5、配置myid​

zookeeper根据配置文件自动创建dataDir、dataLogDir目录，需要自己手动创建

mkdir /usr/local/hadoop/zookeeper/datamkdir /usr/local/hadoop/zookeeper/logs

给每台zk节点配置myid，每个节点myid的值和zoo.cfg中的server.N一致

echo "1" > /usr/local/hadoop/zookeeper/data/myid

echo "2" > /usr/local/hadoop/zookeeper/data/myid

echo "3" > /usr/local/hadoop/zookeeper/data/myid

cat /usr/local/hadoop/zookeeper/data/myid

zookeeper的选举机制和myid密切关联，myid值大的相对更容易成为leader，具体这里就不详细讨论了。

​6、配置zk环境变量​

echo '## zookeeper configexport ZOO_HOME=/usr/local/hadoop/zookeeperexport PATH=$PATH:$ZOO_HOME/bin' >> /etc/profile

tail -n 10 /etc/profile

source /etc/profile

​7、启动zookeeper​

zkServer.sh start

zkServer.sh status

zkCli.sh

ls /zookeeper

quit

六、安装hadoop3.0.0

​​大数据集群搭建之Linux安装hadoop3.0.0_qq262593421的博客​​

​1、下载安装​

tar zxpf hadoop-3.0.0.tar.gz -C /usr/local/hadoop/

​2、创建软链接​

ln -s /usr/local/hadoop/hadoop-3.0.0 /usr/local/hadoop/hadoop

​3、环境变量配置​

echo '## hadoop configexport HADOOP_HOME=/usr/local/hadoop/hadoopexport PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin' >> /etc/profile

tail -n 10 /etc/profile

source /etc/profile

​4、检查磁盘空间​

df -h

​5、修改worker配置​

echo "hadoop01hadoop02hadoop03" > /usr/local/hadoop/hadoop/etc/hadoop/workers

​6、修改core-site.xml配置​

mv /usr/local/hadoop/hadoop/etc/hadoop/core-site.xml /usr/local/hadoop/hadoop/etc/hadoop/core-site.xml.init

vim /usr/local/hadoop/hadoop/etc/hadoop/core-site.xml

fs.defaultFS hdfs://ns1 hadoop.tmp.dir /home/cluster/hadoop/data/tmp io.file.buffer.size 131072 Size of read/write buffer used in SequenceFiles ha.zookeeper.quorum hadoop01:2181,hadoop02:2181,hadoop03:2181 DFSZKFailoverController hadoop.proxyuser.root.hosts * hadoop.proxyuser.root.groups *

​7、修改hadoop-env.sh配置​

echo 'export HDFS_NAMENODE_OPTS="-XX:+UseParallelGC -Xmx4g"export HDFS_NAMENODE_USER=rootexport HDFS_DATANODE_USER=rootexport HDFS_JOURNALNODE_USER=rootexport HDFS_SECONDARYNAMENODE_USER=rootexport YARN_RESOURCEMANAGER_USER=rootexport YARN_NODEMANAGER_USER=rootexport JAVA_HOME=/usr/java/jdk1.8' >> /usr/local/hadoop/hadoop/etc/hadoop/hadoop-env.sh

tail -n 10 /usr/local/hadoop/hadoop/etc/hadoop/hadoop-env.sh

​8、修改hdfs-site.xml配置​

mv /usr/local/hadoop/hadoop/etc/hadoop/hdfs-site.xml /usr/local/hadoop/hadoop/etc/hadoop/hdfs-site.xml.init

vim /usr/local/hadoop/hadoop/etc/hadoop/hdfs-site.xml

dfs.namenode.name.dir /home/cluster/hadoop/data/nn dfs.datanode.data.dir /home/cluster/hadoop/data/dn dfs.journalnode.edits.dir /home/cluster/hadoop/data/jn dfs.nameservices ns1 dfs.ha.namenodes.ns1 hadoop01,hadoop02 dfs.namenode.rpc-address.ns1.hadoop01 hadoop01:8020 dfs.namenode. hadoop01:9870 dfs.namenode.rpc-address.ns1.hadoop02 hadoop02:8020 dfs.namenode. hadoop02:9870 dfs.ha.automatic-failover.enabled.ns1 true dfs.client.failover.proxy.provider.ns1 org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider dfs.permissions.enabled false dfs.replication 2 dfs.blocksize HDFS blocksize of 128MB for large file-systems dfs.namenode.handler.count 100 More NameNode server threads to handle RPCs from large number of DataNodes. dfs.namenode.shared.edits.dir qjournal://hadoop01:8485;hadoop02:8485;hadoop03:8485/ns1 dfs.ha.fencing.methods sshfence dfs.ha.fencing.ssh.private-key-files /root/.ssh/id_rsa

​9、修改mapred-site.xml配置​

mv /usr/local/hadoop/hadoop/etc/hadoop/mapred-site.xml /usr/local/hadoop/hadoop/etc/hadoop/mapred-site.xml.init

vim /usr/local/hadoop/hadoop/etc/hadoop/mapred-site.xml

mapreduce.framework.name yarn Execution framework set to Hadoop YARN. mapreduce.map.memory.mb 4096 Larger resource limit for maps. mapreduce.map.java.opts -Xmx4096M Larger heap-size for child jvms of maps. mapreduce.reduce.memory.mb 4096 Larger resource limit for reduces. mapreduce.reduce.java.opts -Xmx4096M Larger heap-size for child jvms of reduces. mapreduce.task.io.sort.mb 2040 Higher memory-limit while sorting data for efficiency. mapreduce.task.io.sort.factor 400 More streams merged at once while sorting files. mapreduce.reduce.shuffle.parallelcopies 200 Higher number of parallel copies run by reduces to fetch outputs from very large number of maps. mapreduce.jobhistory.address hadoop01:10020 MapReduce JobHistory Server host:port.Default port is 10020 mapreduce.jobhistory.webapp.address hadoop01:19888 MapReduce JobHistory Server Web UI host:port.Default port is 19888. mapreduce.jobhistory.intermediate-done-dir /tmp/mr-history/tmp Directory where history files are written by MapReduce jobs. mapreduce.jobhistory.done-dir /tmp/mr-history/done Directory where history files are managed by the MR JobHistory Server.

​10、修改yarn-site.xml配置​

mv /usr/local/hadoop/hadoop/etc/hadoop/yarn-site.xml /usr/local/hadoop/hadoop/etc/hadoop/yarn-site.xml.init

vim /usr/local/hadoop/hadoop/etc/hadoop/yarn-site.xml

yarn.resourcemanager.ha.enabled true yarn.resourcemanager.ha.automatic-failover.enabled true yarn.resourcemanager.ha.automatic-failover.embedded true yarn.resourcemanager.cluster-id yarn-rm-cluster yarn.resourcemanager.ha.rm-ids rm1,rm2 yarn.resourcemanager.hostname.rm1 hadoop01 yarn.resourcemanager.hostname.rm2 hadoop02 yarn.resourcemanager.recovery.enabled true yarn.resourcemanager.zk.state-store.address hadoop01:2181,hadoop02:2181,hadoop03:2181 yarn.resourcemanager.zk-address hadoop01:2181,hadoop02:2181,hadoop03:2181 yarn.resourcemanager.address.rm1 hadoop01:8032 yarn.resourcemanager.address.rm2 hadoop02:8032 yarn.resourcemanager.scheduler.address.rm1 hadoop01:8034 yarn.resourcemanager.webapp.address.rm1 hadoop01:8088 yarn.resourcemanager.scheduler.address.rm2 hadoop02:8034 yarn.resourcemanager.webapp.address.rm2 hadoop02:8088 yarn.acl.enable true Enable ACLs? Defaults to false. yarn.admin.acl * yarn.log-aggregation-enable false Configuration to enable or disable log aggregation yarn.resourcemanager.hostname hadoop01 host Single hostname that can be set in place of setting all yarn.resourcemanager*address resources. Results in default ports for ResourceManager components. yarn.scheduler.minimum-allocation-mb 1024 saprk调度时一个container能够申请的最小资源，默认值为1024MB yarn.scheduler.maximum-allocation-mb 28672 saprk调度时一个container能够申请的最大资源，默认值为8192MB yarn.nodemanager.resource.memory-mb 28672 nodemanager能够申请的最大内存，默认值为8192MB yarn.app.mapreduce.am.resource.mb 28672 AM能够申请的最大内存，默认值为1536MB yarn.nodemanager.log.retain-seconds 10800 yarn.nodemanager.log-dirs /home/cluster/yarn/log/1,/home/cluster/yarn/log/2,/home/cluster/yarn/log/3 yarn.nodemanager.aux-services mapreduce_shuffle Shuffle service that needs to be set for Map Reduce applications. yarn.log-aggregation.retain-seconds -1 yarn.log-aggregation.retain-check-interval-seconds -1 yarn.app.mapreduce.am.staging-dir hdfs://ns1/tmp/hadoop-yarn/staging The staging dir used while submitting jobs. yarn.application.classpath /usr/local/hadoop/hadoop/etc/hadoop:/usr/local/hadoop/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/hadoop/share/hadoop/common/*:/usr/local/hadoop/hadoop/share/hadoop/hdfs:/usr/local/hadoop/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/hadoop/share/hadoop/yarn:/usr/local/hadoop/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/hadoop/share/hadoop/yarn/* Linux上打 hadoop classpath 找到的所有路径

七、初始化并启动集群

​1、启动zookeeper（hadoop01、hadoop02、hadoop03）​

由于hadoop的HA机制依赖于zookeeper，因此先启动zookeeper集群

zkServer.sh start

zkServer.sh status

​2、在zookeeper中初始化元数据（hadoop01）​

hdfs zkfc -formatZK

​3、启动zkfc（hadoop01、hadoop02）​

hdfs --daemon start zkfc

​4、启动JournalNode（hadoop01、hadoop02、hadoop03）​

格式化NameNode前必须先格式化JournalNode，否则格式化失败

这里配置了3个JournalNode节点，hadoop01、hadoop02、hadoop03

hdfs --daemon start journalnode

​5、格式化NameNode（hadoop01）​

在第一台NameNode节点上执行

hdfs namenode -format

​6、启动hdfs（hadoop01）​

hdfs --daemon start namenode

​7、同步备份NameNode（hadoop02）​

等hdfs初始化完成之后（20秒），在另一台NameNode上执行

hdfs namenode -bootstrapStandby

常见报错（1）如果格式化失败或者出现以下错误，把对应节点上的目录删掉再重新格式化

Directory is in an inconsistent state: Can't format the storage directory because the current directory is not empty.

hdfs --daemon stop journalnoderm -rf /home/cluster/hadoop/data/*hdfs namenode -format

常见报错（2）如果hadoop02无法连接master，可能是由于hosts文件的缘故，需改成精简版

INFO ipc.Client: Retrying connect to server: hadoop01/172.18.0.2:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) FATAL ha.BootstrapStandby: Unable to fetch namespace information from any remote NN. Possible NameNodes: [RemoteNameNodeInfo [nnId=hadoop01, ipcAddress=hadoop01/172.18.0.2:8020, /etc/hosts /etc/hosts.init

echo '127.0.0.1 localhosts172.18.0.2 hadoop01172.18.0.3 hadoop02172.18.0.4 hadoop03' > /etc/hosts

​8、启动备份NameNode（hadoop02）​

同步之后，需要在另一台NameNode节点上启动NameNode进程，然后再启动整个

hdfs --daemon start namenode

start-all.sh

​9、查看集群状态（hadoop01）​

hadoop dfsadmin -report

​10、访问集群​

​​ ​​  ​​--daemon stop namenode

查看standby状态的NameNode ​​--daemon start namenode

​​ ​​ ​​        ​​/etc/hosts文件docker容器中的Linux系统并不会在开机时source /etc/profile文件

echo 'echo "127.0.0.1 localhost::1 localhost ip6-localhost ip6-loopbackfe00::0 ip6-localnetff00::0 ip6-mcastprefixff02::1 ip6-allnodesff02::2 ip6-allrouters172.18.0.1 dockerhost172.18.0.2 hadoop01172.18.0.3 hadoop02172.18.0.4 hadoop03" > /etc/hosts' > /etc/profile.d/hosts.sh

cat /etc/profile.d/hosts.sh

cat /etc/hosts

另外还有一个解决思路，通过 docker link 命令实现 容器name 互联，会自动生成ip和host在配置文件中

​13、生成hadoop初始镜像 ​

mv /home/cluster/hadoop01/* /home/cluster/

docker commit -m="java、zk、hadoop单机已配置完成，预配置为2master3slave一共三台机器" -a="日月星辰TEL" hadoop01 hadoop:v1

docker images

​13、生成hadoop02、hadoop03镜像​

​hadoop集群分配​

主机名称

hadoop角色

主机映射ip+端口

docker ip+端口

hadoop01

Master

127.0.0.1:201

172.18.0.02:22

hadoop02

Master/Slave

127.0.0.1:202

172.18.0.03:22

hadoop03

Slave

127.0.0.1:203

172.18.0.04:22

hadoop01

Zookeeper

​​data-id="t4606665-68eq01Ok" style="min-width: auto; overflow-wrap: break-word; margin: 4px 8px; border: 1px solid rgb(217, 217, 217); padding: 4px 8px; cursor: default; vertical-align: top;">

​​data-id="t6b20032-cUqjfE6e" style="">

hadoop02

Zookeeper

​​data-id="t4606665-uD26kh9e" style="min-width: auto; overflow-wrap: break-word; margin: 4px 8px; border: 1px solid rgb(217, 217, 217); padding: 4px 8px; cursor: default; vertical-align: top;">

​​data-id="t6b20032-mZLQoqSJ" style="">

hadoop03

Zookeeper

​​data-id="t4606665-weDPfwts" style="min-width: auto; overflow-wrap: break-word; margin: 4px 8px; border: 1px solid rgb(217, 217, 217); padding: 4px 8px; cursor: default; vertical-align: top;">

​​data-id="t6b20032-8ln3O0mN" style="">

hadoop01

Namenode

​​data-id="t4606665-vR4ViO96" style="min-width: auto; overflow-wrap: break-word; margin: 4px 8px; border: 1px solid rgb(217, 217, 217); padding: 4px 8px; cursor: default; vertical-align: top;">

​​data-id="t6b20032-t2YJdFcV" style="">

hadoop02

Namenode

​​data-id="t4606665-qFGSrdjs" style="min-width: auto; overflow-wrap: break-word; margin: 4px 8px; border: 1px solid rgb(217, 217, 217); padding: 4px 8px; cursor: default; vertical-align: top;">

​​data-id="t6b20032-c54cm3lV" style="">

hadoop01

Hive Server2

​​data-id="t4606665-LAS3IDKU" style="min-width: auto; overflow-wrap: break-word; margin: 4px 8px; border: 1px solid rgb(217, 217, 217); padding: 4px 8px; cursor: default; vertical-align: top;">

​​data-id="t6b20032-SMcnvRQl" style="">

hadoop01

Yarn Applications

​​data-id="t4606665-YajI2XWv" style="min-width: auto; overflow-wrap: break-word; margin: 4px 8px; border: 1px solid rgb(217, 217, 217); padding: 4px 8px; cursor: default; vertical-align: top;">

​​data-id="t6b20032-6ZdBYO1A" style="">

hadoop01

HBase Master

​​data-id="t4606665-eDSO5UX2" style="min-width: auto; overflow-wrap: break-word; margin: 4px 8px; border: 1px solid rgb(217, 217, 217); padding: 4px 8px; cursor: default; vertical-align: top;">

​​data-id="t6b20032-YaOb11F0" style="">

hadoop02

HBase Backup Master

​​data-id="t4606665-dTfcwrqu" style="min-width: auto; overflow-wrap: break-word; margin: 4px 8px; border: 1px solid rgb(217, 217, 217); padding: 4px 8px; cursor: default; vertical-align: top;">

​​data-id="t6b20032-6OwARgLA" style="">

hadoop01

Spark History Server

​​data-id="t4606665-mvieZr7W" style="min-width: auto; overflow-wrap: break-word; margin: 4px 8px; border: 1px solid rgb(217, 217, 217); padding: 4px 8px; cursor: default; vertical-align: top;">

​​data-id="t6b20032-Qmdxy1XM" style="">

hadoop01

Spark Master

​​data-id="t4606665-Cw2BwTOg" style="min-width: auto; overflow-wrap: break-word; margin: 4px 8px; border: 1px solid rgb(217, 217, 217); padding: 4px 8px; cursor: default; vertical-align: top;">

​​data-id="t6b20032-RtHVziYn" style="">

hadoop02

Spark Master

​​data-id="t4606665-6M6uFhQH" style="min-width: auto; overflow-wrap: break-word; margin: 4px 8px; border: 1px solid rgb(217, 217, 217); padding: 4px 8px; cursor: default; vertical-align: top;">

​​data-id="t6b20032-FJyugHJR" style="">

hadoop01

Flink Web

​​data-id="t4606665-HOhbTYVu" style="min-width: auto; overflow-wrap: break-word; margin: 4px 8px; border: 1px solid rgb(217, 217, 217); padding: 4px 8px; cursor: default; vertical-align: top;">

​​data-id="t6b20032-Ku5nAzxg" style="">

hadoop02

Flink Web

​​data-id="t4606665-bz7oZgpg" style="min-width: auto; overflow-wrap: break-word; margin: 4px 8px; border: 1px solid rgb(217, 217, 217); padding: 4px 8px; cursor: default; vertical-align: top;">

​​run -d -it -p 201:22 -p 2281:2181 \ -p 9871:9870 -p 18088:8080 -p 16011:16010 \ -p 18081:18080 -p 8991:8989 -p 8082:8081 -p 10003:10002 \ -v /home/cluster:/home/volume \ --name=hadoop01 \ --hostname=hadoop01 \ --network=bigdata \ --ip 172.18.0.2 \ --privileged=true \ hadoop:v1 /sbin/init

hadoop02

docker run -d -it -p 202:22 -p 2282:2181 \ -p 9872:9870 -p 16012:16010 \ -p 8992:8989 -p 8083:8081 \ -v /home/cluster:/home/volume \ --name=hadoop02 \ --hostname=hadoop02 \ --network=bigdata \ --ip 172.18.0.3 \ --privileged=true \ hadoop:v1 /sbin/init

hadoop03

docker run -d -it -p 203:22 -p 2283:2181 \ -v /home/cluster:/home/volume \ --name=hadoop03 \ --hostname=hadoop03 \ --network=bigdata \ --ip 172.18.0.4 \ --privileged=true \ hadoop:v1 /sbin/init

docker exec -it hadoop01 /bin/bash

docker exec -it hadoop02 /bin/bash

docker exec -it hadoop03 /bin/bash

​​- 欢迎登录阿里云，安全稳定的云计算服务平台​​

​ 1、将容器打包成镜像​

docker ps -a

docker images

docker镜像生成语法：

docker commit -a "提交作者" -m "镜像描述" 容器id/容器name 镜像名称:镜像版本

docker commit -a "静谧星空℡" -m "zookeeper hadoop hive hbase spark" hadoop01 hadoop01:v1docker commit -a "静谧星空℡" -m "zookeeper hadoop hive hbase spark" hadoop02 hadoop02:v1docker commit -a "静谧星空℡" -m "zookeeper hadoop hive hbase spark" hadoop03 hadoop03:v1

注意事项：如果镜像版本已存在，则会将原来的镜像替换

​ 2、docker登录阿里云​

​​镜像仓库 - 金融分布式架构 SOFAStack - 阿里云​​

docker登录阿里仓库语法：

docker login --username=[TestUser] registry.cn-shanghai.aliyuncs.com

docker login --username=26xxx21@qq.com registry.cn-shanghai.aliyuncs.com

如遇到登录报错：

Error saving credentials: error storing credentials - err: exit status 1, out: `Error spawning command line “dbus-lch --autolaunch= --binary-syntax --close-stderr”: Child process exited with code 1`

解决办法：

sudo apt install gnupg2 pass

​ 3、生成仓库镜像版本号​

生成仓库镜像版本语法1：

docker tag [ImageId] registry.cn-shanghai.aliyuncs.com/[命名空间名称]/[仓库名称]:[镜像版本号]

docker tag 0414bdf69fa4 registry.cn-hangzhou.aliyuncs.com/hadoopcluster/hadoop:v1.1

生成仓库镜像版本语法2：

docker tag [Image:TAG] registry.cn-shanghai.aliyuncs.com/[命名空间名称]/[仓库名称]:[镜像版本号]

docker tag hadoop01:v1 registry.cn-hangzhou.aliyuncs.com/hadoopcluster/hadoop:v1.1docker tag hadoop02:v1 registry.cn-hangzhou.aliyuncs.com/hadoopcluster/hadoop:v2.1docker tag hadoop03:v1 registry.cn-hangzhou.aliyuncs.com/hadoopcluster/hadoop:v3.1

​ 4、推送到阿里云仓库 ​

docker推送镜像语法：

docker push registry.cn-shanghai.aliyuncs.com/[命名空间名称]/[仓库名称]:[镜像版本号]

docker push registry.cn-hangzhou.aliyuncs.com/hadoopcluster/hadoop:v1.1docker push registry.cn-hangzhou.aliyuncs.com/hadoopcluster/hadoop:v2.1docker push registry.cn-hangzhou.aliyuncs.com/hadoopcluster/hadoop:v3.1

注意事项：不能强制将阿里云镜像删除，不然会把依赖镜像一起删除掉的

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。

标签：文件
上一篇:Docker和Ubuntu主机互传复制文件
下一篇:Java十分钟精通内部类的使用
相关文章

 发表评论

暂时没有评论，来抢沙发吧~