如何在Redhat7.7中安装CDP DC7.0.3

如何在Redhat7.7中安装CDP DC7.0.3

如何在Redhat7.7中安装CDP DC7.0.3

标签（空格分隔）： 大数据平台构建

一：关于CDP DC 7.0.3 概述 二： 系统环境初始化 三： 构建CDP DC 7.0.3

一： 关于CDP DC 7.0.3 概述

1.1 CDP dc 7.0.3

CDP DC7.0.3是Cloudera与Hortonworks合并后，第一个融合CDH和HDP所有组件的on-premise版本，CDP Data Center主要由Cloudera Runtime构成，Cloudera Runtime由超过35个开源项目组成，当然CDP Data Center还包括其它功能如管理功能Cloudera Manager，Key Management，专业支持等，如下图所示：

1.2 Cloudera Runtime的主要组件版本如下图所示：

二： 系统环境初始化

2.1 系统环境介绍

OS: rhel7.7x64 cat /etc/hosts --- 192.168.100.21 rhel01.flyfish 192.168.100.22 rhel02.flyfish 192.168.100.23 rhel03.flyfish 192.168.100.24 rhel04.flyfish 192.168.100.25 rhel05.flyfish 192.168.100.26 rhel06.flyfish --- 本次采用 前四台 机器

2.2 无密钥登录配置

做root用户无密钥认证 ssh-keygen ---一直敲回车到最后 cat id_rsa.pub >> authorized_keys 将所有的公钥导入authorized_keys 分发到 所有的 机器的.ssh/ 下面 然后测试

2.3关闭 firewalld，iptables 与 selinux

所有节点都要配置： yum install -y firewall* iptable* (1) 关闭firewalld systemctl stop firewalld.service systemctl disable firewalld.service firewall-cmd --state (2) 关闭iptables iptables -F systemctl stop iptables.service service iptables save systemctl disable iptables.service (3) 禁用 selinux sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config setenforce 0 getenforce 0 sestatus 重启机器 reboot

2.4 所有机器更改RHEL7.7X64的文件句柄数

vim /etc/security/limits.conf --- * soft nofile 65535 * hard nofile 1029345 * soft nproc unlimited * hard nproc unlimited * soft memlock unlimited * hard memlock unlimited ---

ansible cdh7 -m copy -a "src=/etc/security/limits.conf dest=/etc/security/"

2.5 时间同步：

所有机器 yum install chrony* 1. rhel01.flyfish: vim /etc/chrony.conf 增加 ---- 注释掉相关默认的server server ntp1.aliyun.com iburst allow 192.168.100.0/24 service chronyd start chkconfig chronyd on 2. rhel02.flyfish----rhel04.flyfish 同步rhel01.flyfish vim /etc/chrony.conf ---- 注释掉默认的server server 192.168.100.21 iburst ---- service chronyd stop service chronyd start chkconfig chronyd on

2.7 cdp dc 7.0.3 安装所需文件下载

CM 的rpm包： --- cloudera-manager-agent-7.0.3-1635136.el7.x86_64.rpm cloudera-manager-daemons-7.0.3-1635136.el7.x86_64.rpm cloudera-manager-server-7.0.3-1635136.el7.x86_64.rpm cloudera-manager-server-db-2-7.0.3-1635136.el7.x86_64.rpm enterprise-debuginfo-7.0.3-1635136.el7.x86_64.rpm openjdk8-8.0+232_9-cloudera.x86_64.rpm allkeys.asc 下载地址： http://archive.cloudera.com/cm7/7.0.3/redhat7/yum/RPMS/x86_64/ http://archive.cloudera.com/cm7/7.0.3/allkeys.asc ---- cdh7.0.3 的parcel： -- CDH-7.0.3-1.cdh7.0.3.p0.1635019-el7.parcel CDH-7.0.3-1.cdh7.0.3.p0.1635019-el7.parcel.sha256 manifest.json 下载地址： http://archive.cloudera.com/cdh7/7.0.3.0/parcels/ ---

2.8 设置swap

将所有的swap为1，并即时生效 ansible cdh7 -m shell -a "echo '1' > /proc/sys/vm/swappiness" ansible cdh7 -m shell -a "sysctl -a | grep vm.swappiness"

vim /etc/sysctl.conf ---- 最后加上： vm.swappiness = 1 ---- sysctl -p

2.8 设置最大透明数

ansible cdh7 -m shell -a "echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag" ansible cdh7 -m shell -a "echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled" ansible cdh7 -m shell -a "cat /sys/kernel/mm/transparent_hugepage/enabled" ansible cdh7 -m shell -a "cat /sys/kernel/mm/transparent_hugepage/defrag"

修改所有节点的/etc/rc.d/rc.local文件的权限以实现开机执行 ansible cdh7 -m shell -a "chmod +x /etc/rc.d/rc.local" vim /etc/rc.d/rc.local --- 最后加上： if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi --- ansible cdh7 -m copy -a "src=/etc/rc.d/rc.local dest=/etc/rc.d/" ansible cdh7 -m shell -a "tail -f /etc/rc.d/rc.local"

2.9 配置服务器

ansible rhel01.flyfish -m yum -a "state=present name=httpd" ansible rhel01.flyfish -m service -a "name=state=started" ansible rhel01.flyfish -m shell -a "ps -ef |grep /etc/在中修改以下内容 vim /etc/httpd/conf/httpd.conf --- AddType application/x-compress .Z AddType application/x-gzip .gz .tgz .parcel --- service restart

2.10 安装maridb 数据库

yum install mariadb* service mariadb start chkonfig mariadb on mysql_secure_installation

2.11 创建CM，Hive等需要的表

vim cm.sql ---- create database metastore default character set utf8; CREATE USER 'hive'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%'; FLUSH PRIVILEGES; create database cm default character set utf8; CREATE USER 'cm'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%'; FLUSH PRIVILEGES; create database rm default character set utf8; CREATE USER 'rm'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%'; FLUSH PRIVILEGES; create database hue default character set utf8; CREATE USER 'hue'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%'; FLUSH PRIVILEGES; create database oozie default character set utf8; CREATE USER 'oozie'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON oozie. * TO 'oozie'@'%'; FLUSH PRIVILEGES; create database ranger default character set utf8; CREATE USER 'rangeradmin'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON ranger. * TO 'rangeradmin'@'%'; FLUSH PRIVILEGES; ---- mysql -uroot -pflyfish225 source cm.sql

2.12 安装JDBC驱动

mkdir -p /usr/share/java/ tar -zxvf mysql-connector-java-5.1.44.tar.gz mv mysql-connector-java-5.1.44-bin.jar /usr/share/java cd /usr/share/java/ mv mysql-connector-java-5.1.44-bin.jar mysql-connector-java.jar chmod 777 mysql-connector-java.jar

三： 构建CDP DC 7.0.3

3.1 安装 jdk

所有节点 卸载系统自带jdk rpm -qa |grep jdk rpm -e java-1.8.0-openjdk-1.8.0.222.b03-1.el7.x86_64 copy-jdk-configs-3.3-10.el7_5.noarch java-1.8.0-openjdk-headless-1.8.0.222.b03-1.el7.x86_64 --nodeps yum -y install openjdk8-8.0+232_9-cloudera vim /etc/profile --- 到最后加上jdk 的环境变量 export JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar PATH=$PATH:$HOME/bin:$JAVA_HOME/bin ---- source /etc/profile java -version

3.2 配置cdh7.0.3 的源

mv cdh7.0.3 cdh7 cd cdh7 createrepo . cd .. mv cdh7 /var/Manager的repo源 cd /etc/yum.reos.d/ vim cdh.repo --- [cmrepo] name = cm_repo baseurl = http://192.168.100.21/cdh7/ enable = true gpgcheck = false --- ansible cdh7 -m copy -a "src=/etc/repos.d/cdh7.repo dest=/etc/repos.d/" yum repolist

3.3 安装Cloudera Manager Server

yum -y install cloudera-manager-server

初始化数据库 /opt/cloudera/cm/schema/scm_prepare_database.sh mysql cm cm password

启动Cloudera Manager Server systemctl start cloudera-scm-server systemctl status cloudera-scm-server cd /var/log/cloudera-scm-server tail -100f cloudera-scm-server.log netstat -nultp |grep 7180

3.4 通过http://192.168.100.21:7180/cmf/login访问CM

用户名：admin 密码： admin

集群安装：

集群名字

识别集群的主机

Parcel Repository & Network Settings

安装 cloudera-scm-agent

分配parcel包

定义服务：

首先输入CM的用户名和密码admin/admin 以下4个密码，都需要包含至少一个字母与一个数字，为了安装方便都输入admin@123

安装服务初始化

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。