【CentOS】安装 Sqoop 组件

【CentOS】安装 Sqoop 组件

文章目录

​​1、解压 Sqoop 安装包​​​​2、配置 sqoop-env.sh​​​​3、配置系统环境变量​​​​4、sqoop 连接 MySQL​​​​5、sqoop 连接 MySQL 导入到 Hive​​

​​5.1 准备​​​​5.2 MySQL创建模拟数据表（数据源）​​​​5.3 Hive 创建数据表（接收）​​​​5.3 MySQL 导出到 Hive​​

准备环境：Hadoop 完全分布式集群环境Sqoop 安装包：​​Sqoop 安装包

上传本地安装包：

解压后并重命名：

​​返回顶部​​

2、配置 sqoop-env.sh

复制 ​​sqoop-env-template.sh​​ 模板，并将模板重命名为 ​​sqoop-env.sh​​，在 ​​sqoop-env.sh​​ 中添加 ​​Hdoop​​、​​Hbase​​、​​Hive​​、​​HBase​​的安装路径：

#Set path to where bin/hadoop is availableexport HADOOP_COMMON_HOME=/usr/local/src/hadoop#Set path to where hadoop-*-core.jar is availableexport HADOOP_MAPRED_HOME=/usr/local/src/hadoop#set the path to where bin/hbase is availableexport HBASE_HOME=/usr/local/src/hbase#Set the path to where bin/hive is availableexport HIVE_HOME=/usr/local/src/hive#Set the path for where zookeper config dir isexport ZOOCFGDIR=/usr/local/src/zookeeper/conf

​​返回顶部​​

3、配置系统环境变量

配置 Linux 系统环境变量（​​/etc/profile​​），添加 Sqoop 组件的路径：

# set sqoop environmentexport SQOOP_HOME=/usr/local/src/sqoopexport PATH=$PATH:$SQOOP_HOME/binexport CLASSPATH=$CLASSPATH:$SQOOP_HOME/lib

保存退出后 ​​source​​ 使其生效！

​​返回顶部​​

4、sqoop 连接 MySQL

为了使 Sqoop 能够连接 MySQL 数据库，需要将 ​​/../download/mysql-connector-java-5.1.46.jar​​ 文件放入 sqoop 的 ​​lib​​ 目录中：

[root@server download]# cp mysql-connector-java-5.1.46.jar /usr/local/src/sqoop/lib/

执行 Sqoop 前需要先启动 Hadoop 集群，在 ​​server​​ 节点执行 ​​start-all.sh​​ 命令启动 Hadoop 集群，查看进程：

[root@server conf]# jps5392 HMaster1762 ResourceManager4754 QuorumPeerMain10226 Jps1603 SecondaryNameNode1412 NameNode[root@agent1 src]# jps1235 DataNode1863 HRegionServer1306 NodeManager1563 QuorumPeerMain2687 Jps[root@agent2 src]# jps1344 NodeManager1570 QuorumPeerMain1238 DataNode2678 Jps1868

测试 Sqoop 是否能够正常连接 MySQL 数据库：

[root@server download]# cp mysql-connector-java-5.1.46.jar /usr/local/src/sqoop/lib/[root@server download]# sqoop list-databases --connect jdbc:mysql://127.0.0.1:3306/ --username root -PWarning: /usr/local/src/sqoop/../hcatalog does not exist! HCatalog jobs will fail.Please set $HCAT_HOME to the root of your HCatalog installation.Warning: /usr/local/src/sqoop/../accumulo does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.22/02/25 19:35:42 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7Enter password: 22/02/25 19:35:45 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.Fri Feb 25 19:35:46 CST 2022 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

能 够 查 看 到 MySQL 数 据 库 中 的 ​​information_schema​​ 、 ​​hive​​ 、 ​​mysql​​ 、​​performance_schema​​、​​sys​​ 等数据库，说明 Sqoop 可以正常连接 MySQL。

​​返回顶部​​

5、sqoop 连接 MySQL 导入到 Hive

5.1 准备

为了使 Sqoop 能够连接 Hive，需要将 hive 组件 ​​/usr/local/src/hive/lib​​ 目录下的 ​​hive-common-2.3.9.jar​​ 也放入 Sqoop 安装路径的 lib 目录中：

[root@server ~] cp /usr/local/src/hive/lib/hive-common-2.3.9.jar /usr/local/src/sqoop/lib/

5.2 MySQL创建模拟数据表（数据源）

MySQL中创建 ​​test​​ 数据库，在 test 中创建 ​​student​​ 表，在 student 表中插入了 3 条数据：

[root@server lib]# mysql -uroot -pEnter password: Welcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 15Server version: 5.7.22 MySQL Community Server (GPL)Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> create database test;Query OK, 1 row affected (0.00 sec)mysql> use test;Database changedmysql> create table student(number char(9) primary key, name varchar(10));Query OK, 0 rows affected (0.25 sec)mysql> insert into student values('01','zhangsan');Query OK, 1 row affected (0.00 sec)mysql> insert into student values('02','lisi');Query OK, 1 row affected (0.00 sec)mysql> insert into student values('03','wangwu');Query OK, 1 row affected (0.00 sec)mysql> select * from student;+--------+----------+| number | name |+--------+----------+| 01 | zhangsan || 02 | lisi || 03 | wangwu |+--------+----------+3 rows in set (0.00 sec)

5.3 Hive 创建数据表（接收）

在 Hive 中创建 test 数据库和 student 数据表：

hive> create database test;.................OKTime taken: 6.211 secondshive> use test;OKTime taken: 0.018 secondshive> create table student(number STRING, name STRING) > row format delimited > fields terminated by "|" > stored as textfile; OKTime taken: 9.939

5.3 MySQL 导出到 Hive

需要说明该命令的以下几个参数：

–connect：MySQL 数据库连接 URL。–username&–password：MySQL 数据库的用户名和密码。–table：导出的数据表名。–fields-terminated-by：Hive 中字段分隔符。–delete-target-dir：删除导出目的目录。–num-mappers：Hadoop 执行 Sqoop 导入导出启动的 map 任务数。–hive-import --hive-database：导出到 Hive 的数据库名。–hive-table：导出到 Hive 的表名。

[hadoop@master ~]$ sqoop import --connect jdbc:mysql://server:3306/test --username root --password 123456 --table student --fields-terminated-by '|' --delete-target-dir --num-mappers 1 --hive-import --hive-database test --hive-table student..........22/02/25 20:02:10 INFO hive.HiveImport: OK22/02/25 20:02:10 INFO hive.HiveImport: Time taken: 10.6 seconds22/02/25 20:02:13 INFO hive.HiveImport: Loading data to table test.student22/02/25 20:02:14 INFO hive.HiveImport: OK22/02/25 20:02:14 INFO hive.HiveImport: Time taken: 3.846 seconds22/02/25 20:02:15 INFO hive.HiveImport: Hive import complete.22/02/25 20:02:15 INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory.

查看Hive表中的数据：

hive> show tables;OKstudentTime taken: 0.409 seconds, Fetched: 1 row(s)hive> select * from student;OK01 zhangsan02 lisi03 wangwuTime taken: 1.371 seconds, Fetched: 3 row(s)

​​返回顶部​​

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。