Hadoop 问题小结

Hadoop 问题小结

任务无法提交

排查

登录master节点服务，yarn.scheduler.capacity.maximum-am-resource-percent 配置为 0.5 ，表示core的资源使用率不能超过50%

[root@ads-flink-online-master ~]# cat /etc/hadoop/conf.empty/capacity-scheduler.xml yarn.scheduler.capacity.maximum-am-resource-percent 0.5 Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running applications.

解决

将yarn.scheduler.capacity.maximum-am-resource-percent改为0.8

[root@ads-flink-online-master ~]# vim /etc/hadoop/conf.empty/capacity-scheduler.xml yarn.scheduler.capacity.maximum-am-resource-percent 0.8 Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running applications.

Hdfs ACL授权失败

报错信息：setfacl: The ACL operation has been rejected. Support for ACLs has been disabled by setting dfs.namenode.acls.enabled to false.

解决方法：在hdfs-site.xml文件中添加如下配置即可：

dfs.namenode.acls.enabled true

重启服务：

hadoop-hdfs-namenode.service

hdfs权限问题

Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hue_product, access=WRITE, inode="/user":hdfs:hdfsadmingroup:drwxr-xr-x

解决方法：

##检查目录权限 [root@emr001 conf]# sudo -u hdfs hadoop fs -ls /user Found 9 items drwxr-xr-x - hadoop hdfsadmingroup 0 2022-08-04 04:26 /user/hadoop drwxr-xr-x - mapred mapred 0 2022-08-04 04:26 /user/history drwxr-xr-x - hdfs hdfsadmingroup 0 2022-08-04 08:59 /user/hive drwxr-xr-x - hue hue 0 2022-08-04 04:26 /user/hue drwxr-xr-x - livy livy 0 2022-08-04 07:48 /user/livy drwxr-xr-x - oozie oozie 0 2022-08-04 04:27 /user/oozie drwxr-xr-x - root hdfsadmingroup 0 2022-08-04 04:50 /user/root drwxr-xr-x - spark spark 0 2022-08-04 04:26 /user/spark drwxr-xr-x - zeppelin hdfsadmingroup 0 2022-08-04 04:26 /user/zeppelin ##将目录权限设置成777，1表示stick bit(不能删除别人创建的文件/文件夹) sudo -u hdfs hadoop fs -chmod -R 1777 /user

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。