flink从1.10版本升级到1.13.1版本通过flink sink往oss上写数据报错处理

flink从1.10版本升级到1.13.1版本通过flink sink往oss上写数据报错处理

因为需要往oss上数据，发现flink上的报错如下：

java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter at com.aliyun.emr.jfs.flink.sink.common.fs.AbstractFlinkJindoFileSystemFactory.create(AbstractFlinkJindoFileSystemFactory.java:56) at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:506) at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:407) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBucketWriter(StreamingFileSink.java:288) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:298) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:469) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:189) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:171) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96) at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:118) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:290) at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:436) at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:582) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.call(StreamTaskActionExecutor.java:100) at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:562) at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647) at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:537) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 20 more

报错的问题

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter报错的原因是缺少FileOutputCommitter类

解决方案：

1.查看老环境中哪两个jar包里有这个类

[root@dbos-bigdata-flink005 lib]# grep FileOutputCommitter *Binary file flink-shaded-hadoop-2-uber-2.6.5-10.0.jar matchesBinary file jindofs-sdk.jar matches

2.查看新环境中少了flink-shaded-hadoop-2-uber-2.6.5-10.0.jar

[root@dbos-bigdata-flink004 lib]# lltotal 198168-rwxrwxrwx 1 root root 92311 May 25 20:32 flink-csv-1.13.1.jar-rwxrwxrwx 1 root root 115530972 May 25 20:36 flink-dist_2.11-1.13.1.jar-rwxrwxrwx 1 root root 148131 May 25 20:32 flink-json-1.13.1.jar-rwxrwxrwx 1 root root 7709740 Apr 8 2021 flink-shaded-zookeeper-3.4.14.jar-rwxrwxrwx 1 root root 36417228 May 25 20:35 flink-table_2.11-1.13.1.jar-rwxrwxrwx 1 root root 40965908 May 25 20:35 flink-table-blink_2.11-1.13.1.jar-rwxrwxrwx 1 root root 67114 Oct 10 2019 log4j-1.2-api-2.12.1.jar-rwxrwxrwx 1 root root 276771 Oct 10 2019 log4j-api-2.12.1.jar-rwxrwxrwx 1 root root 1674433 Oct 10 2019 log4j-core-2.12.1.jar-rwxrwxrwx 1 root root 23518 Oct 10 2019 log4j-slf4j-impl-2.12.1.jar

3.把此jar放入到flink/lib的文件夹下面后，重新提交flink任务，可以正常使用了。

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。