Yarn 跑 Flink job报错 标配
【摘要】 Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exceptionflinkCaused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.Pathorg.apache.hadoop.ipc.RemoteException: org.ap...
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exception
flinkCaused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.Path
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /user/zkx/.flink/application_1616591628688_0001. Name node is in safe mode.
Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /user/zkx/.flink/application_1616591628688_0001. Name node is in safe mode.
[root@node01 ~]# find / -name hadoop-yarn-api-3.1.2.jar
[root@node01 ~]# ll
total 4
drwxr-xr-x 2 root root 6 Oct 21 11:04 -
-rw-------. 1 root root 1334 Sep 20 2020 anaconda-ks.cfg
drwxr-xr-x 3 root root 22 Dec 4 21:36 datas
drwxr-xr-x 2 root root 6 Oct 21 11:04 p
[root@node01 ~]# find / -name hadoop-yarn-api-*.jar
/opt/module/hadoop-2.7.5/share/hadoop/yarn/hadoop-yarn-api-2.7.5.jar
/opt/module/hadoop-2.7.5/share/hadoop/yarn/sources/hadoop-yarn-api-2.7.5-test-sources.jar
/opt/module/hadoop-2.7.5/share/hadoop/yarn/sources/hadoop-yarn-api-2.7.5-sources.jar
/opt/module/hbase-1.3.1/lib/hadoop-yarn-api-2.5.1.jar
/opt/module/kylin-2.5.1/spark/jars/hadoop-yarn-api-2.7.3.jar
/opt/module/spark-2.4.4-bin-hadoop2.7/jars/hadoop-yarn-api-2.7.3.jar
[root@node01 ~]# cd /opt/module/flink-1.12.1/
You have new mail in /var/spool/mail/root
[root@node01 flink-1.12.1]# cd /opt/module/hadoop-2.7.5/share/hadoop/yarn/
[root@node01 yarn]# scp -r hadoop-yarn-api-2.7.5.jar /opt/module/flink-1.12.1/li
lib/ licenses/
[root@node01 yarn]# scp -r hadoop-yarn-api-2.7.5.jar /opt/module/flink-1.12.1/lib/
You have new mail in /var/spool/mail/root
[root@node01 yarn]# cd sources/
[root@node01 sources]# scp -r hadoop-yarn-api-2.7.5-sources.jar /opt/module/flink-1.12.1/li
lib/ licenses/
[root@node01 sources]# scp -r hadoop-yarn-api-2.7.5-sources.jar /opt/module/flink-1.12.1/lib/
[root@node01 sources]#2021-03-24 21:40:31,863 ERROR org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Error while running the Flink session.
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/Path
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.getLocalFlinkDistPathFromCmd(FlinkYarnSessionCli.java:347) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.applyDescriptorOptionToConfig(FlinkYarnSessionCli.java:473) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.toConfiguration(FlinkYarnSessionCli.java:394) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:571) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:860) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:860) [flink-dist_2.11-1.12.1.jar:1.12.1]
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.Path
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_144]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_144]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) ~[?:1.8.0_144]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_144]
... 7 more
------------------------------------------------------------
The program finished with the following exception:
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/Path
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.getLocalFlinkDistPathFromCmd(FlinkYarnSessionCli.java:347)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.applyDescriptorOptionToConfig(FlinkYarnSessionCli.java:473)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.toConfiguration(FlinkYarnSessionCli.java:394)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:571)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:860)
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:860)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.Path
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:425) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:606) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:860) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_144]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754) ~[flink-shaded-hadoop-2-uber-2.7.5-8.0.jar:2.7.5-8.0]
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:860) [flink-dist_2.11-1.12.1.jar:1.12.1]
Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The cluster does not have the requested resources for the JobManager available!
Maximum Memory: 1024MB Requested: 1600MB. Please check the 'yarn.scheduler.maximum-allocation-mb' and the 'yarn.nodemanager.resource.memory-mb' configuration values
at org.apache.flink.yarn.YarnClusterDescriptor.validateClusterResources(YarnClusterDescriptor.java:644) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:575) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:418) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
... 7 more
------------------------------------------------------------
The program finished with the following exception:
org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:425)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:606)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:860)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:860)
Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The cluster does not have the requested resources for the JobManager available!
Maximum Memory: 1024MB Requested: 1600MB. Please check the 'yarn.scheduler.maximum-allocation-mb' and the 'yarn.nodemanager.resource.memory-mb' configuration values
at org.apache.flink.yarn.YarnClusterDescriptor.validateClusterResources(YarnClusterDescriptor.java:644)
at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:575)
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:418)
... 7 more
开关安全模式、查看安全模式状态：
hdfs dfsadmin -safemode get //查看状态
hdfs dfsadmin -safemode enter //进入安全模式
hdfs dfsadmin -safemode get //查看状态
hdfs dfsadmin -safemode leave //退出安全模式
hdfs dfsadmin -safemode get //查看安全模式
————————————————
[root@node01 ~]# hdfs dfsadmin -safemode get
Safe mode is ON in node01/192.168.10.81:8020
Safe mode is ON in node02/192.168.10.82:8020
[root@node01 ~]# hdfs dfsadmin -safemode enter
Safe mode is ON in node01/192.168.10.81:8020
Safe mode is ON in node02/192.168.10.82:8020
[root@node01 ~]# hdfs dfsadmin -safemode get
Safe mode is ON in node01/192.168.10.81:8020
Safe mode is ON in node02/192.168.10.82:8020
[root@node01 ~]# hdfs dfsadmin -safemode leave
Safe mode is OFF in node01/192.168.10.81:8020
Safe mode is OFF in node02/192.168.10.82:8020
[root@node01 ~]# hdfs dfsadmin -safemode get
Safe mode is OFF in node01/192.168.10.81:8020
Safe mode is OFF in node02/192.168.10.82:8020
[root@node01 ~]#
org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:425) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:606) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:860) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_144]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754) ~[flink-shaded-hadoop-2-uber-2.7.5-8.0.jar:2.7.5-8.0]
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:860) [flink-dist_2.11-1.12.1.jar:1.12.1]
Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The cluster does not have the requested resources for the JobManager available!
Maximum Memory: 1024MB Requested: 1600MB. Please check the 'yarn.scheduler.maximum-allocation-mb' and the 'yarn.nodemanager.resource.memory-mb' configuration values
at org.apache.flink.yarn.YarnClusterDescriptor.validateClusterResources(YarnClusterDescriptor.java:644) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:575) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:418) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
... 7 more
------------------------------------------------------------
The program finished with the following exception:
org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:425)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:606)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:860)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:860)
Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The cluster does not have the requested resources for the JobManager available!
Maximum Memory: 1024MB Requested: 1600MB. Please check the 'yarn.scheduler.maximum-allocation-mb' and the 'yarn.nodemanager.resource.memory-mb' configuration values
at org.apache.flink.yarn.YarnClusterDescriptor.validateClusterResources(YarnClusterDescriptor.java:644)
at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:575)
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:418)
... 7 more
集群没有JobManager所请求的资源可用!
最大内存:1024MB请求:1600MB。请检查Please check the 'yarn.scheduler.maximum-allocation-mb' and the 'yarn.nodemanager.resource.memory-mb' configuration values
修改为2048
切记修改配置需要重启yarn
启动成功!
/opt/module/flink-1.12.1/bin/flink run -m yarn-cluster -yjm 1024mb -ytm 1024mb /opt/module/flink-1.12.1/examples/streaming/WordCount.jar
【版权声明】本文为华为云社区用户原创内容，未经允许不得转载，如需转载请自行联系原作者进行授权。如果您发现本社区中有涉嫌抄袭的内容，欢迎发送邮件进行举报，并提供相关证据，一经查实，本社区将立刻删除涉嫌侵权内容，举报邮箱：
cloudbbs@huaweicloud.com
- 点赞
- 收藏
- 关注作者
评论（0）