开源flink1.14提交任务到mrs315安全集群

1、安装flink：tar -zxvf flink-1.13.5-bin-scala_2.11.tgz

flink1.14.6下载地址：https://archive.apache.org/dist/flink/flink-1.14.6/flink-1.14.6-bin-scala_2.11.tgz

2、将MRS的core-site.xml、hdfs-site.xml、yarn-site.xml加入到 flink/conf/ 下

3、修改flink-conf.yaml文件

添加以下配置：

#可以参考mrs flink客户端的flink-conf.yaml对应的配置项

env.java.opts.jobmanager: -Djava.security.krb5.conf=/opt/Bigdata/FusionInsight_BASE_8.1.2.5/1_5_KerberosClient/etc/kdc.conf

env.java.opts.taskmanager: -Djava.security.krb5.conf=/opt/Bigdata/FusionInsight_BASE_8.1.2.5/1_5_KerberosClient/etc/kdc.conf

security.kerberos.login.keytab: /root/wwk/user.keytab #修改为对应的keytab
security.kerberos.login.principal: flinkuser #修改为对应的用户名
env.java.opts: -Djava.security.krb5.conf=/root/ krb5.conf -Djava.library.path=${HADOOP_COMMON_HOME}/lib/native #修改为对应的krb5文件路径

security.kerberos.login.use-ticket-cache: true
classloader.check-leaked-classloader: false
security.kerberos.login.contexts: Client,KafkaClient
zookeeper.sasl.service-name: zookeeper
zookeeper.sasl.login-context-name: Client

4、如果flink需要开启高可用，需要执行以下步骤

a、需要在flink-conf.yaml加入以下配置

high-availability.job.delay: 10 s
high-availability.storageDir: hdfs:///flink/recovery
high-availability.zookeeper.client.acl: creator
high-availability.zookeeper.client.connection-timeout: 15000
high-availability.zookeeper.client.max-retry-attempts: 3
high-availability.zookeeper.client.retry-wait: 5000
high-availability.zookeeper.client.session-timeout: 60000
high-availability.zookeeper.path.root: /flink
high-availability.zookeeper.quorum: 192.168.0.35:2181,192.168.0.112:2181,192.168.0.210:2181
high-availability: zookeeper

zookeeper.clientPort.quorum: 192.168.0.35:2181,192.168.0.112:2181,192.168.0.210:2181
zookeeper.root.acl: OPEN

zookeeper.sasl.disable: false

zookeeper.secureClientPort.quorum:  192.168.0.35:2181,192.168.0.112:2181,192.168.0.210:2181

zookeeper.server.principal: zookeeper/hadoop.XXXXXXXX.com@XXXXXX.COM

b、替换flink/lib下的flink-shaded-zookeeper-3.4.14.jar包

将flink-shaded-zookeeper-3.4.14.jar包换成flink-shaded-zookeeper-3-3.5.9-14.0.jar

5、设置HADOOP_CLASSPATH

可以直接source /opt/Bigdata/client/bigdata_env,没有客户端时可以执行以下命令进行设置：

export HADOOP_CLASSPATH=/opt/hadoopclient/HDFS/hadoop/etc/hadoop:/opt/hadoopclient/HDFS/hadoop/share/hadoop/common/lib/*:/opt/hadoopclient/HDFS/hadoop/share/hadoop/common/*:/opt/hadoopclient/HDFS/hadoop/share/hadoop/hdfs:/opt/hadoopclient/HDFS/hadoop/share/hadoop/hdfs/lib/*:/opt/hadoopclient/HDFS/hadoop/share/hadoop/hdfs/*:/opt/hadoopclient/HDFS/hadoop/share/hadoop/mapreduce/*:/opt/hadoopclient/HDFS/hadoop/share/hadoop/yarn/lib/*:/opt/hadoopclient/HDFS/hadoop/share/hadoop/yarn/*:/opt/hadoopclient/HDFS/hadoop/share/hadoop/tools/lib/hadoop-huaweicloud-3.1.1-hw-42.jar:/opt/hadoopclient/HDFS/hadoop/share/hadoop/tools/lib/hadoop-huawei-obscommitter-3.1.1-hw-ei-310013.jar:/opt/hadoopclient/HDFS/hadoop/share/hadoop/tools/lib/hadoop-trace-ping-3.1.1-hw-ei-310013.jar

6、测试flink任务

运行任务时日志会出现error，但是不影响