MRS试用（三）在ExaGear环境配置大数据

    MRS基于鲲鹏处理器进行软硬件垂直优化，充分释放硬件算力，实现高性价比。但是我想尝试在鲲鹏版的MRS里通过exagear搭建大数据平台，实现一套鲲鹏云服务器能支持2套大数据平台（arm和x86），为了避免冲突，不能同时使用，只能使用其中一套。而使用exagear环境的只能使用本套服务器的百分之八十的性能。

    在exagear环境配置ssh无秘钥登录，查看是有.ssh目录，没有就创建一个并进其目录。各节点：ssh-keygen -t rsa，出现bash: ssh-keygen: command not found提示，说明没有安装ssh，yum -y install openssh 安装ssh-copy-id node-master1jXHa.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com提示bash: ssh-copy-id: command not found，需要安装yum -y install openssh-clients

ssh-copy-id ip

**注意，生成authorized_keys后，要给authorized_keys分配权限。

chmod 700 ~/.ssh

chmod 600 ~/.ssh/authorized_keys

    之前在虚拟机已经搭建过大数据平台，把相关软件打包上传华为云服务器，在做相应配置的修改即可。

1.Zookeeper分布式集群部署

1). 修改配置

rm -rf /opt/modules/zookeeper/zkData

mkdir -p /opt/modules/zookeeper/zkData

vi /opt/modules/zookeeper/conf/zoo.cfg

server.1=node-master1jXHa.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:2888:3888

server.2=node-master2qxne.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:2888:3888

server.3=node-ana-coreygal.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:2888:3888

server.4=node-ana-coreRDZO.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:2888:3888

server.5=node-ana-corewaOy.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:2888:3888

2).测试Zookeeper服务

(1）各个节点使用如下命令启动Zookeeper服务

cd /opt/modules/zookeeper/ 

/opt/modules/zookeeper/bin/zkServer.sh start

(2）查看各个节点服务状态

/opt/modules/zookeeper/bin/zkServer.sh status

2.hadoop集群配置

1)配置java_home路径

hadoop-env.sh

yarn-env.sh

mapred-env.sh

rm -rf /opt/modules/hadoop/data

mkdir -p /opt/modules/hadoop/data/tmp

mkdir -p /opt/modules/hadoop/data/jn

2)配置core-site.xml

vi /opt/modules/hadoop/etc/hadoop/core-site.xml

    <property>

        <name>hadoop.http.staticuser.user</name>

        <value>root</value>

    </property>

         <property>

        <name>ha.zookeeper.quorum</name>

        <value>node-master1jXHa.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:2181,node-master2qxne.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:2181,node-ana-coreygal.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:2181,node-ana-coreRDZO.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:2181,node-ana-corewaOy.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:2181</value>

    </property>

         <property>

        <name>hadoop.proxyuser.root.hosts</name>

        <value>*</value>

    </property>

    <property>

        <name>hadoop.proxyuser.root.groups</name>

        <value>*</value>

    </property>

3)配置hdfs-site.xml  #<value>1</value>  副本数量1。

vi /opt/modules/hadoop/etc/hadoop/hdfs-site.xml

    <property>

       <name>dfs.namenode.rpc-address.ns.nn1</name>

       <value>node-master1jXHa.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:8020</value>

    </property>

    <property>

       <name>dfs.namenode.rpc-address.ns.nn2</name>

       <value>node-master2qxne.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:8020</value>

    </property>

    <property>

       <name>dfs.namenode.http-address.ns.nn1</name>

       <value>node-master1jXHa.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:50070</value>

    </property>

    <property>

       <name>dfs.namenode.http-address.ns.nn2</name>

       <value>node-master2qxne.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:50070</value>

    </property>

    <property>

       <name>dfs.namenode.shared.edits.dir</name>

       <value>qjournal://node-master1jXHa.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:8485;node-master2qxne.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:8485;node-ana-coreygal.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:8485;node-ana-coreRDZO.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:8485;node-ana-corewaOy.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:8485/ns</value>

    </property>

    <property>

      <name>dfs.ha.fencing.ssh.private-key-files</name>

      <value>/root/.ssh/id_rsa</value>

    </property>

4)配置slaves

vi /opt/modules/hadoop/etc/hadoop/slaves

node-ana-coreygal.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com

node-ana-coreRDZO.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com

node-ana-corewaOy.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com

5）配置yarn

vi /opt/modules/hadoop/etc/hadoop/yarn-site.xml

    <property>

        <name>yarn.resourcemanager.hostname.rm1</name>

        <value>node-master1jXHa.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com</value>

    </property>

    <property>

        <name>yarn.resourcemanager.hostname.rm2</name>

        <value>node-master2qxne.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com</value>

    </property>

    <property>

        <name>yarn.resourcemanager.webapp.address.rm1</name>

        <value>node-master1jXHa.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:8088</value>

    </property>

    <property>

        <name>yarn.resourcemanager.webapp.address.rm2</name>

        <value>node-master2qxne.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:8088</value>

    </property>

    <property>

        <name>yarn.resourcemanager.zk-address</name>

        <value>node-master1jXHa.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:2181,node-master2qxne.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:2181,node-ana-coreygal.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:2181,node-ana-coreRDZO.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:2181,node-ana-corewaOy.80e11d55-8b1a-44d1-b1e0-2d74d2daccb6.com:2181</value>

    </property>

         <property>

        <name>yarn.scheduler.minimum-allocation-mb</name>

        <value>8192</value>

        <description>default value is 1024</description>

    </property>

    原本已经配置好，但是由于在.bashrc里注销source /root/env_file和添加了一行代码source /opt/exagear/images/centos-7-x86_64/root/.bashrc，导致master1主机无法登陆。需要提单解决。等master主机弄好后，继续搭建。