MRS使用之控制impala并发度

一、impalad配置文件添加fair-scheduler.xml和llama-site.xml

备注：所有节点都需要配置，否则修改完impalad服务的--fair_scheduler_allocation_path、--llama_site_path配置后，没有该文件的节点会启动失败。

添加位置：/opt/Bigdata/MRS_2.1.1/1_17_Impalad/etc/

修改相应的权限：

chmod 600 fair-scheduler.xml
chown omm:wheel llama-site.xml
chmod 600 llama-site.xml
chown omm:wheel llama-site.xml

Example（impalad节点4U16G）：

fair-scheduler.xml

<allocations>
    <queue name="root">
        <aclSubmitApps> </aclSubmitApps>
        <queue name="default">
            <maxResources>5120 mb, 0 vcores</maxResources>
            <aclSubmitApps>*</aclSubmitApps>
        </queue>
        <queue name="development">
            <maxResources>2048 mb, 0 vcores</maxResources>
            <aclSubmitApps>admin</aclSubmitApps>
        </queue>
        <queue name="production">
            <maxResources>7168 mb, 0 vcores</maxResources>
            <aclSubmitApps>omm</aclSubmitApps>
        </queue>
    </queue>
    <queuePlacementPolicy>
        <rule name="specified" create="false"/>
        <rule name="default" />
    </queuePlacementPolicy>
</allocations>

 llama-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <property>
    <name>llama.am.throttling.maximum.placed.reservations.root.development</name>
    <value>1</value>
  </property>
  <property>
    <name>llama.am.throttling.maximum.queued.reservations.root.development</name>
    <value>2</value>
  </property>
  <property>
    <name>impala.admission-control.pool-default-query-options.root.development</name>
    <value>mem_limit=128m,query_timeout_s=20,max_io_buffers=10</value>
  </property>
  <property>
    <name>impala.admission-control.pool-queue-timeout-ms.root.development</name>
    <value>30000</value>
  </property>
  <property>
    <name>impala.admission-control.max-query-mem-limit.root.development.regularPool</name>
    <value>4026531840</value><!--3.75GB-->
  </property>
  <property>
    <name>impala.admission-control.min-query-mem-limit.root.development.regularPool</name>
    <value>52428800</value><!--50MB-->
  </property>
  <property>
    <name>impala.admission-control.clamp-mem-limit-query-option.root.development.regularPool</name>
    <value>true</value>
  </property>
</configuration>

二、impalad配置文件中添加这两个文件的位置

--fair_scheduler_allocation_path：/opt/Bigdata/MRS_2.1.1/1_17_Impalad/etc/fair-scheduler.xml

--llama_site_path：/opt/Bigdata/MRS_2.1.1/1_17_Impalad/etc/llama-site.xml

修改完成后保存配置并重启服务

三、并发提交impala查询进行测试

admin用户提交到root.development队列，用法：impala-shell -i ${impalad} -f ${query_file}-Q request_pool=root.development &

• llama-site.xml中的配置控制并发度
  

如llama-site.xml所示root.development队列running的SQL个数不超过1个，queued的SQL个数不超过2个，如上如所示，按照配置生效了。

• 多个队列同时存在
  

• 拦截无权限提交队列的用户

  
  

参考文档：https://impala.apache.org/docs/build/html/topics/impala_admission_config.html