GPU裸金属服务器使用Node Exporter实现RDMA网卡指标监控可视化方案

0. 前置条件

裸金属服务器需要安装nvidia-driver、nvidia-cuda、nvidia-fabric-manager、promtheus、docker软件包;

上述软件安装方法可参考：GPU Ant8裸金属服务器装机和验证-云社区-华为云 (huaweicloud.com)

promtheus和docker安装可以参考: https://bbs.huaweicloud.com/blogs/399818

1. 运行Node Exporter

docker run -d -p 9100:9100 prom/node-exporter

2. 配置Prometheus

将Node expertor的指标获取端口9100添加到Prometheus配置文件中

vim /usr/local/prometheus/prometheus.yml

global:
  scrape_interval: 15s # 采集间隔
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['xx.xx.xx.xx:9100']  # Node-Exporter指标获取端口，替换xx.xx.xx.xx为节点的IP地址

3. 重启Prometheus

a. 找出Prometheus的container ID

docker ps | grep prometheus
>755c1573602c    prom/prometheus   "/bin/prometheus --c…"   2 days ago          Up 2 days

b. 杀掉Prometheus

docker kill 755c1573602c

c. 启动Prometheus

docker run -d \
    -p 9090:9090 \
    -v /usr/local/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
    prom/prometheus

4. 制作Grafana报表

通过 node_infiniBand_port_data_received_bytes_total 和 node_infiniBand_port_data_transmitted_bytes_total 这两个指标来计算N卡的RoCE上行流量和下行流量，详细配置如下：

a. 上线流量

# 查询语句
rate(node_infiniband_port_data_transmitted_bytes_total[$__rate_interval])
# Legend（建议）
node_infiniband_port_data_transmitted_rate.{{device}}.{{port}}

b. 下行流量

# 查询语句
rate(node_infiniband_port_data_received_bytes_total[$__rate_interval])
# Legend（建议）
node_infiniband_port_data_received_rate.{{device}}.{{port}}

c. 效果展示