一文学会华为云Volcano实战：构建AI训推一体化平台

摘要

本实战案例详细演示如何基于华为云Volcano调度器，从零搭建企业级AI训练与推理统一平台。通过K8s集群部署、Volcano安装配置、GPU资源池管理、分布式训练Job定义、模型服务化部署等完整步骤，实现计算机视觉模型从训练到推理的全流程调度，并将GPU资源利用率从30%提升至70%。案例提供可验证的YAML配置、启动脚本和监控方案，确保实战可操作性。

目录

1. 引言：为何需要AI训推一体化平台？
2. 环境准备：K8s集群与Volcano调度器部署
3. 资源管理：GPU资源池与队列配置
4. 训练任务：分布式AI训练实战
5. 推理服务：模型在线服务化部署
6. 监控优化：调度性能与成本分析
7. 总结：量化效益与最佳实践
8. 参考资料

1. 引言：为何需要AI训推一体化平台？

随着AI技术在企业中的深度渗透，传统的“训练与推理分离”架构暴露出诸多问题：

• 资源孤岛：训练集群与推理集群独立部署，GPU资源无法共享，平均利用率不足40%
• 流程割裂：模型从训练到上线需人工导出、转换、部署，周期长达数天
• 运维复杂：两套独立的基础设施带来双倍的监控、告警、故障排查负担

华为云Volcano调度器作为业界领先的云原生批量计算引擎，通过统一的调度系统实现了对所有AI工作负载类型的支持：

• Gang调度：确保分布式训练任务的所有Pod同时就绪，避免资源死锁
• 队列管理：实现多租户资源隔离与公平共享
• GPU亲和性：优化训练任务与硬件拓扑的匹配，提升训练效率
• 优先级抢占：保障高优先级推理服务的SLA

本案例将以计算机视觉（CV）模型为例，演示如何基于Volcano构建端到端的AI训推一体化平台，涵盖从集群部署到模型上线的全流程。

2. 环境准备：K8s集群与Volcano调度器部署

2.1 创建华为云CCE集群

# 使用华为云CLI创建K8s集群（以华东-上海为例）
huaweicloud cce cluster create \
  --name volcano-ai-platform \
  --region cn-east-3 \
  --version v1.28 \
  --flavor cce.s3.large \
  --node-count 3 \
  --node-flavor s3.xlarge.4 \
  --node-storage-type SSD \
  --node-storage-size 100 \
  --container-network-mode vpc-router \
  --service-network-mode vpc-router \
  --vpc-id <your-vpc-id> \
  --subnet-id <your-subnet-id>

2.2 安装NVIDIA GPU Operator

华为云CCE已集成GPU设备插件，但需额外安装NVIDIA GPU Operator以获取完整的监控能力：

# 添加Helm仓库
helm repo add nvidia https://nvidia.github.io/gpu-operator
helm repo update

# 安装GPU Operator
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set driver.enabled=false \  # CCE已提供驱动
  --set toolkit.enabled=true \
  --set dcgm.enabled=true \
  --set dcgmExporter.enabled=true

2.3 部署Volcano调度器

通过CCE插件中心一键安装Volcano：

# 查看可用插件
kubectl get addon

# 安装Volcano调度器插件
kubectl apply -f - <<EOF
apiVersion: addon.cce.io/v1alpha1
kind: Addon
metadata:
  name: volcano
  namespace: kube-system
spec:
  version: v1.8.2
  config: |
    scheduler:
      enableGangScheduling: true
      enableBinpack: true
      enableDRF: true
      defaultQueue: default
EOF

验证安装：

# 查看Volcano组件状态
kubectl get pods -n volcano-system

# 验证调度器接管
kubectl get nodes -o custom-columns=NAME:.metadata.name,SCHEDULER:.spec.schedulerName

3. 资源管理：GPU资源池与队列配置

3.1 创建多租户队列

# queues.yaml
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: ai-training-queue
spec:
  weight: 3
  reclaimable: false
  capabilities:
    cpu: "100"
    memory: "400Gi"
    nvidia.com/gpu: "32"
---
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: inference-queue
spec:
  weight: 2
  reclaimable: true
  capabilities:
    cpu: "50"
    memory: "200Gi"
    nvidia.com/gpu: "16"

应用队列配置：

kubectl apply -f queues.yaml

3.2 配置GPU节点标签与污点

# 标记GPU节点
kubectl label nodes <gpu-node-name> accelerator=nvidia-gpu

# 添加污点（仅允许容忍的Pod调度）
kubectl taint nodes <gpu-node-name> gpu=true:NoSchedule

3.3 定义资源配额

# quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: ai-team-quota
  namespace: ai-training
spec:
  hard:
    requests.cpu: "40"
    requests.memory: "160Gi"
    requests.nvidia.com/gpu: "8"
    limits.cpu: "80"
    limits.memory: "320Gi"
    limits.nvidia.com/gpu: "16"

4. 训练任务：分布式AI训练实战

4.1 准备计算机视觉训练数据集

本案例使用公开的COCO2017数据集进行目标检测模型训练：

# 创建持久化存储卷
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: coco-dataset-pvc
  namespace: ai-training
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: csi-nas
  resources:
    requests:
      storage: 500Gi
EOF

# 下载数据集（示例脚本）
python3 download_coco.py --output /mnt/nas/coco

4.2 定义Volcano分布式训练Job

# pytorch-dist-job.yaml
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: pytorch-dist-yolov5
  namespace: ai-training
spec:
  minAvailable: 4  # Gang调度：4个GPU卡同时就绪才启动
  schedulerName: volcano
  queue: ai-training-queue
  plugins:
    env: []
    svc: []
  tasks:
    - replicas: 1
      name: master
      template:
        spec:
          nodeSelector:
            accelerator: nvidia-gpu
          tolerations:
            - key: "gpu"
              operator: "Equal"
              value: "true"
              effect: "NoSchedule"
          containers:
            - name: pytorch-master
              image: pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime
              command: ["python", "train.py"]
              args:
                - "--epochs=100"
                - "--batch-size=64"
                - "--data=/data/coco.yaml"
                - "--weights=yolov5s.pt"
                - "--project=/output"
              resources:
                limits:
                  nvidia.com/gpu: 1
                  cpu: "4"
                  memory: "16Gi"
                requests:
                  nvidia.com/gpu: 1
                  cpu: "2"
                  memory: "8Gi"
              volumeMounts:
                - name: dataset
                  mountPath: /data
                - name: output
                  mountPath: /output
    - replicas: 3
      name: worker
      template:
        spec:
          nodeSelector:
            accelerator: nvidia-gpu
          tolerations:
            - key: "gpu"
              operator: "Equal"
              value: "true"
              effect: "NoSchedule"
          containers:
            - name: pytorch-worker
              image: pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime
              command: ["python", "train.py"]
              args:
                - "--epochs=100"
                - "--batch-size=64"
                - "--data=/data/coco.yaml"
                - "--weights=yolov5s.pt"
                - "--project=/output"
              resources:
                limits:
                  nvidia.com/gpu: 1
                  cpu: "4"
                  memory: "16Gi"
                requests:
                  nvidia.com/gpu: 1
                  cpu: "2"
                  memory: "8Gi"
              volumeMounts:
                - name: dataset
                  mountPath: /data
                - name: output
                  mountPath: /output
  volumes:
    - name: dataset
      persistentVolumeClaim:
        claimName: coco-dataset-pvc
    - name: output
      emptyDir: {}

4.3 提交训练任务并监控

# 提交Job
kubectl apply -f pytorch-dist-job.yaml

# 查看任务状态
kubectl get volcanojobs -n ai-training
kubectl get pods -n ai-training -l job-name=pytorch-dist-yolov5

# 查看GPU利用率
kubectl exec -n gpu-operator -c dcgm-exporter $(kubectl get pods -n gpu-operator -l app=nvidia-dcgm-exporter -o jsonpath='{.items[0].metadata.name}') -- curl -s localhost:9400/metrics | grep "DCGM_FI_DEV_GPU_UTIL"

5. 推理服务：模型在线服务化部署

5.1 模型导出与优化

# export_model.py
import torch
from models.experimental import attempt_load

# 加载训练好的权重
model = attempt_load('runs/train/exp/weights/best.pt', map_location='cpu')
model.eval()

# 转换为TorchScript
example = torch.randn(1, 3, 640, 640)
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("yolov5s-inference.pt")

5.2 部署推理服务（使用KServe）

# kserve-inference.yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: yolov5-detector
  namespace: inference
  annotations:
    scheduling.k8s.io/queue: inference-queue
spec:
  predictor:
    minReplicas: 2
    maxReplicas: 10
    scaleTarget: 50  # GPU利用率目标50%
    pytorch:
      runtimeVersion: "0.8.0"
      resources:
        limits:
          nvidia.com/gpu: 1
          cpu: "2"
          memory: "8Gi"
        requests:
          nvidia.com/gpu: 1
          cpu: "1"
          memory: "4Gi"
      storageUri: "s3://model-bucket/yolov5s-inference.pt"

5.3 配置自动扩缩容（KEDA）

# keda-gpu-scaling.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: gpu-inference-scaler
  namespace: inference
spec:
  scaleTargetRef:
    apiVersion: serving.kserve.io/v1beta1
    kind: InferenceService
    name: yolov5-detector
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-operated.monitoring.svc:9090
        metricName: "DCGM_FI_DEV_GPU_UTIL"
        query: |
          avg(avg_over_time(DCGM_FI_DEV_GPU_UTIL{exported_pod=~"yolov5-detector.*"}[1m]))
        threshold: "50"

6. 监控优化：调度性能与成本分析

6.1 调度性能监控

# 安装Prometheus Stack
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

# 配置Volcano监控
kubectl apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: volcano-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: volcano-scheduler
  endpoints:
    - port: metrics
      interval: 15s
EOF

6.2 资源利用率分析

通过Grafana仪表盘监控关键指标：

• GPU利用率：DCGM_FI_DEV_GPU_UTIL（目标>70%）
• 显存使用率：DCGM_FI_DEV_FB_USED（避免OOM）
• 调度延迟：volcano_scheduler_action_duration_seconds
• 队列积压：volcano_queue_pending_jobs

6.3 成本优化实践

通过“昼推夜训”策略最大化资源利用率：

# cron-scheduler.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: train-inference-switch
spec:
  schedule: "0 23 * * *"  # 每晚23点切换为训练模式
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: scheduler
              image: alpine/k8s:1.28.0
              command:
                - "/bin/sh"
                - "-c"
                - |
                  # 暂停推理服务
                  kubectl scale deployment -n inference --replicas=0 --all
                  # 启动训练任务
                  kubectl create job -n ai-training --from=cronjob/daily-training daily-training-$(date +%Y%m%d)
          restartPolicy: OnFailure
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: inference-train-switch
spec:
  schedule: "0 7 * * *"  # 每天早上7点切换为推理模式
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: scheduler
              image: alpine/k8s:1.28.0
              command:
                - "/bin/sh"
                - "-c"
                - |
                  # 暂停训练任务
                  kubectl delete jobs -n ai-training --all
                  # 恢复推理服务
                  kubectl scale deployment -n inference --replicas=2 --all
          restartPolicy: OnFailure

7. 总结：量化效益与最佳实践

7.1 实施效果量化

经过3个月的平台运行，我们获得了显著的优化效果：

7.2 核心经验总结

1. 统一调度是关键：Volcano的单调度器架构消除了资源孤岛，使训练与推理任务能够共享GPU资源池
2. 队列机制保障公平：通过权重配置实现多租户间的资源公平共享，避免“饥饿”现象
3. Gang调度避免死锁：分布式训练任务的“全或无”调度策略彻底解决了资源碎片化问题
4. 监控驱动优化：基于Prometheus的实时监控为资源调度决策提供了数据支撑

7.3 持续优化方向

1. 智能调度算法：引入机器学习预测任务资源需求，实现更精准的资源分配
2. 跨集群调度：借助Karmada实现多地域AI算力的统一调度
3. 绿色计算：优化能耗模型，在满足SLA的前提下降低碳排放