FinOPS之 基于节点真实负载情况调度之二：crane-scheduler-plus

crane-scheduler-plus 独立于 Prometheus 实现版本

1. 背景

请查看第一篇：https://kubeservice.cn/2022/11/24/k8s-crane-scheduler-plus/

2. 组件介绍

Scheduler plus 是基于 Kubernetes 原生 Kube-scheduler Extender 机制实现的动态调度器插件，可基于 Node 真实负载进行预选和优选。在 集群中安装该插件后，该插件将与 Kube-scheduler 协同生效，有效避免原生调度器基于 request 和 limit 调度机制带来的节点负载不均问题。

该组件不再依赖 Prometheus 监控组件。并通过特定的node exporter plus实现对监控组件依赖

部署在集群内的 Kubernetes 对象

crane-scheduler-controller 组件负责定期从node-metrics节点负载 metric，同步到节点的 annotation。
crane-scheduler 是一个 scheduler-extender，根据 node annotation 负载数据，在节点预选和优选中进行过滤和评分计算。

3. 应用场景

Kubernetes 原生调度器大部分基于 Pod Request 资源进行调度，并不具备根据 Node 当前和过去一段时间的真实负载情况进行相关调度的决策，因此可能会导致如下问题：

• 集群内部分节点的剩余可调度资源较多（根据节点上运行的 Pod 的 request 和 limit 计算出的值）但真实负载却比较高，而另外节点的剩余可调度资源比较少但真实负载却比较低，此时 Kube-scheduler 会优先将 Pod 调度到剩余资源比较多的节点上（根据 LeastRequestedPriority 策略）。

如下图所示，Kube-Scheduler 会将 Pod 调度到 Node2 上，但明显调度到 Node1（真实负载水位更低）是更优的选择。

4. 防止调度热点

为防止低负载的节点被持续调度 Pod，Crane Scheduler 支持设置防调度热点策略（统计节点过去几分钟调度 Pod 的数量，并相应减小节点在优选阶段的评分）。
当前采取策略如下：

• 如果节点在过去1分钟调度了超过2个 Pod，则优选评分减去1分。
• 如果节点在过去5分钟调度了超过5个 Pod，则优选评分减去1分。

5. 组件原理

动态调度器基于 scheduler extender 扩展机制，从 node-metrics 中获取节点负载数据，开发基于节点实际负载的调度策略，在调度预选和优选阶段进行干预，优先将 Pod 调度到低负载节点上。该组件由 crane-scheduler-controller 和 crane-scheduler 构成。

5.1 crane-scheduler-controller

crane-scheduler-controller 组件负责定期从node-metrics中拉取节点负载 metric，同步到节点的 annotation。如下图所示：

! 组件删除后，crane-scheduler-controller 生成的 annotation 并不会被自动清除。您可根据需要手动清除。

5.2 crane-scheduler

crane-scheduler 是一个 scheduler-extender，根据 node annotation 负载数据，在节点预选和优选中进行过滤和评分计算。

• 5.2.1 预选策略

为了避免 Pod 调度到高负载的 Node 上，需要先通过预选过滤部分高负载的 Node（其中过滤策略和比例可以动态配置如下图所示，Node2 过去5分钟的负载，Node3 过去1小时的负载均超过对应的域值，因此不会参与接下来的优选阶段。

• 5.2.2 优选策略

同时为了使集群各节点的负载尽量均衡，Dynamic-scheduler 会根据 Node 负载数据进行打分，负载越低打分越高。
如下图所示，Node1 的打分最高将会被优先调度（其中打分策略和权重可以动态配置）。

5.3 组件参数说明

1. 预选参数-predicate

2. 优选参数-priority

6. 部署

6.1 安装Node-Metrics, 作为 node-exporter plus

确保您的 kubernetes 集群安装了Node-Metrics。如果没有，请参考Install Node-Metrics。

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: node-metrics
  name: node-metrics
  namespace: crane-system
spec:
  selector:
    matchLabels:
      app: node-metrics
  template:
    metadata:
      labels:
        app: node-metrics
    spec:
      containers:
      - image: dongjiang1989/node-metrics:latest
        name: node-metrics
        args:
        - --web.listen-address=0.0.0.0:19101
        resources:
          limits:
            cpu: 102m
            memory: 180Mi
          requests:
            cpu: 102m
            memory: 180Mi
      hostNetwork: true
      hostPID: true
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master

6.2 安装Scheduler和Controller

有两种选择:

1. Scheduler作为第二调度器:

   git clone git@github.com:kubeservice-stack/crane-scheduler.git
   cd crane-scheduler/deploy/deployment/
   kubectl apply -f .
   

2. 将原生 Kube-scheduler 替换为 scheduler:
   1. 备份 /etc/kubernetes/manifests/kube-scheduler.yaml

   cp /etc/kubernetes/manifests/kube-scheduler.yaml /etc/kubernetes/
   

   2. 修改 kube-scheduler( scheduler-config.yaml) 的配置文件以启用动态调度程序插件并配置插件参数：:

   apiVersion: kubescheduler.config.k8s.io/v1beta2
   kind: KubeSchedulerConfiguration
   ...
   profiles:
   - schedulerName: default-scheduler
     plugins:
       filter:
         enabled:
         - name: Dynamic
       score:
         enabled:
         - name: Dynamic
           weight: 3
     pluginConfig:
     - name: Dynamic
        args:
         policyConfigPath: /etc/kubernetes/policy.yaml
   ...
   

   3. 创建 /etc/kubernetes/policy.yaml, 用作动态插件的调度程序策略:

    apiVersion: scheduler.policy.crane.io/v1alpha1
    kind: DynamicSchedulerPolicy
    spec:
      syncPolicy:
        ##cpu usage
        - name: cpu_usage_avg_5m
          period: 3m
        - name: cpu_usage_max_avg_1h
          period: 15m
        - name: cpu_usage_max_avg_1d
          period: 3h
        ##memory usage
        - name: mem_usage_avg_5m
          period: 3m
        - name: mem_usage_max_avg_1h
          period: 15m
        - name: mem_usage_max_avg_1d
          period: 3h
   
      predicate:
        ##cpu usage
        - name: cpu_usage_avg_5m
          maxLimitPecent: 65
        - name: cpu_usage_max_avg_1h
          maxLimitPecent: 75
        ##memory usage
        - name: mem_usage_avg_5m
          maxLimitPecent: 65
        - name: mem_usage_max_avg_1h
          maxLimitPecent: 75
   
      priority:
        ##cpu usage
        - name: cpu_usage_avg_5m
          weight: 0.2
        - name: cpu_usage_max_avg_1h
          weight: 0.3
        - name: cpu_usage_max_avg_1d
          weight: 0.5
        ##memory usage
        - name: mem_usage_avg_5m
          weight: 0.2
        - name: mem_usage_max_avg_1h
          weight: 0.3
        - name: mem_usage_max_avg_1d
          weight: 0.5
   
      hotValue:
        - timeRange: 5m
          count: 5
        - timeRange: 1m
          count: 2
   

   4. 修改 kube-scheduler.yaml and 并替换 kube-scheduler 镜像为 Crane-scheduler：

   ...
    image: docker.io/dongjiang1989/crane-scheduler:main
   ...
   

   5. 安装 cheduler-controller:

   kubectl apply ./deploy/controller/rbac.yaml && kubectl apply -f ./deploy/controller/deployment.yaml
   

7. 验证和测试

7.1 测试 cpu 压力情况

使用以下示例测试调度程序:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cpu-stress
spec:
  selector:
    matchLabels:
      app: cpu-stress
  replicas: 1
  template:
    metadata:
      labels:
        app: cpu-stress
    spec:
      schedulerName: crane-scheduler
      hostNetwork: true
      tolerations:
      - key: node.kubernetes.io/network-unavailable
        operator: Exists
        effect: NoSchedule
      containers:
      - name: stress
        image: docker.io/dongjiang1989/stress:latest
        command: ["stress", "-c", "1"]
        resources:
          requests:
            memory: "1Gi"
            cpu: "1"
          limits:
            memory: "1Gi"
            cpu: "1"

Note: 更改crane-scheduler为default-scheduler用作默认值。.

如果测试 pod 调度成功，会有如下事件:

Events:
  Type    Reason     Age   From             Message
  ----    ------     ----  ----             -------
  Normal  Scheduled  91s   crane-scheduler  Successfully assigned default/cpu-stress-5c64f4d6fb-wnmsj to kcs-dongjiang-s-xtl6v
  Normal  Pulling    91s   kubelet          Pulling image "docker.io/dongjiang1989/stress:latest"
  Normal  Pulled     5s    kubelet          Successfully pulled image "docker.io/dongjiang1989/stress:latest" in 1m26.001017318s
  Normal  Created    5s    kubelet          Created container stress
  Normal  Started    5s    kubelet          Started container stress

4.2 测试Nginx rolling update案例

apiVersion: apps/v1                                                                                                                                         
kind: Deployment
metadata:
  name: nginxapp
  labels:
    app: nginxapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      schedulerName: crane-scheduler
      containers:
      - name: nginx
        image: nginx:1.11.9-alpine
        ports:
        - containerPort: 80

Note: 更改crane-scheduler为default-scheduler用作默认值。.

$ kubectl get deployment
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
nginxapp         3/3     3            3           1d
$ kubectl scale --replicas=30 deployment/nginxapp 
deployment.apps/nginxapp scaled

如果测试 pod 调度成功，会有如下事件:

Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   nginxapp-57bdf45cbf (30/30 replicas created)
Events:
  Type    Reason             Age                From                   Message
  ----    ------             ----               ----                   -------
  Normal  ScalingReplicaSet  56m                deployment-controller  Scaled up replica set nginxapp-57bdf45cbf to 13
  Normal  ScalingReplicaSet  56m                deployment-controller  Scaled down replica set nginxapp-664b9459f7 to 39
  Normal  ScalingReplicaSet  56m                deployment-controller  Scaled up replica set nginxapp-57bdf45cbf to 26
  Normal  ScalingReplicaSet  51m                deployment-controller  Scaled down replica set nginxapp-664b9459f7 to 3
  Normal  ScalingReplicaSet  51m                deployment-controller  Scaled down replica set nginxapp-57bdf45cbf to 2
  Normal  ScalingReplicaSet  47m                deployment-controller  Scaled down replica set nginxapp-664b9459f7 to 1
  Normal  ScalingReplicaSet  46m (x4 over 47m)  deployment-controller  (combined from similar events): Scaled up replica set nginxapp-57bdf45cbf to 40
  Normal  ScalingReplicaSet  2m52s              deployment-controller  Scaled up replica set nginxapp-57bdf45cbf to 43
  Normal  ScalingReplicaSet  94s                deployment-controller  Scaled down replica set nginxapp-57bdf45cbf to 3
  Normal  ScalingReplicaSet  37s                deployment-controller  Scaled up replica set nginxapp-57bdf45cbf to 30

8. 兼容性矩阵

 KUBE_EDITOR="sed -i 's/v1beta2/v1beta1/g'" kubectl edit cm scheduler-config -n crane-system && kubectl edit deploy crane-scheduler -n crane-system