背景

Nginx Ingress作为K8s流量入口，可以通过域名和path路径转发流量至不同的后端pod。nginx-ingress暴露nginx_ingress_controller_requests指标，以此指标可以采集到不同域名访问的QPS值，可以根据该指标作为对应后端工作负载HPA策略，实现根据域名的访问QPS对关联后端实例进行HPA。

简介

本方案基于开源nginx-ingress默认提供的nginx_ingress_controller_requests指标，该指标包含的标签可以区分每个域名的访问量，再通过Prometheus-adapter转化成QPS指标，HPA利用QPS指标对域名关联的后端工作负载做弹性策略
注意：由于HPA规则中scaleTargetRef和describedObject两个字段关联的资源都无法指定命名空间，所以指标来源、HPA和弹性目标需在同一命名空间，而nginx-ingress和业务工作负载一般处在不同命名空间，本次方案将采用external类型的HPA，可以忽略指标来源的命名空间

操作步骤

1. 创建演示需要的弹性目标工作负载，service以及ingress

   apiVersion: apps/v1 
   kind: Deployment 
   metadata: 
     name: app1
     labels: 
       app: app1 
   spec: 
     replicas: 1 
     selector: 
       matchLabels: 
         app: app1
     template: 
       metadata: 
         labels: 
           app: app1
       spec: 
         containers: 
         - image: nginx:v1
           name: nginx
           ports: 
           - name: http 
             containerPort: 80 
   --- 
   apiVersion: v1 
   kind: Service 
   metadata: 
     name: app1-service 
     namespace: default 
     labels: 
       app: app1-service 
   spec: 
     ports: 
       - port: 8080 
         name: http 
         protocol: TCP 
         targetPort: 80
     selector: 
       app: app1
     type: ClusterIP 
   
   --- 
   apiVersion: apps/v1 
   kind: Deployment 
   metadata: 
     name: app2
     labels: 
       app: app2
   spec: 
     replicas: 1 
     selector: 
       matchLabels: 
         app: app2
     template: 
       metadata: 
         labels: 
           app: app2 
       spec: 
         containers: 
         - image: nginx:v2
           name: nginx
           ports: 
           - name: http 
             containerPort: 80
   --- 
   apiVersion: v1 
   kind: Service 
   metadata: 
     name: app2-service
     namespace: default 
     labels: 
       app: app2-service
   spec: 
     ports: 
       - port: 80 
         name: http 
         protocol: TCP 
         targetPort: 80
     selector: 
       app: app2
     type: ClusterIP 
   --- 
   apiVersion: networking.k8s.io/v1 
   kind: Ingress 
   metadata: 
     name: test-hpa
     namespace: default 
   spec: 
     ingressClassName: nginx
     rules:
     - host: www.test-hpa.com
       http:
         paths:
           - path: /
             pathType: ImplementationSpecific
             backend:
               service:
                 name: app1-service
                 port:
                   number: 8080
             property:
               ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
           - path: /home/
             pathType: ImplementationSpecific
             backend:
               service:
                 name: app2-service
                 port:
                   number: 80
   

2. 分别查询www.test-hpa.com/和www.test-hpa.com/home/的nginx_ingress_controller_requests指标，指标正常
   
   

3. 创建external类型的apiservices资源；创建后apiservices的状态为false是正常现象，添加externalRules后状态变为true

   apiVersion: apiregistration.k8s.io/v1 
   kind: APIService 
   metadata: 
     name: v1beta1.external.metrics.k8s.io 
   spec: 
     group: external.metrics.k8s.io 
     groupPriorityMinimum: 100 
     insecureSkipTLSVerify: true 
     service:         #指定prometheus-adapter对应的service，华为CCE插件中adapter名称为custom-metrics-apiserver 
       name: custom-metrics-apiserver 
       namespace: monitoring 
       port: 443 
     version: v1beta1 
     versionPriority: 100
   

4. 将externalRules规则添加到adapter的configmap中，修改后需要重启prometheus-adapter服务

   kubectl -n monitoring edit configmap user-adapter-config
   

   

   externalRules: 
   - metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>) 
     name: 
       as: ${1}_per_second 
       matches: ^(.*) 
     resources: 
       namespaced: false  #忽略指标来源的命名空间，该配置不适用rules规则 
     seriesQuery: nginx_ingress_controller_requests
   

   seriesQuery：原始指标；可以直接写指标名称，也可以使用{labelKey=labelValue}的方式筛选出原始指标
   metricsQuery：用PromQL对指标进行筛选汇聚；.Series表示原始指标，.LabelMatchers表示对指标进行标签筛选，hpa中可以配置具体的筛选规则
   name：将指标重命名
   resources：hpa查询指标时通过api的方式调用，调用路径为：$\color{orange}{/apis/external.metrics.k8s.io/v1beta1/namespaces/\$\{namespace\}/nginx\_ingress\_controller\_requests\_per\_second}$，而resources的作用就是将指标中命名空间标签的值替换路径中的${namespace}，而我们本次方案需要忽略指标来源命名空间。

5. custom-metrics-apiserver服务重启后需要等待1分钟左右，执行命令查看指标是否正常

   kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/*/nginx_ingress_controller_requests_per_second
   

   

6. 创建HPA规则

   apiVersion: autoscaling/v2 
   kind: HorizontalPodAutoscaler 
   metadata: 
     name: app1-hpa
   spec: 
     scaleTargetRef: 
       apiVersion: apps/v1 
       kind: Deployment 
       name: app1 
     minReplicas: 1 
     maxReplicas: 10 
     metrics: 
       - type: External 
         external: 
           metric: 
             name: nginx_ingress_controller_requests_per_second 
             selector: 
               matchLabels:    #可以通过该字段对指标进行过滤，这里标签筛选条件会加入到externalRules的<<.LabelMatchers>>中。 
                 exported_service: app1-service  #筛选后端服务为sample-app的请求 
                 host: www.test-hpa.com        #筛选访问域名为test.example.com的请求 
           target: 
             type: AverageValue   #External指标类型下只支持Value和AverageValue类型的目标值。 
             averageValue: 30 
   --- 
   apiVersion: autoscaling/v2 
   kind: HorizontalPodAutoscaler 
   metadata: 
     name: app2-hpa 
   spec: 
     scaleTargetRef: 
       apiVersion: apps/v1 
       kind: Deployment 
       name: app2
     minReplicas: 1 
     maxReplicas: 10 
     metrics: 
       - type: External 
         external: 
           metric: 
             name: nginx_ingress_controller_requests_per_second 
             selector: 
               matchLabels: 
                 exported_service: app2-service 
                 host: www.test-hpa.com
           target: 
             type: AverageValue 
             averageValue: 30
   

   

弹性演示

1. 使用命令压测app1对应的访问域名和路径，正常触发弹性；请自行配置域名与ELB地址的映射

   ab -c 50 -n 5000 http://www.test-hpa.com/
   

   

2. 用同样的方式压测app2，正常触发弹性

   ab -c 50 -n 5000 http://www.test-hpa.com/home/