nginx-ingress安装admission webhook增加对ingress资源校验能力

举报
可以交个朋友 发表于 2023/12/12 21:13:31 2023/12/12
【摘要】 nginx-ingress安装admission webhook增加对ingress资源校验能力

一、背景

       nginx-ingress作为南北向流量入口被广泛使用，某些情况下由于疏忽未检查ingress资源的合法性，可能导致ingress资源创建后，ingress-nginx容器中nginx一直reload，导致服务入口异常。

二、方案介绍

       在nginx-ingress中引入admission webhook，在配置到达ingress-nginx之前，对其的合法性进行校验。

       Admission webhook是K8S对请求资源对象的准入控制，Admission webhook有两种控制机制MutatingAdmissionWebhook和 ValidatingAdmissionWebhook，这两个控制器将发送准入请求的 HTTP 回调服务并接收一个准入响应。准入控制器是在对象持久化之前用于对 Kubernetes API Server 的请求进行拦截的代码段，在请求经过身份验证和授权之后放行通过。准入控制器可能正在 validating、 mutating或者都在执行，Mutating 控制器可以修改他们的处理的资源对象，Validating 控制器校验资源合法性，如果任何一个阶段中的任何控制器拒绝了请求，则会立即拒绝整个请求，并将错误返回给最终的用户。

  • MutatingAdmissionWebhook：对请求的对象进行修改，sidecar就是通过此机制实现对请求对象进行修改，如istio 就是通过这种webhook将envoy sidecar容器自动注入pod中的

  • ValidatingAdmissionWebhook：对请求资源对象的合法性进行检查，避免错误的对象被写入导致发生异常，例如ingress就是应用的这种webhook对请求资源进行合法性检查。

三、启用nginx-ingress admission webhook

3.1 helm部署nginx-ingress时开启webhook

      使用helm直接开启webhook能力

  • 在集群中安装helm3工具
# 先在集群中安装helm3部署工具（可忽略）
wget https://get.helm.sh/helm-v3.10.2-linux-amd64.tar.gz
tar -xzvf helm-v3.10.2-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/helm
# 验证helm正确安装，查看helm版本
helm version
version.BuildInfo{Version:"v3.10.2", GitCommit:"50f003e5ee8704ec937a756c646870227d7c8b58", GitTreeState:"clean", GoVersion:"go1.18.8"}

  • 下载ingress-nginx chart包

wget https://github.com/kubernetes/ingress-nginx/releases/download/helm-chart-4.3.0/ingress-nginx-4.3.0.tgz
tar -xf ./ingress-nginx-4.3.0.tgz
cd ./ingress-nginx
  • 创建配置文件，开启admissionWebhook配置
cat > config.yaml << EOF
controller:
  name: controller #自定义名称
# controller镜像地址设定
  image: 
    # name指定了nginx-ingress-controller工作负载的名称
    name: custom
    # repository指定了镜像地址，请根据集群所在region进行替换
    repository: swr.cn-east-3.myhuaweicloud.com/hwofficial/nginx-ingress   
    registry: ""
    image: ""
    # 镜像版本
    tag: "v1.2.1"
    digest: ""
# 设定ingress class和controller api，ingressClass和ingressClassResource.name的值要相同
  ingressClass: ingress-custom
  ingressClassResource:
    name: ingress-custom
    controllerValue: k8s.io/ingress-custom
# 设定controller要使用的svc类型。本例中使用的是LoadBalancer类型
  service: 
    # 注解中写入ELB ID
    # 本例中ELB ID已经省略中间部分，以实际使用的ELB ID为准
    type: LoadBalancer
    annotations:
      kubernetes.io/elb.class: performance
      kubernetes.io/elb.id: 3660...a481
      kubernetes.io/elb.health-check-flag: "on"
      kubernetes.io/elb.health-check-option: '{}'
    externalTrafficPolicy: Local
  resources:
    requests:
      cpu: 200m
      memory: 200Mi    
  config:
    keep-alive-requests: "100"
  admissionWebhooks:   # 开启admissionWebhook配置，helm安装后，自动安装
    enabled: true
    patch:
      enabled: true
      image:
        registry: registry.k8s.io  
        image: ingress-nginx/kube-webhook-certgen
        tag: v1.1.1    
        digest: ""
# 关闭defaultBackend的安装
defaultBackend: 
  enabled: false
EOF

3.2 在部署nginx-ingress后手动通过yaml部署webhook方式

     某些场景在一开始安装ingress-nginx时，没有开启webhook能力，需要手动后面补装，主要步骤如下：

  1. 创建ValidatingWebhookConfiguration资源，确保kube-apiserver收到ingress资源创建时可以回调到对应webhook服务
  2. 创建job生产证书，webhook服务必须是https，并且kube-apiserver调用webhoo服务是单向认证（客户端验证服务端证书）
  3. 创建相应的权限
  4. 为nginx-ingress的validating-webhook服务暴露service
  5. 修改nginx-ingress启动参数，增加validating-webhook能力，加载服务端证书和暴露相关服务端口
  • 创建相关资源，对应上述1-4步内容
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: admission-webhook
  name: ingress-nginx-admission
webhooks:
  - name: validate.nginx.ingress.kubernetes.io
    matchPolicy: Equivalent
    rules:
      - apiGroups:
          - networking.k8s.io
        apiVersions:
          - v1
        operations:
          - CREATE
          - UPDATE
        resources:
          - ingresses
    failurePolicy: Fail   #若admission调用失败(访问异常或处理异常)，Fail拒绝请求，Ignore则继续处理请求但跳过 Webhook
    sideEffects: None
    admissionReviewVersions:
      - v1
    clientConfig:
      service:
        namespace: kube-system
        name: ingress-nginx-controller-admission
        path: /networking/v1/ingresses
---
# Source: ingress-nginx/templates/controller-service-webhook.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller-admission
  namespace: kube-system
spec:
  type: ClusterIP
  ports:
    - name: https-webhook
      port: 443
      targetPort: webhook
      appProtocol: https
  selector:
    app: nginx-ingress
    component: controller
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-nginx-admission
  namespace: kube-system
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: admission-webhook
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: admission-webhook
rules:
  - apiGroups:
      - admissionregistration.k8s.io
    resources:
      - validatingwebhookconfigurations
    verbs:
      - get
      - update
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: admission-webhook
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: kube-system
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ingress-nginx-admission
  namespace: kube-system
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: admission-webhook
rules:
  - apiGroups:
      - ''
    resources:
      - secrets
    verbs:
      - get
      - create
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ingress-nginx-admission
  namespace: kube-system
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: admission-webhook
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: kube-system
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ingress-nginx-admission-create
  namespace: kube-system
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: admission-webhook
spec:
  template:
    metadata:
      name: ingress-nginx-admission-create
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/component: admission-webhook
    spec:
      containers:
        - name: create
          image: swr.cn-north-4.myhuaweicloud.com/hjmtest/kube-webhook-certgen:v1.1.1 #测试镜像（需自行更换）
          imagePullPolicy: IfNotPresent
          args:
            - create
            - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
            - --namespace=$(POD_NAMESPACE)
            - --secret-name=ingress-nginx-admission
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          securityContext:
            allowPrivilegeEscalation: false
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
      imagePullSecrets:
      - name: default-secret
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ingress-nginx-admission-patch
  namespace: kube-system
  annotations:
    helm.sh/hook: post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: admission-webhook
spec:
  template:
    metadata:
      name: ingress-nginx-admission-patch
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/component: admission-webhook
    spec:
      containers:
        - name: patch
          image: swr.cn-north-4.myhuaweicloud.com/hjmtest/kube-webhook-certgen:v1.1.1  #测试镜像（需自行更换）
          imagePullPolicy: IfNotPresent
          args:
            - patch
            - --webhook-name=ingress-nginx-admission
            - --namespace=$(POD_NAMESPACE)
            - --patch-mutating=false
            - --secret-name=ingress-nginx-admission
            - --patch-failure-policy=Fail
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          securityContext:
            allowPrivilegeEscalation: false
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
      imagePullSecrets:
      - name: default-secret
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
        fsGroup: 2000
  • 修改ingress-nginx 启动参数，对应第5步内容
# 修改ingress-nginx controller启动参数，新增validating-webhook、validating-webhook-certificate、validating-webhook-key
  - args:
        - /nginx-ingress-controller
        - --default-backend-service=kube-system/cceaddon-nginx-ingress-default-backend
        - --election-id=ingress-controller-leader
        - --ingress-class=nginx
        - --configmap=kube-system/cceaddon-nginx-ingress-controller
        - --publish-service=$(POD_NAMESPACE)/cceaddon-nginx-ingress-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert       
        - --validating-webhook-key=/usr/local/certificates/key

# 修改ingress-nginx controller port端口
         name: nginx-ingress-controller
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        - containerPort: 8443
          name: webhook
          protocol: TCP
        - containerPort: 10254
          name: prometheus
          protocol: TCP

# 挂载secrets作为服务端证书
        volumeMounts:
        - mountPath: /etc/localtime
          name: localtime
          readOnly: true
        - mountPath: /usr/local/certificates/
          name: webhook-cert
          readOnly: true
…
      volumes:
        - name: webhook-cert
          secret:
            secretName: ingress-nginx-admission
            defaultMode: 420
        - name: localtime
          hostPath:
            path: /etc/localtime
            type: ''

四、多nginx-ingress场景下，通过admissionwebhook作用域限制单个nginx-ingress webhook作用范围

       admission webhook作用域默认是全局的，如果一个集群中有多个nginx-ingress组件（通过ingressClass区域），由于有多个ValidatingAdmissionWebhook，当某个ingress资源创建时，kube-apiserver会调用多个nginx-ingress webhok能力进行校验，当某个nginx-ingress出现故障（可能与该ingress资源并不关联），将会阻塞所有的ingress的写入。

     多nginx-ingress场景下建议通过admissionwebhook 自带的namespaceSelector控制每个nginx-ingress的作用域，使其作用到指定的命名空间，只教验作用域命名空间下的ingresses资源。

 admissionwebhook：
   namespaceSelector: 
      matchExpressions:
      - key: kubernetes.io/metadata.name #需要教验的namespace含有的标签key
        operator: In
        values: ["${namespace}"] # namespace含有的标签volue（默认命名空间本身）
    rules:
    - operations: ["CREATE","UPDATE"]
      apiGroups: ["*"]
      apiVersions: ["*"]
      resources: ["ingresses"] #限制作用资源
      scope: "*" #由于限制了作用资源ingresses，无需配置
    objectSelector: {}


qrcode_for_gh_0412fa5a12f4_344 (5).jpg

订阅本文作者或关注容器魔方

获取更多云原生技术资讯

本文评论区回复“云原生”，即可添加小助手微信k8s2222

领取应季云原生资料一份

【版权声明】本文为华为云社区用户原创内容，转载时必须标注文章的来源（华为云社区）、文章链接、文章作者等基本信息， 否则作者和本社区有权追究责任。如果您发现本社区中有涉嫌抄袭的内容，欢迎发送邮件进行举报，并提供相关证据，一经查实，本社区将立刻删除涉嫌侵权内容，举报邮箱： cloudbbs@huaweicloud.com
Kubernetes 云原生 容器
  • 点赞
  • 收藏
  • 关注作者

评论（0）

0/1000
抱歉，系统识别当前为高风险访问，暂不支持该操作

全部回复

上滑加载中

设置昵称

在此一键设置昵称，即可参与社区互动！

*长度不超过10个汉字或20个英文字符，设置后3个月内不可修改。

*长度不超过10个汉字或20个英文字符，设置后3个月内不可修改。

确认 取消

加入云驻计划，成为创作者

  • 华为云周边好礼
  • 免费体验产品
  • 特殊身份标识
  • 线下官方门票
  • 内部专家零距离
  • 与10000+优质创作者共同成长
立即加入