Kubernetes Addon gatekeeper使用

背景

在Kubernetes中对于集群使用规范是必然的。 但规划停留在纸面，依靠开发人员/运维人员手动确保合规性很容易出错。

将各自规范要求，落地到门禁规范中，自动化策略执行可确保一致性, 通过即时反馈降低开发延迟，并通过允许开发人员在不牺牲合规性的情况下独立操作来帮助提高敏捷性。

Gatekeeper 实现

Gatekeeper通过准入控制器 webhooks将策略决策与 API 服务器的内部工作分离，每当创建、更新或删除资源时都会执行这些决策。 Gatekeeper 是一个验证和变异的 Webhook，它强制执行由Open Policy Agent执行的基于 CRD 的策略，Open Policy Agent 是 CNCF 作为毕业项目托管的云原生环境的策略引擎(是一种基于CRD的标准)。

概念

约束 Constraints

约束模板 Template

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8scontainerlimits
  annotations:
    metadata.gatekeeper.sh/title: "Container Limits"
    metadata.gatekeeper.sh/version: 1.0.1
    description: >-
      Requires containers to have memory and CPU limits set and constrains
      limits to be within the specified maximum values.

      https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
spec:
  crd: # CRD注册
    spec:
      names:
        kind: K8sContainerLimits
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          type: object
          properties:
            exemptImages:
              description: >-
                Any container that uses an image that matches an entry in this list will be excluded
                from enforcement. Prefix-matching can be signified with `*`. For example: `my-image-*`.

                It is recommended that users use the fully-qualified Docker image name (e.g. start with a domain name)
                in order to avoid unexpectedly exempting images from an untrusted repository.
              type: array
              items:
                type: string
            cpu:
              description: "The maximum allowed cpu limit on a Pod, exclusive."
              type: string
            memory:
              description: "The maximum allowed memory limit on a Pod, exclusive."
              type: string
  targets: # 约束条件
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8scontainerlimits

        import data.lib.exempt_container.is_exempt

        missing(obj, field) = true {
          not obj[field]
        }

        missing(obj, field) = true {
          obj[field] == ""
        }

        canonify_cpu(orig) = new {
          is_number(orig)
          new := orig * 1000
        }

        canonify_cpu(orig) = new {
          not is_number(orig)
          endswith(orig, "m")
          new := to_number(replace(orig, "m", ""))
        }

        canonify_cpu(orig) = new {
          not is_number(orig)
          not endswith(orig, "m")
          regex.match("^[0-9]+(\\.[0-9]+)?$", orig)
          new := to_number(orig) * 1000
        }

        # 10 ** 21
        mem_multiple("E") = 1000000000000000000000 { true }

        # 10 ** 18
        mem_multiple("P") = 1000000000000000000 { true }

        # 10 ** 15
        mem_multiple("T") = 1000000000000000 { true }

        # 10 ** 12
        mem_multiple("G") = 1000000000000 { true }

        # 10 ** 9
        mem_multiple("M") = 1000000000 { true }

        # 10 ** 6
        mem_multiple("k") = 1000000 { true }

        # 10 ** 3
        mem_multiple("") = 1000 { true }

        # Kubernetes accepts millibyte precision when it probably shouldn't.
        # https://github.com/kubernetes/kubernetes/issues/28741
        # 10 ** 0
        mem_multiple("m") = 1 { true }

        # 1000 * 2 ** 10
        mem_multiple("Ki") = 1024000 { true }

        # 1000 * 2 ** 20
        mem_multiple("Mi") = 1048576000 { true }

        # 1000 * 2 ** 30
        mem_multiple("Gi") = 1073741824000 { true }

        # 1000 * 2 ** 40
        mem_multiple("Ti") = 1099511627776000 { true }

        # 1000 * 2 ** 50
        mem_multiple("Pi") = 1125899906842624000 { true }

        # 1000 * 2 ** 60
        mem_multiple("Ei") = 1152921504606846976000 { true }

        get_suffix(mem) = suffix {
          not is_string(mem)
          suffix := ""
        }

        get_suffix(mem) = suffix {
          is_string(mem)
          count(mem) > 0
          suffix := substring(mem, count(mem) - 1, -1)
          mem_multiple(suffix)
        }

        get_suffix(mem) = suffix {
          is_string(mem)
          count(mem) > 1
          suffix := substring(mem, count(mem) - 2, -1)
          mem_multiple(suffix)
        }

        get_suffix(mem) = suffix {
          is_string(mem)
          count(mem) > 1
          not mem_multiple(substring(mem, count(mem) - 1, -1))
          not mem_multiple(substring(mem, count(mem) - 2, -1))
          suffix := ""
        }

        get_suffix(mem) = suffix {
          is_string(mem)
          count(mem) == 1
          not mem_multiple(substring(mem, count(mem) - 1, -1))
          suffix := ""
        }

        get_suffix(mem) = suffix {
          is_string(mem)
          count(mem) == 0
          suffix := ""
        }

        canonify_mem(orig) = new {
          is_number(orig)
          new := orig * 1000
        }

        canonify_mem(orig) = new {
          not is_number(orig)
          suffix := get_suffix(orig)
          raw := replace(orig, suffix, "")
          regex.match("^[0-9]+(\\.[0-9]+)?$", raw)
          new := to_number(raw) * mem_multiple(suffix)
        }

        violation[{"msg": msg}] {
          general_violation[{"msg": msg, "field": "containers"}]
        }

        violation[{"msg": msg}] {
          general_violation[{"msg": msg, "field": "initContainers"}]
        }

        # Ephemeral containers not checked as it is not possible to set field.

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          not is_exempt(container)
          cpu_orig := container.resources.limits.cpu
          not canonify_cpu(cpu_orig)
          msg := sprintf("container <%v> cpu limit <%v> could not be parsed", [container.name, cpu_orig])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          not is_exempt(container)
          mem_orig := container.resources.limits.memory
          not canonify_mem(mem_orig)
          msg := sprintf("container <%v> memory limit <%v> could not be parsed", [container.name, mem_orig])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          not is_exempt(container)
          not container.resources
          msg := sprintf("container <%v> has no resource limits", [container.name])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          not is_exempt(container)
          not container.resources.limits
          msg := sprintf("container <%v> has no resource limits", [container.name])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          not is_exempt(container)
          missing(container.resources.limits, "cpu")
          msg := sprintf("container <%v> has no cpu limit", [container.name])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          not is_exempt(container)
          missing(container.resources.limits, "memory")
          msg := sprintf("container <%v> has no memory limit", [container.name])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          not is_exempt(container)
          cpu_orig := container.resources.limits.cpu
          cpu := canonify_cpu(cpu_orig)
          max_cpu_orig := input.parameters.cpu
          max_cpu := canonify_cpu(max_cpu_orig)
          cpu > max_cpu
          msg := sprintf("container <%v> cpu limit <%v> is higher than the maximum allowed of <%v>", [container.name, cpu_orig, max_cpu_orig])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          not is_exempt(container)
          mem_orig := container.resources.limits.memory
          mem := canonify_mem(mem_orig)
          max_mem_orig := input.parameters.memory
          max_mem := canonify_mem(max_mem_orig)
          mem > max_mem
          msg := sprintf("container <%v> memory limit <%v> is higher than the maximum allowed of <%v>", [container.name, mem_orig, max_mem_orig])
        }
      libs:
        - |
          package lib.exempt_container

          is_exempt(container) {
              exempt_images := object.get(object.get(input, "parameters", {}), "exemptImages", [])
              img := container.image
              exemption := exempt_images[_]
              _matches_exemption(img, exemption)
          }

          _matches_exemption(img, exemption) {
              not endswith(exemption, "*")
              exemption == img
          }

          _matches_exemption(img, exemption) {
              endswith(exemption, "*")
              prefix := trim_suffix(exemption, "*")
              startswith(img, prefix)
          }

参数配置 CRD

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sContainerLimits # 定义月嫂
metadata:
  name: container-must-have-limits
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    cpu: "200m"
    memory: "1Gi"

部署

dongjiang@MacBook Pro:~ $ helm install -n gatekeeper-system gatekeeper gatekeeper/gatekeeper --create-namespace

NAME: gatekeeper
LAST DEPLOYED: Wed Apr 24 13:53:16 2024
NAMESPACE: gatekeeper-system
STATUS: deployed
REVISION: 1
TEST SUITE: None

结果:

dongjiang@MacBook Pro:replicalimits $ kubectl get pods -n gatekeeper-system
NAME                                             READY   STATUS    RESTARTS   AGE
gatekeeper-audit-5b55979884-rg5z6                1/1     Running   0          64m
gatekeeper-controller-manager-69d88fcd4f-5kbjz   1/1     Running   0          64m
gatekeeper-controller-manager-69d88fcd4f-6xhnd   1/1     Running   0          64m
gatekeeper-controller-manager-69d88fcd4f-jkc6n   1/1     Running   0          64m

部署规则

部署 副本数限制 管理： https://github.com/open-policy-agent/gatekeeper-library/tree/master/library/general/replicalimits

dongjiang@MacBook Pro:replicalimits $ kubectl get crd | grep templates.gatekeeper
constrainttemplates.templates.gatekeeper.sh          2024-04-24T05:53:15Z
dongjiang@MacBook Pro:replicalimits $ kubectl get crd | grep constraints.gatekeeper
k8sreplicalimits.constraints.gatekeeper.sh           2024-04-24T07:03:28Z

dongjiang@MacBook Pro:replicalimits $ kubectl get k8sreplicalimits.constraints.gatekeeper.sh
NAME             ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
replica-limits                        5

  1 # Please edit the object below. Lines beginning with a '#' will be ignored,                                                                      
  2 # and an empty file will abort the edit. If an error occurs while saving this file will be
  3 # reopened with the relevant failures.
  4 #
  5 apiVersion: constraints.gatekeeper.sh/v1beta1
  6 kind: K8sReplicaLimits
  7 metadata:
  8   annotations:
  9     kubectl.kubernetes.io/last-applied-configuration: |
 10       {"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sReplicaLimits","metadata":{"annotations":{},"name":"replica-limits"},"spec":{"    match":{"kinds":[{"apiGroups":["apps"],"kinds":["Deployment"]}]},"parameters":{"ranges":[{"max_replicas":50,"min_replicas":3}]}}}
 11   creationTimestamp: "2024-04-24T07:04:00Z"
 12   generation: 1
 13   name: replica-limits
 14   resourceVersion: "2859691"
 15   uid: a933d3f5-e417-4bb4-89fe-bd8d40650a15
 16 spec:
 17   match:
 18     kinds:
 19     - apiGroups:
 20       - apps
 21       kinds:
 22       - Deployment
 23   parameters:
 24     ranges:
 25     - max_replicas: 50
 26       min_replicas: 3
 27 status:
 28   auditTimestamp: "2024-04-24T07:14:01Z"

验证方式

dongjiang@MacBook Pro:replicalimits $ kubectl apply -f example_disallowed.yaml
Error from server (Forbidden): error when creating "example_disallowed.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [replica-limits] The provided number of replicas is not allowed for Deployment: disallowed-deployment. Allowed ranges: {"ranges": [{"max_replicas": 50, "min_replicas": 3}]}

dongjiang@MacBook Pro:replicalimits $ kubectl apply -f example_allowed.yaml 
deployment.apps/allowed-deployment created

建议默认部署

dongjiang@MacBook Pro:general $ tree -L 1
.
|-- allowedrepos                        ✓
|-- automount-serviceaccount-token      ✓
|-- block-endpoint-edit-default-role    x
|-- block-loadbalancer-services         x
|-- block-nodeport-services             ✓
|-- block-wildcard-ingress              x
|-- containerlimits                     ✓
|-- containerrequests                   ✓
|-- containerresourceratios             ?
|-- containerresources                  ✓
|-- disallowanonymous                   ✓
|-- disallowedrepos
|-- disallowedtags
|-- disallowinteractive
|-- ephemeralstoragelimit
|-- externalip
|-- horizontalpodautoscaler             x
|-- httpsonly
|-- imagedigests                        ✓
|-- kustomization.yaml
|-- noupdateserviceaccount
|-- poddisruptionbudget
|-- replicalimits                       ✓
|-- requiredannotations
|-- requiredlabels
|-- requiredprobes
|-- storageclass
|-- uniqueingresshost
|-- uniqueserviceselector
`-- verifydeprecatedapi

高级

Rego 语言语法

Rego 语言语法

自定义： 约束 Constraints 和 约束模板 Template

我们对 pod volume size做一个范围limit限制，请参考：feat(general): Add volumeresources emptyDir sizelimit

其他

• Rego 语言语法
• OPA
• General策略
• google例子