关于 Kubernetes中Service使用Metallb实现LoadBalancer的一个Demo

写在前面

• 学习K8s中Service遇到，单独整理分享给小伙伴
• 博文内容涉及：
• Metallb的创建
• LoadBalancer类型的Service的创建

有些人心如花木，皆向阳而生 ——烽火戏诸侯《剑来》

Service 负载均衡问题

NodePort还没有完全解决外部访问Service的所有问题,比如负载均衡问题,假如我们的集群中有10个Node,则此时最好有一个负载均衡器,外部的请求只需访问此负载均衡器的IP地址,由负载均衡器负责转发流量到后面某个Node的NodePort上。如图

NodePort的负载均衡

LoadBalancer方式

Load balancer组件独立于Kubernetes集群之外,通常是一个硬件的负载均衡器,或者是以软件方式实现的,例如HAProxy或者Nginx。对于每个Service,我们通常需要配置一个对应的Load balancer实例来转发流量到后端的Node上

Kubernetes提供了自动化的解决方案,如果我们的集群运行在谷歌的GCE公有云上,那么只要我们把Service的type-NodePort改为type-LoadBalancer,此时Kubernetes会自动创建一个对应的Load balancer实例并返回它的IP地址供外部客户端使用。在没有公有云的情况下，我们也可以用一些插件来实现，如metallb等

LoadBalancer 需要建立服务之外的负载池。然后给Service分配一个IP。

我们直接创建一个LoadBalancer的Service的时候，会一直处于pending状态，是因为我们没有对应的云负载均衡器

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$kubectl expose  --name=blogsvc pod blog --port=80 --type=LoadBalancer
service/blogsvc exposed
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$kubectl  get svc -o wide | grep blogsvc
blogsvc   LoadBalancer   10.106.28.175   <pending>     80:32745/TCP   26s   run=blog
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$

Metallb创建

Metallb可以通过k8s原生的方式提供LB类型的Service支持

使用： metallb https://metallb.universe.tf/
资源文件https://github.com/metallb/metallb/blob/main/manifests/metallb.yaml

创建命名空间

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$kubectl  create  ns metallb-system
namespace/metallb-system created
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$kubectl  config set-context  $(kubectl config current-context) --namespace=metallb-system
Context "kubernetes-admin@kubernetes" modified.
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$

:set paste 解决粘贴混乱的问题

创建metallb

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl  apply  -f metallb.yaml
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl  get pods -o wide
NAME                         READY   STATUS    RESTARTS   AGE     IP               NODE                         NOMINATED NODE   READINESS GATES
controller-66d9554cc-8rxq8   1/1     Running   0          3m36s   10.244.171.170   vms82.liruilongs.github.io   <none>           <none>
speaker-bbl94                1/1     Running   0          3m36s   192.168.26.83    vms83.liruilongs.github.io   <none>           <none>
speaker-ckbzj                1/1     Running   0          3m36s   192.168.26.81    vms81.liruilongs.github.io   <none>           <none>
speaker-djmpr                1/1     Running   0          3m36s   192.168.26.82    vms82.liruilongs.github.io   <none>           <none>
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$

创建地址池- 192.168.26.240-192.168.26.250

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$vim pool.yaml
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl apply  -f pool.yaml
configmap/config created

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.26.240-192.168.26.250

使用type=LoadBalancer的配置通过metallb分配192.168.26.240这个地址给blogsvc

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl  get svc
No resources found in metallb-system namespace.
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl config  set-context  $(kubectl config current-context) --namespace=liruilong-svc-create
Context "kubernetes-admin@kubernetes" modified.

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl  get svc
NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
dbsvc   ClusterIP   10.102.137.59   <none>        3306/TCP   101m
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl  expose  --name=blogsvc pod blog --port=80 --type=LoadBalancer
service/blogsvc exposed
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl  get svc -o wide
NAME      TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)        AGE    SELECTOR
blogsvc   LoadBalancer   10.108.117.197   192.168.26.240   80:30230/TCP   9s     run=blog
dbsvc     ClusterIP      10.102.137.59    <none>           3306/TCP       101m   run=dbpod

直接访问192.168.26.240就可以了

在创建一个也可以访问

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl  expose  --name=blogsvc-1 pod blog --port=80 --type=LoadBalancer
service/blogsvc-1 exposed
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl  get svc -o wide
NAME        TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)        AGE    SELECTOR
blogsvc     LoadBalancer   10.108.117.197   192.168.26.240   80:30230/TCP   11m    run=blog
blogsvc-1   LoadBalancer   10.110.58.143    192.168.26.241   80:31827/TCP   3s     run=blog
dbsvc       ClusterIP      10.102.137.59    <none>           3306/TCP       113m   run=dbpod
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$

也可以访问

相关的资源文件

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$cat namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: metallb-system
  labels:
    app: metallb

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$cat pool.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.26.240-192.168.26.250

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$cat metallb.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  labels:
    app: metallb
  name: controller
  namespace: metallb-system
spec:
  allowPrivilegeEscalation: false
  allowedCapabilities: []
  allowedHostPaths: []
  defaultAddCapabilities: []
  defaultAllowPrivilegeEscalation: false
  fsGroup:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  hostIPC: false
  hostNetwork: false
  hostPID: false
  privileged: false
  readOnlyRootFilesystem: true
  requiredDropCapabilities:
  - ALL
  runAsUser:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  volumes:
  - configMap
  - secret
  - emptyDir
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  labels:
    app: metallb
  name: speaker
  namespace: metallb-system
spec:
  allowPrivilegeEscalation: false
  allowedCapabilities:
  - NET_RAW
  allowedHostPaths: []
  defaultAddCapabilities: []
  defaultAllowPrivilegeEscalation: false
  fsGroup:
    rule: RunAsAny
  hostIPC: false
  hostNetwork: true
  hostPID: false
  hostPorts:
  - max: 7472
    min: 7472
  - max: 7946
    min: 7946
  privileged: true
  readOnlyRootFilesystem: true
  requiredDropCapabilities:
  - ALL
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - configMap
  - secret
  - emptyDir
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: metallb
  name: controller
  namespace: metallb-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: metallb
  name: speaker
  namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: metallb
  name: metallb-system:controller
rules:
- apiGroups:
  - ''
  resources:
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ''
  resources:
  - services/status
  verbs:
  - update
- apiGroups:
  - ''
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - policy
  resourceNames:
  - controller
  resources:
  - podsecuritypolicies
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: metallb
  name: metallb-system:speaker
rules:
- apiGroups:
  - ''
  resources:
  - services
  - endpoints
  - nodes
  verbs:
  - get
  - list
  - watch
- apiGroups: ["discovery.k8s.io"]
  resources:
  - endpointslices
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ''
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - policy
  resourceNames:
  - speaker
  resources:
  - podsecuritypolicies
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app: metallb
  name: config-watcher
  namespace: metallb-system
rules:
- apiGroups:
  - ''
  resources:
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app: metallb
  name: pod-lister
  namespace: metallb-system
rules:
- apiGroups:
  - ''
  resources:
  - pods
  verbs:
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app: metallb
  name: controller
  namespace: metallb-system
rules:
- apiGroups:
  - ''
  resources:
  - secrets
  verbs:
  - create
- apiGroups:
  - ''
  resources:
  - secrets
  resourceNames:
  - memberlist
  verbs:
  - list
- apiGroups:
  - apps
  resources:
  - deployments
  resourceNames:
  - controller
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: metallb
  name: metallb-system:controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metallb-system:controller
subjects:
- kind: ServiceAccount
  name: controller
  namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: metallb
  name: metallb-system:speaker
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metallb-system:speaker
subjects:
- kind: ServiceAccount
  name: speaker
  namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app: metallb
  name: config-watcher
  namespace: metallb-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: config-watcher
subjects:
- kind: ServiceAccount
  name: controller
- kind: ServiceAccount
  name: speaker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app: metallb
  name: pod-lister
  namespace: metallb-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-lister
subjects:
- kind: ServiceAccount
  name: speaker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app: metallb
  name: controller
  namespace: metallb-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: controller
subjects:
- kind: ServiceAccount
  name: controller
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: metallb
    app.kubernetes.io/component: speaker
  name: speaker
  namespace: metallb-system
spec:
  selector:
    matchLabels:
      app: metallb
      app.kubernetes.io/component: speaker
  template:
    metadata:
      annotations:
        prometheus.io/port: '7472'
        prometheus.io/scrape: 'true'
      labels:
        app: metallb
        app.kubernetes.io/component: speaker
    spec:
      containers:
      - args:
        - --port=7472
        - --config=config
        - --log-level=info
        env:
        - name: METALLB_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: METALLB_HOST
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: METALLB_ML_BIND_ADDR
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        # needed when another software is also using memberlist / port 7946
        # when changing this default you also need to update the container ports definition
        # and the PodSecurityPolicy hostPorts definition
        #- name: METALLB_ML_BIND_PORT
        #  value: "7946"
        - name: METALLB_ML_LABELS
          value: "app=metallb,app.kubernetes.io/component=speaker"
        - name: METALLB_ML_SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: memberlist
              key: secretkey
        image: quay.io/metallb/speaker:main
        name: speaker
        ports:
        - containerPort: 7472
          name: monitoring
        - containerPort: 7946
          name: memberlist-tcp
        - containerPort: 7946
          name: memberlist-udp
          protocol: UDP
        livenessProbe:
          httpGet:
            path: /metrics
            port: monitoring
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /metrics
            port: monitoring
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 3
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_RAW
            drop:
            - ALL
          readOnlyRootFilesystem: true
      hostNetwork: true
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: speaker
      terminationGracePeriodSeconds: 2
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: metallb
    app.kubernetes.io/component: controller
  name: controller
  namespace: metallb-system
spec:
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: metallb
      app.kubernetes.io/component: controller
  template:
    metadata:
      annotations:
        prometheus.io/port: '7472'
        prometheus.io/scrape: 'true'
      labels:
        app: metallb
        app.kubernetes.io/component: controller
    spec:
      containers:
      - args:
        - --port=7472
        - --config=config
        - --log-level=info
        env:
        - name: METALLB_ML_SECRET_NAME
          value: memberlist
        - name: METALLB_DEPLOYMENT
          value: controller
        image: quay.io/metallb/controller:main
        name: controller
        ports:
        - containerPort: 7472
          name: monitoring
        livenessProbe:
          httpGet:
            path: /metrics
            port: monitoring
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /metrics
            port: monitoring
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 3
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - all
          readOnlyRootFilesystem: true
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        fsGroup: 65534
      serviceAccountName: controller
      terminationGracePeriodSeconds: 0