kubernetes 1.20 二进制方式高可用部署之部署Worker Node（5）

六、部署Worker Node

6.1 创建工作目录并拷贝文件

--------------------node1、node2节点-------------------
[root@k8s-node1 ~]# mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}

#master将kubelet和kube-proxy拷贝给node1、node2节点
[root@k8s-master k8s]# cd ~/kubernetes/server/bin/
[root@k8s-master bin]# scp kubelet kube-proxy k8s-node1:/opt/kubernetes/bin/

#本地拷贝
[root@k8s-master1 bin]# cp kubelet kube-proxy /opt/kubernetes/bin

上传到/opt/kubernetes/bin下

6.2 部署kubelet

----------------------下面这些操作在master节点完成：---------------------------
将kubelet-bootstrap用户绑定到系统集群角色
[root@k8s-master1 ~]# /opt/kubernetes/bin/kubectl create clusterrolebinding kubelet-bootstrap \
  --clusterrole=system:node-bootstrapper \
  --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created

创建kubeconfig文件:
在生成kubernetes证书的目录下执行以下命令生成kubeconfig文件：
[root@k8s-master1 ~]# cd /opt/crt/
指定apiserver 内网负载均衡地址
[root@k8s-master1 crt]# KUBE_APISERVER="https://192.168.246.162:6443"  #写你master的ip地址，集群中就写负载均衡的ip地址
[root@k8s-master1 crt]# BOOTSTRAP_TOKEN=674c457d4dcf2eefe4920d7dbb6b0ddc

# 设置集群参数
[root@k8s-master1 crt]# /opt/kubernetes/bin/kubectl config set-cluster kubernetes \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig

# 设置客户端认证参数
[root@k8s-master crt]# /opt/kubernetes/bin/kubectl config set-credentials kubelet-bootstrap \
  --token=${BOOTSTRAP_TOKEN} \
  --kubeconfig=bootstrap.kubeconfig

# 设置上下文参数
[root@k8s-master crt]# /opt/kubernetes/bin/kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

# 设置默认上下文
[root@k8s-master crt]# /opt/kubernetes/bin/kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

#====================================================================================

# 创建kube-proxy kubeconfig文件

[root@k8s-master1 crt]# /opt/kubernetes/bin/kubectl config set-cluster kubernetes \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig

[root@k8s-master1 crt]# /opt/kubernetes/bin/kubectl config set-credentials kube-proxy \
  --client-certificate=kube-proxy.pem \
  --client-key=kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

[root@k8s-master1 crt]# /opt/kubernetes/bin/kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

[root@k8s-master1 crt]# /opt/kubernetes/bin/kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig


[root@k8s-master1 crt]# ls
bootstrap.kubeconfig  kube-proxy.kubeconfig

#必看：将这两个文件拷贝到Node节点/opt/kubernetes/cfg目录下。
[root@k8s-master1 crt]# scp *.kubeconfig k8s-node1:/opt/kubernetes/cfg/
[root@k8s-master1 crt]# scp *.kubeconfig k8s-node2:/opt/kubernetes/cfg/

下面这些操作在node节点完成

部署kubelet组件
#将前面下载的二进制包中的kubelet和kube-proxy拷贝到/opt/kubernetes/bin目录下。
将master上面的包拷贝过去
[root@k8s-master1 ~]# scp kubernetes-server-linux-amd64.tar.gz k8s-node1:/root/
[root@k8s-master1 ~]# scp kubernetes-server-linux-amd64.tar.gz k8s-node2:/root/
[root@k8s-node1 ~]# tar xzf kubernetes-server-linux-amd64.tar.gz
[root@k8s-node1 ~]# cd kubernetes/server/bin/
[root@k8s-node1 bin]# cp kubelet kube-proxy /opt/kubernetes/bin/
#=====================================================================================
在两个node节点创建kubelet配置文件：
[root@k8s-node1 ~]# vim /opt/kubernetes/cfg/kubelet
KUBELET_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.1.2* \   #每个节点自己的ip地址
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet.config \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"  #这个镜像需要提前下载

[root@k8s-node1 ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
[root@k8s-node2 ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0

参数说明：

• –hostname-override 在集群中显示的主机名
• –kubeconfig 指定kubeconfig文件位置，会自动生成
• –bootstrap-kubeconfig 指定刚才生成的bootstrap.kubeconfig文件
• –cert-dir 颁发证书存放位置
• –pod-infra-container-image 管理Pod网络的镜像

其中/opt/kubernetes/cfg/kubelet.config配置文件如下：

[root@k8s-node1 ~]# vim /opt/kubernetes/cfg/kubelet.config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 192.168.1.2*   #写你机器的ip地址
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS: ["10.0.0.2"]      #不要改，就是这个ip地址
clusterDomain: cluster.local.
failSwapOn: false
authentication:
  anonymous:
    enabled: true
  webhook:
    enabled: false
    
systemd管理kubelet组件：
# vim /usr/lib/systemd/system/kubelet.service 
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet
ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target

启动：
# systemctl daemon-reload
# systemctl enable kubelet
# systemctl start kubelet
[root@k8s-master ~]# /opt/kubernetes/bin/kubectl get csr
NAME                                                   AGE       REQUESTOR           CONDITION
node-csr-F5AQ8SeoyloVrjPuzSbzJnFKQaUsier7EGvNFXLKTqM   17s       kubelet-bootstrap   Pending
node-csr-bjeHSWXOuUDSHganJPL_hDz_8jjYhM2FQyTkbA9pM0Q   18s       kubelet-bootstrap   Pending

在Master审批Node加入集群：
启动后还没加入到集群中，需要手动允许该节点才可以。在Master节点查看请求签名的Node：
[root@k8s-master1 ~]# /opt/kubernetes/bin/kubectl certificate approve XXXXID
注意：xxxid 指的是上面的NAME这一列
[root@k8s-master1 ~]# /opt/kubernetes/bin/kubectl get csr
NAME                                                   AGE       REQUESTOR           CONDITION
node-csr--1TVDzcozo7NoOD3WS2t9xLQqNunsVXj_i2AQ5x1mbs   1m        kubelet-bootstrap   Approved,Issued
node-csr-L0wqvr69oy8rzXwFm1u1uNx4aEMOOvd_RWPxaAERn_w   27m       kubelet-bootstrap   Approved,Issued


查看集群节点信息：
[root@k8s-master1 ~]# /opt/kubernetes/bin/kubectl get node
NAME              STATUS    ROLES     AGE       VERSION
192.168.1.2*   Ready     <none>    1m        v1.11.10
192.168.246.165   Ready     <none>    17s       v1.11.10

6.3 部署kube-proxy组件

创建kube-proxy配置文件：还是在所有node节点
[root@k8s-node1 ~]# vim /opt/kubernetes/cfg/kube-proxy
# cat /opt/kubernetes/cfg/kube-proxy
KUBE_PROXY_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.1.2* \   #写每个node节点ip
--cluster-cidr=10.0.0.0/24 \           //不要改，就是这个ip
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"

systemd管理kube-proxy组件：
[root@k8s-node1 ~]# cd /usr/lib/systemd/system
# cat /usr/lib/systemd/system/kube-proxy.service 
[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

启动：

 systemctl daemon-reload
 systemctl enable kube-proxy
 systemctl start kube-proxy

在master查看集群状态
[root@k8s-master1 ~]# /opt/kubernetes/bin/kubectl get node
NAME              STATUS    ROLES     AGE       VERSION
192.168.1.2*   Ready     <none>    19m       v1.11.10
192.168.246.165   Ready     <none>    18m       v1.11.10

查看集群状态
[root@k8s-master1 ~]# /opt/kubernetes/bin/kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok                   
controller-manager   Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   
etcd-1               Healthy   {"health": "true"}   
etcd-2               Healthy   {"health": "true"}
=====================================================================================

6.4 部署网络组件（Calico、flanneld二选一）

Calico
Calico是一个纯三层的数据中心网络方案，是目前Kubernetes主流的网络方案

#上传yaml文件
链接： https://pan.baidu.com/s/1jPzSdsnFKSFxkVQzc2lQ9g?pwd=x311
提取码：x311

#部署Calico
[root@k8s-master1 k8s]# cd /opt/kubernetes/cfg/

*上传至/opt/kubernetes/cfg/

[root@k8s-master1 cfg]# kubectl apply -f calico.yaml
[root@k8s-master1 cfg]# kubectl get pods -n kube-system

waiting…

[root@k8s-master1 cfg]# kubectl get pods -n kube-system
NAME                                      READY   STATUS    RESTARTS   AGE
calico-kube-controllers-97769f7c7-9d49d   1/1     Running   0          9m16s
calico-node-8djzj                         1/1     Running   0          9m16s
[root@k8s-master1 cfg]# kubectl get node
NAME          STATUS   ROLES    AGE   VERSION
k8s-master1   Ready    <none>   23m   v1.20.4

flanneld
master1节点

/opt/etcd/bin/etcdctl \
--cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem \
--endpoints="https://192.168.1.20:2379,https://192.168.1.21:2379,https://192.168.1.22:2379" \
put /coreos.com/network/config  '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'

master1、node1、node2节点

#下载二进制包
链接：https://pan.baidu.com/s/1vSxuXNQZU8yXkcCvDED1Pw?pwd=0ind
提取码：0ind

[root@k8s-node1 ~]# tar zvxf flannel-v0.13.0-linux-amd64.tar.gz
[root@k8s-node1 ~]# mv flanneld mk-docker-opts.sh /opt/kubernetes/bin
#配置Flannel
[root@k8s-node1 ~]# vim /opt/kubernetes/cfg/flanneld
FLANNEL_OPTIONS="--etcd-endpoints=https://192.168.1.20:2379,https://192.168.1.21:2379,https://192.168.1.22:2379 -etcd-cafile=/opt/etcd/ssl/ca.pem -etcd-certfile=/opt/etcd/ssl/server.pem -etcd-keyfile=/opt/etcd/ssl/server-key.pem"


#systemd管理Flannel
[root@k8s-node1 ~]# vim /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure

[Install]
WantedBy=multi-user.target

启动flannel和docker
systemctl daemon-reload
systemctl start flanneld
systemctl enable flanneld
systemctl daemon-reload
systemctl restart docker

报错（Couldn‘t fetch network config）
原因：flanneld目前不能与etcdV3直接交互

参考：https://blog.51cto.com/u_8355320/2564588

master、node1、node2节点

#开启etcd 支持V2api功能，在etcd启动参数中加入 --enable-v2参数，并重启etcd2


[root@k8s-master etcd]# vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--logger=zap \
--enable-v2     #此处添加
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

[root@master1 ~]# systemctl daemon-reload
[root@master1 ~]# systemctl restart etcd

·master查看etcd集群健康状况
[root@k8s-master etcd]# ETCDCTL_API=2 /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.1.20:2379,https://192.168.1.21:2379,https://192.168.1.22:2379" cluster-health

#删除原来写入的子网信息
[root@k8s-master etcd]# /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.1.20:2379,https://192.168.1.21:2379,https://192.168.1.22:2379" del /coreos.com/network/config

#重新使用V2写入子网信息
ETCDCTL_API=2 /opt/etcd/bin/etcdctl \
--ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem \
--endpoints="https://192.168.1.20:2379,https://192.168.1.21:2379,https://192.168.1.22:2379" \
set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'

#重启flanneld服务
systemctl daemon-reload
systemctl start flanneld
systemctl enable flanneld

#修改docker文件
[root@k8s-node1 ~]# vim /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

#重启docker
systemctl daemon-reload
systemctl restart docker

6.5 授权apiserver访问kubelet

#应用场景：例如kubectl logs

[root@k8s-master1 cfg]# vim apiserver-to-kubelet-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes


[root@k8s-master1 cfg]# kubectl apply -f apiserver-to-kubelet-rbac.yaml

6.6 新增加Worker Node(如果需要)

master1节点

#在Master1节点将Worker Node涉及文件拷贝到node1
[root@k8s-master1 cfg]# scp -r /opt/kubernetes k8s-node1:/opt/
[root@k8s-master1 cfg]# scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service k8s-node1:/usr/lib/systemd/system
[root@k8s-master1 cfg]# scp /opt/kubernetes/ssl/ca.pem k8s-node1:/opt/kubernetes/ssl/

node1节点

#删除kubelet证书和kubeconfig文件
[root@k8s-node1 ~]# rm -f /opt/kubernetes/cfg/kubelet.kubeconfig
[root@k8s-node1 ~]# rm -f /opt/kubernetes/ssl/kubelet*

*注：这几个文件是证书申请审批后自动生成的，每个Node不同，必须删除


#修改对应文件的主机名
[root@k8s-node1 ~]# vim /opt/kubernetes/cfg/kubelet.conf
--hostname-override=k8s-node1
[root@k8s-node1 ~]# vim /opt/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: k8s-node1

#启动并设置开机启动
systemctl daemon-reload
systemctl enable kubelet kube-proxy
systemctl start kubelet kube-proxy

master1节点

#在Master1上批准新Node kubelet证书申请

*查看证书请求
[root@k8s-master1 cfg]# kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-0UTuuhUTPbL02uDpLinrwBc_YDnmXj3t-JjUqMM247I   41m   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Approved,Issued
node-csr-xQsFeNF5nlB-rZQr2cIxFB18ET3kAGYHSC9GMSKDoI8   41s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

*授权请求（Pending，一定要是待定状态的）
[root@k8s-master1 cfg]# kubectl certificate approve node-csr-xQsFeNF5nlB-rZQr2cIxFB18ET3kAGYHSC9GMSKDoI8
certificatesigningrequest.certificates.k8s.io/node-csr-xQsFeNF5nlB-rZQr2cIxFB18ET3kAGYHSC9GMSKDoI8 approved

#查看Node状态
[root@k8s-master1 cfg]# kubectl get node
NAME          STATUS     ROLES    AGE   VERSION
k8s-master1   Ready      <none>   41m   v1.20.4
k8s-node1     NotReady   <none>   61s   v1.20.4

发现新加入的k8s-node1节点是NotReady状态，不要急，等待一下，跟master节点一样，Calico在做初始化

这个时候可以去做node2，跟加入node1的步骤一样，当然最后也要等！！！

最终加入结果

#查看名称空间kube-system中的pod情况
[root@k8s-master1 cfg]# kubectl get pods -n kube-system
NAME                                      READY   STATUS    RESTARTS   AGE
calico-kube-controllers-97769f7c7-9d49d   1/1     Running   0          42m
calico-node-8djzj                         1/1     Running   0          42m
calico-node-h6ghf                         1/1     Running   0          15m
calico-node-nj9l7                         1/1     Running   0          8m1s


#查看Node状态
[root@k8s-master1 cfg]# kubectl get node
NAME          STATUS   ROLES    AGE     VERSION
k8s-master1   Ready    <none>   56m     v1.20.4
k8s-node1     Ready    <none>   16m     v1.20.4
k8s-node2     Ready    <none>   8m25s   v1.20.4