虚拟机安装kubesphere3.0

具体步骤

1，背景 

3台 linux 虚机 操作系统 centos7 

10.10.10.104

10.10.10.106

10.10.10.108

cat /etc/redhat-release

CentOS Linux release 7.7.1908 (Core)

2，  配置 hosts文件

cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

# kubekey hosts BEGIN

10.10.10.104  kubesphere.localdomain.cluster.local kubesphere.localdomain

10.10.10.104  lb.kubesphere.local

10.10.10.104  blockdeviceclaims.openebs.io

10.10.10.106  worker1

10.10.10.108  worker2

10.10.10.104 kubesphere

#10.10.10.106 worker1 worker1.localdomain

#10.10.10.108  worker2 woker2.localdoomain

#10.10.10.104  kubesphere.localdomain.cluster.local kubesphere.localdomain

#10.10.10.104  lb.kubesphere.local

#10.10.10.104  blockdeviceclaims.openebs.io

# kubekey hosts END

3，一些 必要部署

 setenforce 0

 getenforce

 vi /etc/selinux/config

  systemctl status firewalld

   systemctl stop firewalld

  systemctl disabled firewalld

 docker pull golang:1.14.7

  yum install -y socat conntrack ebtables ipset 

4， 安装docker

curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun 

systemctl start docker

systemctl enable docker

设置 docker 镜像加速

     sudo tee /etc/docker/daemon.json <<-'EOF'

     {

       "registry-mirrors": ["https://3iv1aqw6.mirror.aliyuncs.com"]

     }

     EOF

     sudo systemctl daemon-reload

     dockr info 

#   32  ./build.sh -p

5, 下载kk 和离线安装包

 yum install wget

 wget https://github.com/kubesphere/kubekey/releases/download/v1.0.0/kubekey-v1.0.0-linux-amd64.tar.gz

 wget https://kubesphere-installer.pek3b.qingstor.com/offline/v3.0.0/kubesphere-all-v3.0.0-offline-linux-amd64.tar.gz

  tar xvf kubesphere-all-v3.0.0-offline-linux-amd64.tar.gz

   cd kubesphere-all-v3.0.0-offline-linux-amd64

   ls

   cd kubesphere-images-v3.0.0/ 

6，加载相关的包

 docker  load < *.tar

 docker  load < csi_images.tar

 docker  load < example_images.tar  

 docker load < k8s-images.tar 

 docker load < nfs-client-image.tar 

 docker load < ks-minimal-images.tar

 chmod +x kk

7，用 kk 安装cluster 

  ./kk create cluster --with-kubernetes v1.18.6 --with-kubesphere v3.0.0

  

kubectl get nodes

发现 集群节点不ready, flannel 没有部署上，kubesphere 也就没有部署上

 cat /var/log/messages

 ls -al /etc/cni/net.d/

 kubectl  describe node kubesphere.localdomain

 kubectl  describe node kubesphere.localdomain | grep tai

 kubectl  version

 systemctl statuskubelet

 systemctl status kubelet  

 

8，解决flannel 问题 

cat <<EOF> /etc/cni/net.d/10-flannel.conf

 {

“name”: “cbr0”,

“type”: “flannel”,

“delegate”: {

“isDefaultGateway”: true

 }

 }

 EOF

 mkdir /usr/share/oci-umount/oci-umount.d -p

 mkdir /run/flannel/

  cat <<EOF> /run/flannel/subnet.env

 FLANNEL_NETWORK=10.244.0.0/16

 FLANNEL_SUBNET=10.244.1.0/24

 FLANNEL_MTU=1450

 FLANNEL_IPMASQ=true

 EOF

部署 flannel 

 kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 

9，部署 openebs

  kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml

  kubectl get pv

  kubectl get pv -A

  kubectl get pods -A

  kubectl describe pods maya-apiserver-697d7997f7-hx7xg -n openebs

  kubectl get pods -A

  kubectl describe pods openebs-admission-server-68c6d59d59-5qtc4

  kubectl describe pods openebs-admission-server-68c6d59d59-5qtc4 -n openebs

  kubectl get pods -A

  kubectl describe pods maya-apiserver-697d7997f7-hx7xg -n openebs

  

 patch storageclass

kubectl patch storageclass openebs-hostpath   -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

10，部署kubesphere 3.0

kubectl apply -f https://raw.githubusercontent.com/kubesphere/ks-installer/v3.0.0/deploy/kubesphere-installer.yaml

kubectl apply -f https://raw.githubusercontent.com/kubesphere/ks-installer/v3.0.0/deploy/cluster-configuration.yaml

 kubectl get pods -A

  kubectl logs ks-installer-7cb866bd-fxhrz -f -n kubesphere-system

11，部署nfs，供其它几个节点访问

  yum -y install nfs-utils rpcbind

  systemctl enable nfs

  systemctl enable rpcbind

  systemctl enable rpcbind

  vi /etc/exports

   ls -al

   vi /etc/exports

  

 10.10.10.0/24(rw,sync)

  /opt/luo** 10.10.10.106(rw,sync)

 /opt/luo*** 10.10.10.108(rw,sync)

  systemctl start rpcbind

  systemctl start nfs

  netstat -antp | grep rpcbind

    yum install netstat

   showmount -e 

#showmount -e

Export list for kubesphere.localdomain:

/root/kubesphere 10.10.10.0/24

/opt/luopeng     10.10.10.108,10.10.10.106

12, 其它两个节点 安装系统

1） 每个节点上要安装kubeadm，kubelet，kubectl

2） worker 节点 从nfs 访问 数据等

    yum -y install nfs-utils rpcbind

    showmount -e 10.10.10.104 

    mount -t nfs4 -o intr,nolock  10.10.10.104:/kubesphere /mnt/media

有时候遇到 nfs4的问题

 yum -y install nfs-utils portmap

systemctl  restart  nfs

systemctl  restart  rpcbind 

3）kubeadm join

kubeadm get token

kubeadm token create

kubeadm token list  | awk -F" " '{print $1}' |tail -n 1

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed  's/^ .* //'  

 kubeadm join 10.10.10.104:6443 --token qy76h0.m61c8jnxr7etecwb   --discovery-token-ca-cert-hash sha256:11ddb475bc6d1efc39e250224e72295b0e04d31c3696d71e7d960e4d6748d0e7 --v=5  

3）都要解决flannel 插件问题

4）

cat > kubernetes.conf <<EOF

net.bridge.bridge-nf-call-iptables=1

net.bridge.bridge-nf-call-ip6tables=1

net.ipv4.ip_forward=1

net.ipv4.tcp_tw_recycle=0

vm.swappiness=0

vm.overcommit_memory=1

vm.panic_on_oom=0

fs.inotify.max_user_watches=89100

fs.file-max=52706963

fs.nr_open=52706963

net.ipv6.conf.all.disable_ipv6=1

net.netfilter.nf_conntrack_max=2310720

EOF 

sysctl -p kubernetes.conf  

5）worker 节点

docker load < k8s-images.tar

docker load < nfs-client-image.tar

 docker load < ks-minimal-images.tar

kubeadm join 的时候如果报证书 已有，删掉再来

 mv  /etc/kubernetes/pki  /etc/kubernetes/pkiold

6）worker 节点 copy cni二进制文件等

 scp 10.10.10.104:/opt/cni/bin /opt/cni/bin

 scp -r 10.10.10.104:/opt/cni/bin /opt/cni/bin

 systemctl restart kubelet

systemctl status kubelet 

13，测试

kubectl create deployment nginx --dry-run -o yaml > nginx

kubectl create deployment nginx --image=nginx --dry-run -o yaml > nginx 

 kubectl get pods

kubectl describe pods  nginx-××××-×××× 

15， 节点加入完成以后，可以通过https://nodeip:30080 访问kubepshre3.0的界面  

16，中间遇到问题

1）查看 flannel 的configmap

kubectl get configmap -o yaml  -n kube-system kube-flannel-cfg

2）安装brctl 工具 查看 网桥

yum install brige-util* 

3）查看 kubelet 的状态

 systemctl status kubelet.service

 cat /etc/systemd/system/kubelet.service

 cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf