生产k8s三主三从高可用集群搭建

本文阅读量: 2023-06-04

一部署规划与准备

系统：CentOS7.9

K8S版本：1.15.12

K8S机器规划：

服务器	节点名称	其他	备注
192.168.0.101	k8s-master01	keepalived、haproxy、VIP(192.168.0.200)	主节点
192.168.0.102	k8s-master02	keepalived、haproxy、VIP(192.168.0.200)	主节点
192.168.0.103	k8s-master03	keepalived、haproxy、VIP(192.168.0.200)	主节点
192.168.0.104	k8s-worker01		工作节点
192.168.0.105	k8s-worker02		工作节点
192.168.0.106	k8s-worker03		工作节点

每台机器更改hosts

cat > /etc/hosts << EOF
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
 
192.168.0.101   k8s-master01
192.168.0.102   k8s-master02
192.168.0.103   k8s-master03
192.168.0.104   k8s-worker01
192.168.0.105   k8s-worker02
192.168.0.106   k8s-worker03
192.168.0.200   k8s_vip
EOF

禁用SELINUX并关闭Swap、关闭防火墙

setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
sed -ri 's/.*swap.*/#&/' /etc/fstab
 
systemctl stop firewalld
systemctl disable firewalld

安装的docker版本：18.09.9

yum -y install yum-utils
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install docker-ce-cli-18.09.9-3.el7
yum install docker-ce-18.09.9-3.el7
 
systemctl daemon-reload
systemctl enable docker
systemctl start docker

修改 /etc/docker/daemon.json 如下：

1
2
3

cat > /etc/docker/daemon.json << EOF
{"insecure-registries": ["dockerhub:5000"],"log-opts": {"max-size": "200m","max-file": "3"},"exec-opts": ["native.cgroupdriver=systemd"]}
EOF

insecure-registries 配置主要是因为我们使用的是自建的docker镜像仓库；

log-opts 配置主要是限制每个pod日志的大小以避免被日志写满磁盘；

exec-opts 配置主要是告诉 Docker 守护进程使用 Systemd Cgroup 驱动程序来管理容器的 Cgroups

取其中一台如 192.168.0.101 作为时钟服务器

yum install chronyd -y

vim /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
# server 0.centos.pool.ntp.org iburst
# server 1.centos.pool.ntp.org iburst
# server 2.centos.pool.ntp.org iburst
# server 3.centos.pool.ntp.org iburst
server 0.asia.pool.ntp.org iburst
server 1.asia.pool.ntp.org iburst
server 2.asia.pool.ntp.org iburst
server 3.asia.pool.ntp.org iburst
server 192.168.0.101
local stratum 10

# Record the rate at which the system clock gains/losses time.
driftfile /var/lib/chrony/drift

# Allow the system clock to be stepped in the first three updates
# if its offset is larger than 1 second.
makestep 1.0 3

# Enable kernel synchronization of the real-time clock (RTC).
rtcsync

# Enable hardware timestamping on all interfaces that support it.
#hwtimestamp *

# Increase the minimum number of selectable sources required to adjust
# the system clock.
#minsources 2

# Allow NTP client access from local network.
allow 192.168.0.0/16

# Serve time even if not synchronized to a time source.
#local stratum 10

# Specify file containing keys for NTP authentication.
#keyfile /etc/chrony.keys

# Specify directory for log files.
logdir /var/log/chrony

# Select which information is logged.
#log measurements statistics tracking

另外5台直接同步以上机器的时钟

yum install chronyd -y
 
vim /etc/chronyd.conf
# 注释以下配置
# server 0.centos.pool.ntp.org iburst
# server 1.centos.pool.ntp.org iburst
# server 2.centos.pool.ntp.org iburst
# server 3.centos.pool.ntp.org iburst
# 添加以下配置
server 192.168.0.101 iburst
 
systemctl start chronyd
systemctl enable chronyd

K8S相关的yum源配置

cat << EOF > kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
 
yum-config-manager --add-repo kubernetes.repo

修改所有节点内核参数

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = ``1
net.bridge.bridge-nf-call-ip6tables = ``1
net.bridge.bridge-nf-call-iptables = ``1
EOF

1	modprobe br_netfilter && sysctl -p /etc/sysctl.d/k8s.conf

二安装k8s（安装keepalived、 haproxy、kubeadm-1.15.12、kubectl-1.15.12、kubelet-1.15.12）

1 2	# master节点安装 yum -y install keepalived haproxy kubeadm-``1.15``.``12` `kubectl-``1.15``.``12` `kubelet-``1.15``.``12

1 2	# worker节点安装 yum -y install kubeadm-``1.15``.``12` `kubectl-``1.15``.``12` `kubelet-``1.15``.``12

haproxy配置

1	# /etc/haproxy/haproxy.cfg 后面追加

#---------------------------------------------------------------------
frontend k8s-api
  ``bind *:``8443
  ``mode tcp
  ``default_backend       apiserver
#---------------------------------------------------------------------
backend apiserver
  ``balance   roundrobin
  ``mode tcp
  ``server k8s-master01 k8s-master01:``6443` `check weight ``1` `maxconn ``2000` `check inter ``2000` `rise ``2` `fall ``3
  ``server k8s-master02 k8s-master02:``6443` `check weight ``1` `maxconn ``2000` `check inter ``2000` `rise ``2` `fall ``3
  ``server k8s-master03 k8s-master03:``6443` `check weight ``1` `maxconn ``2000` `check inter ``2000` `rise ``2` `fall ``3

配置keepalived

配置/etc/keepalived/keepalived.conf ( priority 192.168.0.103:99、192.168.0.102:90、192.168.0.101:80 state 192.168.0.103:MASTER、192.168.0.102:BACKUP、192.168.0.101:BACKUP）

cat <<EOF > /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
   router_id VIP_200
}
 
vrrp_instance VI_1 {
    state MASTER
    interface bond1
    virtual_router_id 200
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
      192.168.0.200
    }
}
EOF

启动 haproxy、keepalived

systemctl daemon-reload
systemctl enable haproxy
systemctl enable keepalived
 
systemctl start haproxy
systemctl start keepalived

k8s的master安装、worker安装

# 生成预处理文件，kubeadm-init.yaml，是我们初始化使用的文件，里面大概修改这几项参数
kubeadm config print init-defaults > kubeadm-init.yaml
 
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.0.101
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-master01
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  # 添加如下两行信息
  certSANs:
  - "192.168.0.200"         # VIP地址
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager:
  extraArgs:
    allocate-node-cidrs: "true"
    cluster-cidr: "10.244.0.0/16"
    horizontal-pod-autoscaler-sync-period: "10s"   
    node-monitor-grace-period: "10s"
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
controlPlaneEndpoint: "192.168.0.200:8443"
kind: ClusterConfiguration
kubernetesVersion: v1.15.12
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}

三个master节点提前拉取镜像

# kubeadm init --config kubeadm-init.yaml --upload-certs
[init] Using Kubernetes version: v1.15.12
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.0.101 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.0.101 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.101 192.168.0.200 192.168.0.200]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 34.003308 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
261bc852612b0055e57a2a829a44b7c7b44812d5480a9fba54175cb352ed5c50
[mark-control-plane] Marking the node k8s-master01 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy
 
Your Kubernetes control-plane has initialized successfully!
 
To start using your cluster, you need to run the following as a regular user:
 
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
 
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
 
You can now join any number of the control-plane node running the following command on each as root:
 
kubeadm join 192.168.0.200:8443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:dea0850253e82618e9629aa7871bbbb69ed7abcfab13a21dbd572b7409368bf4 \
--control-plane --certificate-key 261bc852612b0055e57a2a829a44b7c7b44812d5480a9fba54175cb352ed5c50
 
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
 
Then you can join any number of worker nodes by running the following on each as root:
 
kubeadm join 192.168.0.200:8443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:dea0850253e82618e9629aa7871bbbb69ed7abcfab13a21dbd572b7409368bf4

在192.168.0.102节点执行以下命令增加 k8s-master02

kubeadm join 192.168.0.200:8443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:dea0850253e82618e9629aa7871bbbb69ed7abcfab13a21dbd572b7409368bf4 \
--control-plane --certificate-key 261bc852612b0055e57a2a829a44b7c7b44812d5480a9fba54175cb352ed5c50 \
--node-name k8s-master02

在192.168.0.103节点执行以下命令增加 k8s-master02

kubeadm join 192.168.0.200:8443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:dea0850253e82618e9629aa7871bbbb69ed7abcfab13a21dbd572b7409368bf4 \
--control-plane --certificate-key 261bc852612b0055e57a2a829a44b7c7b44812d5480a9fba54175cb352ed5c50 \
--node-name k8s-master03

三个master节点执行以下命令

1
2
3

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

三个worker节点执行以下命令

# 192.168.0.104
kubeadm join 192.168.0.200:8443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:dea0850253e82618e9629aa7871bbbb69ed7abcfab13a21dbd572b7409368bf4 \
--node-name k8s-worker01
 
# 192.168.0.105
kubeadm join 192.168.0.200:8443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:dea0850253e82618e9629aa7871bbbb69ed7abcfab13a21dbd572b7409368bf4 \
--node-name k8s-worker02
 
# 192.168.0.106
kubeadm join 192.168.0.200:8443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:dea0850253e82618e9629aa7871bbbb69ed7abcfab13a21dbd572b7409368bf4 \
--node-name k8s-worker03

安装flannel，添加以下yaml

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
  - configMap
  - secret
  - emptyDir
  - hostPath
  allowedHostPaths:
  - pathPrefix: "/etc/cni/net.d"
  - pathPrefix: "/etc/kube-flannel"
  - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs: ['use']
  resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.13.0
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.13.0
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg

特别注意：由于cni1.0版本后 /opt/cni/bin 缺少 flannell，所以需要在每个节点下的 /opt/cni/bin 下加入以下文件（解压后加入）

github下载地址：https://github.com/containernetworking/plugins/releases/tag/v0.8.6

如果需要让k8s的master参与调度，执行一下命令：

# 查看污点
kubectl describe node k8s-master |grep Taints
Taints:    node-role.kubernetes.io/master:NoSchedule
 
# 删除污点
kubectl taint nodes --all node-role.kubernetes.io/master-

注意：通过kubeadm安装的k8s集群，证书只有一年有效期，到期后需要刷新证书，刷新证书的命令如下：

# 证书有效期查询 -- 在master节点
kubeadm alpha certs check-expiration
 
# 直接通过以下命令更新证书 在其中一台master执行以下命令
kubeadm alpha certs renew all

# 在每台master节点再执行以下命令 则每个master节点都能正常使用kubectl命令
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
# 重启kubelet
systemctl restart kubelet
# 重启kube-apiserver、kube-controller-manage、kube-scheduler
docker ps |grep kube-apiserver|grep -v pause|awk '{print $1}'|xargs -i docker restart {}
docker ps |grep kube-controller-manage|grep -v pause|awk '{print $1}'|xargs -i docker restart {}
docker ps |grep kube-scheduler|grep -v pause|awk '{print $1}'|xargs -i docker restart {}

一 部署规划与准备

二 安装k8s（安装keepalived、 haproxy、kubeadm-1.15.12、kubectl-1.15.12、kubelet-1.15.12）

一部署规划与准备

二安装k8s（安装keepalived、 haproxy、kubeadm-1.15.12、kubectl-1.15.12、kubelet-1.15.12）