记录一次K8S的kubelet启动失败分析

一 错误说明

一个k8s的worker节点需要升级CPU，于是先驱逐完pod之后先停机，更改了配置后，重启起来，机器起来之后，发现无法连接到k8s集群了，节点一直都是NotReady，于是查看节点的kubelet，发现起不来，如下：
github他人提的issue：https://github.com/kubernetes/kubernetes/issues/76820

$ systemctl status kubelet kube-proxy -l  
● kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Tue 2022-01-11 11:43:15 CST; 2s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 10418 ExecStart=/usr/bin/kubelet $KUBE_LOGTOSTDERR $KUBE_LOG_LEVEL $KUBELET_API_SERVER $KUBELET_ADDRESS $KUBELET_PORT $KUBELET_HOSTNAME $KUBE_ALLOW_PRIV $KUBELET_POD_INFRA_CONTAINER $KUBELET_ARGS (code=exited, status=255)
 Main PID: 10418 (code=exited, status=255)

Jan 11 11:38:36 k8s-node03 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Jan 11 11:38:36 k8s-node03 systemd[1]: Unit kubelet.service entered failed state.
Jan 11 11:38:36 k8s-node03 systemd[1]: kubelet.service failed.
Jan 11 11:38:36 k8s-node03 systemd[1]: kubelet.service holdoff time over, scheduling restart.
Jan 11 11:38:36 k8s-node03 systemd[1]: start request repeated too quickly for kubelet.service
Jan 11 11:38:36 k8s-node03 systemd[1]: Failed to start Kubernetes Kubelet Server.
Jan 11 11:38:36 k8s-node03 systemd[1]: Unit kubelet.service entered failed state.
Jan 11 11:38:36 k8s-node03 systemd[1]: kubelet.service failed.

查看了日志，发现报了如下错误：

$ sudo journalctl -xeu kubelet
.....
Jan 11 11:38:36 k8s-node03 kubelet[10418]: F0111 11:38:36.770078   10418 node_container_manager_linux.go:50] Failed to create ["kubepods"] cgroup
Jan 11 11:38:36 k8s-node03 kubelet[10418]: F0111 11:38:36.770102   10418 kubelet.go:1359] Failed to start ContainerManager Cannot set property TasksAccounting, or unknown property.
Jan 11 11:38:36 k8s-node03 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Jan 11 11:38:36 k8s-node03 systemd[1]: Unit kubelet.service entered failed state.
Jan 11 11:38:36 k8s-node03 systemd[1]: kubelet.service failed.

于是去github找相关issue查看原因，确实也有不少这类报错，找到一个解决方案。

二 解决方案

重新install一下systemd，看issue说是因为systemd版本过旧的原因导致的。

yum install systemd -y

然后重启kubelet服务 systemctl restart kubelet则恢复正常了。

另外，看issue还有其他解决方法，如yum update升级一下所有包。

仅此记录一下，以后遇到同样的问题，可以快速修复。
<!–stackedit_data:

eyJoaXN0b3J5IjpbLTE5NjYwOTQ3MTIsLTE3ODc2NzMwNTZdfQ

–>