Kubernetes:nodeはNotReady状態にあり、リカバリ方法

ノードのステータスの表示

 kubectl get nodes
NAME       STATUS     ROLES     AGE       VERSION
docker01   Ready      master    1y        v1.9.0
docker04   Ready          1y        v1.9.0
docker06   NotReady       1y        v1.9.0
docker08   Ready          89d       v1.9.0

ノードがNotReady状態にあるなんて、不快だ.
クラスタに追加されたコマンドを実行して問題を見てみましょう
docker 06に切り替える

ssh root@docker06

クラスタへの追加コマンドの実行

# kubeadm join --token 6be0d2.121fb2825cd41f64 192.168.100.61:6443 --discovery-token-ca-cert-hash sha256:4e671bcabdf9e35491c1e9b51ce06dc6900bdd5b53ad48a13419051b5f1382f6
[preflight] Running pre-flight checks.
	[WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Some fatal errors occurred:
	[ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists
	[ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
	[ERROR Swap]: running with swap on is not supported. Please disable swap

主に二つの問題です
1:証明書が存在する場合
2:仮想ストレージswapがオープン
ステップ1:swapを閉じる

vim /etc/fstab

SWAPパーティション項目の注記

#
# /etc/fstab
# Created by anaconda on Thu Apr  9 22:39:56 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/cl-root     /                       xfs     defaults        0 0
UUID=e4976f5b-c44e-4fba-b0c7-3b10bb939db2 /boot                   ext4    defaults        1 2
/dev/mapper/cl-home     /home                   xfs     defaults        0 0
#/dev/mapper/cl-swap     swap                    swap    defaults        0 0

コマンドライン実行

swapoff -a

ステップ2:node上のk 8 sを再起動
再起動コマンド

kubeadm reset
[preflight] Running pre-flight checks.
[reset] Stopping the kubelet service.
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Removing kubernetes-managed containers.
[reset] No etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml". Assuming external etcd.
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

 
ステップ3:クラスタへの追加コマンドを再実行

 kubeadm join --token 6be0d2.121fb2825cd41f64 192.168.100.61:6443 --discovery-token-ca-cert-hash sha256:4e671bcabdf9e35491c1e9b51ce06dc6900bdd5b53ad48a13419051b5f1382f6
[preflight] Running pre-flight checks.
	[WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Starting the kubelet service
[discovery] Trying to connect to API Server "192.168.100.61:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.100.61:6443"
[discovery] Failed to connect to API Server "192.168.100.61:6443": there is no JWS signed token in the cluster-info ConfigMap. This token id "6be0d2" is invalid for this cluster, can't connect
[discovery] Trying to connect to API Server "192.168.100.61:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.100.61:6443"
[discovery] Failed to connect to API Server "192.168.100.61:6443": there is no JWS signed token in the cluster-info ConfigMap. This token id "6be0d2" is invalid for this cluster, can't connect
[discovery] Trying to connect to API Server "192.168.100.61:6443"

また新しい問題で、
なぜならk 8 sのtokenの有効期間は24時間しかないので、tokenを再構築します.
ステップ4:tokenを新規作成する

kubeadm token create

永久tokenになるには次のコマンドを使います.

kubeadm token create --ttl 0

tokenの表示
 

# kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION   EXTRA GROUPS
dxnj79.rnj561a137ri76ym      2018-11-02T14:06:43+08:00   authentication,signing           system:bootstrappers:kubeadm:default-node-token
o4avtg.65ji6b778nyacw68                           authentication,signing           system:bootstrappers:kubeadm:default-node-token

 
ステップ5:ca証明書sha 256符号化hash値を取得する

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
2cc3029123db737f234186636330e87b5510c173c669f513a9c0e0da395515b0

 
ステップ6:nodeノードの追加

kubeadm join 10.167.11.153:6443 --token o4avtg.65ji6b778nyacw68 --discovery-token-ca-cert-hash sha256:2cc3029123db737f234186636330e87b5510c173c669f513a9c0e0da395515b0

kubeadm join 10.167.11.153:6443 --token o4avtg.65ji6b778nyacw68 --discovery-token-ca-cert-hash sha256:2cc3029123db737f234186636330e87b5510c173c669f513a9c0e0da395515b0
注意:赤い部分は上の2つのコマンドの結果に対応します.
実行結果は、クラスタへの正常な参加を示します.

[preflight] Running pre-flight checks.
	[WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Starting the kubelet service
[discovery] Trying to connect to API Server "192.168.100.61:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.100.61:6443"
[discovery] Requesting info from "https://192.168.100.61:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.100.61:6443"
[discovery] Successfully established connection with API Server "192.168.100.61:6443"

This node has joined the cluster:
* Certificate signing request was sent to master and a response
  was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

ステップ7:グループに入るノードの状態を見てみましょう

]# kubectl get nodes
NAME       STATUS    ROLES     AGE       VERSION
docker01   Ready     master    1y        v1.9.0
docker04   Ready         1y        v1.9.0
docker06   Ready         1y        v1.9.0
docker08   Ready         89d       v1.9.0

もう正常です.