Kubernetes本番実践シリーズの7:etcdによるKubernetesクラスタのバックアップとリカバリ
9289 ワード
一、前言
厳密には、Kubernetesのすべての構築は無状態であり、これらの構築の状態には、後で導入されたさまざまなリソースの状態がetcdクラスタに格納されているため、etcdをバックアップすることで、災害下でクラスタとクラスタ上のアプリケーションを迅速にリカバリすることができます.
転載先https://blog.csdn.net/cloudvtec
二、クラスタバックアップを行う
2.1クラスタの現在のステータスの表示
kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
default demo-deployment-7c687dbbfd-jxvnw 1/1 Running 0 17m 10.254.66.8 k8s-node-02
default demo-deployment-7c687dbbfd-l4pzv 1/1 Running 0 17m 10.254.95.250 k8s-node-01
default demo-deployment-7c687dbbfd-pjz9q 1/1 Running 0 17m 10.254.95.251 k8s-node-01
default demo-deployment-7c687dbbfd-xzb6b 1/1 Running 0 17m 10.254.95.249 k8s-node-01
kube-system calico-kube-controllers-64b4dd5f65-5r6sj 1/1 Running 0 17m 10.0.2.15 k8s-node-01
kube-system calico-node-bdv8q 2/2 Running 0 17m 10.0.2.15 k8s-node-01
kube-system calico-node-v4d25 2/2 Running 0 17m 10.0.2.15 k8s-node-02
kube-system coredns-794cc4cddd-jt4jf 1/1 Running 0 1m 10.254.66.15 k8s-node-02
kube-system coredns-794cc4cddd-k5n88 1/1 Running 0 1m 10.254.95.253 k8s-node-01
ETCDCTL_API=3 etcdctl --cert=/opt/etcd/ca/client.pem --key=/opt/etcd/ca/client-key.pem --cacert=/opt/etcd/ca/ca.pem --endpoints=https://k8s-master-01:2379,https://k8s-master-02:2379,https://k8s-master-03:2379 get / --prefix --keys-only
2.2 etcdバックアップ
etcdクラスタには一般的に複数のノードがあり、そのうちの1つを選択してバックアップすればよい.
Step 1:APIServerが動作していることを確認する
[root@k8s-master-01 coredns]# ps -ef | grep kube-api
kube 2218 1 1 00:33 ? 00:07:02 /usr/bin/kube-apiserver --logtostderr=true --v=2 --etcd-servers=https://192.168.56.101:2379,https://192.168.56.102:2379,https://192.168.56.103:2379 --advertise-address=192.168.56.101 --insecure-bind-address=127.0.0.1 --bind-address=192.168.56.101 --insecure-port=8080 --secure-port=6443 --allow-privileged=true --service-cluster-ip-range=10.254.0.0/16 --admission-control=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota --authorization-mode=RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --anonymous-auth=false --kubelet-https=true --enable-bootstrap-token-auth=true --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-50000 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/k8s-root-ca.pem --service-account-key-file=/etc/kubernetes/ssl/k8s-root-ca.pem --etcd-quorum-read=true --storage-backend=etcd3 --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/client.pem --etcd-keyfile=/etc/kubernetes/ssl/client-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/log/kube-audit/audit.log --event-ttl=1h --external-hostname=k8s-master-01
Step 2:バックアップを行う
export ETCD_SERVERS=$(ps -ef|grep apiserver|grep -Eo "etcd-servers=.*2379"|awk -F= '{print $NF}')
mkdir /opt/etcd/backup
ETCDCTL_API=3 etcdctl snapshot --endpoints=$ETCD_SERVERS --cacert=/opt/etcd/ca/ca.pem --cert=/opt/etcd/ca/client.pem --key=/opt/etcd/ca/client-key.pem save /opt/etcd/backup/backup_$(date "+%Y%m%d%H%M%S").db
Snapshot saved at /opt/etcd/backup/backup_20181031080729.db
cp -a /etc/kubernetes/ /etc/kubernetes.bak
Step 3:現在のクラスタからdemo-deploymentというdeploymentを削除してクラスタ状態を変化させる
[root@k8s-master-01 coredns]# kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
default demo-deployment-7c687dbbfd-l4pzv 0/1 Terminating 0 1h k8s-node-01
default demo-deployment-7c687dbbfd-pjz9q 0/1 Terminating 0 1h k8s-node-01
default demo-deployment-7c687dbbfd-xzb6b 0/1 Terminating 0 1h k8s-node-01
kube-system calico-kube-controllers-64b4dd5f65-5r6sj 1/1 Running 0 1h 10.0.2.15 k8s-node-01
kube-system calico-node-bdv8q 2/2 Running 0 1h 10.0.2.15 k8s-node-01
kube-system calico-node-v4d25 2/2 Running 0 1h 10.0.2.15 k8s-node-02
kube-system coredns-794cc4cddd-jt4jf 1/1 Running 0 1h 10.254.66.15 k8s-node-02
kube-system coredns-794cc4cddd-k5n88 1/1 Running 0 1h 10.254.95.253 k8s-node-01 転載先https://blog.csdn.net/cloudvtec
三、クラスタがバックアップした状態を回復する
Step 1:すべてのAPIServerを停止する
systemctl stop kube-apiserver
Step 2:バックアップファイルを各etcdノードに同期する
scp backup_20181031080729.db root@k8s-master-02:/opt/etcd/
scp backup_20181031080729.db root@k8s-master-03:/opt/etcd/
Step 3:etcdバックアップファイルから次の方法で復元する
etcdctl snapshot restore backup_20181031080729.db \
> --endpoints=192.168.56.10{1|2|3}:2379 \
> --name=k8s-master-0{1|2|3} \
> --cacert=/opt/etcd/ca/ca.pem \
> --key=/opt/etcd/ca/server-key.pem \
> --cert=/opt/etcd/ca/server.pem \
> --initial-advertise-peer-urls=https://k8s-master-0{1|2|3}:2380 \
> --initial-cluster-token=wayz-etcd-cluster-token \
> --initial-cluster=k8s-master-01=https://k8s-master-01:2380,k8s-master-02=https://k8s-master-02:2380,k8s-master-03=https://k8s-master-03:2380 \
> --data-dir=/opt/etcd/data
2018-10-31 08:24:08.639390 I | pkg/netutil: resolving k8s-master-01:2380 to 192.168.56.101:2380
2018-10-31 08:24:08.639628 I | pkg/netutil: resolving k8s-master-01:2380 to 192.168.56.101:2380
2018-10-31 08:24:08.674823 I | mvcc: restore compact to 38176
2018-10-31 08:24:08.684389 I | etcdserver/membership: added member 666e3882c4f82f71 [https://k8s-master-03:2380] to cluster a374d6d95deea33d
2018-10-31 08:24:08.684424 I | etcdserver/membership: added member 679ae639419c436f [https://k8s-master-02:2380] to cluster a374d6d95deea33d
2018-10-31 08:24:08.684436 I | etcdserver/membership: added member f21bce98b30c6f30 [https://k8s-master-01:2380] to cluster a374d6d95deea33dStep 4:etcdを再起動して状態を確認する
systemctl start etcd
ETCDCTL_API=2 etcdctl --cert-file=/opt/etcd/ca/client.pem --key-file=/opt/etcd/ca/client-key.pem --ca-file=/opt/etcd/ca/ca.pem --endpoints=https://k8s-master-01:2379,https://k8s-master-02:2379,https://k8s-master-03:2379 cluster-health
member 666e3882c4f82f71 is healthy: got healthy result from https://192.168.56.103:2379
member 679ae639419c436f is healthy: got healthy result from https://192.168.56.102:2379
member f21bce98b30c6f30 is healthy: got healthy result from https://192.168.56.101:2379
ETCDCTL_API=3 etcdctl --cert=/opt/etcd/ca/client.pem --key=/opt/etcd/ca/client-key.pem --cacert=/opt/etcd/ca/ca.pem --endpoints=https://k8s-master-01:2379,https://k8s-master-02:2379,https://k8s-master-03:2379 endpoint health
https://k8s-master-03:2379 is healthy: successfully committed proposal: took = 10.454518ms
https://k8s-master-01:2379 is healthy: successfully committed proposal: took = 8.020988ms
https://k8s-master-02:2379 is healthy: successfully committed proposal: took = 8.316226msStep 5:APIServerの再起動
systemctl start kube-apiserver
Step 6:クラスタの状態を確認し、demo-deploymentがリカバリ導入中であることを期待
[root@k8s-master-01 backup]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default demo-deployment-7c687dbbfd-jxvnw 0/1 ContainerCreating 0 1h
default demo-deployment-7c687dbbfd-l4pzv 0/1 ContainerCreating 0 1h
default demo-deployment-7c687dbbfd-pjz9q 0/1 ContainerCreating 0 1h
default demo-deployment-7c687dbbfd-xzb6b 0/1 ContainerCreating 0 1h
kube-system calico-kube-controllers-64b4dd5f65-5r6sj 0/1 Running 0 1h
kube-system calico-node-bdv8q 2/2 Running 0 1h
kube-system calico-node-v4d25 2/2 Running 0 1h
kube-system coredns-794cc4cddd-jt4jf 1/1 Running 0 1h
kube-system coredns-794cc4cddd-k5n88 1/1 Running 0 1h
転載先https://blog.csdn.net/cloudvtec