kubernetesにHeketiとGlusterFSを配備(二)
kubernetesに導入するHeketiとGlusterFS(二)前節では、Heketiの導入方式はまだ生産環境では使用できないが、Heketi Podのデータは永続化していないため、heketiのデータが失われやすく、Heketiのデータは/var/lib/heketi/heketiに保存されている.dbファイルにあるため、このディレクトリをGlusterFS分散ストレージにマウントする必要があります.
前節の手順に従ってheketi-cli topology load--json=topology-sampleを実行する.json
次に実行:heketi-storageが生成されます.jsonファイル.
setup-openshift-heketi-storageサブコマンドを実行している間にheketi-cliが「スペースなし」エラーを報告した場合:$heketi-cli setup-openshift-heketi-storageError:Failed to allocate new volume:No spaceがtopology loadコマンドを何気なく実行している場合、サービス側とheketi-cliのバージョンが一致しないためです.実行中のheketi pod:kubectl scale deployment deploy-heketi--replicas=0 ストレージブロックデバイス内の任意の署名を手動で削除する:トポロジをロードする操作はglusterにPeerを追加するので、手動detach peer が必要である.その後、heketi pod:kubectl scale deployment deploy-heketi--replicas=1の実行を続行します. は、一致するバージョンのheketi-cliでトポロジを再ロードし、ステップを再試行します.
実行後、Pod deploy-heketiログ情報を確認し、どのようなことをしたかを確認します.
任意のGlusterFS Podにアクセスしてボリューム情報を表示します.
分析してjson:
作成:
ジョブが実行されると削除できます.
jobが完了するまでbootstrap Heketiインスタンスに関連するコンポーネントを削除します.
以前に作成したdeploy-heketiというpodは、サービスが削除されました.
最後にheketi-deploymentを用いる.jsonファイル再配置heketi
heketi-deployment.jsonファイルは、次のリソースを作成します.
導入:
heketiがgluster volumeで使用されているかどうかを確認します.
これで、heketi dbはGlusterFSボリュームを正しく構成しました.
前節の手順に従ってheketi-cli topology load--json=topology-sampleを実行する.json
$ echo $HEKETI_CLI_SERVER
http://10.254.49.43:8080
$ heketi-cli topology load --json=topology-sample.json
Creating cluster ... ID: 5b930ef6081fd22e895c25a3dfb0c516
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node 10.30.1.15 ... ID: b120572be40db6c1d979c3903876430b
Adding device /dev/sdb ... OK
Creating node 10.30.1.16 ... ID: 7ce13ffc5eabe64a3791e93233fd3c1a
Adding device /dev/sdb ... OK
Creating node 10.30.1.17 ... ID: f9abdc2e5d4cfa17c035a97f984a1a3b
Adding device /dev/sdb ... OK
次に実行:heketi-storageが生成されます.jsonファイル.
$ heketi-cli setup-openshift-heketi-storage
Saving heketi-storage.json
setup-openshift-heketi-storageサブコマンドを実行している間にheketi-cliが「スペースなし」エラーを報告した場合:$heketi-cli setup-openshift-heketi-storageError:Failed to allocate new volume:No spaceがtopology loadコマンドを何気なく実行している場合、サービス側とheketi-cliのバージョンが一致しないためです.
実行後、Pod deploy-heketiログ情報を確認し、どのようなことをしたかを確認します.
# , glusterfs Pod brick 3 Replicate volume, volume heketidbstorage
[kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir -p /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6
Result:
[kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.15 Pod: glusterfs-c4859 Command: mkdir -p /var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4
Result:
[kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: mkdir -p /var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa
[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick
[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick
[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.15 Pod: glusterfs-c4859 Command: mkdir /var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick
[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: mkdir /var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick
Result:
[cmdexec] INFO 2018/07/09 07:07:26 Creating volume heketidbstorage replica 3
[kubeexec] DEBUG 2018/07/09 07:07:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: gluster --mode=script volume create heketidbstorage replica 3 10.30.1.16:/var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick 10.30.1.17:/var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick 10.30.1.15:/var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick
Result: volume create: heketidbstorage: success: please start the volume to access data
任意のGlusterFS Podにアクセスしてボリューム情報を表示します.
$ kubectl exec glusterfs-25cm8 -it bash
[root@ubuntu16 /]# gluster volume info
Volume Name: heketidbstorage
Type: Replicate
Volume ID: c8da2a4a-3066-4708-a59d-201d22decd92
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.30.1.16:/var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick
Brick2: 10.30.1.17:/var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick
Brick3: 10.30.1.15:/var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
[root@ubuntu16 /]#
分析してjson:
# :
Endpoints:
name:heketi-storage-endpoints
10.30.1.16:1 10.30.1.15:1 10.30.1.17:1
Service:
name: heketi-storage-endpoints
job:
name: heketi-storage-copy-job
images: heketi/heketi:dev
volume:heketi-storage
"volumes": [
{
"name": "heketi-storage",
"glusterfs": {
"endpoints": "heketi-storage-endpoints",
"path": "heketidbstorage"
}
},
/heketi:
volumeMounts": [
{
"name": "heketi-storage",
"mountPath": "/heketi"
},
:cp /db/heketi.db /heketi
# , job heketi /heketi, /heketi heketi-storage , heketi-storage volume "heketi-cli setup-openshift-heketi-storage"
作成:
$ kubectl create -f heketi-storage.json
secret "heketi-storage-secret" created
endpoints "heketi-storage-endpoints" created
service "heketi-storage-endpoints" created
job "heketi-storage-copy-job" created
ジョブが実行されると削除できます.
$ kubectl get job
NAME DESIRED SUCCESSFUL AGE
heketi-storage-copy-job 1 1 1m
jobが完了するまでbootstrap Heketiインスタンスに関連するコンポーネントを削除します.
# heketi-bootstrap.json
$ kubectl delete all,service,jobs,deployment,secret --selector="deploy-heketi"
deployment "deploy-heketi" deleted
job "heketi-storage-copy-job" deleted
pod "deploy-heketi-69bfbd4bbd-q8tsk" deleted
service "deploy-heketi" deleted
secret "heketi-storage-secret" deleted
以前に作成したdeploy-heketiというpodは、サービスが削除されました.
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
glusterfs-25cm8 1/1 Running 1 1h
glusterfs-8qrpt 1/1 Running 1 1h
glusterfs-c4859 1/1 Running 1 1h
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
heketi-storage-endpoints ClusterIP 10.254.191.233 1/TCP 4m
最後にheketi-deploymentを用いる.jsonファイル再配置heketi
$ cat heketi-deployment.json
{
"kind": "List",
"apiVersion": "v1",
"items": [
{
"kind": "Secret",
"apiVersion": "v1",
"metadata": {
"name": "heketi-db-backup",
"labels": {
"glusterfs": "heketi-db",
"heketi": "db"
}
},
"data": {
},
"type": "Opaque"
},
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "heketi",
"labels": {
"glusterfs": "heketi-service",
"deploy-heketi": "support"
},
"annotations": {
"description": "Exposes Heketi Service"
}
},
"spec": {
"selector": {
"name": "heketi"
},
"ports": [
{
"name": "heketi",
"port": 8080,
"targetPort": 8080
}
]
}
},
{
"kind": "Deployment",
"apiVersion": "extensions/v1beta1",
"metadata": {
"name": "heketi",
"labels": {
"glusterfs": "heketi-deployment"
},
"annotations": {
"description": "Defines how to deploy Heketi"
}
},
"spec": {
"replicas": 1,
"template": {
"metadata": {
"name": "heketi",
"labels": {
"name": "heketi",
"glusterfs": "heketi-pod"
}
},
"spec": {
"serviceAccountName": "heketi-service-account",
"containers": [
{
"image": "heketi/heketi:dev",
"imagePullPolicy": "Always",
"name": "heketi",
"env": [
{
"name": "HEKETI_EXECUTOR",
"value": "kubernetes"
},
{
"name": "HEKETI_DB_PATH",
"value": "/var/lib/heketi/heketi.db"
},
{
"name": "HEKETI_FSTAB",
"value": "/var/lib/heketi/fstab"
},
{
"name": "HEKETI_SNAPSHOT_LIMIT",
"value": "14"
},
{
"name": "HEKETI_KUBE_GLUSTER_DAEMONSET",
"value": "y"
}
],
"ports": [
{
"containerPort": 8080
}
],
"volumeMounts": [
{
"mountPath": "/backupdb",
"name": "heketi-db-secret"
},
{
"name": "db",
"mountPath": "/var/lib/heketi"
},
{
"name": "config",
"mountPath": "/etc/heketi"
}
],
"readinessProbe": {
"timeoutSeconds": 3,
"initialDelaySeconds": 3,
"httpGet": {
"path": "/hello",
"port": 8080
}
},
"livenessProbe": {
"timeoutSeconds": 3,
"initialDelaySeconds": 30,
"httpGet": {
"path": "/hello",
"port": 8080
}
}
}
],
"volumes": [
{
"name": "db",
"glusterfs": {
"endpoints": "heketi-storage-endpoints",
"path": "heketidbstorage"
}
},
{
"name": "heketi-db-secret",
"secret": {
"secretName": "heketi-db-backup"
}
},
{
"name": "config",
"secret": {
"secretName": "heketi-config-secret"
}
}
]
}
}
}
}
]
}
heketi-deployment.jsonファイルは、次のリソースを作成します.
Service:
name: heketi
port: 8080
Deployment:
name: heketi
replicas: 1
image: heketi/heketi:dev
volumeMounts:
name: db mountPath: /var/lib/heketi
volumes:
endpoints: heketi-storage-endpoints # heketi-storage.json
path:heketidbstorage # gluster volume , volume "heketi-cli setup-openshift-heketi-storage" 。
# heketi /var/lib/heketi GlusterFS volume “heketidbstorage” 。
導入:
$ kubectl create -f heketi-deployment.json
secret "heketi-db-backup" created
service "heketi" created
deployment "heketi" created
$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
heketi 1 1 1 1 45s
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
heketi ClusterIP 10.254.239.189 8080/TCP 51s
heketi-storage-endpoints ClusterIP 10.254.191.233 1/TCP 31m
heketiがgluster volumeで使用されているかどうかを確認します.
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
glusterfs-25cm8 1/1 Running 1 1h
glusterfs-8qrpt 1/1 Running 1 1h
glusterfs-c4859 1/1 Running 1 1h
heketi-7898db85dd-nb6kn 1/1 Running 0 1m
$ kubectl exec heketi-7898db85dd-nb6kn -it bash
[root@heketi-7898db85dd-nb6kn /]# mount |grep heketi
10.30.1.15:heketidbstorage on /var/lib/heketi type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
これで、heketi dbはGlusterFSボリュームを正しく構成しました.