kubernetesにHeketiとGlusterFSを配備(二)


kubernetesに導入するHeketiとGlusterFS(二)前節では、Heketiの導入方式はまだ生産環境では使用できないが、Heketi Podのデータは永続化していないため、heketiのデータが失われやすく、Heketiのデータは/var/lib/heketi/heketiに保存されている.dbファイルにあるため、このディレクトリをGlusterFS分散ストレージにマウントする必要があります.
前節の手順に従ってheketi-cli topology load--json=topology-sampleを実行する.json
$ echo $HEKETI_CLI_SERVER
http://10.254.49.43:8080

$ heketi-cli topology load --json=topology-sample.json
Creating cluster ... ID: 5b930ef6081fd22e895c25a3dfb0c516
    Allowing file volumes on cluster.
    Allowing block volumes on cluster.
    Creating node 10.30.1.15 ... ID: b120572be40db6c1d979c3903876430b
        Adding device /dev/sdb ... OK
    Creating node 10.30.1.16 ... ID: 7ce13ffc5eabe64a3791e93233fd3c1a
        Adding device /dev/sdb ... OK
    Creating node 10.30.1.17 ... ID: f9abdc2e5d4cfa17c035a97f984a1a3b
        Adding device /dev/sdb ... OK

次に実行:heketi-storageが生成されます.jsonファイル.
$ heketi-cli setup-openshift-heketi-storage
Saving heketi-storage.json

setup-openshift-heketi-storageサブコマンドを実行している間にheketi-cliが「スペースなし」エラーを報告した場合:$heketi-cli setup-openshift-heketi-storageError:Failed to allocate new volume:No spaceがtopology loadコマンドを何気なく実行している場合、サービス側とheketi-cliのバージョンが一致しないためです.
  • 実行中のheketi pod:kubectl scale deployment deploy-heketi--replicas=0
  • ストレージブロックデバイス内の任意の署名を手動で削除する:トポロジをロードする操作はglusterにPeerを追加するので、手動detach peer
  • が必要である.
  • その後、heketi pod:kubectl scale deployment deploy-heketi--replicas=1の実行を続行します.
  • は、一致するバージョンのheketi-cliでトポロジを再ロードし、ステップを再試行します.

  • 実行後、Pod deploy-heketiログ情報を確認し、どのようなことをしたかを確認します.
    #        ,           glusterfs Pod  brick          3 Replicate volume, volume  heketidbstorage
    [kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir -p /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6
    Result: 
    [kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.15 Pod: glusterfs-c4859 Command: mkdir -p /var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4
    Result: 
    [kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: mkdir -p /var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa
    
    [kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick
    
    [kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick
    
    [kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.15 Pod: glusterfs-c4859 Command: mkdir /var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick
    
    [kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: mkdir /var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick
    Result: 
    [cmdexec] INFO 2018/07/09 07:07:26 Creating volume heketidbstorage replica 3
    [kubeexec] DEBUG 2018/07/09 07:07:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: gluster --mode=script volume create heketidbstorage replica 3 10.30.1.16:/var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick 10.30.1.17:/var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick 10.30.1.15:/var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick
    Result: volume create: heketidbstorage: success: please start the volume to access data

    任意のGlusterFS Podにアクセスしてボリューム情報を表示します.
    $ kubectl exec glusterfs-25cm8 -it bash
    [root@ubuntu16 /]# gluster volume info
    
    Volume Name: heketidbstorage
    Type: Replicate
    Volume ID: c8da2a4a-3066-4708-a59d-201d22decd92
    Status: Started
    Snapshot Count: 0
    Number of Bricks: 1 x 3 = 3
    Transport-type: tcp
    Bricks:
    Brick1: 10.30.1.16:/var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick
    Brick2: 10.30.1.17:/var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick
    Brick3: 10.30.1.15:/var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick
    Options Reconfigured:
    transport.address-family: inet
    nfs.disable: on
    [root@ubuntu16 /]# 
    

    分析してjson:
    #         :
    Endpoints:
        name:heketi-storage-endpoints
             10.30.1.16:1 10.30.1.15:1 10.30.1.17:1
    Service:    
        name: heketi-storage-endpoints  
    job: 
       name: heketi-storage-copy-job
       images: heketi/heketi:dev
           volume:heketi-storage 
             "volumes": [
                                    {
                                        "name": "heketi-storage",
                                        "glusterfs": {
                                            "endpoints": "heketi-storage-endpoints",
                                            "path": "heketidbstorage"
                                        }
                                    },
             /heketi:
                 volumeMounts": [
                                                {
                                                    "name": "heketi-storage",
                                                    "mountPath": "/heketi"
                                                },
    
              :cp /db/heketi.db /heketi 
       #    , job       heketi        /heketi, /heketi       heketi-storage , heketi-storage volume     "heketi-cli setup-openshift-heketi-storage"      

    作成:
    $ kubectl create -f heketi-storage.json
    secret "heketi-storage-secret" created
    endpoints "heketi-storage-endpoints" created
    service "heketi-storage-endpoints" created
    job "heketi-storage-copy-job" created

    ジョブが実行されると削除できます.
    $ kubectl get job 
    NAME                      DESIRED   SUCCESSFUL   AGE
    heketi-storage-copy-job   1         1            1m

    jobが完了するまでbootstrap Heketiインスタンスに関連するコンポーネントを削除します.
    #    heketi-bootstrap.json       
    $ kubectl delete all,service,jobs,deployment,secret --selector="deploy-heketi"
    deployment "deploy-heketi" deleted
    job "heketi-storage-copy-job" deleted
    pod "deploy-heketi-69bfbd4bbd-q8tsk" deleted
    service "deploy-heketi" deleted
    secret "heketi-storage-secret" deleted

    以前に作成したdeploy-heketiというpodは、サービスが削除されました.
    $ kubectl get pod
    NAME              READY     STATUS              RESTARTS   AGE
    glusterfs-25cm8   1/1       Running             1          1h
    glusterfs-8qrpt   1/1       Running             1          1h
    glusterfs-c4859   1/1       Running             1          1h
    
    $ kubectl get svc
    NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
    heketi-storage-endpoints      ClusterIP   10.254.191.233           1/TCP      4m

    最後にheketi-deploymentを用いる.jsonファイル再配置heketi
    $ cat heketi-deployment.json 
    {
      "kind": "List",
      "apiVersion": "v1",
      "items": [
        {
          "kind": "Secret",
          "apiVersion": "v1",
          "metadata": {
            "name": "heketi-db-backup",
            "labels": {
              "glusterfs": "heketi-db",
              "heketi": "db"
            }
          },
          "data": {
          },
          "type": "Opaque"
        },
        {
          "kind": "Service",
          "apiVersion": "v1",
          "metadata": {
            "name": "heketi",
            "labels": {
              "glusterfs": "heketi-service",
              "deploy-heketi": "support"
            },
            "annotations": {
              "description": "Exposes Heketi Service"
            }
          },
          "spec": {
            "selector": {
              "name": "heketi"
            },
            "ports": [
              {
                "name": "heketi",
                "port": 8080,
                "targetPort": 8080
              }
            ]
          }
        },
        {
          "kind": "Deployment",
          "apiVersion": "extensions/v1beta1",
          "metadata": {
            "name": "heketi",
            "labels": {
              "glusterfs": "heketi-deployment"
            },
            "annotations": {
              "description": "Defines how to deploy Heketi"
            }
          },
          "spec": {
            "replicas": 1,
            "template": {
              "metadata": {
                "name": "heketi",
                "labels": {
                  "name": "heketi",
                  "glusterfs": "heketi-pod"
                }
              },
              "spec": {
                "serviceAccountName": "heketi-service-account",
                "containers": [
                  {
                    "image": "heketi/heketi:dev",
                    "imagePullPolicy": "Always",
                    "name": "heketi",
                    "env": [
                      {
                        "name": "HEKETI_EXECUTOR",
                        "value": "kubernetes"
                      },
                      {
                        "name": "HEKETI_DB_PATH",
                        "value": "/var/lib/heketi/heketi.db"
                      },
                      {
                        "name": "HEKETI_FSTAB",
                        "value": "/var/lib/heketi/fstab"
                      },
                      {
                        "name": "HEKETI_SNAPSHOT_LIMIT",
                        "value": "14"
                      },
                      {
                        "name": "HEKETI_KUBE_GLUSTER_DAEMONSET",
                        "value": "y"
                      }
                    ],
                    "ports": [
                      {
                        "containerPort": 8080
                      }
                    ],
                    "volumeMounts": [
                      {
                        "mountPath": "/backupdb",
                        "name": "heketi-db-secret"
                      },
                      {
                        "name": "db",
                        "mountPath": "/var/lib/heketi"
                      },
                      {
                        "name": "config",
                        "mountPath": "/etc/heketi"
                      }
                    ],
                    "readinessProbe": {
                      "timeoutSeconds": 3,
                      "initialDelaySeconds": 3,
                      "httpGet": {
                        "path": "/hello",
                        "port": 8080
                      }
                    },
                    "livenessProbe": {
                      "timeoutSeconds": 3,
                      "initialDelaySeconds": 30,
                      "httpGet": {
                        "path": "/hello",
                        "port": 8080
                      }
                    }
                  }
                ],
                "volumes": [
                  {
                    "name": "db",
                    "glusterfs": {
                      "endpoints": "heketi-storage-endpoints",
                      "path": "heketidbstorage"
                    }
                  },
                  {
                    "name": "heketi-db-secret",
                    "secret": {
                      "secretName": "heketi-db-backup"
                    }
                  },
                  {
                    "name": "config",
                    "secret": {
                      "secretName": "heketi-config-secret"
                    }
                  }
                ]
              }
            }
          }
        }
      ]
    }
    

    heketi-deployment.jsonファイルは、次のリソースを作成します.
    Service:
         name: heketi
         port: 8080
    Deployment:
        name: heketi
        replicas: 1
        image: heketi/heketi:dev
        volumeMounts: 
               name: db   mountPath: /var/lib/heketi 
       volumes: 
          endpoints: heketi-storage-endpoints   # heketi-storage.json    
           path:heketidbstorage  #  gluster volume , volume  "heketi-cli setup-openshift-heketi-storage"    。
        #  heketi    /var/lib/heketi       GlusterFS volume “heketidbstorage” 。

    導入:
    $ kubectl create -f heketi-deployment.json 
    secret "heketi-db-backup" created
    service "heketi" created
    deployment "heketi" created
    
    $ kubectl get deployment 
    NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    heketi    1         1         1            1           45s
    $ kubectl get svc 
    NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
    heketi                        ClusterIP   10.254.239.189           8080/TCP   51s
    heketi-storage-endpoints      ClusterIP   10.254.191.233           1/TCP      31m

    heketiがgluster volumeで使用されているかどうかを確認します.
    $ kubectl get pod
    NAME                      READY     STATUS              RESTARTS   AGE
    glusterfs-25cm8           1/1       Running             1          1h
    glusterfs-8qrpt           1/1       Running             1          1h
    glusterfs-c4859           1/1       Running             1          1h
    heketi-7898db85dd-nb6kn   1/1       Running             0          1m
    
    $ kubectl exec heketi-7898db85dd-nb6kn -it bash
    [root@heketi-7898db85dd-nb6kn /]# mount |grep heketi
    10.30.1.15:heketidbstorage on /var/lib/heketi type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

    これで、heketi dbはGlusterFSボリュームを正しく構成しました.