kubernetes ceph is locked by other nodesエラー解決策

1.問題の説明

pod       ContainerCreating  

[root@master ~]# kubectl get pods -n wangweidd | grep -v Running
NAME         READY     STATUS              RESTARTS   AGE
enncloud-1   0/1       ContainerCreating   0          14d

[root@master ~]# kubectl describe pod enncloud-1 -n wangweidd
...
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason      Message
  --------- --------    -----   ----            -------------   --------    ------      -------
  14d       1m      10013   kubelet, test           Warning     FailedMount MountVolume.SetUp failed for volume "wangweidd.datadir-enncloud-2" : rbd: image wangweidd.CID-516874818ed4.datadir-enncloud-2 is locked by other nodes
  14d       54s     8993    kubelet, test           Warning     FailedMount Unable to mount volumes for pod "enncloud-1_wangweidd(69b763cd-f9a3-11e7-9d36-5254b24cbf5e)": timeout expired waiting for volumes to attach/mount for pod "wangweidd"/"enncloud-1". list of unattached/unmounted volumes=[datadir]
  14d       54s     8993    kubelet, test           Warning     FailedSync  Error syncing pod

エラーメッセージ

 rbd: image wangweidd.CID-516874818ed4.datadir-enncloud-2 is locked by other nodes

2.問題解決

 1.       ceph     
    ：
  rbd -p test-pool status  enngastest.CID-516874818ed4.datadir-mi-es-pro-1   
  Watchers:
           watcher=10.39.1.95:0/3703494529 client.104824 cookie=2
        : 
    10.39.1.95     

2. rbd  showmapped            rbd        
    rbd showmapped 
    2017-07-13 14:10:15.704627 7f824e7657c0 -1 asok(0x3791960) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/rbd-clients/ceph-client.admin.8255.58268576.asok': (2) No such file or directory
   id pool      image                                           snap device    
     0  tenx-pool lixueyang.CID-516874818ed4.datadir-iauc-crm-0   -    /dev/rbd0 
     1  tenx-pool enngastest.CID-516874818ed4.datadir-mi-es-pro-1 -   /dev/rbd1 
     2  tenx-pool ceres-prod.CID-516874818ed4.ceres-etcd-0        -      /dev/rbd2 
  
rbd lock list tenx-pool/kubernetes-dynamic-pvc-d1043bbb-d5e4-11e7-a259-5254b24cbf5e Locker ID Address
client.811885 kubelet_lock_magic_paas-slave-20-48 10.39.20.48:0/2805197474

   rbd unmap /dev/rbd2

Ceph rbd拡張

rbd showmapped   
df -h     /dev/rbdx       
    
rbd  resize --size    1024   foo   
rbd resize --size  1024  liuguanghuif.CID-516874818ed4.datadir-lk-test-1  -p tenx-pool 
df -Th           
          
blockdev --getsize64 /dev/rbd4 
resize2fs /dev/rbd4 

    ok！ 

  :  xfs   resize        xfs_growfs /mnt