Cephクラスタの導入
Cephの最下層はRADOSで「A reliable,autonomous,distributed object storage」という意味です.
RADOSは2つのコンポーネントで構成されています.
•
OSD:Object Storage Device、ストレージリソースを提供します.すなわちosd
•
Monitor:Cephクラスタ全体のグローバル状態を維持します.すなわちmon
RADOSは拡張性とプログラム性に優れており、CephはRADOSに基づいてObject Storage、Block Storage、FileSystemを開発した.
Cephの他の2つのコンポーネントは次のとおりです.
•
MDS:CephFSのメタデータを保存します.すなわちmds
•
RADOS Gateway:RESTインタフェースを対外的に提供し、S 3とSwiftのAPIに対応する.すなわちrgw
環境準備
1,ubuntu 1404 serverを4台用意
192.168.3.106 Client(ceph-deployの導入)
192.168.3.8 node1 (mon,mdsの配備)
192.168.3.9 node2 (配置osd)
192.168.3.10 node3 (配置osd)
2,hostsを1台ずつ配置する
3,rootユーザーsshを開く
vi/etc/ssh/sshd_config
PermitRootLogin yesの変更
サービスssh restartの実行 sshサービスの再起動
4,クライアントから各ノードへのパスワードなしログインの構成
Node 1,node 2,node 3実行
ssh-keygen
クライアント実行:
ssh-keygen
ssh-copy-id node1
ssh-copy-id node2
ssh-copy-id node3
ssh mon、ログイン成功!
cephクラスタのインストール
1,ceph-deployのインストール
実行:apt-get install ceph-deploy
2 monノードnodeを構成する
ceph-deploy new node 1を実行します.ここではホスト名を使用する必要があります.
実行に成功したディレクトリは、次の3つのファイルを生成します.
ceph.mon.keyring ,ceph.conf,ceph.log
ceph.conf追加の編集 osd_pool_default_size = 2
3、cephを3つのノードにインストール
ceph-deploy install client node 1の実行 node2 node3
各nodeにcephを実行する --version cephインストール結果の確認
モニタノードのアクティブ化
ceph.bootstrap-mds.keyring,ceph.bootstrap-osd.keyring,ceph.client.admin.keyring
monノードnode 1にceph_monプロセスがあります
OSdノードの構成
node 2、node 3にそれぞれハードディスク8 Gを追加し、パーティション/dev/sdb 1を作成し、xfsファイルシステムにフォーマットします.
OSdノードの追加
ceph-deploy osd prepare node 2:/dev/sdb 1 node 3:/dev/sdb 1を実行
OSdノードをアクティブにする
実行 ceph-deploy osd activate node2:/dev/sdb1 node3:/dev/sdb1
OSdノードでのアクティブ化の検証に成功しました
実行 ps aux | grep ceph-osd|grep -v grep
それぞれ2つのosdノードで、/dev/sdb 1パーティションを/etc/fstabに追加して電源を入れて自動的にマウントします.
追加 /dev/sdb1 /var/lib/ceph/osd/ceph-0 xfs defaults 0 0
node 1に追加 metadata server
ceph-deploy mds create node 1の実行
mdsプロセスの確認
cephファイルシステムの作成
コンフィギュレーションファイルの同期
実行 ceph-deploy admin node1 node2 node3
rbdの使用
新しいceph pool rados mkpool test poolにミラー(ipはmonノード)rbd create test-1--size 4096-p test-m 172.17.106.122 poolブロックデバイスにミラーをマッピングrbd map test-1-p test--name client.admin rbdのマッピング関係rbd showmappedを表示
新しいミラーcephブロックをフォーマットする
マウント
root@node1:~# mount/dev/rbd0/mnt
RADOSは2つのコンポーネントで構成されています.
•
OSD:Object Storage Device、ストレージリソースを提供します.すなわちosd
•
Monitor:Cephクラスタ全体のグローバル状態を維持します.すなわちmon
RADOSは拡張性とプログラム性に優れており、CephはRADOSに基づいてObject Storage、Block Storage、FileSystemを開発した.
Cephの他の2つのコンポーネントは次のとおりです.
•
MDS:CephFSのメタデータを保存します.すなわちmds
•
RADOS Gateway:RESTインタフェースを対外的に提供し、S 3とSwiftのAPIに対応する.すなわちrgw
環境準備
1,ubuntu 1404 serverを4台用意
192.168.3.106 Client(ceph-deployの導入)
192.168.3.8 node1 (mon,mdsの配備)
192.168.3.9 node2 (配置osd)
192.168.3.10 node3 (配置osd)
2,hostsを1台ずつ配置する
root@node2:~# cat /etc/hosts
127.0.0.1 localhost
192.168.3.8 node1
192.168.3.9 node2
192.168.3.10 node3
192.168.3.106 client
3,rootユーザーsshを開く
vi/etc/ssh/sshd_config
PermitRootLogin yesの変更
サービスssh restartの実行 sshサービスの再起動
4,クライアントから各ノードへのパスワードなしログインの構成
Node 1,node 2,node 3実行
ssh-keygen
クライアント実行:
ssh-keygen
ssh-copy-id node1
ssh-copy-id node2
ssh-copy-id node3
root@client:~# ssh-copy-id node1
The authenticity of host 'node1 (192.168.3.8)' can't be established.
ECDSA key fingerprint is 1b:16:32:16:e8:92:4e:f2:8b:98:1f:1a:9b:e4:27:e9.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@osd's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'node1'"
and check to make sure that only the key(s) you wanted were added.
テストパスワードなしログインssh mon、ログイン成功!
root@client:~# ssh node1
Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.19.0-25-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Sun Jan 10 11:09:47 CST 2016
System load: 0.01 Processes: 93
Usage of /: 19.7% of 5.51GB Users logged in: 1
Memory usage: 5% IP address for eth0: 192.168.3.8
Swap usage: 0%
Graph this data and manage this system at:
https://landscape.canonical.com/
Last login: Sun Jan 10 10:34:52 2016
root@node1:~#
cephクラスタのインストール
1,ceph-deployのインストール
実行:apt-get install ceph-deploy
2 monノードnodeを構成する
ceph-deploy new node 1を実行します.ここではホスト名を使用する必要があります.
root@client:~/ceph# ceph-deploy new node1
[ceph_deploy.cli][INFO ] Invoked (1.4.0): /usr/bin/ceph-deploy new node1
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][DEBUG ] Resolving host node1
[ceph_deploy.new][DEBUG ] Monitor node1 at 192.168.3.8
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[node1][DEBUG ] connected to host: client
[node1][INFO ] Running command: ssh -CT -o BatchMode=yes node1
[ceph_deploy.new][DEBUG ] Monitor initial members are ['node1']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.3.8']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
実行に成功したディレクトリは、次の3つのファイルを生成します.
ceph.mon.keyring ,ceph.conf,ceph.log
ceph.conf追加の編集 osd_pool_default_size = 2
root@client:~/ceph# vi ceph.conf
[global]
fsid = 3a562301-bd64-45c5-aaa0-ef57e3dfd76f
mon_initial_members = node1
mon_host = 192.168.3.8
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd_pool_default_size = 2
3、cephを3つのノードにインストール
ceph-deploy install client node 1の実行 node2 node3
root@client:~/ceph# ceph-deploy install client node1 node2 node3
[ceph_deploy.cli][INFO ] Invoked (1.4.0): /usr/bin/ceph-deploy install client node1 node2 node3
[ceph_deploy.install][DEBUG ] Installing stable version emperor on cluster ceph hosts client node1 node2 node3
[ceph_deploy.install][DEBUG ] Detecting platform for host client ...
[client][DEBUG ] connected to host: client
[client][DEBUG ] detect platform information from remote host
[client][DEBUG ] detect machine type
[ceph_deploy.install][INFO ] Distro info: Ubuntu 14.04 trusty
[client][INFO ] installing ceph on client
[client][INFO ] Running command: env DEBIAN_FRONTEND=noninteractive apt-get -q install --assume-yes ca-certificates
[client][DEBUG ] Reading package lists...
[client][DEBUG ] Building dependency tree...
[node3][DEBUG ] Unpacking ceph-mds (0.80.10-0ubuntu1.14.04.3) ...
[node3][DEBUG ] Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
[node3][DEBUG ] Processing triggers for ureadahead (0.100.0-16) ...
[node3][DEBUG ] ureadahead will be reprofiled on next reboot
[node3][DEBUG ] Setting up ceph-fs-common (0.80.10-0ubuntu1.14.04.3) ...
[node3][DEBUG ] Setting up ceph-mds (0.80.10-0ubuntu1.14.04.3) ...
[node3][DEBUG ] ceph-mds-all start/running
[node3][DEBUG ] Processing triggers for ureadahead (0.100.0-16) ...
[node3][INFO ] Running command: ceph --version
[node3][DEBUG ] ceph version 0.80.10 (ea6c958c38df1216bf95c927f143d8b13c4a9e70)
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr
各nodeにcephを実行する --version cephインストール結果の確認
root@node2:~# ceph --version
ceph version 0.80.10 (ea6c958c38df1216bf95c927f143d8b13c4a9e70)
モニタノードのアクティブ化
root@client:~/ceph# ceph-deploy mon create-initial
[ceph_deploy.cli][INFO ] Invoked (1.4.0): /usr/bin/ceph-deploy mon create-initial
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts node1
[ceph_deploy.mon][DEBUG ] detecting platform for host node1 ...
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: Ubuntu 14.04 trusty
[node1][DEBUG ] determining if provided host has same hostname in remote
[node1][DEBUG ] get remote short hostname
[node1][DEBUG ] deploying mon to node1
[node1][DEBUG ] get remote short hostname
[node1][DEBUG ] remote hostname: node1
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node1][DEBUG ] create the mon path if it does not exist
[node1][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-node1/done
[node1][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-node1/done
[node1][INFO ] creating keyring file: /var/lib/ceph/tmp/ceph-node1.mon.keyring
[node1][DEBUG ] create the monitor keyring file
[node1][INFO ] Running command: ceph-mon --cluster ceph --mkfs -i node1 --keyring /var/lib/ceph/tmp/ceph-node1.mon.keyring
[node1][DEBUG ] ceph-mon: mon.noname-a 192.168.3.8:6789/0 is local, renaming to mon.node1
[node1][DEBUG ] ceph-mon: set fsid to 3a562301-bd64-45c5-aaa0-ef57e3dfd76f
[node1][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-node1 for mon.node1
[node1][INFO ] unlinking keyring file /var/lib/ceph/tmp/ceph-node1.mon.keyring
[node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[node1][DEBUG ] create the init path if it does not exist
[node1][DEBUG ] locating the `service` executable...
[node1][INFO ] Running command: initctl emit ceph-mon cluster=ceph id=node1
[node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status
[node1][DEBUG ] ********************************************************************************
[node1][DEBUG ] status for monitor: mon.node1
[node1][DEBUG ] {
[node1][DEBUG ] "election_epoch": 2,
[node1][DEBUG ] "extra_probe_peers": [],
[node1][DEBUG ] "monmap": {
[node1][DEBUG ] "created": "0.000000",
[node1][DEBUG ] "epoch": 1,
[node1][DEBUG ] "fsid": "3a562301-bd64-45c5-aaa0-ef57e3dfd76f",
[node1][DEBUG ] "modified": "0.000000",
[node1][DEBUG ] "mons": [
[node1][DEBUG ] {
[node1][DEBUG ] "addr": "192.168.3.8:6789/0",
[node1][DEBUG ] "name": "node1",
[node1][DEBUG ] "rank": 0
[node1][DEBUG ] }
[node1][DEBUG ] ]
[node1][DEBUG ] },
[node1][DEBUG ] "name": "node1",
[node1][DEBUG ] "outside_quorum": [],
[node1][DEBUG ] "quorum": [
[node1][DEBUG ] 0
[node1][DEBUG ] ],
[node1][DEBUG ] "rank": 0,
[node1][DEBUG ] "state": "leader",
[node1][DEBUG ] "sync_provider": []
[node1][DEBUG ] }
[node1][DEBUG ] ********************************************************************************
[node1][INFO ] monitor: mon.node1 is running
[node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status
[ceph_deploy.mon][INFO ] processing monitor mon.node1
[node1][DEBUG ] connected to host: node1
[node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status
[ceph_deploy.mon][INFO ] mon.node1 monitor has reached quorum!
[ceph_deploy.mon][INFO ] all initial monitors are running and have formed quorum
[ceph_deploy.mon][INFO ] Running gatherkeys...
[ceph_deploy.gatherkeys][DEBUG ] Checking node1 for /etc/ceph/ceph.client.admin.keyring
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.client.admin.keyring key from node1.
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
[ceph_deploy.gatherkeys][DEBUG ] Checking node1 for /var/lib/ceph/bootstrap-osd/ceph.keyring
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-osd.keyring key from node1.
[ceph_deploy.gatherkeys][DEBUG ] Checking node1 for /var/lib/ceph/bootstrap-mds/ceph.keyring
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-mds.keyring key from node1.
実行に成功すると3つのkeyファイルがローカルに生成され、ceph.bootstrap-mds.keyring,ceph.bootstrap-osd.keyring,ceph.client.admin.keyring
root@client:~/ceph# ll
total 100
drwxr-xr-x 2 root root 4096 Jan 12 14:07 ./
drwx------ 5 root root 4096 Jan 12 13:53 ../
-rw-r--r-- 1 root root 71 Jan 12 14:07 ceph.bootstrap-mds.keyring
-rw-r--r-- 1 root root 71 Jan 12 14:07 ceph.bootstrap-osd.keyring
-rw-r--r-- 1 root root 63 Jan 12 14:07 ceph.client.admin.keyring
-rw-r--r-- 1 root root 251 Jan 12 13:53 ceph.conf
-rw-r--r-- 1 root root 66652 Jan 12 14:07 ceph.log
-rw-r--r-- 1 root root 73 Jan 12 13:52 ceph.mon.keyring
monノードnode 1にceph_monプロセスがあります
root@node1:~# ps aux|grep ceph
root 1518 0.2 1.8 154700 19284 ? Ssl 14:48 0:00 /usr/bin/ceph-mon --cluster=ceph -i node1 -f
root 1846 0.0 0.2 11748 2208 pts/0 S+ 14:49 0:00 grep --color=auto ceph
OSdノードの構成
node 2、node 3にそれぞれハードディスク8 Gを追加し、パーティション/dev/sdb 1を作成し、xfsファイルシステムにフォーマットします.
root@node3:~# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xe5f14ca9.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): p
Disk /dev/sdb: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xe5f14ca9
Device Boot Start End Blocks Id System
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
Using default value 1
First sector (2048-16777215, default 2048): 2048
Last sector, +sectors or +size{K,M,G} (2048-16777215, default 16777215): 16000000
Command (m for help): p
Disk /dev/sdb: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xe5f14ca9
Device Boot Start End Blocks Id System
/dev/sdb1 2048 16000000 7998976+ 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
root@node3:~# mkfs.xfs /dev/sdb1
meta-data=/dev/sdb1 isize=256 agcount=4, agsize=499936 blks
= sectsz=512 attr=2, projid32bit=0
data = bsize=4096 blocks=1999744, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@node3:~#
OSdノードの追加
ceph-deploy osd prepare node 2:/dev/sdb 1 node 3:/dev/sdb 1を実行
root@client:~/ceph# ceph-deploy osd prepare node2:/dev/sdb1
[ceph_deploy.cli][INFO ] Invoked (1.4.0): /usr/bin/ceph-deploy osd prepare node2:/dev/sdb1
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks node2:/dev/sdb1:
[node2][DEBUG ] connected to host: node2
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[ceph_deploy.osd][INFO ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.osd][DEBUG ] Deploying osd to node2
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node2][INFO ] Running command: udevadm trigger --subsystem-match=block --action=add
[ceph_deploy.osd][DEBUG ] Preparing host node2 disk /dev/sdb1 journal None activate False
[node2][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdb1
[node2][WARNIN] Error: Partition(s) 1 on /dev/sdb1 have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use. As a result, the old partition(s) will remain in use. You should reboot now before making further changes.
[node2][DEBUG ] meta-data=/dev/sdb1 isize=2048 agcount=4, agsize=499936 blks
[node2][DEBUG ] = sectsz=512 attr=2, projid32bit=0
[node2][DEBUG ] data = bsize=4096 blocks=1999744, imaxpct=25
[node2][DEBUG ] = sunit=0 swidth=0 blks
[node2][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0
[node2][DEBUG ] log =internal log bsize=4096 blocks=2560, version=2
[node2][DEBUG ] = sectsz=512 sunit=0 blks, lazy-count=1
[node2][DEBUG ] realtime =none extsz=4096 blocks=0, rtextents=0
[ceph_deploy.osd][DEBUG ] Host node2 is now ready for osd use.
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr
OSdノードをアクティブにする
実行 ceph-deploy osd activate node2:/dev/sdb1 node3:/dev/sdb1
root@client:~/ceph# ceph-deploy osd activate node2:/dev/sdb1 node3:/dev/sdb1
[ceph_deploy.cli][INFO ] Invoked (1.4.0): /usr/bin/ceph-deploy osd activate node2:/dev/sdb1 node3:/dev/sdb1
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks node2:/dev/sdb1: node3:/dev/sdb1:
[node2][DEBUG ] connected to host: node2
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[ceph_deploy.osd][INFO ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.osd][DEBUG ] activating host node2 disk /dev/sdb1
[ceph_deploy.osd][DEBUG ] will use init type: upstart
[node2][INFO ] Running command: ceph-disk-activate --mark-init upstart --mount /dev/sdb1
[node2][WARNIN] got monmap epoch 1
[node2][WARNIN] 2016-01-12 15:26:22.781005 7f1967a62800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
[node2][WARNIN] 2016-01-12 15:26:23.132088 7f1967a62800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
[node2][WARNIN] 2016-01-12 15:26:23.136228 7f1967a62800 -1 filestore(/var/lib/ceph/tmp/mnt.LRhdRc) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
[node2][WARNIN] 2016-01-12 15:26:23.386713 7f1967a62800 -1 created object store /var/lib/ceph/tmp/mnt.LRhdRc journal /var/lib/ceph/tmp/mnt.LRhdRc/journal for osd.0 fsid 3a562301-bd64-45c5-aaa0-ef57e3dfd76f
[node2][WARNIN] 2016-01-12 15:26:23.386942 7f1967a62800 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.LRhdRc/keyring: can't open /var/lib/ceph/tmp/mnt.LRhdRc/keyring: (2) No such file or directory
[node2][WARNIN] 2016-01-12 15:26:23.387056 7f1967a62800 -1 created new key in keyring /var/lib/ceph/tmp/mnt.LRhdRc/keyring
[node2][WARNIN] added key for osd.0
[node3][DEBUG ] connected to host: node3
[node3][DEBUG ] detect platform information from remote host
[node3][DEBUG ] detect machine type
[ceph_deploy.osd][INFO ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.osd][DEBUG ] activating host node3 disk /dev/sdb1
[ceph_deploy.osd][DEBUG ] will use init type: upstart
[node3][INFO ] Running command: ceph-disk-activate --mark-init upstart --mount /dev/sdb1
[node3][WARNIN] got monmap epoch 1
[node3][WARNIN] 2016-01-12 15:26:27.482403 7f5efe4de800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
[node3][WARNIN] 2016-01-12 15:26:28.286329 7f5efe4de800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
[node3][WARNIN] 2016-01-12 15:26:28.416139 7f5efe4de800 -1 filestore(/var/lib/ceph/tmp/mnt.l2aD7V) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
[node3][WARNIN] 2016-01-12 15:26:29.444301 7f5efe4de800 -1 created object store /var/lib/ceph/tmp/mnt.l2aD7V journal /var/lib/ceph/tmp/mnt.l2aD7V/journal for osd.1 fsid 3a562301-bd64-45c5-aaa0-ef57e3dfd76f
[node3][WARNIN] 2016-01-12 15:26:29.444558 7f5efe4de800 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.l2aD7V/keyring: can't open /var/lib/ceph/tmp/mnt.l2aD7V/keyring: (2) No such file or directory
[node3][WARNIN] 2016-01-12 15:26:29.444832 7f5efe4de800 -1 created new key in keyring /var/lib/ceph/tmp/mnt.l2aD7V/keyring
[node3][WARNIN] added key for osd.1
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr
OSdノードでのアクティブ化の検証に成功しました
実行 ps aux | grep ceph-osd|grep -v grep
root@node2:~# ps aux | grep ceph-osd|grep -v grep
root 1752 3.9 2.9 518216 29572 ? Ssl 15:26 0:03 /usr/bin/ceph-osd --cluster=ceph -i 0 -f
root@node2:~#
表示/dev/sdb 1パーティションがマウントされましたroot@node2:~# df -TH
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/ubuntu1404--vg-root ext4 6.0G 1.4G 4.3G 24% /
none tmpfs 4.1k 0 4.1k 0% /sys/fs/cgroup
udev devtmpfs 510M 4.1k 510M 1% /dev
tmpfs tmpfs 105M 476k 104M 1% /run
none tmpfs 5.3M 0 5.3M 0% /run/lock
none tmpfs 521M 0 521M 0% /run/shm
none tmpfs 105M 0 105M 0% /run/user
/dev/sda1 ext2 247M 40M 195M 17% /boot
/dev/sdb1 xfs 8.2G 5.5G 2.8G 67% /var/lib/ceph/osd/ceph-0
それぞれ2つのosdノードで、/dev/sdb 1パーティションを/etc/fstabに追加して電源を入れて自動的にマウントします.
追加 /dev/sdb1 /var/lib/ceph/osd/ceph-0 xfs defaults 0 0
root@node2:~# vi /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#
/dev/mapper/ubuntu1404--vg-root / ext4 errors=remount-ro 0 1
# /boot was on /dev/sda1 during installation
UUID=54b04b5b-52ce-485f-95ef-70c667cfd8b3 /boot ext2 defaults 0 2
/dev/mapper/ubuntu1404--vg-swap_1 none swap sw 0 0
/dev/sdb1 /var/lib/ceph/osd/ceph-0 xfs defaults 0 0
~
mdsサービスの構成node 1に追加 metadata server
ceph-deploy mds create node 1の実行
root@client:~/ceph# ceph-deploy mds create node1
[ceph_deploy.cli][INFO ] Invoked (1.4.0): /usr/bin/ceph-deploy mds create node1
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node1:node1
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[ceph_deploy.mds][INFO ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.mds][DEBUG ] remote host will use upstart
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node1
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node1][DEBUG ] create path if it doesn't exist
[node1][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node1 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node1/keyring
[node1][INFO ] Running command: initctl emit ceph-mds cluster=ceph id=node1
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr
mdsプロセスの確認
root@node1:~# ps aux|grep mds
root 2025 0.1 1.6 165288 16280 ? Ssl 15:34 0:00 /usr/bin/ceph-mds --cluster=ceph -i node1 -f
root 2046 0.0 0.2 11748 2216 pts/0 S+ 15:36 0:00 grep --color=auto mds
cephファイルシステムの作成
root@node1:~# ceph osd pool create cephfs_data 10
pool 'cephfs_data' created
root@node1:~# ceph osd pool create cephfs_metadata 10
pool 'cephfs_metadata' created
root@node1:~# ceph fs new cephfs cephfs_metadata cephfs_data
no valid command found; 10 closest matches:
fsid
Error EINVAL: invalid command
root@node1:~# ceph fs ls
no valid command found; 10 closest matches:
fsid
Error EINVAL: invalid command
root@node1:~# ceph mds stat
e4: 1/1/1 up {0=node1=up:active}
root@node1:~#
コンフィギュレーションファイルの同期
実行 ceph-deploy admin node1 node2 node3
root@client:~/ceph# ceph-deploy admin node1 node2 node3
[ceph_deploy.cli][INFO ] Invoked (1.4.0): /usr/bin/ceph-deploy admin node1 node2 node3
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node1
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] get remote short hostname
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node2
[node2][DEBUG ] connected to host: node2
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[node2][DEBUG ] get remote short hostname
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node3
[node3][DEBUG ] connected to host: node3
[node3][DEBUG ] detect platform information from remote host
[node3][DEBUG ] detect machine type
[node3][DEBUG ] get remote short hostname
[node3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
クラスタの健康状態の表示root@node1:~# ceph health
HEALTH_OK
rbdの使用
新しいceph pool rados mkpool test poolにミラー(ipはmonノード)rbd create test-1--size 4096-p test-m 172.17.106.122 poolブロックデバイスにミラーをマッピングrbd map test-1-p test--name client.admin rbdのマッピング関係rbd showmappedを表示
root@node1:~# rados mkpool test
successfully created pool test
root@node1:~# lsmod |grep rbd
root@node1:~# rbd create test-1 --size 4096 -p test -m 192.168.3.8
root@node1:~# rbd map test-1 -p test --name client.admin
root@node1:~# rbd showmapped
id pool image snap device
0 test test-1 - /dev/rbd0
新しいミラーcephブロックをフォーマットする
root@node1:~# mkfs.ext4 -m0 /dev/rbd0
mke2fs 1.42.9 (4-Feb-2014)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=1024 blocks, Stripe width=1024 blocks
262144 inodes, 1048576 blocks
0 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1073741824
32 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
マウント
root@node1:~# mount/dev/rbd0/mnt
root@node1:~# df -TH
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/ubuntu1404--vg-root ext4 6.0G 1.4G 4.3G 24% /
none tmpfs 4.1k 0 4.1k 0% /sys/fs/cgroup
udev devtmpfs 510M 4.1k 510M 1% /dev
tmpfs tmpfs 105M 472k 104M 1% /run
none tmpfs 5.3M 0 5.3M 0% /run/lock
none tmpfs 521M 0 521M 0% /run/shm
none tmpfs 105M 0 105M 0% /run/user
/dev/sda1 ext2 247M 40M 195M 17% /boot
/dev/rbd0 ext4 4.1G 8.4M 4.1G 1% /mnt
/mntディレクトリが使用可能になりました