ceph初窥之安装篇

2013-08-02 | pyinx | 未分类

系统架构图
角色 Ip地址主机名备注
ceph_mon 100.0.0.2 ceph1
Ubuntu server 12.04
关闭selinux和iptables
ceph_mds 100.0.0.2 ceph1
ceph_osd0 100.0.0.2 ceph1
ceph_osd1 100.0.0.19 ceph2
ceph_client 100.0.0.20 client

2. 统一hosts文件
100.0.0.2 ceph1
100.0.0.19 ceph2
两台ceph server之间做ssh互相（基于主机名而不是ip）
ceph服务器配置
A. 安装ceph软件包（ceph1、ceph2和client上都执行这步）

#sudo wget -q -O- ‘https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc‘ | sudo
apt-key add -

#sudo echo deb http://ceph.com/debian/ $(lsb_release -sc) main | sudo tee
/etc/apt/sources.list.d/ceph.list

sudo apt-get update && sudo apt-get install ceph

B. 在ceph1上修改配置文件/etc/ceph/ceph.conf，内容如下：
[global]
auth cluster required = none
auth service required = none
auth client required = none
[osd]
osd journal size = 1000
filestore xattr use omap = true
osd mkfs type = xfs
[mon]
mon data = /var/log/ceph/mon$id
debug ms = 1
[mon.a]
host = ceph1
mon addr = 100.0.0.2:6789
[osd.0]
host = ceph1
devs = /dev/vdb1
[osd.1]
host = ceph2
devs = /dev/vdb1
[mds.a]
host = ceph1
[group everyone]
addr = 0.0.0.0/0
[mount /]
allow = %everyone
C. 将配置文件拷贝到ceph2上
scp /etc/ceph/ceph.conf ceph2:/etc/ceph/

D. 创建目录、挂载分区

Ceph1：
mkdir -p /var/lib/ceph/osd/ceph-0
mkdir -p /var/lib/ceph/mon/ceph-a
mkdir -p /var/lib/ceph/mds/ceph-a
mkfs.xfs -f /dev/sdb1
mount /dev/sdb1 /var/lib/ceph/osd/ceph-0

Ceph2：
mkdir -p /var/lib/ceph/osd/ceph-1
mkfs.xfs -f /dev/sdb1
mount /dev/sdb1 /var/lib/ceph/osd/ceph-1
在ceph1上初始化
mkcephfs -a -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.keyring

temp dir is /tmp/mkcephfs.dDN21h8na7
preparing monmap in /tmp/mkcephfs.dDN21h8na7/monmap
/usr/bin/monmaptool –create –clobber –add a 100.0.0.2:6789 –print /tmp/mkcephfs.dDN21h8na7/monmap
/usr/bin/monmaptool: monmap file /tmp/mkcephfs.dDN21h8na7/monmap
/usr/bin/monmaptool: generated fsid e0015518-37a0-4b43-a0bf-d347f9d42b38
epoch 0
fsid e0015518-37a0-4b43-a0bf-d347f9d42b38
last_changed 2013-08-02 07:32:08.071444
created 2013-08-02 07:32:08.071444
0: 100.0.0.2:6789/0 mon.a
/usr/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.dDN21h8na7/monmap (1 monitors)

WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see:
http://github.com/ceph/ceph-deploy
=== osd.0 ===
2013-08-02 07:32:08.189925 7f21c3879780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
2013-08-02 07:32:08.225072 7f21c3879780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
2013-08-02 07:32:08.226385 7f21c3879780 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2013-08-02 07:32:08.255304 7f21c3879780 -1 created object store /var/lib/ceph/osd/ceph-0 journal /var/lib/ceph/osd/ceph-0/journal for osd.0 fsid e0015518-37a0-4b43-a0bf-d347f9d42b38
2013-08-02 07:32:08.255688 7f21c3879780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can’t open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory
2013-08-02 07:32:08.256205 7f21c3879780 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring

WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see:
http://github.com/ceph/ceph-deploy
=== osd.1 ===
pushing conf and monmap to ceph2:/tmp/mkfs.ceph.CXXnr0kCxyhjeKgzi3iLbYjXEUHCKjGz
2013-08-02 07:32:10.572172 7f461acd0780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
2013-08-02 07:32:10.613648 7f461acd0780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
2013-08-02 07:32:10.615259 7f461acd0780 -1 filestore(/var/lib/ceph/osd/ceph-1) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2013-08-02 07:32:10.646996 7f461acd0780 -1 created object store /var/lib/ceph/osd/ceph-1 journal /var/lib/ceph/osd/ceph-1/journal for osd.1 fsid e0015518-37a0-4b43-a0bf-d347f9d42b38
2013-08-02 07:32:10.647225 7f461acd0780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can’t open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory
2013-08-02 07:32:10.647536 7f461acd0780 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring

WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see:
http://github.com/ceph/ceph-deploy
collecting osd.1 key
=== mds.a ===
creating private key for mds.a keyring /var/lib/ceph/mds/ceph-a/keyring
creating /var/lib/ceph/mds/ceph-a/keyring

WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see:
http://github.com/ceph/ceph-deploy
Building generic osdmap from /tmp/mkcephfs.dDN21h8na7/conf
/usr/bin/osdmaptool: osdmap file ‘/tmp/mkcephfs.dDN21h8na7/osdmap’
/usr/bin/osdmaptool: writing epoch 1 to /tmp/mkcephfs.dDN21h8na7/osdmap
Generating admin key at /tmp/mkcephfs.dDN21h8na7/keyring.admin
creating /tmp/mkcephfs.dDN21h8na7/keyring.admin
Building initial monitor keyring
added entity mds.a auth auth(auid = 18446744073709551615 key=AQB7YPtRaJadIhAAiZghsDnRrkOuTE+0wtXotg== with 0 caps)
added entity osd.0 auth auth(auid = 18446744073709551615 key=AQB4YPtR4OtCDxAAk5oOBSowHA19DkCoMos4uw== with 0 caps)
added entity osd.1 auth auth(auid = 18446744073709551615 key=AQB6YPtR6G6WJhAAj6UiKhCYrYN/eIB3x6bbAQ== with 0 caps)

WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see:
http://github.com/ceph/ceph-deploy
=== mon.a ===
/usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a

WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see:
http://github.com/ceph/ceph-deploy
placing client.admin keyring in /etc/ceph/ceph.keyring
启动服务

#service ceph -a start
=== mon.a ===
Starting Ceph mon.a on ceph1…
Starting ceph-create-keys on ceph1…
=== mds.a ===
Starting Ceph mds.a on ceph1…
starting mds.a at :/0
=== osd.0 ===
Mounting xfs on ceph1:/var/lib/ceph/osd/ceph-0
osd.0 does not exist. create it before updating the crush map
Starting Ceph osd.0 on ceph1…
starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
=== osd.1 ===
Mounting xfs on ceph2:/var/lib/ceph/osd/ceph-1
osd.1 does not exist. create it before updating the crush map
Starting Ceph osd.1 on ceph2…
starting osd.1 at :/0 osd_data /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1/journal
检查服务是否正常

#ceph health （服务启动完后等会再试）
HEALTH_OK
客户端挂载
mount -t ceph 100.0.0.2:6789:/ /mnt