Ceph 命令介绍 | 谢先斌的博客

记录 ceph 常用命令；ceph 快照管理方法；qemu 使用 ceph rbd 块设备方法；rbd 对接 openstack cinder 和 glance 方法

进程相关命令

查看集群的状态

    
    bash
  
ceph health (detail)或 ceph -w

ceph 进程操作

启动/停止/重启 ceph 进程（当前节点）

bash

sudo /etc/init.d/ceph start/stop/restart

启动/停止/重启 ceph 进程（所有节点）

bash

sudo /etc/init.d/ceph -a start/stop/restart

ceph 守护进程类型：mon osd mds

通过守护进程类型重启服务

语法：

    
    bash
  
sudo /etc/init.d/ceph [options] [start|restart] [daemonType|daemonID]

示例：

启动/停止指定类型的 ceph 守护进程（当前节点）

bash

sudo /etc/init.d/ceph start/stop mon/osd/mds

启动/停止指定类型的 ceph 守护进程（所有节点）

bash

sudo /etc/init.d/ceph -a start/stop mon/osd/mds

或采用：

    
    bash
  
sudo service ceph [options] [start|restart] [daemonType|daemonID]

启动或停止一个守护进程

语法：

    
    bash
  
sudo /etc/init.d/ceph start/stop {daemon-type}.{instance}

或

    
    bash
  
sudo /etc/init.d/ceph -a start/stop {daemon-type}.{instance}

或

    
    bash
  
sudo service ceph start {daemon-type}.{instance}

示例：

当前节点

bash

sudo /etc/init.d/ceph start osd.0

所有节点

bash

sudo /etc/init.d/ceph -a start osd.0

monitor 相关命令

监控包括：osd，mon，pg，mds

交互模式命令如下：

    
    bash
  
# ceph
ceph> health
ceph> status
ceph> quorum_status
ceph> mon_status
ceph quorum_status --format json-pretty

监视集群状态

bash

ceph -w

检测集群资源使用状态

    
    bash
  
ceph df
ceph status
ceph -s

检测 osd 的状态命令

    
    bash
  
ceph osd stat
ceph osd dump
ceph osd tree
ceph osd df

检测 mon 的状态命令

bash

ceph mon stat
ceph mon dump

查看 mon 选举状态

bash

ceph quorum_status

检测 mds 的状态命令

bash

ceph mds stat
ceph mds dump

注释：metadta services 有两个状态集 up|down active|inactive

rbd 相关命令

准备

    
    bash
  
ceph osd pool create xiexianbin 16
ceph osd pool set rbd pgp_num 128 pg_num 128

创建 block device image

text

rbd create bar --size 10240 --pool xiexianbin

list block device image

text

rbd ls {poolname}
rbd ls xiexianbin

查看指定 image 的详细信息

text

rbd --image {image-name} -p {pool-name} info
rbd --image bar -p xiexianbin info

resize a block device image

text

rbd resize --image bar -p xiexianbin --size 20480

removing a block device image

text

rbd rm {image-name} -p {pool-name}
rbd rm bar -p xiexianbin

内核模块操作

准备：

text

rbd create image-block --size 10240  --pool xiexianbin

列出指定 pool 中的所有 images

text

rbd ls/list {poolname}

映射块设备

    
    bash
  
sudo rbd map {image-name} --pool {pool-name} --id {user-name} --keyring {keyring_path}
sudo rbd map --pool xiexianbin image-block

存在问题：若块设备名称为 image-block-1 将会失败。命令如下：

    
    bash
  
sudo rbd map image-block-1 --pool xiexianbin(错误)

查看块设备映射

text

rbd showmapped

解除块设备映射

    
    bash
  
sudo rbd unmap /dev/rbd/{poolname}/{imagename}
sudo rbd unmap /dev/rbd1

mount 命令

    
    bash
  
sudo mkfs.ext4 /dev/rbd1
mkdir /home/ceph/xxb/rbd1
sudo mount /dev/rbd1 /home/ceph/xxb/rbd1

快照

注意：为 image 创建快照是，image 应处于一个稳定的状态（即没有数据操作）。

创建

text

rbd --pool {pool-name} snap create --snap {snap-name} {image-name}
rbd snap create {pool-name}/{image-name}@{snap-name}

例如：

text

rbd --pool xiexianbin snap create --snap bak1 image-block
rbd snap create xiexianbin/image-block@bak2

查询

    
    text
  
rbd --pool {pool-name} snap ls {image-name}
rbd snap ls {pool-name}/{image-name}
rbd --pool xiexianbin snap ls/list image-block
rbd snap ls xiexianbin/image-block

回滚

    
    text
  
rbd --pool {pool-name} snap rollback --snap {snap-name} {image-name}
rbd snap rollback {pool-name}/{image-name}@{snap-name}
rbd --pool xiexianbin snap rollback --snap bak1 image-block
rbd snap rollback xiexianbin/image-block@bak2

删除

    
    text
  
rbd --pool {pool-name} snap rm --snap {snap-name} {image-name}
rbd snap rm {pool-name}/{image-name}@{snap-name}
rbd --pool xiexianbin snap rm --snap bak2 image-block
rbd snap rm xiexianbin/image-block@bak1

清空快照

    
    text
  
rbd --pool {pool-name} snap purge {image-name}
rbd snap purge {pool-name}/{image-name}
rbd --pool xiexianbin snap purge image-block
rbd snap purge xiexianbin/image-block

qemu

常用命令

    
    bash
  
qemu-img {command} [options] rbd:glance-pool/maipo:id=glance:conf=/etc/ceph/ceph.conf

qemu-img create -f raw rbd:{pool-name}/{image-name} {size}

qemu-img create -f raw rbd:data/foo 10G

qemu-img resize rbd:{pool-name}/{image-name} {size}

qemu-img resize rbd:data/foo 10G

qemu-img info rbd:{pool-name}/{image-name}

qemu-img info rbd:data/foo

RUNNING QEMU WITH RBD

You can use qemu-img to convert existing virtual machine images to Ceph block device images. For example, if you have a qcow2 image, you could run:

bash

qemu-img convert -f qcow2 -O raw debian_squeeze.qcow2 rbd:data/squeeze

To run a virtual machine booting from that image, you could run:

text

qemu -m 1024 -drive format=raw,file=rbd:data/squeeze

RBD caching can significantly improve performance. Since QEMU 1.2, QEMU’s cache options control librbd caching:

text

qemu -m 1024 -drive format=rbd,file=rbd:data/squeeze,cache=writeback

libvirt

configure ceph

http://docs.ceph.com/docs/v0.94.4/rbd/libvirt/

Create a pool

    
    bash
  
ceph osd pool create libvirt-pool 64 64

Create a Ceph User

    
    bash
  
ceph auth get-or-create client.libvirt mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=libvirt-pool'

Use QEMU to create an image in your RBD pool

bash

qemu-img create -f rbd rbd:libvirt-pool/new-libvirt-image 2G

rbd 对接 openstack

创建池

    
    bash
  
ceph osd pool create volumes 32 32
ceph osd pool create images 16 16
ceph osd pool create backups 32 32
ceph osd pool create vms 32 32

配置 OpenStack ceph 客户端

http://docs.ceph.com/docs/v0.94.4/rbd/rbd-openstack/

    
    bash
  
ssh {your-openstack-server} sudo tee /etc/ceph/ceph.conf </etc/ceph/ceph.conf

对于 glance-api 端，安装 rbd

bash

sudo yum install python-rbd

对于 nova-compute、cinder-backups 和 cinder-volume 端安装 ceph

bash

sudo yum install ceph

创建用户

    
    bash
  
ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images'
ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
ceph auth get-or-create client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'

将 keyring 同步到指定机器

    
    bash
  
ceph auth get-or-create client.glance | ssh {your-glance-api-server} sudo tee /etc/ceph/ceph.client.glance.keyring
ssh {your-glance-api-server} sudo chown glance:glance /etc/ceph/ceph.client.glance.keyring
ceph auth get-or-create client.cinder | ssh {your-volume-server} sudo tee /etc/ceph/ceph.client.cinder.keyring
ssh {your-cinder-volume-server} sudo chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring
ceph auth get-or-create client.cinder-backup | ssh {your-cinder-backup-server} sudo tee /etc/ceph/ceph.client.cinder-backup.keyring
ssh {your-cinder-backup-server} sudo chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring

获取 key

    
    bash
  
ceph auth get-key client.cinder | ssh {your-compute-node} tee client.cinder.key

配置 secret

text

# uuidgen
457eb676-33da-42ec-9a8c-9293d545c337

    
      
      
    
    bash
  
cat > secret.xml <<EOF
<secret ephemeral='no' private='no'>
  <uuid>457eb676-33da-42ec-9a8c-9293d545c337</uuid>
  <usage type='ceph'>
    <name>client.cinder secret</name>
  </usage>
</secret>
EOF

bash

sudo virsh secret-define --file secret.xml
Secret 457eb676-33da-42ec-9a8c-9293d545c337 created

    
    bash
  
sudo virsh secret-set-value --secret 457eb676-33da-42ec-9a8c-9293d545c337 --base64 $(cat client.cinder.key) && rm client.cinder.key secret.xml

其他

查看集群的详细配置

bash

ceph daemon mon.compute-192-168-2-202 config show | more

查看 ceph log 日志所在的目录

text

ceph-conf --name mon.compute-192-168-2-202 --show-config-value log_file

参照：

http://docs.ceph.com/docs/master/rados/operations/operating/#running-ceph-with-sysvinit