CentOS 6.6 默认安装的libvirt-0.10.2是不支持storage-rbd,若要支持Ceph Pool功能,需要重新编译libvirt。

重新编译libvirt

CentOS 6.6 的src.rpm包下载 libvirt-0.10.2-46.el6_6.6.src.rpm,然后执行编译:

rpmbuild --rebuild libvirt-0.10.2-46.el6_6.6.src.rpm

然后提示一堆需要的依赖包,自己安装进去。当然不是直接编译这个libvirt-0.10.2-46.el6_6.6.src.rpm。巧妙之处是,在编译过程中,Ctrl+c终止这个过程,然后修改配置文件

/root/rpmbuild/SPECS/libvirt.spec
79 %if 0%{?fedora} >= 16 `|| 0%{?centos} >= 6`
80 %define with_storage_rbd 0%{!?_without_storage_rbd:%{server_drivers}}
81 %else
82 %define with_storage_rbd 0
83 %endif

需要增加|| 0%{?centos} >= 6。OK,好,可以继续编译,编译命令是

rpmbuild --bb /root/rpmbuild/SPECS/libvirt.spec

不过又会遇到问题,你需要先安装ceph-devel包。后面继续报错,就是storage_backend_rbd.c里面定义了一个结构state,这个和include进来的头文件冲突,解决方法是把storage_backend_rbd.c定义那个:

state

更改为:

clusterstate

好,这样在执行

rpmbuild --bb /root/rpmbuild/SPECS/libvirt.spec

应该没有问题了。生成文件:

[root@ceph-libvirt x86_64]# ll
total 12012
-rw-r--r--. 1 root root 2928971 May 27 16:15 libvirt-0.10.2-46.0.1.el6.6.x86_64.rpm
-rw-r--r--. 1 root root 7256817 May 27 16:15 libvirt-client-0.10.2-46.0.1.el6.6.x86_64.rpm
-rw-r--r--. 1 root root 1044981 May 27 16:15 libvirt-devel-0.10.2-46.0.1.el6.6.x86_64.rpm
-rw-r--r--. 1 root root 275625 May 27 16:15 libvirt-lock-sanlock-0.10.2-46.0.1.el6.6.x86_64.rpm
-rw-r--r--. 1 root root 781811 May 27 16:15 libvirt-python-0.10.2-46.0.1.el6.6.x86_64.rpm

然后就是对接Ceph/RBD。先创建一个Pool:

[root@ceph-mon-1 libvirtd]# ceph osd pool create storage1 128 128
pool 'storage1' created
[root@ceph-mon-1 libvirtd]# ceph auth get-or-create client.storage1 mon 'allow r' osd 'allow rwx pool=storage1'
[client.storage1]
key = AQD4gWVVSFNRHRAA4WOd8unlNCxnbgPhBIhTlA==
[root@ceph-mon-1 libvirtd]#

这里创建了Pool并且生成了一个访问凭证。在计算节点编辑文件secret-storage1.xml

<secret ephemeral='no' private='no'>
<usage type='ceph'>
<name>storage1</name>
</usage>
</secret>

然后就是导入:

[root@ceph-libvirt ceph]# virsh secret-define --file secret-storage1.xml 
Secret de3ea4e8-fb19-9dfb-fe01-b0de5aa172fd created

然后就是填充密钥:

[root@ceph-libvirt ceph]# virsh 
Welcome to virsh, the virtualization interactive terminal.

Type: 'help' for help with commands
'quit' to quit

virsh # secret-list 
UUID Usage
de3ea4e8-fb19-9dfb-fe01-b0de5aa172fd Unused

virsh # secret-set-value de3ea4e8-fb19-9dfb-fe01-b0de5aa172fd AQD4gWVVSFNRHRAA4WOd8unlNCxnbgPhBIhTlA==
Secret value set

virsh #

编译一个pool文件storage1.xml:

<pool type='rbd'>
<name>storage1</name>
<source>
<host name='10.89.13.71' port='6789'/>
<name>storage1</name>
<auth type='ceph' username='storage1'>
<secret uuid='de3ea4e8-fb19-9dfb-fe01-b0de5aa172fd'/>
</auth>
</source>
</pool>

然后定义这个pool文件:

virsh pool-define storage1.xml

[root@ceph-libvirt ceph]# virsh 
Welcome to virsh, the virtualization interactive terminal.

Type: 'help' for help with commands
'quit' to quit

virsh # pool-list --all
Name State Autostart
storage1 inactive no 

virsh # pool-
pool-autostart pool-create-as pool-delete pool-edit pool-name pool-undefine

pool-build pool-define pool-destroy pool-info pool-refresh pool-uuid

pool-create pool-define-as pool-dumpxml pool-list pool-start

virsh # pool-start storage1
Pool storage1 started

virsh # pool-list --all
Name State Autostart
storage1 active no 

virsh # pool-dumpxml storage1
<pool type='rbd'>
<name>storage1</name>
<uuid>107550ae-3d6c-38de-9a78-55069ec62d03</uuid>
<capacity unit='bytes'>1173478178816</capacity>
<allocation unit='bytes'>0</allocation>
<available unit='bytes'>1170975944704</available>
<source>
<host name='10.89.13.71' port='6789'/>
<name>storage1</name>
<auth username='storage1' type='ceph'>
<secret uuid='de3ea4e8-fb19-9dfb-fe01-b0de5aa172fd'/>
</auth>
</source>
</pool>

咔咔,泄露了容量信息 .>_<. 然后创建一个块存储:

[root@ceph-libvirt ceph]# virsh 
Welcome to virsh, the virtualization interactive terminal.

Type: 'help' for help with commands
'quit' to quit

virsh # vol-create-as --pool 107550ae-3d6c-38de-9a78-55069ec62d03 --name image1 --capacity 10G --backing-vol-format 2
virsh # vol-list --pool 107550ae-3d6c-38de-9a78-55069ec62d03
Name Path
image1 storage1/image1 

virsh # vol-dumpxml --pool 107550ae-3d6c-38de-9a78-55069ec62d03 image1
<volume>
<name>image1</name>
<key>storage1/image1</key>
<source>
</source>
<capacity unit='bytes'>10737418240</capacity>
<allocation unit='bytes'>10737418240</allocation>
<target>
<path>storage1/image1</path>
<format type='unknown'/>
<permissions>
<mode>0600</mode>
<owner>4294967295</owner>
<group>4294967295</group>
</permissions>
</target>
</volume>

virsh #

看看Ceph那边:

[root@ceph-mon-1 libvirtd]# rbd ls storage1
image1
[root@ceph-mon-1 libvirtd]# rbd info storage1/image1
rbd image 'image1':
size 10240 MB in 2560 objects
order 22 (4096 kB objects)
block_name_prefix: rb.0.224fbb.6b12cdbd
format: 1

聪明的cepher估计已经看到一些瑕疵地方,估计在新版本会修复吧。

完毕。