search.xml

<?xml version="1.0" encoding="utf-8"?>
<search>
  
    
    <entry>
      <title><![CDATA[CentOS7.2安装“X Window System”之后，启动进不了系统]]></title>
      <url>%2F2017%2F12%2F04%2Fcentos-install-xwindow-error%2F</url>
      <content type="text"><![CDATA[CentOS7.2安装“X Window System”之后，启动进不了系统，出现以下提示信息：1234Initial setup of CentOS Linux 7 (Core)1) [!] License information 2) [ ] User creation (License not accepted) (no user will be created)Please make your choice from above ['q' to quit | 'c' to continue | 'r' to refresh]: 解决方法其实解决方法比较显然，根据提示一步步操作即可： 输入1，回车显示License信息 根据提示信息，输入2，接受授权 键入q，离开设置; 输入yes，保存信息即可进入CentOS7.2的登录界面。 参考http://www.hawkers.cc/2015/12/CentOS-initial-setup.html]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[CentOS7.2安装Oracle 12c]]></title>
      <url>%2F2017%2F11%2F30%2Fcentos-install-oracle%2F</url>
      <content type="text"><![CDATA[在CentOS7.2上安装Oracle数据库12c。 环境 CentOS Linux release 7.2.1511 (Core) Mini版 本地 Win7 安装软件包安装所需软件包（可以先执行 yum update -y 更新系统，也可以不更新）1[root@localhost ~]# yum install -y binutils.x86_64 compat-libcap1.x86_64 gcc.x86_64 gcc-c++.x86_64 glibc.i686 glibc.x86_64 glibc-devel.i686 glibc-devel.x86_64 ksh compat-libstdc++-33 libaio.i686 libaio.x86_64 libaio-devel.i686 libaio-devel.x86_64 libgcc.i686 libgcc.x86_64 libstdc++.i686 libstdc++.x86_64 libstdc++-devel.i686 libstdc++-devel.x86_64 libXi.i686 libXi.x86_64 libXtst.i686 libXtst.x86_64 make.x86_64 sysstat.x86_64 smartmontools net-tools zip unzip 配置桌面安装X Window系统1[root@localhost ~]# yum groupinstall -y "X Window System" Oracle配置创建用户及群组123456789[root@localhost ~]# groupadd oinstall[root@localhost ~]# groupadd dba[root@localhost ~]# useradd -g oinstall -G dba oracle[root@localhost ~]# passwd oracleChanging password for user oracle.New password:BAD PASSWORD: The password is shorter than 8 charactersRetype new password:passwd: all authentication tokens updated successfully. 修改/etc/sysctl.conf文件，增加以下内核参数配置 fs.aio-max-nr = 1048576 fs.file-max = 6815744 kernel.shmall = 2097152 kernel.shmmax = 1987162112 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 net.ipv4.ip_local_port_range = 9000 65500 net.core.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048586 应用新的内核参数值，并显示所有内核参数 123456789101112131415[root@localhost ~]# sysctl -pfs.aio-max-nr = 1048576fs.file-max = 6815744kernel.shmall = 2097152kernel.shmmax = 1987162112kernel.shmmni = 4096kernel.sem = 250 32000 100 128net.ipv4.ip_local_port_range = 9000 65500net.core.rmem_default = 262144net.core.rmem_max = 4194304net.core.wmem_default = 262144net.core.wmem_max = 1048586[root@localhost ~]# sysctl -a........ 修改 /etc/security/limits.conf，增加以下配置为oracle设置上限 oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536 创建/stage目录，并更改用户和组12[root@localhost ~]# mkdir /stage/[root@localhost ~]# chown -R oracle:oinstall /stage/ 为Oracle安装文件和Oracle数据库文件创建新目录，并赋权1234[root@localhost ~]# mkdir -p /u01 /u02[root@localhost ~]# chown -R oracle:oinstall /u01 /u02[root@localhost ~]# chmod -R 775 /u01 /u02[root@localhost ~]# chmod g+s /u01 /u02 下载&amp;解压Oracle下载Oracle Database 12c Release 2的Linux x86-64版本：linuxx64_12201_database.zip 解压oracle安装文件1[root@localhost ~]# unzip linuxx64_12201_database.zip -d /stage/ GUI安装Oracle运行Xming下载Xming http://sourceforge.net/projects/xming/files/latest/download，安装并运行Xming SSH连接打开PuTTY， 在PuTTY的设置窗口，选择 Connection –&gt; SSH –&gt; X11，勾选“Enable X11 forwarding”选项 点击Session，输入连接的主机名称或IP地址 使用oracle用户登录 安装Oracle切换到/stage/database/目录，执行runInstaller脚本安装Oracle12345678[oracle@localhost ~]$ cd /stage/database/[oracle@localhost database]$ ./runInstallerStarting Oracle Universal Installer...Checking Temp space: must be greater than 500 MB. Actual 39203 MB PassedChecking swap space: must be greater than 150 MB. Actual 7807 MB PassedChecking monitor: must be configured to display at least 256 colors. Actual 16777216 PassedPreparing to launch Oracle Universal Installer from /tmp/OraInstall2017-11-29_03-01-59AM. Please wait ...[oracle@localhost database]$ 图形化安装过程上面执行runInstaller脚本之后，弹出Oracle图形化安装界面，点击“Next&gt;” 在“Installation Option”中，选择“Create and configure a database”，然后点击“Next&gt;” 在“System Class”中，选择“Desktop class”，然后点击“Next&gt;” 在“Typical Installation”中，设置： Oracle base：/u01/app/oracle Software location： /u01/app/oracle/product/12.2.0/dbhome_1 Database file location： /u02 OSDBA group： dba Global database name：输入名称 Password：输入密码 Confirm password：再次输入 取消选中“Create as Container database” 这里告警提示管理密码:[INS-30011]输入的 ADMIN 口令不符合 Oracle 建议的标准，可以直接忽略点击“Next&gt;”。输入密码遵循“小写字母+数字+大写字母”的方式就可以消除告警，然后点击“Next&gt;” 在“Create Inventory”中，设置： Inventory Directory： /u01/app/oraInventory oraInventory Group Name：使用’ oinstall ‘组 然后点击“Next&gt;” 在“Prerequisite Checks”中，勾选“Ignore All”（这里如果有其他的check比如smartmontools、net-tools包没装，需要先安装好），然后点击“Next&gt;” 在“Summary”中，如果有修改的可以在这里编辑相应配置，点击“Next&gt;” 安装进度 需要以root用户执行以下两个sh脚本 执行orainstRoot.sh脚本1234567[root@localhost ~]# sh /u01/app/oraInventory/orainstRoot.shChanging permissions of /u01/app/oraInventory.Adding read,write permissions for group.Removing read,write,execute permissions for world.Changing groupname of /u01/app/oraInventory to oinstall.The execution of the script is complete. 执行root.sh脚本，然后回车键设置local bin directory，选择安装Oracle Trace File Analyzer1234567891011121314151617181920212223[root@localhost ~]# sh /u01/app/oracle/product/12.2.0/dbhome_1/root.shPerforming root user operation.The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/app/oracle/product/12.2.0/dbhome_1Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ...Creating /etc/oratab file...Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinished running generic part of root script.Now product-specific root actions will be performed.Do you want to setup Oracle Trace File Analyzer (TFA) now ? yes|[no] :yesInstalling Oracle Trace File Analyzer (TFA).Log File: /u01/app/oracle/product/12.2.0/dbhome_1/install/root_localhost.localdomain_2017-11-29_05-16-20-360969054.logFinished installing Oracle Trace File Analyzer (TFA) 安装完成 测试sqlplus登录使用oracle用户登录数据库服务器，设置oracle环境，使用sqlplus登录orcl，可以执行SQL语句来查看“select * from v$instance”12345678910111213141516[root@localhost ~]# su oracle[oracle@localhost root]$ export ORACLE_SID=orcl[oracle@localhost root]$ export ORACLE_HOME=/u01/app/oracle/product/12.2.0/dbhome_1/[oracle@localhost root]$ export PATH=$PATH:$ORACLE_HOME/bin[oracle@localhost root]$ sqlplus system@orclSQL*Plus: Release 12.2.0.1.0 Production on Thu Nov 30 03:09:17 2017Copyright (c) 1982, 2016, Oracle. All rights reserved.Enter password:Connected to:Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit ProductionSQL&gt; select * from v$instance; Web登录先关闭防火墙1[root@localhost ~]# systemctl stop firewalld.service 浏览器访问 https://[oracle ip]:5500/em/，使用system用户，在“Typical Installation”中设置的密码登录 登录之后 参考https://wiki.centos.org/HowTos/Oracle12onCentos7 https://wiki.centos.org/zh/HowTos/Xming 问题安装“X Window System”之后，启动进不了系统 请参考CentOS7.2安装“X Window System”之后，启动进不了系统]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[CentOS7.2执行sh脚本出错 /usr/bin/expect^M: bad interpreter: No such file or directory]]></title>
      <url>%2F2017%2F11%2F24%2Fcentos-sh%2F</url>
      <content type="text"><![CDATA[在CentOS7.2上执行CeTune/deploy/prepare-scripts/configure_autossh.sh脚本出错，错误信息如下 12[root@node210 prepare-scripts]# ./configure_autossh.sh 127.0.0.1 root123-bash: ./configure_autossh.sh: /usr/bin/expect^M: bad interpreter: No such file or directory 分析这是不同系统编码格式引起的。在windows系统中编辑的.sh文件可能有不可见字符，所以在Linux系统下执行会报以上异常信息。 解决方法在Linux中转换首先要确保文件有可执行权限1[root@node210 prepare-scripts]# chmod a+x configure_autossh.sh 然后修改文件格式1[root@node210 prepare-scripts]# vi configure_autossh.sh 利用如下命令查看文件格式1:set ff 或 :set fileformat 可以看到 fileformat=dos 或 fileformat=unix利用如下命令修改文件格式12:set ff=unix 或 :set fileformat=unix :wq (存盘退出) 最后再执行文件1[root@node210 prepare-scripts]# ./configure_autossh.sh 127.0.0.1 root123 在windows下转换利用一些编辑器如UltraEdit或EditPlus等工具先将脚本编码转换成UNIX，再上传到Linux。]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[CentOS7.2安装systemtap]]></title>
      <url>%2F2017%2F09%2F26%2Fcentos-install-systemtap%2F</url>
      <content type="text"><![CDATA[systemtap是监控和跟踪运行中的 Linux 内核的操作的动态方法。 环境 CentOS Linux release 7.2.1511 (Core) Min版 1234[root@node ~]# cat /etc/redhat-releaseCentOS Linux release 7.2.1511 (Core)[root@node ~]# uname -r3.10.0-327.el7.x86_64 安装依赖安装systemtap依赖的rpm包12[root@node ~]# yum install --downloadonly --downloaddir=/tmp/other gcc zip unzip perl mokutil libdwarf boost-system boost-thread kernel-devel-uname-r[root@node ~]# yum localinstall -yC --disablerepo=* /tmp/other/*.rpm 安装systemtap12[root@node ~]# yum install --downloadonly --downloaddir=/tmp/systemtap systemtap[root@node ~]# yum localinstall -yC --disablerepo=* /tmp/systemtap/*.rpm 安装内核调试包查看内核版本 12[root@node ~]# uname -r3.10.0-327.el7.x86_64 确认下载与内核版本一致的调试包，安装debuginfo包需要在yum命令中添加 --enablerepo=*-debuginfo1[root@node ~]# yum --enablerepo=*-debuginfo install --downloadonly --downloaddir=/tmp/kernel-debuginfo kernel-debuginfo-3.10.0-327.el7 kernel-debuginfo-common-x86_64-3.10.0-327.el7 由于kernel-debuginfo、kernel-debuginfo-common包比较大，可以直接下载http://debuginfo.centos.org/7/x86_64/kernel-debuginfo-3.10.0-327.el7.x86_64.rpmhttp://debuginfo.centos.org/7/x86_64/kernel-debuginfo-common-x86_64-3.10.0-327.el7.x86_64.rpm kernel-devel下载地址，下载之后放入/tmp/kernel-debuginfo/目录http://rpm.pbone.net/index.php3/stat/4/idpl/31979931/dir/centos_7/com/kernel-devel-3.10.0-327.el7.x86_64.rpm.html 安装kernel-debuginfo、kernel-debuginfo-common、kernel-devel包1[root@node ~]# yum localinstall -yC --disablerepo=* /tmp/kernel-debuginfo/*.rpm 安装验证验证，如果没有提示错误，就是安装成功了。12345678[root@node ~]# stap -v -e 'probe vfs.read&#123;printf("read performed\n"); exit()&#125;'Pass 1: parsed user script and 466 library scripts using 226048virt/39160res/3304shr/36016data kb, in 420usr/40sys/623real ms.Pass 2: analyzed script: 1 probe, 1 function, 7 embeds, 0 globals using 352356virt/166700res/4596shr/162324data kb, in 2120usr/720sys/5246real ms.Pass 3: translated to C into "/tmp/stapgSQMsO/stap_4a25503e3b30a75bc7067bab5f510801_2661_src.c" using 352356virt/166964res/4860shr/162324data kb, in 10usr/50sys/64real ms.Pass 4: compiled C into "stap_4a25503e3b30a75bc7067bab5f510801_2661.ko" in 8550usr/1530sys/10476real ms.Pass 5: starting run.read performedPass 5: run completed in 0usr/40sys/346real ms. 安装ceph调试信息包下载ceph-debuginfo包，包比较大，有1G多，从网易下载对应ceph版本的ceph-debuginfohttp://mirrors.163.com/ceph/rpm-luminous/el7/x86_64/ceph-debuginfo-12.2.0-0.el7.x86_64.rpm 安装ceph-debuginfo 1[root@node ~]# yum localinstall -yC --disablerepo=* ceph-debuginfo/*.rpm]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[CentOS7.2编译nbd内核模块]]></title>
      <url>%2F2017%2F09%2F14%2Fcentos-build-nbd%2F</url>
      <content type="text"><![CDATA[CentOS7.2默认没有nbd的内核模块，需要手动编译nbd内核模块。 环境CentOS7.212[root@localhost ~]# cat /etc/redhat-releaseCentOS Linux release 7.2.1511 (Core) 内核版本12[root@localhost ~]# uname -r3.10.0-327.el7.x86_64 安装rpmbuild和编译的相关依赖12[root@localhost ~]# yum install rpm-build -y[root@localhost ~]# yum install m4 net-tools bc xmlto asciidoc hmaccalc newt-devel perl pesign elfutils-devel binutils-devel bison audit-libs-devel numactl-devel pciutils-devel ncurses-devel libtiff perl-ExtUtils-Embed -y 安装内核rpm从 http://vault.centos.org 下载CentOS7.2的kernel src rpm包并使用builder用户安装。 添加builder用户和组12[root@localhost opt]# useradd builder[root@localhost opt]# groupadd builder 下载CentOS7.2最新的kernel rpm123[root@localhost ~]# cd /opt/[root@localhost opt]# wget http://vault.centos.org/7.2.1511/updates/Source/SPackages/kernel-3.10.0-327.36.3.el7.src.rpm[root@localhost opt]# cp kernel-3.10.0-327.36.3.el7.src.rpm /home/builder 安装kernel-3.10.0-327.36.3.el7.src.rpm123[root@localhost opt]# su builder[builder@localhost opt]$ cd ~[builder@localhost ~]$ rpm -ivh kernel-3.10.0-327.36.3.el7.src.rpm 准备编译创建rpmbuild目录。kernel-3.10.0-327.36.3.el7.src.rpm安装后的源码和kernel.spec文件分别自动生成到~/rpmbuild/SOURCES/目录和~/rpmbuild/SPECS/目录123456789[builder@localhost ~]$ mkdir -p ~/rpmbuild/&#123;BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS&#125;[builder@localhost ~]$ ls ~/rpmbuild/SOURCES/centos.cer cpupower.service kernel-3.10.0-ppc64.config kernel-3.10.0-s390x-debug.config linux-3.10.0-327.36.3.el7.tar.xz Module.kabi_s390xcentos-kpatch.x509 debrand-rh-i686-cpu.patch kernel-3.10.0-ppc64-debug.config kernel-3.10.0-s390x-kdump.config linux-kernel-test.patch Module.kabi_x86_64centos-ldup.x509 debrand-rh_taint.patch kernel-3.10.0-ppc64le.config kernel-3.10.0-x86_64.config Makefile.common secureboot.cercheck-kabi debrand-single-cpu.patch kernel-3.10.0-ppc64le-debug.config kernel-3.10.0-x86_64-debug.config Module.kabi_ppc64 sign-modulescpupower.config extra_certificates kernel-3.10.0-s390x.config kernel-abi-whitelists-327.tar.bz2 Module.kabi_ppc64le x509.genkey[builder@localhost ~]$ ls ~/rpmbuild/SPECS/kernel.spec 解包源代码123[builder@localhost ~]$ echo '%_topdir %(echo $HOME)/rpmbuild' &gt; ~/.rpmmacros[builder@localhost ~]$ cd ~/rpmbuild/SPECS[builder@localhost ~]$ rpmbuild -bp --target=$(uname -m) kernel.spec 编译执行make menuconfig配置内核配置信息，在弹出的窗口中选择Device Driver -&gt; Block devices -&gt; Network block device support，并在“Network block device support”中输入“M”，保存退出12[builder@localhost ~]$ cd ~/rpmbuild/BUILD/kernel-3.10.0-327.36.3.el7/linux-3.10.0-327.36.3.el7.x86_64[builder@localhost linux-3.10.0-327.36.3.el7.x86_64]$ make menuconfig 编译12[builder@localhost linux-3.10.0-327.36.3.el7.x86_64]$ make prepare &amp;&amp; make modules_prepare &amp;&amp; make[builder@localhost linux-3.10.0-327.36.3.el7.x86_64]$ make M=drivers/block -j8 查看nbd模块，并复制到内核目录，加载nbd12345678910111213141516[root@localhost linux-3.10.0-327.36.3.el7.x86_64]# su builder[builder@localhost linux-3.10.0-327.36.3.el7.x86_64]$ modinfo drivers/block/nbd.kofilename: /home/builder/rpmbuild/BUILD/kernel-3.10.0-327.36.3.el7/linux-3.10.0-327.36.3.el7.x86_64/drivers/block/nbd.kolicense: GPLdescription: Network Block Devicerhelversion: 7.2srcversion: D8C4A5FDDF180E1B9264E35depends:vermagic: 3.10.0 SMP mod_unload modversionsparm: nbds_max:number of network block devices to initialize (default: 16) (int)parm: max_part:number of partitions per device (default: 0) (int)parm: debugflags:flags for controlling debug output (int)[builder@localhost linux-3.10.0-327.36.3.el7.x86_64]$ su root[root@localhost linux-3.10.0-327.36.3.el7.x86_64]# cp drivers/block/nbd.ko /lib/modules/3.10.0-327.el7.x86_64/extra/[root@localhost linux-3.10.0-327.36.3.el7.x86_64]# depmod -a &amp;&amp; modprobe nbd nbd.ko共享复制nbd.ko 到其他CentOS7.2主机上的 /lib/modules/3.10.0-327.el7.x86_64/extra/目录。比如在节点node232上，加载nbd内核，查看内核123456789101112131415[root@node232 ~]# depmod -a &amp;&amp; modprobe nbd[root@node232 ~]# modinfo nbdfilename: /lib/modules/3.10.0-327.el7.x86_64/extra/nbd.kolicense: GPLdescription: Network Block Devicerhelversion: 7.2srcversion: D8C4A5FDDF180E1B9264E35depends:vermagic: 3.10.0 SMP mod_unload modversionsparm: nbds_max:number of network block devices to initialize (default: 16) (int)parm: max_part:number of partitions per device (default: 0) (int)parm: debugflags:flags for controlling debug output (int)[root@node232 ~]# lsmod | grep nbdnbd 17603 0 参考http://purplepalmdash.github.io/2015/08/13/build-nbd-kernel-module-on-centos7]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[CentOS7.2部署Luminous版Ceph-12.2.0]]></title>
      <url>%2F2017%2F09%2F13%2Fceph-deploy-luminous%2F</url>
      <content type="text"><![CDATA[在CentOS7.2上安装部署Luminous版Ceph-12.2.0。由于ceph的Luminous版本默认使用bluestore作为后端存储类型，也新增了mgr功能，所以使用ceph-deploy的1.5.38版本来部署集群、创建MON、OSD、MGR等。 环境每台主机 CentOS Linux release 7.2.1511 (Core) Mini版 两个100G的磁盘做OSD 12345678910111213[root@localhost ~]# cat /etc/redhat-releaseCentOS Linux release 7.2.1511 (Core)[root@localhost ~]# lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTsr0 11:0 1 1024M 0 romxvda 202:0 0 10G 0 disk├─xvda1 202:1 0 500M 0 part /boot└─xvda2 202:2 0 9.5G 0 part ├─centos-root 253:0 0 8.5G 0 lvm / └─centos-swap 253:1 0 1G 0 lvm [SWAP]xvdb 202:16 0 100G 0 diskxvdc 202:32 0 100G 0 disk 主机node232作为管理节点，部署ceph-deploy。三台主机配置如下 主机 IP 安装组件 node232 192.168.217.232 ceph-deploy、mon、osd、mgr、ntp node233 192.168.217.233 mon、osd、ntpdate node234 192.168.217.234 mon、osd、ntpdate 域名解析修改每台主机的hostname1[root@localhost ~]# hostnamectl set-hostname [hostname] hostname分别为node232、node233、node234 修改每台主机的/etc/hosts1[root@localhost ~]# vi /etc/hosts 添加192.168.217.232 node232192.168.217.233 node233192.168.217.234 node234 关闭selinux关闭每台主机的selinux12[root@node232 ~]# sed -i 's/SELINUX=.*/SELINUX=disabled/' /etc/selinux/config[root@node232 ~]# setenforce 0 配置无密码访问在node232节点上执行配置。 生成本机密钥1[root@node232 ~]# ssh-keygen -t rsa 将本机密钥copy到node233、node23412[root@node232 ~]# ssh-copy-id root@node233[root@node232 ~]# ssh-copy-id root@node234 验证配置是否成功12[root@node232 ~]# ssh root@node233[root@node232 ~]# ssh root@node234 NTP 配置NTP配置请参考 CentOS7.2配置NTP服务 node232作为NTP服务端，node233、node234作为NTP客户端 yum源配置配置Luminous版本的阿里云或网易的yum源，可以参考 CentOS7.2部署Jewel版Ceph-10.2.7 【yum源配置】部分，Jewel改成相应的Luminous 安装ceph下载ceph的相关rpm到本地1[root@node232 ~]# yum install --downloadonly --downloaddir=/tmp/ceph ceph 在每台主机上安装ceph1[root@node232 ~]# yum localinstall -C -y --disablerepo=* /tmp/ceph/*.rpm 安装成功，查看ceph版本12[root@node232 ~]# ceph -vceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc) 部署ceph在管理节点node232上执行。 安装ceph-deploy下载ceph-deploy-1.5.381[root@node232 ~]# yum install --downloadonly --downloaddir=/tmp/ceph-deploy/ ceph-deploy 安装ceph-deploy1[root@node232 ~]# yum localinstall -C -y --disablerepo=* /tmp/ceph-deploy/*.rpm 安装成功，查看ceph-deploy版本12[root@node232 ~]# ceph-deploy --version1.5.38 部署集群创建部署目录，部署集群123[root@node232 ~]# mkdir ceph-cluster[root@node232 ~]# cd ceph-cluster[root@node232 ceph-cluster]# ceph-deploy new node232 node233 node234 部署mon1[root@node232 ceph-cluster]# ceph-deploy mon create-initial 执行ceph -s会出错，是由于缺少/etc/ceph/ceph.client.admin.keyring文件12345[root@node232 ceph-cluster]# ceph -s2017-09-13 12:12:18.772214 7f3d3fc3f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory2017-09-13 12:12:18.772260 7f3d3fc3f700 -1 monclient: ERROR: missing keyring, cannot use cephx for authentication2017-09-13 12:12:18.772263 7f3d3fc3f700 0 librados: client.admin initialization error (2) No such file or directory[errno 2] error connecting to the cluster 手工复制ceph-cluster目录下ceph.client.admin.keyring文件到/etc/ceph/目录下或者执行 ceph-deploy admin命令自动复制ceph.client.admin.keyring文件 1[root@node232 ceph-cluster]# ceph-deploy admin node232 node233 node234 查看集群123456789101112131415[root@node232 ceph-cluster]# ceph -s cluster: id: 988e29ea-8b2c-4fa7-a808-e199f2e6a334 health: HEALTH_OK services: mon: 3 daemons, quorum node232,node233,node234 mgr: no daemons active osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 bytes usage: 0 kB used, 0 kB / 0 kB avail pgs: 创建osd1[root@node232 ceph-cluster]# ceph-deploy --overwrite-conf osd prepare node232:/dev/xvdb node232:/dev/xvdc node233:/dev/xvdb node233:/dev/xvdc node234:/dev/xvdb node234:/dev/xvdc --zap-disk 激活osd1[root@node232 ceph-cluster]# ceph-deploy --overwrite-conf osd activate node232:/dev/xvdb1 node232:/dev/xvdc1 node233:/dev/xvdb1 node233:/dev/xvdc1 node234:/dev/xvdb1 node234:/dev/xvdc1 查看集群12345678910111213141516[root@node232 ceph-cluster]# ceph -s cluster: id: 988e29ea-8b2c-4fa7-a808-e199f2e6a334 health: HEALTH_WARN no active mgr services: mon: 3 daemons, quorum node232,node233,node234 mgr: no daemons active osd: 6 osds: 6 up, 6 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 bytes usage: 0 kB used, 0 kB / 0 kB avail pgs: 配置mgr在node232上创建名称为foo的mgr服务1[root@node232 ceph-cluster]# ceph-deploy mgr create node232:foo 启用dashboard1[root@node232 ceph-cluster]# ceph mgr module enable dashboard 通过 http://192.168.217.232:7000 访问dashboard dashboard的port默认为7000，可以执行ceph config-key set mgr/dashboard/server_port $PORT修改port。也可以执行ceph config-key set mgr/dashboard/server_addr $IP指定dashboard的访问IP。]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[CentOS7.2上Jewel版ceph-10.2.7 rbd map failed (6) No such device or address]]></title>
      <url>%2F2017%2F08%2F01%2Fceph-rbdmap-error-1%2F</url>
      <content type="text"><![CDATA[rbd map错误信息12345[root@node210 ~]# rbd map rbd/rbdtestrbd: sysfs write failedRBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable".In some cases useful info is found in syslog - try "dmesg | tail" or so.rbd: map failed: (6) No such device or address ceph集群dmesg查看1234567891011[root@ceph01 ceph]# dmesg | tail[258857.936153] libceph: mon0 192.168.212.201:6789 feature set mismatch, my 4a042a42 &lt; server's 2004a042a42, missing 20000000000[258857.936165] libceph: mon0 192.168.212.201:6789 socket error on read[258867.936151] libceph: mon0 192.168.212.201:6789 feature set mismatch, my 4a042a42 &lt; server's 2004a042a42, missing 20000000000[258867.936163] libceph: mon0 192.168.212.201:6789 socket error on read[258877.936155] libceph: mon0 192.168.212.201:6789 feature set mismatch, my 4a042a42 &lt; server's 2004a042a42, missing 20000000000[258877.936167] libceph: mon0 192.168.212.201:6789 socket error on read[258887.936153] libceph: mon0 192.168.212.201:6789 feature set mismatch, my 4a042a42 &lt; server's 2004a042a42, missing 20000000000[258887.936165] libceph: mon0 192.168.212.201:6789 socket error on read[258897.936139] libceph: mon0 192.168.212.201:6789 feature set mismatch, my 4a042a42 &lt; server's 2004a042a42, missing 20000000000[258897.936150] libceph: mon0 192.168.212.201:6789 socket error on read 环境 CentOS Linux release 7.2.1511 (Core) ceph version 10.2.7 CentOS7.2的内核版本3.1012[root@node210 ~]# uname -r3.10.0-327.el7.x86_64 分析ceph支持rbd块两种格式： format 1：新建 rbd 映像时使用最初的格式。此格式兼容所有版本的 librbd 和内核模块，但是不支持较新的功能，像克隆。 format 2：使用第二版 rbd 格式， librbd 和 3.11 版以上内核模块才支持（除非是分拆的模块）。此格式增加了克隆支持，使得扩展更容易，还允许以后增加新功能。 Jewel版10.2.7新建rbd默认使用格式2，默认全部开启如下特性： layering: 支持分层 striping: 支持条带化 v2 exclusive-lock: 支持独占锁 object-map: 支持对象映射（依赖 exclusive-lock ） fast-diff: 快速计算差异（依赖 object-map ） deep-flatten: 支持快照扁平化操作 journaling: 支持记录 IO 操作（依赖独占锁） 查看下支持了哪些特性12345678[root@node210 ~]# rbd info rbd/rbdtestrbd image 'rbdtest': size 102400 MB in 25600 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.fa6d2ae8944a format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten flags: CentOS7.2内核版本 3.10，仅支持此特性（layering），所以映射失败。其它特性需要使用更高版本内核，或者从新编译内核加载特性模块才行。 解决方法对于当前rbd，直接disable这个rbd不支持的特性exclusive-lock、 object-map、 fast-diff、 deep-flatten1234[root@node210 ~]# rbd feature disable rbd/rbdtest fast-diff[root@node210 ~]# rbd feature disable rbd/rbdtest deep-flatten[root@node210 ~]# rbd feature disable rbd/rbdtest object-map[root@node210 ~]# rbd feature disable rbd/rbdtest exclusive-lock 再次查看rbd信息，只有layering特性12345678[root@node210 ~]# rbd info rbd/rbdtestrbd image 'rbdtest': size 102400 MB in 25600 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.fa6d2ae8944a format: 2 features: layering flags: rbd map成功12[root@node210 ~]# rbd map rbd/rbdtest/dev/rbd1 对于后续其他rbd的操作，可以使用以下方法： 1、创建rbd时使用 --image-feature 选项指定需要的特性，不用全部开启，开启layering即可。1[root@node210 ~]# rbd create rbdtest2 --size 100G --image-feature layering 2、修改/etc/ceph/ceph.conf，增加或修改以下配置 rbd default features = 3 只开启layering，禁用exclusive-lock、 object-map、fast-diff、 deep-flatten这些特性 参考http://cephnotes.ksperis.com/blog/2014/01/21/feature-set-mismatch-error-on-ceph-kernel-client http://www.zphj1987.com/2016/06/07/rbd%E6%97%A0%E6%B3%95map-rbd-feature-disable/]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[CentOS7.2编译Jewel版ceph-10.2.7]]></title>
      <url>%2F2017%2F08%2F01%2Fceph-jewel-build%2F</url>
      <content type="text"><![CDATA[在CentOS7.2下编译Jewel版ceph-10.2.7，通过解压ceph-10.2.7.tar.gz来编译ceph。 环境CentOS Linux release 7.2.1511 (Core) Mini版 配置yum使用网易的源CentOS-Base.repo，并配置epel.repo 安装依赖下载依赖的rpm到本地1[root@node156 ~]# yum install --downloadonly --downloaddir=/tmp/ceph-10.2.7 make automake autoconf boost-devel fuse-devel gcc-c++ libtool libuuid-devel libblkid-devel keyutils-libs-devel cryptopp-devel fcgi-devel libcurl-devel expat-devel gperftools-devel libedit-devel libatomic_ops-devel snappy-devel leveldb-devel libaio-devel xfsprogs-devel libudev-devel btrfs-progs Cython python-devel openssl-devel openldap-devel python-pip python-virtualenv net-tools 本地安装依赖的rpm包1[root@node156 ~]# yum localinstall -C -y --disablerepo=* /tmp/ceph-10.2.7/*.rpm 编译安装下载ceph源码tar.gz文件下载地址： https://download.ceph.com/tarballs/ceph-10.2.7.tar.gz http://pkgs.fedoraproject.org/repo/pkgs/ceph/ceph-10.2.7.tar.gz https://codeload.github.com/ceph/ceph/tar.gz/v10.2.7 解压[root@node156 ~]# tar xzvf ceph-10.2.7.tar.gz 源码编译123[root@node156 ~]# cd ceph-10.2.7[root@node156 ceph-10.2.7]# ./autogen.sh[root@node156 ceph-10.2.7]# ./configure 中间会有一些warning，只要没有error就是成功了。1[root@node156 ceph-10.2.7]# make -j4 启动停止集群进入src目录，启动集群，默认3个MON，3个OSD12[root@node156 ceph-10.2.7]# cd src[root@node156 src]# ./vstart.sh -d -n -x 指定1个MON，0个MDS，启动集群1[root@node156 src]# MON=1 MDS=0 ./vstart.sh -d -n -x 查看集群1234567891011121314[root@node156 src]# ./ceph -s*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***2017-07-13 04:24:35.083777 7f84942f8700 -1 WARNING: the following dangerous and experimental features are enabled: *2017-07-13 04:24:35.087659 7f84942f8700 -1 WARNING: the following dangerous and experimental features are enabled: * cluster 0e262580-8dbd-4ac4-ad98-c5966aa7c9ad health HEALTH_OK monmap e1: 3 mons at &#123;a=127.0.0.1:6789/0,b=127.0.0.1:6790/0,c=127.0.0.1:6791/0&#125; election epoch 4, quorum 0,1,2 a,b,c fsmap e6: 1/1/1 up &#123;0=c=up:active&#125;, 2 up:standby osdmap e11: 3 osds: 3 up, 3 in flags sortbitwise,require_jewel_osds pgmap v25: 24 pgs, 3 pools, 2068 bytes data, 20 objects 60178 MB used, 88675 MB / 145 GB avail 24 active+clean 停止集群1[root@node156 src]# ./stop.sh 编译错误信息处理安装 Cython1configure:17667: error: cython not found 安装 python-devel1configure:17667: error: python-config not found 安装 openssl-devel1configure:20311: error: radosgw selected but OpenSSL not found 安装 openldap-devel1configure: error: ldap.h not found (openldap-dev, openldap-devel) 安装 python-virtualenv12345678/bin/sh: /tmp/ceph-disk-virtualenv/bin/pip: No such file or directorymake[3]: *** [/tmp/ceph-disk-virtualenv] Error 127make[3]: Leaving directory `/root/ceph-10.2.7/src'make[2]: *** [all-recursive] Error 1make[2]: Leaving directory `/root/ceph-10.2.7/src'make[1]: *** [all] Error 2make[1]: Leaving directory `/root/ceph-10.2.7/src'make: *** [all-recursive] Error 1]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[MON数据恢复]]></title>
      <url>%2F2017%2F07%2F24%2Fceph-monstore-tool-1%2F</url>
      <content type="text"><![CDATA[MON的数据丢失或毁坏的情况下，使用ceph-monstore-tool工具来恢复MON。ceph-monstore-tool工具需要手动安装ceph-test包才有，例如ceph-10.2.7版本的ceph-test-10.2.7-0.el7.x86_64.rpm。 环境 CentOS Linux release 7.2.1511 (Core) Mini版 Jewel版ceph-10.2.7 测试步骤毁坏数据停止MON1[root@node201 ~]# systemctl stop ceph-mon.target 制造mon毁坏1[root@node201 ~]# mv /var/lib/ceph/mon/ceph-node201/ /var/lib/ceph/mon/ceph-node201_bak OSD收集MON数据创建临时目录1[root@node201 ~]# mkdir /tmp/mon-store 停止OSD12[root@node201 ~]# systemctl stop ceph-osd@0[root@node201 ~]# systemctl stop ceph-osd@1 从OSD收集MON相关的数据，存放到/tmp/mon-store/。 如果有多台服务器，需要依次将/tmp/mon-store/上传到这些服务器，从OSD收集MON相关的数据 将最终的/tmp/mon-store/放到一台需要恢复的服务器上1234567[root@node201 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --op update-mon-db --mon-store-path /tmp/mon-store/osd.0 : 0 osdmaps trimmed, 56 osdmaps added. 64 pgs added.[root@node201 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --op update-mon-db --mon-store-path /tmp/mon-store/osd.1 : 0 osdmaps trimmed, 0 osdmaps added. 0 pgs added. 重建MON数据ceph-monstore-tool恢复MON数据1[root@node201 ~]# ceph-monstore-tool /tmp/mon-store rebuild 创建MON目录1[root@node201 ~]# mkdir /var/lib/ceph/mon/ceph-node201 copy MON数据文件1[root@node201 ~]# cp -ra /tmp/mon-store/* /var/lib/ceph/mon/ceph-node201 创建MON的done、systemd文件12[root@node201 ~]# touch /var/lib/ceph/mon/ceph-node201/done[root@node201 ~]# touch /var/lib/ceph/mon/ceph-node201/systemd 重建kering如果未开启cephx，那么不需要重建kering。开启cephx，需要做以下操作 查看client.admin.keyring123[root@node201 ~]# cat /etc/ceph/ceph.client.admin.keyring[client.admin] key = AQB7snVZ1Ha/MRAAOmQNSoLuoep/qcbZiNvBIg== 设置client.admin.keyring的caps1234567[root@node201 ~]# ceph-authtool /etc/ceph/ceph.client.admin.keyring -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *'[root@node201 ~]# cat /etc/ceph/ceph.client.admin.keyring[client.admin] key = AQB7snVZ1Ha/MRAAOmQNSoLuoep/qcbZiNvBIg== caps mds = "allow *" caps mon = "allow *" caps osd = "allow *" 创建MON keyring123456[root@node201 ~]# ceph-authtool --create-keyring /var/lib/ceph/mon/ceph-node201/keyring --gen-key -n mon. --cap mon 'allow *'creating /var/lib/ceph/mon/ceph-node201/keyring[root@node201 ~]# cat /var/lib/ceph/mon/ceph-node201/keyring[mon.] key = AQBQoHVZRQ8KIBAA+Ml7P3qXIPw1h+xCFkzxWA== caps mon = "allow *" 导入client.admin.keyring到MON的keyring123456789101112[root@node201 ~]# ceph-authtool /var/lib/ceph/mon/ceph-node201/keyring --import-keyring /etc/ceph/ceph.client.admin.keyringimporting contents of /etc/ceph/ceph.client.admin.keyring into /var/lib/ceph/mon/ceph-node201/keyring[root@node201 ~]# cat /var/lib/ceph/mon/ceph-node201/keyring[mon.] key = AQBQoHVZRQ8KIBAA+Ml7P3qXIPw1h+xCFkzxWA== caps mon = "allow *"[client.admin] key = AQB7snVZ1Ha/MRAAOmQNSoLuoep/qcbZiNvBIg== auid = 0 caps mds = "allow *" caps mon = "allow *" caps osd = "allow *" 启动MON设置MON的目录权限1[root@node201 ~]# chown ceph:ceph -R /var/lib/ceph/mon/ 启动MON1[root@node201 ~]# systemctl start ceph-mon.target 参考http://www.zphj1987.com/2016/09/20/Ceph%E7%9A%84Mon%E6%95%B0%E6%8D%AE%E9%87%8D%E6%96%B0%E6%9E%84%E5%BB%BA%E5%B7%A5%E5%85%B7/ http://letus.club/2017/03/30/ceph-rebuild-mon-data-from-osds/ https://gist.github.com/jingniao/839c6fe3b635a90b0fcfafa19c1b1557]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[ceph-deploy激活OSD出错 experimental feature 'rocksdb' is not enabled]]></title>
      <url>%2F2017%2F07%2F21%2Fceph-deploy-error-2%2F</url>
      <content type="text"><![CDATA[使用 ceph-deploy --overwrite-conf osd prepare --bluestore 的OSD，在激活OSD时出错 experimental feature ‘rocksdb’ is not enabled ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.ioG4dF: (5) Input/output error 环境 CentOS Linux release 7.2.1511 (Core) Jewel版ceph-10.2.7 ceph-deploy-1.5.37 创建OSD成功1[root@node201 ceph-cluster]# ceph-deploy --overwrite-conf osd prepare --bluestore node201:/dev/sdb 激活OSD出错123456789101112131415161718192021222324252627282930313233343536373839404142434445464748[root@node201 ceph-cluster]# ceph-deploy --overwrite-conf osd activate node201:/dev/sdb1......[node201][WARNIN] command: Running command: /usr/bin/timeout 300 ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /var/lib/ceph/tmp/mnt.ioG4dF/activate.monmap --osd-data /var/lib/ceph/tmp/mnt.ioG4dF --osd-uuid 110bea02-b6ff-4ae1-9a76-536933fec1a2 --keyring /var/lib/ceph/tmp/mnt.ioG4dF/keyring --setuser ceph --setgroup ceph[node201][WARNIN] mount_activate: Failed to activate[node201][WARNIN] unmount: Unmounting /var/lib/ceph/tmp/mnt.ioG4dF[node201][WARNIN] command_check_call: Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.ioG4dF[node201][WARNIN] Traceback (most recent call last):[node201][WARNIN] File "/usr/sbin/ceph-disk", line 9, in &lt;module&gt;[node201][WARNIN] load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5047, in run[node201][WARNIN] main(sys.argv[1:])[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4998, in main[node201][WARNIN] args.func(args)[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3357, in main_activate[node201][WARNIN] reactivate=args.reactivate,[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3114, in mount_activate[node201][WARNIN] (osd_id, cluster) = activate(path, activate_key_template, init)[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3290, in activate[node201][WARNIN] keyring=keyring,[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2756, in mkfs[node201][WARNIN] '--setgroup', get_ceph_user(),[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2720, in ceph_osd_mkfs[node201][WARNIN] raise Error('%s failed : %s' % (str(arguments), error))[node201][WARNIN] ceph_disk.main.Error: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', u'0', '--monmap', '/var/lib/ceph/tmp/mnt.ioG4dF/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.ioG4dF', '--osd-uuid', u'110bea02-b6ff-4ae1-9a76-536933fec1a2', '--keyring', '/var/lib/ceph/tmp/mnt.ioG4dF/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : 2017-07-20 17:15:38.195590 7ff0f5078800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore[node201][WARNIN] 2017-07-20 17:15:38.195842 7ff0f5078800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore[node201][WARNIN] 2017-07-20 17:15:38.195953 7ff0f5078800 -1 WARNING: experimental feature 'bluestore' is enabled[node201][WARNIN] Please be aware that this feature is experimental, untested,[node201][WARNIN] unsupported, and may result in data corruption, data loss,[node201][WARNIN] and/or irreparable damage to your cluster. Do not use[node201][WARNIN] feature with important data.[node201][WARNIN][node201][WARNIN] 2017-07-20 17:15:38.219843 7ff0f5078800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore[node201][WARNIN] 2017-07-20 17:15:38.229498 7ff0f5078800 -1 *** experimental feature 'rocksdb' is not enabled ***[node201][WARNIN] This feature is marked as experimental, which means it[node201][WARNIN] - is untested[node201][WARNIN] - is unsupported[node201][WARNIN] - may corrupt your data[node201][WARNIN] - may break your cluster is an unrecoverable fashion[node201][WARNIN] To enable this feature, add this to your ceph.conf:[node201][WARNIN] enable experimental unrecoverable data corrupting features = rocksdb[node201][WARNIN][node201][WARNIN] 2017-07-20 17:15:38.229514 7ff0f5078800 -1 bluestore(/var/lib/ceph/tmp/mnt.ioG4dF) _open_db error creating db[node201][WARNIN] 2017-07-20 17:15:38.724290 7ff0f5078800 -1 OSD::mkfs: ObjectStore::mkfs failed with error -5[node201][WARNIN] 2017-07-20 17:15:38.724341 7ff0f5078800 -1 ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.ioG4dF: (5) Input/output error[node201][WARNIN][node201][ERROR ] RuntimeError: command returned non-zero exit status: 1[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/sdb1 解决分析错误提示需要在ceph.conf文件中配置rocksdb enable_experimental_unrecoverable_data_corrupting_features = bluestore rocksdb 修改ceph.conf之后，重启ceph，再重新激活1[root@node201 ceph-cluster]# systemctl restart ceph.target]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[ceph-deploy激活OSD出错 experimental feature 'bluestore' is not enabled]]></title>
      <url>%2F2017%2F07%2F21%2Fceph-deploy-error-1%2F</url>
      <content type="text"><![CDATA[使用 ceph-deploy --overwrite-conf osd prepare --bluestore 的OSD，在激活OSD时出错 experimental feature ‘bluestore’ is not enabled 环境 CentOS Linux release 7.2.1511 (Core) Jewel版ceph-10.2.7 ceph-deploy-1.5.37 创建OSD成功1[root@node201 ceph-cluster]# ceph-deploy --overwrite-conf osd prepare --bluestore node201:/dev/sdb 激活OSD出错12345678910111213141516171819202122232425262728293031323334353637[root@node201 ceph-cluster]# ceph-deploy --overwrite-conf osd activate node201:/dev/sdb1......[node201][WARNIN] command: Running command: /usr/bin/timeout 300 ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /var/lib/ceph/tmp/mnt.RwxMi7/activate.monmap --osd-data /var/lib/ceph/tmp/mnt.RwxMi7 --osd-uuid 74872e01-68c2-4050-b8b5-7f20a3d0a88a --keyring /var/lib/ceph/tmp/mnt.RwxMi7/keyring --setuser ceph --setgroup ceph[node201][WARNIN] mount_activate: Failed to activate[node201][WARNIN] unmount: Unmounting /var/lib/ceph/tmp/mnt.RwxMi7[node201][WARNIN] command_check_call: Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.RwxMi7[node201][WARNIN] Traceback (most recent call last):[node201][WARNIN] File "/usr/sbin/ceph-disk", line 9, in &lt;module&gt;[node201][WARNIN] load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5047, in run[node201][WARNIN] main(sys.argv[1:])[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4998, in main[node201][WARNIN] args.func(args)[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3357, in main_activate[node201][WARNIN] reactivate=args.reactivate,[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3114, in mount_activate[node201][WARNIN] (osd_id, cluster) = activate(path, activate_key_template, init)[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3290, in activate[node201][WARNIN] keyring=keyring,[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2756, in mkfs[node201][WARNIN] '--setgroup', get_ceph_user(),[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2720, in ceph_osd_mkfs[node201][WARNIN] raise Error('%s failed : %s' % (str(arguments), error))[node201][WARNIN] ceph_disk.main.Error: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', u'0', '--monmap', '/var/lib/ceph/tmp/mnt.RwxMi7/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.RwxMi7', '--osd-uuid', u'74872e01-68c2-4050-b8b5-7f20a3d0a88a', '--keyring', '/var/lib/ceph/tmp/mnt.RwxMi7/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : 2017-07-20 17:01:21.852764 7fa0be2b7800 -1 *** experimental feature 'bluestore' is not enabled ***[node201][WARNIN] This feature is marked as experimental, which means it[node201][WARNIN] - is untested[node201][WARNIN] - is unsupported[node201][WARNIN] - may corrupt your data[node201][WARNIN] - may break your cluster is an unrecoverable fashion[node201][WARNIN] To enable this feature, add this to your ceph.conf:[node201][WARNIN] enable experimental unrecoverable data corrupting features = bluestore[node201][WARNIN][node201][WARNIN] 2017-07-20 17:01:21.852769 7fa0be2b7800 -1 unable to create object store[node201][WARNIN][node201][ERROR ] RuntimeError: command returned non-zero exit status: 1[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/sdb1 解决分析错误提示需要在ceph.conf文件中配置bluestore enable_experimental_unrecoverable_data_corrupting_features = bluestore 修改ceph.conf之后，重启ceph，再重新激活1[root@node201 ceph-cluster]# systemctl restart ceph.target]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[ceph-disk源码分析]]></title>
      <url>%2F2017%2F06%2F23%2Fceph-disk-1%2F</url>
      <content type="text"><![CDATA[ceph-disk是一个用于部署osd数据及journal分区或目录的工具，基于Python开发。ceph-disk工具放在ceph-base包中，安装ceph-base rpm包将默认安装此工具，比如Jewel版ceph-10.2.7中的ceph-base-10.2.7-0.el7.x86_64.rpm。 ceph-disk命令行ceph-disk 命令格式如下123456ceph-disk [-h] [-v] [--log-stdout] [--prepend-to-path PATH] [--statedir PATH] [--sysconfdir PATH] [--setuser USER] [--setgroup GROUP] &#123;prepare,activate,activate-lockbox,activate-block,activate-journal,activate-all,list,suppress-activate,unsuppress-activate,deactivate,destroy,zap,trigger&#125; ... prepare：使用一个文件目录或磁盘来准备创建OSD activate：激活一个OSD activate-lockbox：激活一个lockbox activate-block：通过块设备激活一个OSD activate-journal：通过journal激活一个OSD activate-all：激活所有标记的OSD分区 list：列出磁盘、分区和OSD suppress-activate：禁止激活一个设备 unsuppress-activate：停止禁止激活一个设备 deactivate：停用一个OSD destroy：销毁一个OSD zap：清除设备分区 trigger：激活任何设备（由udev调用） ceph-disk工作机制通过ceph-disk创建osd, 数据分区和journal分区将自动mount。创建osd，主要是prepare和activate。 假设/dev/sdb是OSD要使用的数据盘，OSD要使用的journal分区在/dev/sdc上创建，/dev/sdc是SSD, 创建激活OSD的命令如下： ceph-disk prepare /dev/sdb /dev/sdc ceph-disk activate /dev/sdb1 sgdisk 命令参考 https://linux.die.net/man/8/sgdiskudevadm 命令参考 https://linux.die.net/man/8/udevadm prepare过程 使用sgdisk命令销毁数据盘/dev/sdb的GPT和MBR，清除所有分区 获取osd_journal_size大小，默认5120M，可以指定设置，准备journal分区 一个SSD很可能被多个OSD共享来划分各自的journal分区，/dev/sdc上已分好的区不变，使用sgdisk在分区上增加新的分区作为journal，不影响原来的分区，如果不指定创建分区的uuid，自动为journal分区生成一个journal_uuid，journal分区的typecode为45b0969e-9b03-4f30-b4c6-b4b80ceff106，journal是一个link,指向一个固定的位置，再由这个link指向真正的journal分区，这样可以解决盘符漂移带来的问题 使用sgdisk创建数据分区，使用--largest-new来使用磁盘最大可能空间，即将所有的空间用来创建数据分区/dev/sdb1 格式化数据分区/dev/sdb1为xfs 挂载数据分区/dev/sdb1到临时目录 在临时目录下写入ceph_fsid、fsid、magic、journal_uuid四个临时文件，文件内容相应写入 创建journal链接，使用ln -s将sdc新建的journal分区连接到临时目录journal文件 卸载、删除临时目录 修改OSD分区的typecode为4fbd7e29-9d25-41b8-afd0-062c0ceff05d udevadm trigger强制内核触发设备事件 activate过程/lib/udev/rules.d/目录下的两个rules文件60-ceph-by-parttypeuuid.rules、95-ceph-osd.rules 其实并不需要显式的调用activate这个命令。是因为prepare最后的udevadm trigger强制内核触发设备事件，udev event调用了ceph-disk trigger命令，分析该分区的typecode，是ceph OSD的数据分区，会自动调用ceph-disk activate /dev/sdb1。typecode为journal的分区则通过ceph-disk activate-journal来激活。 12345678910[root@ceph ~]# cat /lib/udev/rules.d/95-ceph-osd.rules# OSD_UUIDACTION=="add", SUBSYSTEM=="block", \ ENV&#123;DEVTYPE&#125;=="partition", \ ENV&#123;ID_PART_ENTRY_TYPE&#125;=="4fbd7e29-9d25-41b8-afd0-062c0ceff05d", \ OWNER:="ceph", GROUP:="ceph", MODE:="660", \ RUN+="/usr/sbin/ceph-disk --log-stdout -v trigger /dev/$name"ACTION=="change", SUBSYSTEM=="block", \ ENV&#123;ID_PART_ENTRY_TYPE&#125;=="4fbd7e29-9d25-41b8-afd0-062c0ceff05d", \ OWNER="ceph", GROUP="ceph", MODE="660" 获取文件系统类型xfs、osd_mount_options_xfs、osd_fs_mount_options_xfs 挂载/dev/sdb1到临时目录 卸载、删除临时目录 启动OSD进程 源码结构ceph-disk就两个文件 __init__.py：空白初始化文件 main.py：所有的ceph-disk命令操作在这个文件中，代码超5000行 类图main.py所有类图：ceph-disk.png Prepare类Prepare是准备OSD的操作。两个子类PrepareBluestore、PrepareFilestore分别对应Bluestore、Filestore。目前Jewel10.2.7默认Filestore PrepareData类PrepareData是准备OSD的数据操作，磁盘数据分区、Journal分区。两个子类PrepareFilestoreData、PrepareBluestoreData PrepareSpace类PrepareSpace是用来获取磁盘分区大小。两个子类PrepareJournal、PrepareBluestoreBlock DevicePartition类DevicePartition是设备分区的加密模式。四个子类DevicePartitionCrypt、DevicePartitionCryptLuks、DevicePartitionCryptPlain、DevicePartitionMultipath对应四种不同的dmcrypt OSD管理创建OSD主要是分为prepare与activate两个操作。 main.py主函数123if __name__ == '__main__': main(sys.argv[1:]) warned_about = &#123;&#125; main函数1234567891011121314151617181920212223242526def main(argv): # 命令行解析 args = parse_args(argv) #设置日志级别 setup_logging(args.verbose, args.log_stdout) if args.prepend_to_path != '': path = os.environ.get('PATH', os.defpath) os.environ['PATH'] = args.prepend_to_path + ":" + path # 设置ceph-disk.prepare.lock、ceph-disk.activate.lock的目录/var/lib/ceph/tmp setup_statedir(args.statedir) # 设置配置文件目录/etc/ceph/ setup_sysconfdir(args.sysconfdir) global CEPH_PREF_USER CEPH_PREF_USER = args.setuser global CEPH_PREF_GROUP CEPH_PREF_GROUP = args.setgroup # 执行子命令函数 if args.verbose: args.func(args) else: main_catch(args.func, args) parse_args函数解析子命令123456789101112131415161718192021222324def parse_args(argv): parser = argparse.ArgumentParser( 'ceph-disk', )......... # prepare 子命令解析 Prepare.set_subparser(subparsers) # activate 子命令解析 make_activate_parser(subparsers) make_activate_lockbox_parser(subparsers) make_activate_block_parser(subparsers) make_activate_journal_parser(subparsers) make_activate_all_parser(subparsers) make_list_parser(subparsers) make_suppress_parser(subparsers) make_deactivate_parser(subparsers) make_destroy_parser(subparsers) make_zap_parser(subparsers) make_trigger_parser(subparsers) args = parser.parse_args(argv) return args main_catch函数12345678910111213141516171819202122def main_catch(func, args): try: func(args) except Error as e: raise SystemExit( '&#123;prog&#125;: &#123;msg&#125;'.format( prog=args.prog, msg=e, ) ) except CephDiskException as error: exc_name = error.__class__.__name__ raise SystemExit( '&#123;prog&#125; &#123;exc_name&#125;: &#123;msg&#125;'.format( prog=args.prog, exc_name=exc_name, msg=error, ) ) prepareceph-disk prepare命令行格式为：123456789ceph-disk prepare [-h] [--cluster NAME] [--cluster-uuid UUID] [--osd-uuid UUID] [--dmcrypt] [--dmcrypt-key-dir KEYDIR] [--prepare-key PATH] [--fs-type FS_TYPE] [--zap-disk] [--data-dir] [--data-dev] [--lockbox LOCKBOX] [--lockbox-uuid UUID] [--journal-uuid UUID] [--journal-file] [--journal-dev] [--bluestore] [--block-uuid UUID] [--block-file] [--block-dev] DATA [JOURNAL] [BLOCK] Prepare类set_subparser函数解析子命令，默认函数是main123456789101112131415161718@staticmethod def set_subparser(subparsers): parents = [ Prepare.parser(), PrepareData.parser(), Lockbox.parser(), ] parents.extend(PrepareFilestore.parent_parsers()) parents.extend(PrepareBluestore.parent_parsers()) parser = subparsers.add_parser( 'prepare', parents=parents, help='Prepare a directory or disk for a Ceph OSD', ) parser.set_defaults( func=Prepare.main, ) return parser 调用factory函数123@staticmethod def main(args): Prepare.factory(args).prepare() 默认PrepareFilestore123456@staticmethod def factory(args): if args.bluestore: return PrepareBluestore(args) else: return PrepareFilestore(args) PrepareFilestore类初始化 PrepareFilestoreData初始化，继承PrepareData PrepareJournal初始化12345def __init__(self, args): if args.dmcrypt: self.lockbox = Lockbox(args) self.data = PrepareFilestoreData(args) self.journal = PrepareJournal(args) PrepareData初始化，获取fsid、生成新的osd_uuid 执行/usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid获取fsid 生成新的osd_uuid12345678910def __init__(self, args): self.args = args self.partition = None self.set_type() if self.args.cluster_uuid is None: self.args.cluster_uuid = get_fsid(cluster=self.args.cluster) if self.args.osd_uuid is None: self.args.osd_uuid = str(uuid.uuid4()) PrepareJournal初始化，继承PrepareSpace类 调用check_journal_reqs函数，获取check-allows-journal、check-wants-journal、check-needs-journal并校验 调用父类初始化12345678910def __init__(self, args): self.name = 'journal' (self.allows_journal, self.wants_journal, self.needs_journal) = check_journal_reqs(args) if args.journal and not self.allows_journal: raise Error('journal specified but not allowed by osd backend') super(PrepareJournal, self).__init__(args) PrepareSpace类初始化 调用get_space_size函数获取osd_journal_size大小12345678def __init__(self, args): self.args = args self.set_type() self.space_size = self.get_space_size() if getattr(self.args, self.name + '_uuid') is None: setattr(self.args, self.name + '_uuid', str(uuid.uuid4())) self.space_symlink = None self.space_dmcrypt = None 子类PrepareJournal的get_space_size函数 执行/usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size命令12345def get_space_size(self): return int(get_conf_with_default( cluster=self.args.cluster, variable='osd_journal_size', )) 由于PrepareFilestore、PrepareBluestore继承Prepare类，prepare函数在Prepare类中定义123def prepare(self): with prepare_lock: self.prepare_locked() PrepareFilestore类的prepare_locked函数，调用PrepareFilestoreData的prepare函数，PrepareJournal作为参数1234def prepare_locked(self): if self.data.args.dmcrypt: self.lockbox.prepare() self.data.prepare(self.journal) PrepareFilestoreData的父类PrepareData的prepare函数，如果是device设备，调用prepare_device函数1234567def prepare(self, *to_prepare_list): if self.type == self.DEVICE: self.prepare_device(*to_prepare_list) elif self.type == self.FILE: self.prepare_file(*to_prepare_list) else: raise Error('unexpected type ', self.type) PrepareFilestoreData类的prepare_device函数 调用父类PrepareData的prepare_device函数 调用set_data_partition函数 调用populate_data_path_device函数1234567891011def prepare_device(self, *to_prepare_list): # 父类PrepareData的prepare_device函数 super(PrepareFilestoreData, self).prepare_device(*to_prepare_list) for to_prepare in to_prepare_list: # PrepareJournal类的prepare函数，调用prepare_device函数创建journal分区 to_prepare.prepare() # 设置创建数据分区 self.set_data_partition() # 创建OSD self.populate_data_path_device(*to_prepare_list) PrepareData的prepare_device函数 调用sanity_checks函数校验设备是否已使用 调用set_variables函数设置变量 调用zap函数清除分区，并使分区生效12345678def prepare_device(self, *to_prepare_list): # 校验device self.sanity_checks() # 设置变量 self.set_variables() if self.args.zap_disk is not None: # 清除分区，并使分区生效 zap(self.args.data) 调用zap函数清除分区，并使分区生效。[dev]为设备，比如/dev/sdb /usr/sbin/sgdisk --zap-all -- [dev] /usr/sbin/sgdisk --clear --mbrtogpt -- [dev] /usr/bin/udevadm settle --timeout=600 /usr/bin/flock -s [dev] /usr/sbin/partprobe [dev] /usr/bin/udevadm settle --timeout=60012345678910111213141516171819202122232425262728293031323334353637383940def zap(dev): """ Destroy the partition table and content of a given disk. """ dev = os.path.realpath(dev) dmode = os.stat(dev).st_mode if not stat.S_ISBLK(dmode) or is_partition(dev): raise Error('not full block device; cannot zap', dev) try: LOG.debug('Zapping partition table on %s', dev) # try to wipe out any GPT partition table backups. sgdisk # isn't too thorough. lba_size = 4096 size = 33 * lba_size with open(dev, 'wb') as dev_file: dev_file.seek(-size, os.SEEK_END) dev_file.write(size * b'\0') # 清除分区 command_check_call( [ 'sgdisk', '--zap-all', '--', dev, ], ) command_check_call( [ 'sgdisk', '--clear', '--mbrtogpt', '--', dev, ], ) # 使分区生效 update_partition(dev, 'zapped') PrepareJournal类的prepare函数123456789def prepare(self): if self.type == self.DEVICE: self.prepare_device() elif self.type == self.FILE: self.prepare_file() elif self.type == self.NONE: pass else: raise Error('unexpected type ', self.type) prepare_device函数，调用Device类的create_partition函数创建journal分区1234567891011...... device = Device.factory(getattr(self.args, self.name), self.args) # 创建journal分区 num = device.create_partition( uuid=getattr(self.args, self.name + '_uuid'), name=self.name, size=self.space_size, num=num)...... create_partition函数创建journal分区 调用ptype_tobe_for_name函数，获取journal的typecode：45b0969e-9b03-4f30-b4c6-b4b80ceff106 创建journal分区 /usr/sbin/sgdisk --new=2:0:+5120M --change-name=2:ceph journal --partition-guid=2:f693b826-e070-4b42-af3e-07d011994583 --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdb 分区生效 /usr/bin/udevadm settle --timeout=600 /usr/bin/flock -s /dev/sdb /usr/sbin/partprobe /dev/sdb /usr/bin/udevadm settle --timeout=600123456789101112131415161718192021222324252627282930313233def create_partition(self, uuid, name, size=0, num=0): ptype = self.ptype_tobe_for_name(name) if num == 0: num = get_free_partition_index(dev=self.path) if size &gt; 0: new = '--new=&#123;num&#125;:0:+&#123;size&#125;M'.format(num=num, size=size) if size &gt; self.get_dev_size(): LOG.error('refusing to create %s on %s' % (name, self.path)) LOG.error('%s size (%sM) is bigger than device (%sM)' % (name, size, self.get_dev_size())) raise Error('%s device size (%sM) is not big enough for %s' % (self.path, self.get_dev_size(), name)) else: new = '--largest-new=&#123;num&#125;'.format(num=num) LOG.debug('Creating %s partition num %d size %d on %s', name, num, size, self.path) command_check_call( [ 'sgdisk', new, '--change-name=&#123;num&#125;:ceph &#123;name&#125;'.format(num=num, name=name), '--partition-guid=&#123;num&#125;:&#123;uuid&#125;'.format(num=num, uuid=uuid), '--typecode=&#123;num&#125;:&#123;uuid&#125;'.format(num=num, uuid=ptype), '--mbrtogpt', '--', self.path, ] ) # 使分区生效 update_partition(self.path, 'created') return num set_data_partition函数,调用create_data_partition函数创建数据分区123456789101112131415def set_data_partition(self): if is_partition(self.args.data): LOG.debug('OSD data device %s is a partition', self.args.data) self.partition = DevicePartition.factory( path=None, dev=self.args.data, args=self.args) ptype = self.partition.get_ptype() ready = Ptype.get_ready_by_name('osd') if ptype not in ready: LOG.warning('incorrect partition UUID: %s, expected %s' % (ptype, str(ready))) else: LOG.debug('Creating osd partition on %s', self.args.data) self.partition = self.create_data_partition() 调用Device类的create_partition创建数据分区并使分区生效 /usr/sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:1b9521d7-ee24-4043-96a7-1a3140bbff27 --typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be --mbrtogpt -- /dev/sdb /usr/bin/udevadm settle --timeout=600 /usr/bin/flock -s /dev/sdb /usr/sbin/partprobe /dev/sdb /usr/bin/udevadm settle --timeout=60012345678def create_data_partition(self): device = Device.factory(self.args.data, self.args) partition_number = 1 device.create_partition(uuid=self.args.osd_uuid, name='data', num=partition_number, size=self.get_space_size()) return device.get_partition(partition_number) populate_data_path_device函数创建OSD 格式化数据分区为xfs 创建临时目录并挂载 ceph_fsid、fsid、magic、journal_uuid文件写入OSD的临时文件 执行restorecon命令，恢复文件安全 卸载、删除临时目录 更改OSD分区的typecode为4fbd7e29-9d25-41b8-afd0-062c0ceff05d，对应为ready 使分区生效 强制内核触发设备事件1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768def populate_data_path_device(self, *to_prepare_list): partition = self.partition if isinstance(partition, DevicePartitionCrypt): partition.map() try: args = [ 'mkfs', '-t', self.args.fs_type, ] if self.mkfs_args is not None: args.extend(self.mkfs_args.split()) if self.args.fs_type == 'xfs': args.extend(['-f']) # always force else: args.extend(MKFS_ARGS.get(self.args.fs_type, [])) args.extend([ '--', partition.get_dev(), ]) try: LOG.debug('Creating %s fs on %s', self.args.fs_type, partition.get_dev()) # 格式化数据分区为xfs command_check_call(args) except subprocess.CalledProcessError as e: raise Error(e) # 挂载临时目录 path = mount(dev=partition.get_dev(), fstype=self.args.fs_type, options=self.mount_options) try: # OSD的ceph_fsid、fsid、magic、journal_uuid文件写入临时文件 self.populate_data_path(path, *to_prepare_list) finally: # 执行restorecon命令，恢复文件安全 path_set_context(path) # 卸载临时目录，并删除临时目录 unmount(path) finally: if isinstance(partition, DevicePartitionCrypt): partition.unmap() if not is_partition(self.args.data): try: # 更改OSD分区的typecode为4fbd7e29-9d25-41b8-afd0-062c0ceff05d，对应为ready command_check_call( [ 'sgdisk', '--typecode=%d:%s' % (partition.get_partition_number(), partition.ptype_for_name('osd')), '--', self.args.data, ], ) except subprocess.CalledProcessError as e: raise Error(e) # 使分区生效 update_partition(self.args.data, 'prepared') # 强制内核触发设备事件 command_check_call(['udevadm', 'trigger', '--action=add', '--sysname-match', os.path.basename(partition.rawdev)]) activateceph-disk activate命令行格式为：12345ceph-disk activate [-h] [--mount] [--activate-key PATH] [--mark-init INITSYSTEM] [--no-start-daemon] [--dmcrypt] [--dmcrypt-key-dir KEYDIR] [--reactivate] PATH activate子命令解析make_activate_parser函数，默认的执行函数是main_activate。 调用mount_activate函数，挂载OSD 获取挂载点，校验journal文件 启动OSD进程12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273def main_activate(args): cluster = None osd_id = None LOG.info('path = ' + str(args.path)) if not os.path.exists(args.path): raise Error('%s does not exist' % args.path) if is_suppressed(args.path): LOG.info('suppressed activate request on %s', args.path) return # ceph-disk.activate.lock文件：/var/lib/ceph/tmp/ceph-disk.activate.lock with activate_lock: mode = os.stat(args.path).st_mode if stat.S_ISBLK(mode): if (is_partition(args.path) and (get_partition_type(args.path) == PTYPE['mpath']['osd']['ready']) and not is_mpath(args.path)): raise Error('%s is not a multipath block device' % args.path) # 挂载数据分区 (cluster, osd_id) = mount_activate( dev=args.path, activate_key_template=args.activate_key_template, init=args.mark_init, dmcrypt=args.dmcrypt, dmcrypt_key_dir=args.dmcrypt_key_dir, reactivate=args.reactivate, ) # 获取挂载点 osd_data = get_mount_point(cluster, osd_id) elif stat.S_ISDIR(mode): (cluster, osd_id) = activate_dir( path=args.path, activate_key_template=args.activate_key_template, init=args.mark_init, ) osd_data = args.path else: raise Error('%s is not a directory or block device' % args.path) # exit with 0 if the journal device is not up, yet # journal device will do the activation # 校验journal文件 osd_journal = '&#123;path&#125;/journal'.format(path=osd_data) if os.path.islink(osd_journal) and not os.access(osd_journal, os.F_OK): LOG.info("activate: Journal not present, not starting, yet") return if (not args.no_start_daemon and args.mark_init == 'none'): command_check_call( [ 'ceph-osd', '--cluster=&#123;cluster&#125;'.format(cluster=cluster), '--id=&#123;osd_id&#125;'.format(osd_id=osd_id), '--osd-data=&#123;path&#125;'.format(path=osd_data), '--osd-journal=&#123;journal&#125;'.format(journal=osd_journal), ], ) if (not args.no_start_daemon and args.mark_init not in (None, 'none')): # 启动OSD进程 start_daemon( cluster=cluster, osd_id=osd_id, ) mount_activate函数123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133def mount_activate( dev, activate_key_template, init, dmcrypt, dmcrypt_key_dir, reactivate=False,): if dmcrypt: # 获取分区UUID part_uuid = get_partition_uuid(dev) dev = dmcrypt_map(dev, dmcrypt_key_dir) try: # 获取文件系统类型xfs fstype = detect_fstype(dev=dev) except (subprocess.CalledProcessError, TruncatedLineError, TooManyLinesError) as e: raise FilesystemTypeError( 'device &#123;dev&#125;'.format(dev=dev), e, ) # TODO always using mount options from cluster=ceph for # now; see http://tracker.newdream.net/issues/3253 # 获取osd_mount_options_xfs mount_options = get_conf( cluster='ceph', variable='osd_mount_options_&#123;fstype&#125;'.format( fstype=fstype, ), ) if mount_options is None: # 获取osd_fs_mount_options_xfs mount_options = get_conf( cluster='ceph', variable='osd_fs_mount_options_&#123;fstype&#125;'.format( fstype=fstype, ), ) # remove whitespaces from mount_options if mount_options is not None: mount_options = "".join(mount_options.split()) # 挂载临时目录 path = mount(dev=dev, fstype=fstype, options=mount_options) # check if the disk is deactive, change the journal owner, group # mode for correct user and group. if os.path.exists(os.path.join(path, 'deactive')): # logging to syslog will help us easy to know udev triggered failure if not reactivate: unmount(path) # we need to unmap again because dmcrypt map will create again # on bootup stage (due to deactivate) if '/dev/mapper/' in dev: part_uuid = dev.replace('/dev/mapper/', '') dmcrypt_unmap(part_uuid) LOG.info('OSD deactivated! reactivate with: --reactivate') raise Error('OSD deactivated! reactivate with: --reactivate') # flag to activate a deactive osd. deactive = True else: deactive = False osd_id = None cluster = None try: # 挂载OSD (osd_id, cluster) = activate(path, activate_key_template, init) # Now active successfully # If we got reactivate and deactive, remove the deactive file if deactive and reactivate: os.remove(os.path.join(path, 'deactive')) LOG.info('Remove `deactive` file.') # check if the disk is already active, or if something else is already # mounted there active = False other = False src_dev = os.stat(path).st_dev # 校验是否已经激活（挂载到正确目录） try: dst_dev = os.stat((STATEDIR + '/osd/&#123;cluster&#125;-&#123;osd_id&#125;').format( cluster=cluster, osd_id=osd_id)).st_dev if src_dev == dst_dev: active = True else: parent_dev = os.stat(STATEDIR + '/osd').st_dev if dst_dev != parent_dev: other = True elif os.listdir(get_mount_point(cluster, osd_id)): LOG.info(get_mount_point(cluster, osd_id) + " is not empty, won't override") other = True except OSError: pass if active: LOG.info('%s osd.%s already mounted in position; unmounting ours.' % (cluster, osd_id)) # 卸载临时目录，并删除临时目录 unmount(path) elif other: raise Error('another %s osd.%s already mounted in position ' '(old/different cluster instance?); unmounting ours.' % (cluster, osd_id)) else: move_mount( dev=dev, path=path, cluster=cluster, osd_id=osd_id, fstype=fstype, mount_options=mount_options, ) return cluster, osd_id except: LOG.error('Failed to activate') unmount(path) raise finally: # remove our temp dir # 删除临时目录 if os.path.exists(path): os.rmdir(path) 手工管理OSD准备OSD以 /usr/sbin/ceph-disk -v prepare --zap-disk --cluster ceph --fs-type xfs -- /dev/sdb为例，ceph-disk prepare命令执行过程如下。 查看journal参数123456[root@ceph-231 ~]# /usr/bin/ceph-osd --check-allows-journal -i 0 --cluster ceph --setuser ceph --setgroup cephyes[root@ceph-231 ~]# /usr/bin/ceph-osd --check-wants-journal -i 0 --cluster ceph --setuser ceph --setgroup cephyes[root@ceph-231 ~]# /usr/bin/ceph-osd --check-needs-journal -i 0 --cluster ceph --setuser ceph --setgroup cephno 查看已挂载的设备，/dev/sdb未被挂载，可以用来创建OSD123456789[root@ceph-231 ~]# cat /proc/mounts....../dev/sda1 / ext3 rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered 0 0....../dev/sda5 /var/log ext3 rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered 0 0...... 清除分区12345[root@ceph-231 ~]# /usr/sbin/sgdisk --zap-all -- /dev/sdb[root@ceph-231 ~]# /usr/sbin/sgdisk --clear --mbrtogpt -- /dev/sdb[root@ceph-231 ~]# /usr/bin/udevadm settle --timeout=600[root@ceph-231 ~]# /usr/bin/flock -s /dev/sdb /usr/sbin/partprobe /dev/sdb[root@ceph-231 ~]# /usr/bin/udevadm settle --timeout=600 获取osd_journal_size，默认5120M12[root@ceph-231 ~]# /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size5120 生成journal_uuid12[root@ceph-231 ~]# uuidgenf693b826-e070-4b42-af3e-07d011994583 创建journal分区，{num}用具体数字替换 如果数据盘与journal分区是同一个磁盘，{num}为2 如果数据盘与journal分区不在同一个磁盘，查看journal盘的分区信息，{num}为分区数+1 执行 parted –machine – /dev/sdb print 查看journal盘分区信息 1234[root@ceph-231 ~]# /usr/sbin/sgdisk --new=&#123;num&#125;:0:+5120M --change-name=&#123;num&#125;:"ceph journal" --partition-guid=&#123;num&#125;:f693b826-e070-4b42-af3e-07d011994583 --typecode=&#123;num&#125;:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdb[root@ceph-231 ~]# /usr/bin/udevadm settle --timeout=600[root@ceph-231 ~]# /usr/bin/flock -s /dev/sdb /usr/sbin/partprobe /dev/sdb[root@ceph-231 ~]# /usr/bin/udevadm settle --timeout=600 生成data分区uuid12[root@ceph-231 ~]# uuidgen1b9521d7-ee24-4043-96a7-1a3140bbff27 创建data分区1234[root@ceph-231 ~]# /usr/sbin/sgdisk --largest-new=1 --change-name=1:"ceph data" --partition-guid=1:1b9521d7-ee24-4043-96a7-1a3140bbff27 --typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be --mbrtogpt -- /dev/sdb[root@ceph-231 ~]# /usr/bin/udevadm settle --timeout=600[root@ceph-231 ~]# /usr/bin/flock -s /dev/sdb /usr/sbin/partprobe /dev/sdb[root@ceph-231 ~]# /usr/bin/udevadm settle --timeout=600 格式化数据分区为xfs1[root@ceph-231 ~]# /usr/sbin/mkfs -t xfs -f -i size=2048 -- /dev/sdb1 查看挂载属性1234[root@ceph-231 ~]# /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs[root@ceph-231 ~]# /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs[root@ceph-231 ~]# /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs[root@ceph-231 ~]# /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs osd_mkfs_options_xfs、osd_fs_mkfs_options_xfs、osd_mount_options_xfs、osd_fs_mount_options_xfs四个值均为空，xfs的默认挂载属性为noatime,inode64，挂载临时目录123[root@ceph-231 ~]# mkdir /var/lib/ceph/tmp/mnt.uCrLyH[root@ceph-231 ~]# /usr/bin/mount -t xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.uCrLyH[root@ceph-231 ~]# /usr/sbin/restorecon /var/lib/ceph/tmp/mnt.uCrLyH 获取集群fsid12[root@ceph-231 ~]# /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsidad3bdf51-ae79-44c3-b634-0c9f4995bbf5 集群fs_id写入ceph_fsid临时文件1234[root@ceph-231 ~]# vi /var/lib/ceph/tmp/mnt.uCrLyH/ceph_fsid.1308.tmp[root@ceph-231 ~]# /usr/sbin/restorecon -R /var/lib/ceph/tmp/mnt.uCrLyH/ceph_fsid.1308.tmp[root@ceph-231 ~]# /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.uCrLyH/ceph_fsid.1308.tmp[root@ceph-231 ~]# mv /var/lib/ceph/tmp/mnt.uCrLyH/ceph_fsid.1308.tmp /var/lib/ceph/tmp/mnt.uCrLyH/ceph_fsid 生成osd_uuid12[root@ceph-231 ~]# uuidgen410fa9bc-cdbf-469e-a08a-c246048d5e9b osd_uuid写入fsid文件临时文件1234[root@ceph-231 ~]# vi /var/lib/ceph/tmp/mnt.uCrLyH/fsid.1308.tmp[root@ceph-231 ~]# /usr/sbin/restorecon -R /var/lib/ceph/tmp/mnt.uCrLyH/fsid.1308.tmp[root@ceph-231 ~]# /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.uCrLyH/fsid.1308.tmp[root@ceph-231 ~]# mv /var/lib/ceph/tmp/mnt.uCrLyH/fsid.1308.tmp /var/lib/ceph/tmp/mnt.uCrLyH/fsid 写入magic临时文件，内容为 ceph osd volume v0261234[root@ceph-231 ~]# vi /var/lib/ceph/tmp/mnt.uCrLyH/magic.1308.tmp[root@ceph-231 ~]# /usr/sbin/restorecon -R /var/lib/ceph/tmp/mnt.uCrLyH/magic.1308.tmp[root@ceph-231 ~]# /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.uCrLyH/magic.1308.tmp[root@ceph-231 ~]# mv /var/lib/ceph/tmp/mnt.uCrLyH/magic.1308.tmp /var/lib/ceph/tmp/mnt.uCrLyH/magic 查看journal盘sdb2的uuid12[root@ceph-231 ~]# ll /dev/disk/by-partuuid/ | grep sdb2lrwxrwxrwx 1 root root 10 Jun 27 19:21 f693b826-e070-4b42-af3e-07d011994583 -&gt; ../../sdb2 journal_uuid写入临时文件1234[root@ceph-231 ~]# vi /var/lib/ceph/tmp/mnt.uCrLyH/journal_uuid.1308.tmp[root@ceph-231 ~]# /usr/sbin/restorecon -R /var/lib/ceph/tmp/mnt.uCrLyH/journal_uuid.1308.tmp[root@ceph-231 ~]# /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.uCrLyH/journal_uuid.1308.tmp[root@ceph-231 ~]# mv /var/lib/ceph/tmp/mnt.uCrLyH/journal_uuid.1308.tmp /var/lib/ceph/tmp/mnt.uCrLyH/journal_uuid 创建journal链接1[root@ceph-231 ~]# ln -s /dev/disk/by-partuuid/f693b826-e070-4b42-af3e-07d011994583 /var/lib/ceph/tmp/mnt.uCrLyH/journal restorecon命令，恢复文件安全12[root@ceph-231 ~]# /usr/sbin/restorecon -R /var/lib/ceph/tmp/mnt.uCrLyH[root@ceph-231 ~]# /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.uCrLyH 卸载、删除临时目录12[root@ceph-231 ~]# /bin/umount -- /var/lib/ceph/tmp/mnt.uCrLyH[root@ceph-231 ~]# rm -rf /var/lib/ceph/tmp/mnt.uCrLyH 修改OSD分区的typecode为4fbd7e29-9d25-41b8-afd0-062c0ceff05d，对应为ready1234[root@ceph-231 ~]# /usr/sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sdb[root@ceph-231 ~]# /usr/bin/udevadm settle --timeout=600[root@ceph-231 ~]# /usr/bin/flock -s /dev/sdb /usr/sbin/partprobe /dev/sdb[root@ceph-231 ~]# /usr/bin/udevadm settle --timeout=600 强制内核触发设备事件1[root@ceph-231 ~]# /usr/bin/udevadm trigger --action=add --sysname-match sdb1 激活OSD以/usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/sdb1为例，ceph-disk activate命令执行过程如下。 获取文件系统类型xfs12[root@ceph-231 ~]# /sbin/blkid -p -s TYPE -o value -- /dev/sdb1xfs 获取osd_mount_options_xfs1[root@ceph-231 ~]# /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs 获取osd_fs_mount_options_xfs1[root@ceph-231 ~]# /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs osd_mount_options_xfs与osd_fs_mount_options_xfs为空，xfs的默认挂载属性为noatime,inode64，挂载临时目录/var/lib/ceph/tmp/mnt.GoeBOu123[root@ceph-231 ~]# mkdir /var/lib/ceph/tmp/mnt.GoeBOu[root@ceph-231 ~]# /usr/bin/mount -t xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.GoeBOu[root@ceph-231 ~]# /usr/sbin/restorecon /var/lib/ceph/tmp/mnt.GoeBOu 卸载、删除临时目录12[root@ceph-231 ~]# /bin/umount -- /var/lib/ceph/tmp/mnt.GoeBOu[root@ceph-231 ~]# rm -rf /var/lib/ceph/tmp/mnt.GoeBOu 启动OSD进程，0为osd id123[root@ceph-231 ~]# /usr/bin/systemctl disable ceph-osd@0[root@ceph-231 ~]# /usr/bin/systemctl enable --runtime ceph-osd@0[root@ceph-231 ~]# /usr/bin/systemctl start ceph-osd@0]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[ceph-deploy源码分析（四）——osd模块]]></title>
      <url>%2F2017%2F06%2F21%2Fceph-deploy-osd%2F</url>
      <content type="text"><![CDATA[ceph-deploy的osd.py模块是用来管理osd守护进程，主要是创建与激活OSD。 osd 子命令格式如下1ceph-deploy osd [-h] &#123;list,create,prepare,activate&#125; ... list: 显示osd列表信息 create: 创建OSD，包含prepare与activate prepare: 准备OSD，通过格式化/分区磁盘 activate: 激活准备的OSD OSD管理make函数 priority为50 osd子命令默认执行函数为osd123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133@priority(50)def make(parser): """ Prepare a data disk on remote host. """ sub_command_help = dedent(""" Manage OSDs by preparing a data disk on remote host. For paths, first prepare and then activate: ceph-deploy osd prepare &#123;osd-node-name&#125;:/path/to/osd ceph-deploy osd activate &#123;osd-node-name&#125;:/path/to/osd For disks or journals the `create` command will do prepare and activate for you. """ ) parser.formatter_class = argparse.RawDescriptionHelpFormatter parser.description = sub_command_help osd_parser = parser.add_subparsers(dest='subcommand') osd_parser.required = True osd_list = osd_parser.add_parser( 'list', help='List OSD info from remote host(s)' ) osd_list.add_argument( 'disk', nargs='+', metavar='HOST:DISK[:JOURNAL]', type=colon_separated, help='remote host to list OSDs from' ) osd_create = osd_parser.add_parser( 'create', help='Create new Ceph OSD daemon by preparing and activating disk' ) osd_create.add_argument( '--zap-disk', action='store_true', help='destroy existing partition table and content for DISK', ) osd_create.add_argument( '--fs-type', metavar='FS_TYPE', choices=['xfs', 'btrfs' ], default='xfs', help='filesystem to use to format DISK (xfs, btrfs)', ) osd_create.add_argument( '--dmcrypt', action='store_true', help='use dm-crypt on DISK', ) osd_create.add_argument( '--dmcrypt-key-dir', metavar='KEYDIR', default='/etc/ceph/dmcrypt-keys', help='directory where dm-crypt keys are stored', ) osd_create.add_argument( '--bluestore', action='store_true', default=None, help='bluestore objectstore', ) osd_create.add_argument( 'disk', nargs='+', metavar='HOST:DISK[:JOURNAL]', type=colon_separated, help='host and disk to prepare', ) osd_prepare = osd_parser.add_parser( 'prepare', help='Prepare a disk for use as Ceph OSD by formatting/partitioning disk' ) osd_prepare.add_argument( '--zap-disk', action='store_true', help='destroy existing partition table and content for DISK', ) osd_prepare.add_argument( '--fs-type', metavar='FS_TYPE', choices=['xfs', 'btrfs' ], default='xfs', help='filesystem to use to format DISK (xfs, btrfs)', ) osd_prepare.add_argument( '--dmcrypt', action='store_true', help='use dm-crypt on DISK', ) osd_prepare.add_argument( '--dmcrypt-key-dir', metavar='KEYDIR', default='/etc/ceph/dmcrypt-keys', help='directory where dm-crypt keys are stored', ) osd_prepare.add_argument( '--bluestore', action='store_true', default=None, help='bluestore objectstore', ) osd_prepare.add_argument( 'disk', nargs='+', metavar='HOST:DISK[:JOURNAL]', type=colon_separated, help='host and disk to prepare', ) osd_activate = osd_parser.add_parser( 'activate', help='Start (activate) Ceph OSD from disk that was previously prepared' ) osd_activate.add_argument( 'disk', nargs='+', metavar='HOST:DISK[:JOURNAL]', type=colon_separated, help='host and disk to activate', ) parser.set_defaults( func=osd, ) osd函数，osd子命令list,create,prepare,activate分别对应的函数为osd_list、prepare、prepare、activate。 1234567891011121314def osd(args): cfg = conf.ceph.load(args) if args.subcommand == 'list': osd_list(args, cfg) elif args.subcommand == 'prepare': prepare(args, cfg, activate_prepared_disk=False) elif args.subcommand == 'create': prepare(args, cfg, activate_prepared_disk=True) elif args.subcommand == 'activate': activate(args, cfg) else: LOG.error('subcommand %s not implemented', args.subcommand) sys.exit(1) OSD列表命令行格式为：ceph-deploy osd list [-h] HOST:DISK[:JOURNAL] [HOST:DISK[:JOURNAL] …] osd_list函数 执行ceph --cluster=ceph osd tree --format=json命令获取OSD信息 执行ceph-disk list命令获取磁盘、分区信息 根据两个命令结果以及osd目录下文件信息，组装输出OSD列表数据123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778def osd_list(args, cfg): monitors = mon.get_mon_initial_members(args, error_on_empty=True, _cfg=cfg) # get the osd tree from a monitor host mon_host = monitors[0] distro = hosts.get( mon_host, username=args.username, callbacks=[packages.ceph_is_installed] ) # 执行ceph --cluster=ceph osd tree --format=json命令获取osd信息 tree = osd_tree(distro.conn, args.cluster) distro.conn.exit() interesting_files = ['active', 'magic', 'whoami', 'journal_uuid'] for hostname, disk, journal in args.disk: distro = hosts.get(hostname, username=args.username) remote_module = distro.conn.remote_module #获取OSD的目录/var/run/ceph/osd下的osd名称 osds = distro.conn.remote_module.listdir(constants.osd_path) # 执行ceph-disk list命令获取磁盘、分区信息 ceph_disk_executable = system.executable_path(distro.conn, 'ceph-disk') output, err, exit_code = remoto.process.check( distro.conn, [ ceph_disk_executable, 'list', ] ) # 循环OSD for _osd in osds: # osd路径，比如/var/run/ceph/osd/ceph-0 osd_path = os.path.join(constants.osd_path, _osd) # journal路径 journal_path = os.path.join(osd_path, 'journal') # OSD的id _id = int(_osd.split('-')[-1]) # split on dash, get the id osd_name = 'osd.%s' % _id metadata = &#123;&#125; json_blob = &#123;&#125; # piggy back from ceph-disk and get the mount point # ceph-disk list的结果与osd名称匹配，获取磁盘设备 device = get_osd_mount_point(output, osd_name) if device: metadata['device'] = device # read interesting metadata from files # 获取OSD下的active, magic, whoami, journal_uuid文件信息 for f in interesting_files: osd_f_path = os.path.join(osd_path, f) if remote_module.path_exists(osd_f_path): metadata[f] = remote_module.readline(osd_f_path) # do we have a journal path? # 获取 journal path if remote_module.path_exists(journal_path): metadata['journal path'] = remote_module.get_realpath(journal_path) # is this OSD in osd tree? for blob in tree['nodes']: if blob.get('id') == _id: # matches our OSD json_blob = blob # 输出OSD信息 print_osd( distro.conn.logger, hostname, osd_path, json_blob, metadata, ) distro.conn.exit() 创建OSD&amp;准备OSD创建OSD的命令行格式为：ceph-deploy osd create [-h] [--zap-disk] [--fs-type FS_TYPE] [--dmcrypt] [--dmcrypt-key-dir KEYDIR] [--bluestore] HOST:DISK[:JOURNAL] [HOST:DISK[:JOURNAL] …] 准备OSD的命令行格式为：ceph-deploy osd prepare [-h] [--zap-disk] [--fs-type FS_TYPE] [--dmcrypt] [--dmcrypt-key-dir KEYDIR] [--bluestore] HOST:DISK[:JOURNAL] [HOST:DISK[:JOURNAL] …] prepare函数，参数activate_prepared_disk为True是创建OSD，为False是准备OSD 调用exceeds_max_osds函数，单台主机超过20个OSD，将会warning 调用get_bootstrap_osd_key函数，获取当前目录下的ceph.bootstrap-osd.keyring 循环disk 配置写入 /etc/ceph/ceph.conf 创建并写入 /var/lib/ceph/bootstrap-osd/ceph.keyring 调用prepare_disk函数，准备OSD 校验OSD状态，并将信息非正常状态信息写入warning 1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889def prepare(args, cfg, activate_prepared_disk): LOG.debug( 'Preparing cluster %s disks %s', args.cluster, ' '.join(':'.join(x or '' for x in t) for t in args.disk), ) # 单台主机超过20个OSD，将会warning hosts_in_danger = exceeds_max_osds(args) if hosts_in_danger: LOG.warning('if ``kernel.pid_max`` is not increased to a high enough value') LOG.warning('the following hosts will encounter issues:') for host, count in hosts_in_danger.items(): LOG.warning('Host: %8s, OSDs: %s' % (host, count)) # 获取当前目录下的ceph.bootstrap-osd.keyring key = get_bootstrap_osd_key(cluster=args.cluster) bootstrapped = set() errors = 0 for hostname, disk, journal in args.disk: try: if disk is None: raise exc.NeedDiskError(hostname) distro = hosts.get( hostname, username=args.username, callbacks=[packages.ceph_is_installed] ) LOG.info( 'Distro info: %s %s %s', distro.name, distro.release, distro.codename ) if hostname not in bootstrapped: bootstrapped.add(hostname) LOG.debug('Deploying osd to %s', hostname) conf_data = conf.ceph.load_raw(args) # 配置写入/etc/ceph/ceph.conf distro.conn.remote_module.write_conf( args.cluster, conf_data, args.overwrite_conf ) # 创建并写入 /var/lib/ceph/bootstrap-osd/ceph.keyring create_osd_keyring(distro.conn, args.cluster, key) LOG.debug('Preparing host %s disk %s journal %s activate %s', hostname, disk, journal, activate_prepared_disk) storetype = None if args.bluestore: storetype = 'bluestore' # 准备OSD prepare_disk( distro.conn, cluster=args.cluster, disk=disk, journal=journal, activate_prepared_disk=activate_prepared_disk, init=distro.init, zap=args.zap_disk, fs_type=args.fs_type, dmcrypt=args.dmcrypt, dmcrypt_dir=args.dmcrypt_key_dir, storetype=storetype, ) # give the OSD a few seconds to start time.sleep(5) # 校验OSD状态，并将信息非正常状态信息写入warning catch_osd_errors(distro.conn, distro.conn.logger, args) LOG.debug('Host %s is now ready for osd use.', hostname) distro.conn.exit() except RuntimeError as e: LOG.error(e) errors += 1 if errors: raise exc.GenericError('Failed to create %d OSDs' % errors) prepare_disk函数 执行 ceph-disk -v prepare 命令准备OSD 如果activate_prepared_disk为True，设置ceph服务开机启动1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859def prepare_disk( conn, cluster, disk, journal, activate_prepared_disk, init, zap, fs_type, dmcrypt, dmcrypt_dir, storetype): """ Run on osd node, prepares a data disk for use. """ ceph_disk_executable = system.executable_path(conn, 'ceph-disk') args = [ ceph_disk_executable, '-v', 'prepare', ] if zap: args.append('--zap-disk') if dmcrypt: args.append('--dmcrypt') if dmcrypt_dir is not None: args.append('--dmcrypt-key-dir') args.append(dmcrypt_dir) if storetype: args.append('--' + storetype) args.extend([ '--cluster', cluster, '--fs-type', fs_type, '--', disk, ]) if journal is not None: args.append(journal) # 执行 ceph-disk -v prepare 命令 remoto.process.run( conn, args ) # 是否激活，激活即设置ceph服务开机启动 if activate_prepared_disk: # we don't simply run activate here because we don't know # which partition ceph-disk prepare created as the data # volume. instead, we rely on udev to do the activation and # just give it a kick to ensure it wakes up. we also enable # ceph.target, the other key piece of activate. if init == 'systemd': system.enable_service(conn, "ceph.target") elif init == 'sysvinit': system.enable_service(conn, "ceph") 激活OSD命令行格式为：ceph-deploy osd activate [-h] HOST:DISK[:JOURNAL] [HOST:DISK[:JOURNAL] …] activate函数 执行 ceph-disk -v activate 命令激活OSD 校验OSD状态，并将信息非正常状态信息写入warning 设置ceph服务开机启动12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152def activate(args, cfg): LOG.debug( 'Activating cluster %s disks %s', args.cluster, # join elements of t with ':', t's with ' ' # allow None in elements of t; print as empty ' '.join(':'.join((s or '') for s in t) for t in args.disk), ) for hostname, disk, journal in args.disk: distro = hosts.get( hostname, username=args.username, callbacks=[packages.ceph_is_installed] ) LOG.info( 'Distro info: %s %s %s', distro.name, distro.release, distro.codename ) LOG.debug('activating host %s disk %s', hostname, disk) LOG.debug('will use init type: %s', distro.init) ceph_disk_executable = system.executable_path(distro.conn, 'ceph-disk') # 执行 ceph-disk -v activate 命令激活OSD remoto.process.run( distro.conn, [ ceph_disk_executable, '-v', 'activate', '--mark-init', distro.init, '--mount', disk, ], ) # give the OSD a few seconds to start time.sleep(5) # 校验OSD状态，并将信息非正常状态信息写入warning catch_osd_errors(distro.conn, distro.conn.logger, args) # 设置ceph服务开机启动 if distro.init == 'systemd': system.enable_service(distro.conn, "ceph.target") elif distro.init == 'sysvinit': system.enable_service(distro.conn, "ceph") distro.conn.exit() 手工管理OSD以ceph-231上磁盘sdb为例，创建osd。 创建OSD&amp;准备OSD准备OSD1[root@ceph-231 ~]# ceph-disk -v prepare --zap-disk --cluster ceph --fs-type xfs -- /dev/sdb 创建OSD多一个操作，设置ceph服务开机启动1[root@ceph-231 ~]# systemctl enable ceph.target 激活OSD查看init12[root@ceph-231 ~]# cat /proc/1/commsystemd 激活OSD1[root@ceph-231 ~]# ceph-disk -v activate --mark-init systemd --mount /dev/sdb1 设置ceph服务开机启动1[root@ceph-231 ~]# systemctl enable ceph.target]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[执行ceph命令出错：librados: client.admin authentication error (1) Operation not permitted]]></title>
      <url>%2F2017%2F06%2F20%2Fceph-error-authentication%2F</url>
      <content type="text"><![CDATA[执行ceph -s命令出错：123[root@ceph-231 ceph-ceph-231]# ceph -s2017-06-20 17:34:12.196793 7f6744580700 0 librados: client.admin authentication error (1) Operation not permittedError connecting to cluster: PermissionError 解决方法将ceph.client.admin.keyring文件内容copy到mon目录的keyring，重启mon服务。 查看mon的client.admin.keyring1234567[root@ceph-231 ceph]# /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-231/keyring auth get client.adminexported keyring for client.admin[client.admin] key = AQAEI0JZupXTFRAAmFF56vYMzKkzc5nxLit6mA== caps mds = "allow *" caps mon = "allow *" caps osd = "allow *" 如果auth中存在client.admin.keyring，则将client.admin.keyring信息copy到/etc/ceph/ceph.client.admin.keyring与/var/lib/ceph/mon/ceph-ceph-231/keyring文件中。 如果auth中不存在client.admin.keyring，则需创建client.admin.keyring。 创建client.admin.keyring查看/etc/ceph/ceph.client.admin.keyring文件，如果不存在则使用ceph-authtool命令重新生成1[root@ceph-231 ~]# ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' 查看/etc/ceph/ceph.client.admin.keyring文件1234567[root@ceph-231 ~]# cat /etc/ceph/ceph.client.admin.keyring[client.admin] key = AQAEI0JZupXTFRAAmFF56vYMzKkzc5nxLit6mA== auid = 0 caps mds = "allow *" caps mon = "allow *" caps osd = "allow *" 复制keyring将/etc/ceph/ceph.client.admin.keyring文件内容copy到mon目录的keyring12345678910[root@ceph-231 ~]# vi /var/lib/ceph/mon/ceph-ceph-231/keyring[mon.] key = AQBzv0hZAAAAABAAJLiETzmegHWmVO7JwvsMdQ== caps mon = "allow *"[client.admin] key = AQAEI0JZupXTFRAAmFF56vYMzKkzc5nxLit6mA== auid = 0 caps mds = "allow *" caps mon = "allow *" caps osd = "allow *" 重启mon1[root@ceph-231 ~]# systemctl restart ceph-mon@ceph-231]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[ceph-deploy源码分析（三）——mon模块]]></title>
      <url>%2F2017%2F06%2F19%2Fceph-deploy-mon%2F</url>
      <content type="text"><![CDATA[ceph-deploy的mon.py模块是用来管理mon守护进程。 mon 子命令格式如下1ceph-deploy mon [-h] &#123;add,create,create-initial,destroy&#125; ... create: 创建mon add: 添加mon到集群，添加之前先在ceph.conf配置public_network create-initial: 创建mon并初始化 destroy: 删除mon，如果集群只有一个mon则不能删除 在创建mon时，建议使用 ceph-deploy mon create-initial 来创建。 mon管理make函数priority为30，子命令设置的默认函数为mon函数。1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071@priority(30)def make(parser): """ Ceph MON Daemon management """ parser.formatter_class = ToggleRawTextHelpFormatter mon_parser = parser.add_subparsers(dest='subcommand') mon_parser.required = True mon_add = mon_parser.add_parser( 'add', help=('R|Add a monitor to an existing cluster:\n' '\tceph-deploy mon add node1\n' 'Or:\n' '\tceph-deploy mon add --address 192.168.1.10 node1\n' 'If the section for the monitor exists and defines a `mon addr` that\n' 'will be used, otherwise it will fallback by resolving the hostname to an\n' 'IP. If `--address` is used it will override all other options.') ) mon_add.add_argument( '--address', nargs='?', ) mon_add.add_argument( 'mon', nargs=1, ) mon_create = mon_parser.add_parser( 'create', help=('R|Deploy monitors by specifying them like:\n' '\tceph-deploy mon create node1 node2 node3\n' 'If no hosts are passed it will default to use the\n' '`mon initial members` defined in the configuration.') ) mon_create.add_argument( '--keyrings', nargs='?', help='concatenate multiple keyrings to be seeded on new monitors', ) mon_create.add_argument( 'mon', nargs='*', ) mon_create_initial = mon_parser.add_parser( 'create-initial', help=('Will deploy for monitors defined in `mon initial members`, ' 'wait until they form quorum and then gatherkeys, reporting ' 'the monitor status along the process. If monitors don\'t form ' 'quorum the command will eventually time out.') ) mon_create_initial.add_argument( '--keyrings', nargs='?', help='concatenate multiple keyrings to be seeded on new monitors', ) mon_destroy = mon_parser.add_parser( 'destroy', help='Completely remove Ceph MON from remote host(s)' ) mon_destroy.add_argument( 'mon', nargs='+', ) parser.set_defaults( func=mon, ) mon子命令mon 函数，mon有四个subcmd为create、add、destroy、create-initial，分别对应mon_create、mon_add、mon_destroy、mon_create_initial四个函数1234567891011def mon(args): if args.subcommand == 'create': mon_create(args) elif args.subcommand == 'add': mon_add(args) elif args.subcommand == 'destroy': mon_destroy(args) elif args.subcommand == 'create-initial': mon_create_initial(args) else: LOG.error('subcommand %s not implemented', args.subcommand) 创建mon命令行格式为： ceph-deploy mon create [node1] [node2] [node3] … mon_create函数创建mon args参数校验 调用hosts.get函数获取操作系统版本信息，检查是否安装ceph包，目前支持的操作系统为centos/debian/fedora/rhel/suse，如果需要修改代码支持其他操作系统，可以从hosts入手修改，比如增加支持XenServer 调用相应操作系统模块下的create函数创建mon，比如操作系统为centos，即hosts/centos下的mon模块 调用mon_status函数检测mon状态 调用catch_mon_errors函数获取mon的错误信息，写入logger12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364def mon_create(args): # 获取配置文件 cfg = conf.ceph.load(args) if not args.mon: # 参数没指定mon，调用get_mon_initial_members函数从配置文件获取mon_initial_members作为mon args.mon = get_mon_initial_members(args, error_on_empty=True, _cfg=cfg) if args.keyrings: monitor_keyring = concatenate_keyrings(args) else: keyring_path = '&#123;cluster&#125;.mon.keyring'.format(cluster=args.cluster) try: # 获取ceph.mon.keyring文件信息 monitor_keyring = files.read_file(keyring_path) except IOError: LOG.warning('keyring (%s) not found, creating a new one' % keyring_path) new_mon_keyring(args) monitor_keyring = files.read_file(keyring_path) LOG.debug( 'Deploying mon, cluster %s hosts %s', args.cluster, ' '.join(args.mon), ) errors = 0 # 循环mon for (name, host) in mon_hosts(args.mon): try: # TODO add_bootstrap_peer_hint LOG.debug('detecting platform for host %s ...', name) # 获取操作系统版本信息，检查是否安装ceph包，如果需要修改代码支持其他操作系统，可以从hosts入手修改 distro = hosts.get( host, username=args.username, callbacks=[packages.ceph_is_installed] ) LOG.info('distro info: %s %s %s', distro.name, distro.release, distro.codename) rlogger = logging.getLogger(name) # ensure remote hostname is good to go hostname_is_compatible(distro.conn, rlogger, name) rlogger.debug('deploying mon to %s', name) # 创建mon，调用hosts目录的相应操作系列目录，比如系统是centos，那就是hosts/centos下的mon模块 distro.mon.create(distro, args, monitor_keyring) # tell me the status of the deployed mon time.sleep(2) # give some room to start # 检测mon的状态 mon_status(distro.conn, rlogger, name, args) # 获取mon的错误信息，写入logger catch_mon_errors(distro.conn, rlogger, name, cfg, args) distro.conn.exit() except RuntimeError as e: LOG.error(e) errors += 1 if errors: raise exc.GenericError('Failed to create %d monitors' % errors) 以centos为例，hosts/centos下的mon模块的init.py12from ceph_deploy.hosts.common import mon_add as add # noqafrom ceph_deploy.hosts.common import mon_create as create # noqa mon_create函数在hosts/common.py模块 创建/etc/ceph/ceph.conf文件，从cph-deploy new创建的ceph.conf与args中写入数据到/etc/ceph/ceph.conf 将ceph-deploy new创建的ceph.mon.keyring文件内容写入临时文件/var/lib/ceph/tmp/ceph.mon.keyring 创建mon，初始化keyring数据 创建空白的done文件，并将文件的拥有者设置成uid、gid 创建init文件systemd，并将文件的拥有者设置成uid、gid 设置mon服务开机启动，启动mon服务123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687def mon_create(distro, args, monitor_keyring): hostname = distro.conn.remote_module.shortname() logger = distro.conn.logger logger.debug('remote hostname: %s' % hostname) # mon目录，比如：/var/lib/ceph/mon/ceph-1 path = paths.mon.path(args.cluster, hostname) # 获取/var/lib/ceph目录的用户id uid = distro.conn.remote_module.path_getuid(constants.base_path) # 获取/var/lib/ceph目录的用户组gid gid = distro.conn.remote_module.path_getgid(constants.base_path) # 获取mon下的done文件路径，比如：/var/lib/ceph/mon/ceph-1/done done_path = paths.mon.done(args.cluster, hostname) # 获取mon下的systemd文件路径，比如：/var/lib/ceph/mon/ceph-1/systemd init_path = paths.mon.init(args.cluster, hostname, distro.init) # 获取ceph-deploy创建的ceph.conf文件数据 conf_data = conf.ceph.load_raw(args) # write the configuration file # 写入/etc/ceph/ceph.conf distro.conn.remote_module.write_conf( args.cluster, conf_data, args.overwrite_conf, ) # if the mon path does not exist, create it # 如果mon文件目录不存在，创建mon目录，并将目录的拥有者改成uid、gid distro.conn.remote_module.create_mon_path(path, uid, gid) logger.debug('checking for done path: %s' % done_path) if not distro.conn.remote_module.path_exists(done_path): # done文件不存在 logger.debug('done path does not exist: %s' % done_path) if not distro.conn.remote_module.path_exists(paths.mon.constants.tmp_path): # /var/lib/ceph/tmp目录不存在 logger.info('creating tmp path: %s' % paths.mon.constants.tmp_path) # 创建/var/lib/ceph/tmp目录 distro.conn.remote_module.makedir(paths.mon.constants.tmp_path) # 获取/var/lib/ceph/tmp/ceph.mon.keyring keyring = paths.mon.keyring(args.cluster, hostname) logger.info('creating keyring file: %s' % keyring) # 将ceph-deploy new创建的ceph.mon.keyring文件内容写入临时文件/var/lib/ceph/tmp/ceph.mon.keyring distro.conn.remote_module.write_monitor_keyring( keyring, monitor_keyring, uid, gid, ) user_args = [] if uid != 0: user_args = user_args + [ '--setuser', str(uid) ] if gid != 0: user_args = user_args + [ '--setgroup', str(gid) ] # 创建mon remoto.process.run( distro.conn, [ 'ceph-mon', '--cluster', args.cluster, '--mkfs', '-i', hostname, '--keyring', keyring, ] + user_args ) logger.info('unlinking keyring file %s' % keyring) distro.conn.remote_module.unlink(keyring) # create the done file # 创建空白的done文件，并将文件的拥有者设置成uid、gid，表示mon创建完成 distro.conn.remote_module.create_done_path(done_path, uid, gid) # create init path # 创建init文件，并将文件的拥有者设置成uid、gid distro.conn.remote_module.create_init_path(init_path, uid, gid) # start mon service # 启动mon服务 start_mon_service(distro, args.cluster, hostname) 添加mon到集群命令行格式为： ceph-deploy mon add [--address [ADDRESS]] mon … mon_add函数 调用admin模块admin函数（即ceph-deploy admin子命令），远程host上写入/etc/ceph/ceph.conf与/etc/ceph/ceph.client.admin.keyring文件 调用操作系统类型相应的mon的add函数，创建mon 调用catch_mon_errors函数校验error 调用mon_status校验mon的状态1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768def mon_add(args): cfg = conf.ceph.load(args) # args.mon is a list with only one entry mon_host = args.mon[0] # 获取ceph.mon.keyring文件配置信息 try: with open('&#123;cluster&#125;.mon.keyring'.format(cluster=args.cluster), 'rb') as f: monitor_keyring = f.read() except IOError: raise RuntimeError( 'mon keyring not found; run \'new\' to create a new cluster' ) LOG.info('ensuring configuration of new mon host: %s', mon_host) args.client = args.mon # 调用admin模块admin函数，在远程host上写入/etc/ceph/ceph.conf与/etc/ceph/ceph.client.admin.keyring文件 admin.admin(args) LOG.debug( 'Adding mon to cluster %s, host %s', args.cluster, mon_host, ) mon_section = 'mon.%s' % mon_host cfg_mon_addr = cfg.safe_get(mon_section, 'mon addr') # 校验mon ip if args.address: LOG.debug('using mon address via --address %s' % args.address) mon_ip = args.address elif cfg_mon_addr: LOG.debug('using mon address via configuration: %s' % cfg_mon_addr) mon_ip = cfg_mon_addr else: mon_ip = net.get_nonlocal_ip(mon_host) LOG.debug('using mon address by resolving host: %s' % mon_ip) try: LOG.debug('detecting platform for host %s ...', mon_host) distro = hosts.get( mon_host, username=args.username, callbacks=[packages.ceph_is_installed] ) LOG.info('distro info: %s %s %s', distro.name, distro.release, distro.codename) rlogger = logging.getLogger(mon_host) # ensure remote hostname is good to go hostname_is_compatible(distro.conn, rlogger, mon_host) rlogger.debug('adding mon to %s', mon_host) args.address = mon_ip # 添加mon到集群 distro.mon.add(distro, args, monitor_keyring) # tell me the status of the deployed mon time.sleep(2) # give some room to start # 获取mon的错误信息写入logger warning catch_mon_errors(distro.conn, rlogger, mon_host, cfg, args) # 校验mon的状态 mon_status(distro.conn, rlogger, mon_host, args) distro.conn.exit() except RuntimeError as e: LOG.error(e) raise exc.GenericError('Failed to add monitor to host: %s' % mon_host) add函数，以centos为例，在hosts/centos/mon/init.py12from ceph_deploy.hosts.common import mon_add as add # noqafrom ceph_deploy.hosts.common import mon_create as create # noqa hosts/common.py模块的mon_add函数 创建/etc/ceph/ceph.conf文件，从cph-deploy new创建的ceph.conf与args中写入数据到/etc/ceph/ceph.conf 将ceph-deploy new创建的ceph.mon.keyring文件内容写入临时文件/var/lib/ceph/tmp/ceph.mon.keyring 获取mon的monmap 创建mon，初始化monmap和keyring数据 创建空白的done文件，并将文件的拥有者设置成uid、gid 创建init文件systemd，并将文件的拥有者设置成uid、gid 设置mon服务开机启动，启动mon服务 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081def mon_add(distro, args, monitor_keyring): hostname = distro.conn.remote_module.shortname() logger = distro.conn.logger path = paths.mon.path(args.cluster, hostname) uid = distro.conn.remote_module.path_getuid(constants.base_path) gid = distro.conn.remote_module.path_getgid(constants.base_path) monmap_path = paths.mon.monmap(args.cluster, hostname) done_path = paths.mon.done(args.cluster, hostname) init_path = paths.mon.init(args.cluster, hostname, distro.init) conf_data = conf.ceph.load_raw(args) # write the configuration file distro.conn.remote_module.write_conf( args.cluster, conf_data, args.overwrite_conf, ) # if the mon path does not exist, create it distro.conn.remote_module.create_mon_path(path, uid, gid) logger.debug('checking for done path: %s' % done_path) if not distro.conn.remote_module.path_exists(done_path): logger.debug('done path does not exist: %s' % done_path) if not distro.conn.remote_module.path_exists(paths.mon.constants.tmp_path): logger.info('creating tmp path: %s' % paths.mon.constants.tmp_path) distro.conn.remote_module.makedir(paths.mon.constants.tmp_path) keyring = paths.mon.keyring(args.cluster, hostname) logger.info('creating keyring file: %s' % keyring) distro.conn.remote_module.write_monitor_keyring( keyring, monitor_keyring, uid, gid, ) # get the monmap remoto.process.run( distro.conn, [ 'ceph', '--cluster', args.cluster, 'mon', 'getmap', '-o', monmap_path, ], ) # now use it to prepare the monitor's data dir user_args = [] if uid != 0: user_args = user_args + [ '--setuser', str(uid) ] if gid != 0: user_args = user_args + [ '--setgroup', str(gid) ] remoto.process.run( distro.conn, [ 'ceph-mon', '--cluster', args.cluster, '--mkfs', '-i', hostname, '--monmap', monmap_path, '--keyring', keyring, ] + user_args ) logger.info('unlinking keyring file %s' % keyring) distro.conn.remote_module.unlink(keyring) # create the done file distro.conn.remote_module.create_done_path(done_path, uid, gid) # create init path distro.conn.remote_module.create_init_path(init_path, uid, gid) # start mon service start_mon_service(distro, args.cluster, hostname) 删除mon命令行格式为： ceph-deploy mon destroy [-h] mon [mon …] mon_destroy函数，调用destroy_mon函数123456789101112131415161718192021222324252627def mon_destroy(args): errors = 0 for (name, host) in mon_hosts(args.mon): try: LOG.debug('Removing mon from %s', name) distro = hosts.get( host, username=args.username, callbacks=[packages.ceph_is_installed] ) hostname = distro.conn.remote_module.shortname() # 删除mon destroy_mon( distro.conn, args.cluster, hostname, ) distro.conn.exit() except RuntimeError as e: LOG.error(e) errors += 1 if errors: raise exc.GenericError('Failed to destroy %d monitors' % errors) destroy_mon函数 从集群中删除mon 停止mon服务 将mon数据文件移动到/var/lib/ceph/mon-removed目录归档文件123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081def destroy_mon(conn, cluster, hostname): import datetime import time retries = 5 # mon的目录，比如/var/lib/ceph/mon/ceph-node1 path = paths.mon.path(cluster, hostname) if conn.remote_module.path_exists(path): # remove from cluster # 从集群中删除mon remoto.process.run( conn, [ 'ceph', '--cluster=&#123;cluster&#125;'.format(cluster=cluster), '-n', 'mon.', '-k', '&#123;path&#125;/keyring'.format(path=path), 'mon', 'remove', hostname, ], timeout=7, ) # stop if conn.remote_module.path_exists(os.path.join(path, 'upstart')) or system.is_upstart(conn): status_args = [ 'initctl', 'status', 'ceph-mon', 'cluster=&#123;cluster&#125;'.format(cluster=cluster), 'id=&#123;hostname&#125;'.format(hostname=hostname), ] elif conn.remote_module.path_exists(os.path.join(path, 'sysvinit')): status_args = [ 'service', 'ceph', 'status', 'mon.&#123;hostname&#125;'.format(hostname=hostname), ] elif system.is_systemd(conn): # 停止mon服务 status_args = [ 'systemctl', 'stop', 'ceph-mon@&#123;hostname&#125;.service'.format(hostname=hostname), ] else: raise RuntimeError('could not detect a supported init system, cannot continue') while retries: conn.logger.info('polling the daemon to verify it stopped') if is_running(conn, status_args): time.sleep(5) retries -= 1 if retries &lt;= 0: raise RuntimeError('ceph-mon deamon did not stop') else: break # archive old monitor directory fn = '&#123;cluster&#125;-&#123;hostname&#125;-&#123;stamp&#125;'.format( hostname=hostname, cluster=cluster, stamp=datetime.datetime.utcnow().strftime("%Y-%m-%dZ%H:%M:%S"), ) # 创建/var/lib/ceph/mon-removed目录 remoto.process.run( conn, [ 'mkdir', '-p', '/var/lib/ceph/mon-removed', ], ) # 将mon数据文件移动到/var/lib/ceph/mon-removed目录归档文件 conn.remote_module.make_mon_removed_dir(path, fn) 创建mon并初始化命令行格式为： ceph-deploy mon create-initial [-h] [--keyrings [KEYRINGS]] mon_create_initial函数 调用mon_create函数，创建mon，可参考create部分 判断集群中的mon_in_quorum与ceph.conf中的mon_initial_members是否完全匹配 调用gatherkeys模块gatherkeys函数，收集用于配置新节点的keys12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152def mon_create_initial(args): # 获取ceph.conf中的mon_initial_members mon_initial_members = get_mon_initial_members(args, error_on_empty=True) # create them normally through mon_create args.mon = mon_initial_members # 创建mon mon_create(args) # make the sets to be able to compare late mon_in_quorum = set([]) mon_members = set([host for host in mon_initial_members]) for host in mon_initial_members: mon_name = 'mon.%s' % host LOG.info('processing monitor %s', mon_name) sleeps = [20, 20, 15, 10, 10, 5] tries = 5 rlogger = logging.getLogger(host) distro = hosts.get( host, username=args.username, callbacks=[packages.ceph_is_installed] ) while tries: # 获取mon的状态 status = mon_status_check(distro.conn, rlogger, host, args) has_reached_quorum = status.get('state', '') in ['peon', 'leader'] if not has_reached_quorum: LOG.warning('%s monitor is not yet in quorum, tries left: %s' % (mon_name, tries)) tries -= 1 sleep_seconds = sleeps.pop() LOG.warning('waiting %s seconds before retrying', sleep_seconds) time.sleep(sleep_seconds) # Magic number else: mon_in_quorum.add(host) LOG.info('%s monitor has reached quorum!', mon_name) break distro.conn.exit() # 集群中的mon_in_quorum与ceph.conf中的mon_initial_members完全匹配 if mon_in_quorum == mon_members: LOG.info('all initial monitors are running and have formed quorum') LOG.info('Running gatherkeys...') # 调用gatherkeys模块gatherkeys函数，收集用于配置新节点的keys gatherkeys.gatherkeys(args) else: LOG.error('Some monitors have still not reached quorum:') for host in mon_members - mon_in_quorum: LOG.error('%s', host) raise SystemExit('cluster may not be in a healthy state') 手工管理mon创建mon以ceph-deploy创建mon：ceph-deploy mon create ceph-231为例，对应的手工操作。 生成fsid12[root@ceph-231 ~]# uuidgena3b9b0aa-01ab-4e1b-bba3-6f5317b0795b 创建ceph.conf12345678[root@ceph-231 ~]# vi /etc/ceph/ceph.conf[global]fsid = a3b9b0aa-01ab-4e1b-bba3-6f5317b0795bmon_initial_members = ceph-231mon_host = 192.168.217.231auth_cluster_required = cephxauth_service_required = cephxauth_client_required = cephx 创建ceph-ceph-231.mon.keyring创建keyring12345[root@ceph-231 ~]# ceph-authtool --create-keyring /var/lib/ceph/tmp/ceph-ceph-231.mon.keyring --gen-key -n mon. --cap mon 'allow *'[root@ceph-231 ~]# cat /var/lib/ceph/tmp/ceph-ceph-231.mon.keyring[mon.] key = AQBzv0hZAAAAABAAJLiETzmegHWmVO7JwvsMdQ== caps mon = "allow *" 改变文件拥有者为ceph1[root@ceph-231 ~]# chown ceph:ceph /var/lib/ceph/tmp/ceph-ceph-231.mon.keyring 创建mon获取ceph用户的uid、gid12[root@ceph-231 ~]# id cephuid=167(ceph) gid=167(ceph) groups=167(ceph) 创建mon1[root@ceph-231 ~]# ceph-mon --cluster ceph --mkfs -i ceph-231 --keyring /var/lib/ceph/tmp/ceph-ceph-231.mon.keyring --setuser 167 --setgroup 167 创建done文件12[root@ceph-231 ~]# touch /var/lib/ceph/mon/ceph-ceph-231/done[root@ceph-231 ~]# chown ceph:ceph /var/lib/ceph/mon/ceph-ceph-231/done 创建init文件查看init12[root@ceph-231 ~]# cat /proc/1/commsystemd 创建systemd文件12[root@ceph-231 ~]# touch /var/lib/ceph/mon/ceph-ceph-231/systemd[root@ceph-231 ~]# chown ceph:ceph /var/lib/ceph/mon/ceph-ceph-231/systemd 启动mon123[root@ceph-231 ~]# systemctl enable ceph.target[root@ceph-231 ~]# systemctl enable ceph-mon@ceph-231[root@ceph-231 ~]# systemctl start ceph-mon@ceph-231 查看mon状态 rank大于等于0，表示mon正在运行 rank等于-1，state为mon的状态1234567891011121314151617181920212223242526[root@ceph-231 ceph-ceph-231]# ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-231.asok mon_status&#123; "name": "ceph-231", "rank": 0, "state": "leader", "election_epoch": 3, "quorum": [ 0 ], "outside_quorum": [], "extra_probe_peers": [], "sync_provider": [], "monmap": &#123; "epoch": 1, "fsid": "a3b9b0aa-01ab-4e1b-bba3-6f5317b0795b", "modified": "2017-06-20 15:53:44.533604", "created": "2017-06-20 15:53:44.533604", "mons": [ &#123; "rank": 0, "name": "ceph-231", "addr": "192.168.217.231:6789\/0" &#125; ] &#125;&#125; 添加mon到集群集群ceph-231上添加ceph-232这个mon。 ceph.conf添加public_network修改ceph.conf文件，添加 public_network=192.168.217.0/241[root@ceph-231 ~]# vi /etc/ceph/ceph.conf copy配置与keyring复制/etc/ceph/ceph.conf与/etc/ceph/ceph.client.admin.keyring文件12[root@ceph-231 ~]# scp /etc/ceph/ceph.conf root@ceph-232:/etc/ceph[root@ceph-231 ~]# scp /etc/ceph/ceph.client.admin.keyring root@ceph-232:/etc/ceph 复制mon的keyring1[root@ceph-231 ~]# scp /var/lib/ceph/mon/ceph-ceph-231/keyring root@ceph-232:/var/lib/ceph/tmp/ceph-ceph-232.mon.keyring 设置keyring的用户权限1[root@ceph-232 ~]# chown ceph:ceph /var/lib/ceph/tmp/ceph-ceph-232.mon.keyring 创建mon获取ceph-232的monmap1[root@ceph-232 ~]# ceph --cluster ceph mon getmap -o /var/lib/ceph/tmp/ceph.ceph-232.monmap 获取用户ceph的uid、gid12[root@ceph-232 ~]# id cephuid=167(ceph) gid=167(ceph) groups=167(ceph) 创建ceph-232的mon1[root@ceph-232 ~]# ceph-mon --cluster ceph --mkfs -i ceph-232 --monmap /var/lib/ceph/tmp/ceph.ceph-232.monmap --keyring /var/lib/ceph/tmp/ceph-ceph-232.mon.keyring --setuser 167 --setgroup 167 创建done文件12[root@ceph-232 ~]# touch /var/lib/ceph/mon/ceph-ceph-232/done[root@ceph-232 ~]# chown ceph:ceph /var/lib/ceph/mon/ceph-ceph-232/done 创建init文件查看init12[root@ceph-232 ~]# cat /proc/1/commsystemd 创建systemd文件12[root@ceph-232 ~]# touch /var/lib/ceph/mon/ceph-ceph-232/systemd[root@ceph-232 ~]# chown ceph:ceph /var/lib/ceph/mon/ceph-ceph-232/systemd 启动mon123[root@ceph-232 ~]# systemctl enable ceph.target[root@ceph-232 ~]# systemctl enable ceph-mon@ceph-232[root@ceph-232 ~]# systemctl start ceph-mon@ceph-232 查看mon状态1234567891011121314151617181920212223242526272829303132[root@ceph-232 ~]# ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-232.asok mon_status&#123; "name": "ceph-232", "rank": 1, "state": "peon", "election_epoch": 6, "quorum": [ 0, 1 ], "outside_quorum": [], "extra_probe_peers": [], "sync_provider": [], "monmap": &#123; "epoch": 4, "fsid": "e1d69cc3-f40c-44c7-a758-fdfd4e5ef43e", "modified": "2017-06-21 16:46:01.518137", "created": "2017-06-21 16:16:34.913008", "mons": [ &#123; "rank": 0, "name": "ceph-231", "addr": "192.168.217.231:6789\/0" &#125;, &#123; "rank": 1, "name": "ceph-232", "addr": "192.168.217.232:6789\/0" &#125; ] &#125;&#125; 删除mon如果集群只有一个mon则不能删除mon12[root@ceph-232 ~]# ceph --cluster=ceph -n mon. -k /var/lib/ceph/mon/ceph-ceph-232/keyring mon remove ceph-232removing mon.ceph-232 at 192.168.217.232:6789/0, there will be 1 monitors 创建mon并初始化创建mon创建mon与前面【创建mon】相同 初始化初始化，创建keys client.admin1234567[root@ceph-231 ~]# /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-231/keyring auth get client.adminexported keyring for client.admin[client.admin] key = AQAEI0JZupXTFRAAmFF56vYMzKkzc5nxLit6mA== caps mds = "allow *" caps mon = "allow *" caps osd = "allow *" client.bootstrap-mds12345[root@ceph-231 ~]# /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-231/keyring auth get client.bootstrap-mdsexported keyring for client.bootstrap-mds[client.bootstrap-mds] key = AQBjzklZ5D6oLRAAMSwQ169JjzNnPBMzIv6vCw== caps mon = "allow profile bootstrap-mds" 如果client.bootstrap-mds在keyring中不存在，则创建1[root@ceph-231 ~]# /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-231/keyring auth get-or-create client.bootstrap-mds mon "allow profile bootstrap-mds" client.bootstrap-osd12345[root@ceph-231 ~]# /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-231/keyring auth get client.bootstrap-osdexported keyring for client.bootstrap-osd[client.bootstrap-osd] key = AQBkzklZf/NpKhAAeJwteZDEGfZj66BUnbxC1Q== caps mon = "allow profile bootstrap-osd" 如果client.bootstrap-osd在keyring中不存在，则创建1[root@ceph-231 ~]# /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-231/keyring auth get-or-create client.bootstrap-osd mon "allow profile bootstrap-osd" client.bootstrap-rgw12345[root@ceph-231 ~]# /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-231/keyring auth get client.bootstrap-rgwexported keyring for client.bootstrap-rgw[client.bootstrap-rgw] key = AQBlzklZT3BEIRAA9w+/G+Sp6zJ+aTh+VQwTUQ== caps mon = "allow profile bootstrap-rgw" 如果client.bootstrap-rgw在keyring中不存在，则创建1[root@ceph-231 ~]# /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-231/keyring auth get-or-create client.bootstrap-rgw mon "allow profile bootstrap-rgw" mon文件结构mon目录，比如/var/lib/ceph/mon/ceph-ceph-231有四个文件 done： 空白文件，是否创建完成的标志 keyring： mon的keys store.db：mon数据库文件 systemd：空白文件，init的标志文件（如果Centos上的init为sysvinit，那么这个文件名称就是sysvinit）]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[ceph-deploy源码分析（二）——new模块]]></title>
      <url>%2F2017%2F06%2F19%2Fceph-deploy-new%2F</url>
      <content type="text"><![CDATA[ceph-deploy的new.py模块是用来开始部署新集群，创建ceph.conf、ceph.mon.keyring文件。 new 子命令格式如下1234ceph-deploy new [-h] [--no-ssh-copykey] [--fsid FSID] [--cluster-network CLUSTER_NETWORK] [--public-network PUBLIC_NETWORK] MON [MON ...] 部署集群make函数priority为10，子命令设置的默认函数为new函数。1234567891011121314151617181920212223242526272829303132333435363738394041@priority(10)def make(parser): """ Start deploying a new cluster, and write a CLUSTER.conf and keyring for it. """ parser.add_argument( 'mon', metavar='MON', nargs='+', help='initial monitor hostname, fqdn, or hostname:fqdn pair', type=arg_validators.Hostname(), ) parser.add_argument( '--no-ssh-copykey', dest='ssh_copykey', action='store_false', default=True, help='do not attempt to copy SSH keys', ) parser.add_argument( '--fsid', dest='fsid', help='provide an alternate FSID for ceph.conf generation', ) parser.add_argument( '--cluster-network', help='specify the (internal) cluster network', type=arg_validators.Subnet(), ) parser.add_argument( '--public-network', help='specify the public network for a cluster', type=arg_validators.Subnet(), ) parser.set_defaults( func=new, ) 部署新集群new 函数开始部署新集群 创建ceph.conf文件,写入[global]fsid、mon_initial_members、mon_host、auth_cluster_required、auth_service_required、auth_client_required，如果参数中有public_network、cluster_network写入配置文件 调用new_mon_keyring函数创建ceph.mon.keyring文件123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105def new(args): if args.ceph_conf: raise RuntimeError('will not create a Ceph conf file if attemtping to re-use with `--ceph-conf` flag') LOG.debug('Creating new cluster named %s', args.cluster) # 生成配置 cfg = conf.ceph.CephConf() cfg.add_section('global') # 获取参数中的额fsid或者自动生成 fsid = args.fsid or uuid.uuid4() cfg.set('global', 'fsid', str(fsid)) # if networks were passed in, lets set them in the # global section if args.public_network: cfg.set('global', 'public network', str(args.public_network)) if args.cluster_network: cfg.set('global', 'cluster network', str(args.cluster_network)) # mon节点 mon_initial_members = [] # mon主机 mon_host = [] # 循环host for (name, host) in mon_hosts(args.mon): # Try to ensure we can ssh in properly before anything else # ssh key copy if args.ssh_copykey: ssh_copy_keys(host, args.username) # Now get the non-local IPs from the remote node # 连接远程主机 distro = hosts.get(host, username=args.username) # 获取主机的IP地址 remote_ips = net.ip_addresses(distro.conn) # custom cluster names on sysvinit hosts won't work if distro.init == 'sysvinit' and args.cluster != 'ceph': LOG.error('custom cluster names are not supported on sysvinit hosts') raise exc.ClusterNameError( 'host %s does not support custom cluster names' % host ) distro.conn.exit() # Validate subnets if we received any if args.public_network or args.cluster_network: # 校验IP地址 validate_host_ip(remote_ips, [args.public_network, args.cluster_network]) # Pick the IP that matches the public cluster (if we were told to do # so) otherwise pick the first, non-local IP LOG.debug('Resolving host %s', host) if args.public_network: ip = get_public_network_ip(remote_ips, args.public_network) else: ip = net.get_nonlocal_ip(host) LOG.debug('Monitor %s at %s', name, ip) mon_initial_members.append(name) try: socket.inet_pton(socket.AF_INET6, ip) mon_host.append("[" + ip + "]") LOG.info('Monitors are IPv6, binding Messenger traffic on IPv6') cfg.set('global', 'ms bind ipv6', 'true') except socket.error: mon_host.append(ip) LOG.debug('Monitor initial members are %s', mon_initial_members) LOG.debug('Monitor addrs are %s', mon_host) # mon_initial_members 有多个的话，中间用空格隔开 cfg.set('global', 'mon initial members', ', '.join(mon_initial_members)) # no spaces here, see http://tracker.newdream.net/issues/3145 # mon_host 有多个的话，中间没有空格 cfg.set('global', 'mon host', ','.join(mon_host)) # override undesirable defaults, needed until bobtail # http://tracker.ceph.com/issues/6788 cfg.set('global', 'auth cluster required', 'cephx') cfg.set('global', 'auth service required', 'cephx') cfg.set('global', 'auth client required', 'cephx') path = '&#123;name&#125;.conf'.format( name=args.cluster, ) # 创建mon keyring new_mon_keyring(args) LOG.debug('Writing initial config to %s...', path) tmp = '%s.tmp' % path with open(tmp, 'w') as f: # 保存ceph配置文件 cfg.write(f) try: os.rename(tmp, path) except OSError as e: if e.errno == errno.EEXIST: raise exc.ClusterExistsError(path) else: raise 注意： mon_initial_members 有多个的话，中间用空格隔开 mon_host 有多个的话，中间没有空格 创建ceph.mon.keyring文件new_mon_keyring函数创建ceph.mon.keyring文件12345678910111213141516171819202122def new_mon_keyring(args): LOG.debug('Creating a random mon key...') mon_keyring = '[mon.]\nkey = %s\ncaps mon = allow *\n' % generate_auth_key() keypath = '&#123;name&#125;.mon.keyring'.format( name=args.cluster, ) oldmask = os.umask(0o77) LOG.debug('Writing monitor keyring to %s...', keypath) try: tmp = '%s.tmp' % keypath with open(tmp, 'w', 0o600) as f: f.write(mon_keyring) try: os.rename(tmp, keypath) except OSError as e: if e.errno == errno.EEXIST: raise exc.ClusterExistsError(keypath) else: raise finally: os.umask(oldmask) 手工部署集群以ceph-deploy部署集群：ceph-deploy new ceph-231为例，对应的手工操作。 获取ip地址执行以下命令，通过正则表达式获取IP地址192.168.217.231123456789[root@ceph-231 ceph-cluster]# /usr/sbin/ip link show1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:002: eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast master ovs-system state UP mode DEFAULT qlen 1000 link/ether 02:03:e7:fc:dc:36 brd ff:ff:ff:ff:ff:ff3: ovs-system: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN mode DEFAULT link/ether 86:f4:14:e3:1b:b2 brd ff:ff:ff:ff:ff:ff4: xenbr0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT link/ether 02:03:e7:fc:dc:36 brd ff:ff:ff:ff:ff:ff 12345678910111213[root@ceph-231 ceph-cluster]# /usr/sbin/ip addr show1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever2: eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast master ovs-system state UP qlen 1000 link/ether 02:03:e7:fc:dc:36 brd ff:ff:ff:ff:ff:ff3: ovs-system: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN link/ether 86:f4:14:e3:1b:b2 brd ff:ff:ff:ff:ff:ff4: xenbr0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue state UNKNOWN link/ether 02:03:e7:fc:dc:36 brd ff:ff:ff:ff:ff:ff inet 192.168.217.231/24 brd 192.168.217.255 scope global xenbr0 valid_lft forever preferred_lft forever 创建ceph.conf12345678910[root@ceph-231 ceph-cluster]# vi ceph.conf[global]fsid = a3b9b0aa-01ab-4e1b-bba3-6f5317b0795bmon_initial_members = ceph-231mon_host = 192.168.217.231auth_cluster_required = cephxauth_service_required = cephxauth_client_required = cephxpublic_network = 192.168.217.231 创建ceph.mon.keyring可以通过ceph-authtool命令生成 123456[root@ceph-231 ceph-cluster]# ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'creating /tmp/ceph.mon.keyring[root@ceph-231 ~]# cat /tmp/ceph.mon.keyring[mon.] key = AQCzxEhZC7tICxAAuHK5GipD96enMuhv82CCLg== caps mon = "allow *" 将/tmp/ceph.mon.keyring内容复制到ceph.mon.keyring1234[root@ceph-231 ceph-cluster]# vi ceph.mon.keyring[mon.]key = AQCzxEhZC7tICxAAuHK5GipD96enMuhv82CCLg==caps mon = allow *]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[ceph-deploy源码分析（一）——源码结构与cli]]></title>
      <url>%2F2017%2F06%2F15%2Fceph-deploy-cli%2F</url>
      <content type="text"><![CDATA[ceph-deploy是部署Ceph集群的工具，可以通过SSH方式往远程主机上安装 Ceph 软件包、创建集群、增加监视器、收集（或销毁）密钥、增加 OSD 和元数据服务器、配置管理主机，甚至拆除集群。 ceph-deploy使用Python开发，GitHub为 https://github.com/ceph/ceph-deploy。本次源码分析的ceph-deploy版本为1.5.37。 源码结构ceph-deploy-1.5.37源码目录123456789101112├── ceph_deploy # ceph-deploy源码目录├── ceph_deploy.egg-info # egg-info目录├── CONTRIBUTING.rst # 贡献指南├── LICENSE # MIT LICENSE├── MANIFEST.in # 打包规则├── PKG-INFO # PKG-INFO文件├── README.rst # ceph-deploy介绍├── scripts # 启动脚本目录├── setup.cfg # setup.py配置├── setup.py # ceph-deploy安装脚本├── tox.ini # 标准化测试└── vendor.py ceph_deploy源码目录文件12345678910111213141516171819202122232425├── admin.py # 子命令admin模块，将ceph.conf和client.admin key push到远程主机├── calamari.py # 子命令calamari模块，连接calamari master├── cli.py # CLI入口├── cliutil.py # 为装饰器函数增加priority├── conf # ceph.conf与cephdeploy.conf读取与写入相关操作目录 ├── config.py # 子命令config模块，push ceph.conf文件到远程主机；从远程主机 pull ceph.conf文件├── connection.py # 连接本地主机、远程主机├── exc.py # 异常处理Error├── forgetkeys.py # 子命令forgetkeys模块，本地移除authentication keys├── gatherkeys.py # 子命令gatherkeys模块，从mon主机上拉取authentication keys├── hosts # ceph-deploy在不同操作系统（centos、debian、fedora、rhel、suse）的操作├── __init__.py # 初始化版本号信息，当前版本1.5.37├── install.py # 子命令install模块，安装、卸载ceph包，清除数据├── lib # vendor类库├── mds.py # 子命令mds模块，mds管理├── misc.py # 其他工具类，比如：mon host组装tuples├── mon.py # 子命令mon模块，mon管理├── new.py # 子命令new模块，部署集群├── osd.py # 子命令osd模块，osd管理├── pkg.py # 子命令pkg模块，逗号分隔的包安装、卸载├── repo.py # 子命令repo模块，添加 yum repo├── rgw.py # 子命令rgw模块，rgw管理├── tests # 测试文件目录├── util # util目录└── validate.py # 参数校验函数 源码入口script目录下的ceph-deploy文件是ceph-deploy的入口，安装之后是/usr/bin/ceph-deploy。 ceph-deploy的__main__函数调用ceph_deploy.cli的main函数123456......from ceph_deploy.cli import mainif __name__ == '__main__': sys.exit(main()) cli模块cli.py是命令行操作模块。 main函数调用_main函数12345678910111213141516def main(args=None, namespace=None): try: _main(args=args, namespace=namespace) finally: # This block is crucial to avoid having issues with # Python spitting non-sense thread exceptions. We have already # handled what we could, so close stderr and stdout. if not os.environ.get('CEPH_DEPLOY_TEST'): try: sys.stdout.close() except: pass try: sys.stderr.close() except: pass _main函数 设置日志：Console Logger与File Logger添加到root_logger 调用argparse模块，解析cli参数 调用conf目录下的ceph-deploy模块set_overrides函数，从当前目录的cephdeploy.conf或~/.cephdeploy.conf文件获取ceph-deploy-global、ceph-deploy-[subcmd]配置项写入args 调用执行subcmd相应的模块123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869@catches((KeyboardInterrupt, RuntimeError, exc.DeployError,), handle_all=True)def _main(args=None, namespace=None): # Set console logging first with some defaults, to prevent having exceptions # before hitting logging configuration. The defaults can/will get overridden # later. # Console Logger # 命令行控制台日志 sh = logging.StreamHandler() # 不同级别的日志，使用不同的颜色区别：DEBUG蓝色；WARNIN黄色；ERROR红色；INFO白色 sh.setFormatter(log.color_format()) # 设置日志级别为WARNING sh.setLevel(logging.WARNING) # because we're in a module already, __name__ is not the ancestor of # the rest of the package; use the root as the logger for everyone # root_logger日志 root_logger = logging.getLogger() # allow all levels at root_logger, handlers control individual levels # 设置root_logger日志级别为DEBUG root_logger.setLevel(logging.DEBUG) # 将 sh添加到root_logger root_logger.addHandler(sh) # 获取解析cli的argparse，调用argparse模块 parser = get_parser() if len(sys.argv) &lt; 2: parser.print_help() sys.exit() else: # 解析获取sys.argv中的ceph-deploy子命令和参数 args = parser.parse_args(args=args, namespace=namespace) # 设置日志级别 console_loglevel = logging.DEBUG # start at DEBUG for now if args.quiet: console_loglevel = logging.WARNING if args.verbose: console_loglevel = logging.DEBUG # Console Logger sh.setLevel(console_loglevel) # File Logger # 文件日志 fh = logging.FileHandler('ceph-deploy-&#123;cluster&#125;.log'.format(cluster=args.cluster)) fh.setLevel(logging.DEBUG) fh.setFormatter(logging.Formatter(log.FILE_FORMAT)) # 将 fh添加到root_logger root_logger.addHandler(fh) # Reads from the config file and sets values for the global # flags and the given sub-command # the one flag that will never work regardless of the config settings is # logging because we cannot set it before hand since the logging config is # not ready yet. This is the earliest we can do. # 从当前目录的cephdeploy.conf或~/.cephdeploy.conf文件获取ceph-deploy配置覆盖命令行参数 args = ceph_deploy.conf.cephdeploy.set_overrides(args) LOG.info("Invoked (%s): %s" % ( ceph_deploy.__version__, ' '.join(sys.argv)) ) log_flags(args) # args.func为cli中的subcmd子命令，调用相应的模块 return args.func(args) get_parser函数[ceph_deploy.cli]以以下方式配置： 模块名 = 模块包名:执行函数 比如： new = ceph_deploy.new:makenew作为ceph-deploy的子命令，执行ceph-deploy new命令时，执行make函数 其他的模块也类似：mon = ceph_deploy.mon:makeosd = ceph_deploy.osd:makergw = ceph_deploy.rgw:makemds = ceph_deploy.mds:makeconfig = ceph_deploy.config:make 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778def get_parser(): # 调用argparse模块 parser = argparse.ArgumentParser( prog='ceph-deploy', formatter_class=argparse.RawDescriptionHelpFormatter, description='Easy Ceph deployment\n\n%s' % __header__, ) verbosity = parser.add_mutually_exclusive_group(required=False) verbosity.add_argument( '-v', '--verbose', action='store_true', dest='verbose', default=False, help='be more verbose', ) verbosity.add_argument( '-q', '--quiet', action='store_true', dest='quiet', help='be less verbose', ) parser.add_argument( '--version', action='version', version='%s' % ceph_deploy.__version__, help='the current installed version of ceph-deploy', ) parser.add_argument( '--username', help='the username to connect to the remote host', ) parser.add_argument( '--overwrite-conf', action='store_true', help='overwrite an existing conf file on remote host (if present)', ) parser.add_argument( '--cluster', metavar='NAME', help='name of the cluster', type=validate.alphanumeric, ) parser.add_argument( '--ceph-conf', dest='ceph_conf', help='use (or reuse) a given ceph.conf file', ) sub = parser.add_subparsers( title='commands', metavar='COMMAND', help='description', ) sub.required = True # 获取ceph_deploy.cli下的entry_points entry_points = [ (ep.name, ep.load()) for ep in pkg_resources.iter_entry_points('ceph_deploy.cli') ] # 根据priority排序 entry_points.sort( key=lambda name_fn: getattr(name_fn[1], 'priority', 100), ) # 将模块加入到子命令 for (name, fn) in entry_points: p = sub.add_parser( name, description=fn.__doc__, help=fn.__doc__, ) if not os.environ.get('CEPH_DEPLOY_TEST'): p.set_defaults(cd_conf=ceph_deploy.conf.cephdeploy.load()) # flag if the default release is being used p.set_defaults(default_release=False) fn(p) p.required = True parser.set_defaults( cluster='ceph', ) return parser argparse模块cli命令的解析使用了argparse.py模块。argparse是Python标准库中命令行选项、参数和子命令的解析器，其是为替代已经过时的optparse模块，argparse在Python2.7中被引入。 argparse模块请参考https://docs.python.org/2.7/library/argparse.htmlhttp://python.usyiyi.cn/translate/python_278/library/argparse.html set_overrides函数conf目录下的ceph-deploy模块set_overrides函数 调用load()函数 判断ceph-deploy配置文件ceph-deploy-global、ceph-deploy-[subcommand]配置项，调用override_subcommand()函数写入args。12345678910111213141516171819202122def set_overrides(args, _conf=None): """ Read the configuration file and look for ceph-deploy sections to set flags/defaults from the values found. This will alter the ``args`` object that is created by argparse. """ # Get the subcommand name to avoid overwritting values from other # subcommands that are not going to be used subcommand = args.func.__name__ command_section = 'ceph-deploy-%s' % subcommand # 加载ceph-deploy配置 conf = _conf or load() for section_name in conf.sections(): if section_name in ['ceph-deploy-global', command_section]: # ceph-deploy-global、ceph-deploy-[subcommand]配置项写入args override_subcommand( section_name, conf.items(section_name), args ) return args load函数，调用location()函数12345def load(): parser = Conf() # 读取解析ceph-deploy配置文件 parser.read(location()) return parser location函数，调用_locate_or_create()函数123456def location(): """ Find and return the location of the ceph-deploy configuration file. If this file does not exist, create one in a default location. """ return _locate_or_create() _locate_or_create函数，判断当前目录的cephdeploy.conf或~/.cephdeploy.conf文件是否存在。 如果都不存在则调用create_stub函数创建一个~/.cephdeploy.conf文件。这个文件是根据模板创建的，内容为空。 如果存在（提前创建）cephdeploy.conf或~/.cephdeploy.conf文件，可以在文件中配置public_network、cluster_network、overwrite-conf等。123456789101112131415def _locate_or_create(): home_config = path.expanduser('~/.cephdeploy.conf') # With order of importance locations = [ path.join(os.getcwd(), 'cephdeploy.conf'), home_config, ] for location in locations: if path.exists(location): logger.debug('found configuration file at: %s' % location) return location logger.info('could not find configuration file, will create one in $HOME') create_stub(home_config) return home_config]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[CentOS7.2配置NTP服务]]></title>
      <url>%2F2017%2F06%2F12%2Fcentos-ntp-config%2F</url>
      <content type="text"><![CDATA[环境 CentOS Linux release 7.2.1511 (Core) Mini版 NTP服务端设置时区[root@ceph-210 ~]# timedatectl set-timezone Asia/Shanghai[root@ceph-210 ~]# hwclock --systohc 安装NTP12[root@ceph-210 ~]# yum install --downloadonly --downloaddir=/tmp/ntp ntp[root@ceph-210 ~]# yum localinstall -C -y --disablerepo=* /tmp/ntp/*.rpm 修改配置1[root@ceph-210 ~]# vi /etc/ntp.conf 注释或删除1234server 0.centos.pool.ntp.org iburstserver 1.centos.pool.ntp.org iburstserver 2.centos.pool.ntp.org iburstserver 3.centos.pool.ntp.org iburst 增加12server 127.127.1.0 iburst # local clock 使用本机时间作为时间服务的标准fudge 127.127.1.0 stratum 10 #这个值不能太高0-15，太高会报错 启动NTP服务开机启动NTP服务12[root@ceph-210 ~]# systemctl enable ntpdCreated symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service. 启动NTP服务1[root@ceph-210 ~]# systemctl start ntpd 查看NTP服务12345678[root@ceph-210 ~]# systemctl status ntpd● ntpd.service - Network Time Service Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2017-06-12 00:45:05 EDT; 16s ago Process: 2635 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS) Main PID: 2636 (ntpd) CGroup: /system.slice/ntpd.service └─2636 /usr/sbin/ntpd -u ntp:ntp -g 验证NTP服务1234[root@ceph-210 ntp]# ntpq -p remote refid st t when poll reach delay offset jitter==============================================================================*LOCAL(0) .LOCL. 10 l 36 64 1 0.000 0.000 0.000 NTP客户端设置时区[root@ceph-211 ~]# timedatectl set-timezone Asia/Shanghai 安装ntpdate12[root@ceph-211 ~]# yum install --downloadonly --downloaddir=/tmp/ntpdate ntpdate[root@ceph-211 ~]# yum localinstall -C -y --disablerepo=* /tmp/ntpdate/*.rpm 配置crontab每3分钟定时同步NTP服务器192.168.217.210的时间12[root@ceph-211 ~]# crontab -e*/3 * * * * /usr/sbin/ntpdate 192.168.217.210 &gt;&gt; /var/log/ntpdate.log 2&gt;&amp;1 重启crontab1[root@ceph-211 ~]# systemctl restart crond.service]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[CentOS7.2部署Jewel版Ceph-10.2.7]]></title>
      <url>%2F2017%2F06%2F09%2Fceph-deploy-jewel%2F</url>
      <content type="text"><![CDATA[在CentOS7.2上安装部署Jewel版最新Ceph-10.2.7。 环境每台主机 CentOS Linux release 7.2.1511 (Core) Mini版 两个100G的磁盘做OSD 12345678910111213[root@localhost ~]# cat /etc/redhat-releaseCentOS Linux release 7.2.1511 (Core)[root@localhost ~]# lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTsr0 11:0 1 1024M 0 romxvda 202:0 0 10G 0 disk├─xvda1 202:1 0 500M 0 part /boot└─xvda2 202:2 0 9.5G 0 part ├─centos-root 253:0 0 8.5G 0 lvm / └─centos-swap 253:1 0 1G 0 lvm [SWAP]xvdb 202:16 0 100G 0 diskxvdc 202:32 0 100G 0 disk 主机ceph-210作为管理节点，部署ceph-deploy。三台主机配置如下 主机 IP 安装组件 ceph-210 192.168.217.210 ceph-deploy、mon、osd、ntp ceph-211 192.168.217.211 mon、osd、ntpdate ceph-212 192.168.217.212 mon、osd、ntpdate 域名解析修改每台主机的hostname1[root@localhost ~]# hostnamectl set-hostname [hostname] hostname分别为ceph-210、ceph-211、ceph-212 修改/etc/hosts1[root@localhost ~]# vi /etc/hosts 添加192.168.217.210 ceph-210192.168.217.211 ceph-211192.168.217.212 ceph-212 NTP 配置NTP配置请参考 CentOS7.2配置NTP服务 ceph-210作为NTP服务端，ceph-211、ceph-212作为NTP客户端 yum源配置在每台主机上执行。 删除默认的源1234567891011[root@localhost ~]# yum clean all已加载插件：fastestmirrorRepository base is listed more than once in the configurationRepository updates is listed more than once in the configurationRepository extras is listed more than once in the configurationRepository centosplus is listed more than once in the configuration正在清理软件源： Ceph Ceph-noarch base extras updatesCleaning up everythingCleaning up list of fastest mirrors[root@localhost ~]# rm -rf /etc/yum.repos.d/*.repo 下载阿里云的base源1234567891011[root@localhost ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo--2017-06-07 14:26:39-- http://mirrors.aliyun.com/repo/Centos-7.repo正在解析主机 mirrors.aliyun.com (mirrors.aliyun.com)... 112.124.140.210, 115.28.122.210正在连接 mirrors.aliyun.com (mirrors.aliyun.com)|112.124.140.210|:80... 已连接。已发出 HTTP 请求，正在等待回应... 200 OK长度：2573 (2.5K) [application/octet-stream]正在保存至: “/etc/yum.repos.d/CentOS-Base.repo”100%[===================================================================================================================================&gt;] 2,573 --.-K/s 用时 0s2017-06-07 14:26:39 (28.0 MB/s) - 已保存 “/etc/yum.repos.d/CentOS-Base.repo” [2573/2573]) 下载阿里云的epel源1234567891011[root@localhost ~]# wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo--2017-06-07 14:27:08-- http://mirrors.aliyun.com/repo/epel-7.repo正在解析主机 mirrors.aliyun.com (mirrors.aliyun.com)... 115.28.122.210, 112.124.140.210正在连接 mirrors.aliyun.com (mirrors.aliyun.com)|115.28.122.210|:80... 已连接。已发出 HTTP 请求，正在等待回应... 200 OK长度：1084 (1.1K) [application/octet-stream]正在保存至: “/etc/yum.repos.d/epel.repo”100%[===================================================================================================================================&gt;] 1,084 --.-K/s 用时 0s2017-06-07 14:27:08 (121 MB/s) - 已保存 “/etc/yum.repos.d/epel.repo” [1084/1084]) 修改里面的系统版本为7.3.1611，当前用的CentOS7.2.1511版本的yum源已经清空了123[root@localhost ~]# sed -i '/aliyuncs/d' /etc/yum.repos.d/CentOS-Base.repo[root@localhost ~]# sed -i '/aliyuncs/d' /etc/yum.repos.d/epel.repo[root@localhost ~]# sed -i 's/$releasever/7.3.1611/g' /etc/yum.repos.d/CentOS-Base.repo 添加网易的ceph源（阿里云的ceph源没有最新的10.2.7版本）1[root@localhost ~]# vi /etc/yum.repos.d/ceph.repo 添加[ceph]name=cephbaseurl=http://mirrors.163.com/ceph/rpm-jewel/el7/x86_64/gpgcheck=0[ceph-noarch]name=cephnoarchbaseurl=http://mirrors.163.com/ceph/rpm-jewel/el7/noarch/gpgcheck=0 进行yum的makecache1[root@localhost ~]# yum makecache 安装ceph下载ceph的相关rpm到本地1[root@localhost ~]# yum install --downloadonly --downloaddir=/tmp/ceph ceph 在每台主机上安装ceph1[root@localhost ~]# yum localinstall -C -y --disablerepo=* /tmp/ceph/*.rpm 安装成功，查看ceph版本12[root@localhost yum.repos.d]# ceph -vceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185) 关闭selinux关闭每台主机的selinux12[root@localhost ~]# sed -i 's/SELINUX=.*/SELINUX=disabled/' /etc/selinux/config[root@localhost ~]# setenforce 0 部署ceph在管理节点ceph-210上执行。 配置无密码访问生成本机密钥1[root@ceph-210 ~]# ssh-keygen -t rsa 将本机密钥copy到ceph-211、ceph-21212[root@ceph-210 ~]# ssh-copy-id root@ceph-211[root@ceph-210 ~]# ssh-copy-id root@ceph-212 验证配置是否成功12[root@ceph-210 ~]# ssh root@ceph-211[root@ceph-210 ~]# ssh root@ceph-212 安装ceph-deploy下载ceph-deploy1[root@ceph-210 ~]#yum install --downloadonly --downloaddir=/tmp/ceph-deploy/ ceph-deploy 安装ceph-deploy1[root@ceph-210 ~]#yum localinstall -C -y --disablerepo=* /tmp/ceph-deploy/*.rpm 安装成功，查看ceph-deploy版本12[root@ceph-210 yum.repos.d]# ceph-deploy --version1.5.37 部署集群创建部署目录，部署集群123[root@ceph-210 ~]# mkdir ceph-cluster[root@ceph-210 ~]# cd ceph-cluster[root@ceph-210 ceph-cluster]# ceph-deploy new ceph-210 ceph-211 ceph-212 修改ceph.conf文件，添加 public_network=192.168.217.0/241[root@ceph-210 ~]# vi ceph.conf 部署mon1[root@ceph-210 ceph-cluster]# ceph-deploy mon create-initial 创建osd1[root@ceph-210 ceph-cluster]# ceph-deploy --overwrite-conf osd prepare ceph-210:/dev/xvdb ceph-210:/dev/xvdc ceph-211:/dev/xvdb ceph-211:/dev/xvdc ceph-212:/dev/xvdb ceph-212:/dev/xvdc --zap-disk 激活osd1[root@ceph-210 ceph-cluster]# ceph-deploy --overwrite-conf osd activate ceph-210:/dev/xvdb1 ceph-210:/dev/xvdc1 ceph-211:/dev/xvdb1 ceph-211:/dev/xvdc1 ceph-212:/dev/xvdb1 ceph-212:/dev/xvdc1 查看集群12345678910[root@ceph-210 ceph-cluster]# ceph -s cluster b1945f0d-38df-4f17-83df-42b4749d2bed health HEALTH_OK monmap e1: 3 mons at &#123;ceph-210=192.168.217.210:6789/0,ceph-211=192.168.217.211:6789/0,ceph-212=192.168.217.212:6789/0&#125; election epoch 10, quorum 0,1,2 ceph-210,ceph-211,ceph-212 osdmap e51: 6 osds: 6 up, 6 in flags sortbitwise,require_jewel_osds pgmap v214: 64 pgs, 1 pools, 0 bytes data, 0 objects 204 MB used, 569 GB / 569 GB avail 64 active+clean 参考 http://www.xuxiaopang.com/2016/10/09/ceph-quick-install-el7-jewel/ http://linux.it.net.cn/CentOS/course/2016/1213/25330.html]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[Error: libselinux conflicts with systemd-219-19.el7.x86_64]]></title>
      <url>%2F2017%2F06%2F09%2Fceph-libselinux-conflicts%2F</url>
      <content type="text"><![CDATA[在CentOS7.2上安装Jewel版ceph-10.2.7时，出现错误信息：libselinux-2.5-6.el7.x86_64 conflicts systemd &lt; 219-20。由于CentOS7.2上的systemd版本为219-19，刚好与libselinux-2.5-6.el7.x86_64冲突。 在CentOS7.2上安装Ceph-10.2.7时具体错误信息12345--&gt; Processing Conflict: libselinux-2.5-6.el7.x86_64 conflicts systemd &lt; 219-20--&gt; Finished Dependency ResolutionError: libselinux conflicts with systemd-219-19.el7.x86_64 You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest 环境 CentOS Linux release 7.2.1511 (Core) ceph-10.2.7 解决方法 方法一：修改libselinux-2.5-6.el7.x86_64依赖，重新打包 方法二：升级systemd，将systemd.x86_64 0:219-19.el7更新到systemd.x86_64 0:219-30.el7_3.9 方法三：修改ceph-10.2.7依赖，去掉selinux，重新打包 本文使用方法一来解决此问题。 编译libselinuxrpm编译打包可以参考 ceph-deploy源码制作rpm包 使用root用户打rpm包12[root@localhost ~]# cd /root/[root@localhost ~]# vi .rpmmacrosvi 添加以下内容%_topdir /root/rpmbuild/ 创建编译目录123456789[root@localhost ~]# mkdir rpmbuild[root@localhost ~]# cd rpmbuild[root@localhost rpmbuild]# mkdir -pv &#123;BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS&#125;mkdir: created directory ‘BUILD’mkdir: created directory ‘BUILDROOT’mkdir: created directory ‘RPMS’mkdir: created directory ‘SOURCES’mkdir: created directory ‘SPECS’mkdir: created directory ‘SRPMS’ 修改libselinux-2.5-6.el7.src目录下的libselinux.spec文件将1Conflicts: filesystem &lt; 3 systemd &lt; 219-20 修改成1Conflicts: filesystem &lt; 3 systemd &lt; 219-19 将libselinux-2.5-6.el7.src目录下的libselinux.spec复制到/root/rpmbuild/SPECS目录，其他文件复制到/root/rpmbuild/SOURCES目录12345[root@localhost ~]# cp /opt/libselinux.spec /root/rpmbuild/SPECS[root@localhost ~]# cp /opt/libselinux-2.5.tar.gz /root/rpmbuild/SOURCES[root@localhost ~]# cp /opt/libselinux-rhel.patch /root/rpmbuild/SOURCES[root@localhost ~]# cp /opt/selinuxconlist.8 /root/rpmbuild/SOURCES[root@localhost ~]# cp /opt/selinuxdefcon.8 /root/rpmbuild/SOURCES 编译打包12[root@localhost ~]# cd /root/rpmbuild/SPECS[root@localhost SPECS]# rpmbuild -ba libselinux.spec 将/root/rpmbuild/RPMS/x86_64/生成二进制包copy出来libselinux-2.5-6.el7.centos.x86_64.rpmlibselinux-utils-2.5-6.el7.centos.x86_64.rpm]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[ceph-deploy源码制作rpm包]]></title>
      <url>%2F2017%2F06%2F09%2Fceph-deploy-rpmbuild%2F</url>
      <content type="text"><![CDATA[rpm编译打包的步骤： 修改代码中的setup.cfg文件中的版本号 代码setup.cfg同级目录下增加PKG-INFO文件，并修改版本号与setup.cfg中保持一致（PKG-INFO文件可以从项目的.egg-info文件夹获取，或者从centsos的rpm源码参考） 将代码加上版本号重名并打成.tar.gz包 将.tar.gz文件、spec脚本项目下的文件（.spec文件除外）copy到用户的rpmbuild/SOURCES目录 将.spec文件copy到用户的rpmbuild/SPECS 执行 rpmbuild –ba [spec文件] 来打包二进制包和源码包 环境准备 CentOS Linux release 7.2.1511 (Core) ceph-deploy-1.5.36-0.src.rpm 创建用户为了安全单独创建一个用户ceph来制作rpm1[root@node ~]# useradd ceph 设置用户ceph的密码123456[root@node ~]# passwd cephChanging password for user ceph.New password:BAD PASSWORD: The password fails the dictionary check - it is too simplistic/systematicRetype new password:passwd: all authentication tokens updated successfully. 创建rpmbuild目录切换帐号1[root@node ~]# su ceph 添加一个配置文件，并设置编译的根目录为 ~/rpmbuild1[root@node root]$ vi ~/.rpmmacros 添加内容%_topdir /home/ceph/rpmbuild/:wq保存退出 在编译根目录下创建我们需要的目录12345678910[root@node root]$ cd ~[root@node ~]$ mkdir rpmbuild[root@node ~]$ cd ~/rpmbuild[root@node rpmbuild]$ mkdir -pv &#123;BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS&#125;mkdir: created directory ‘BUILD’mkdir: created directory ‘BUILDROOT’mkdir: created directory ‘RPMS’mkdir: created directory ‘SOURCES’mkdir: created directory ‘SPECS’mkdir: created directory ‘SRPMS’ rpmbuild目录下的六个文件目录分别为 BUILD 源代码解压后的存放目录 BUILDROOT 编译临时目录 RPMS 制作完成后的RPM包存放目录，里面有与平台相关的子目录 SOURCES 收集的源材料，补丁的存放位置 SPECS SPEC文件存放目录 SRMPS 存放SRMPS生成的目录 编译rpm包将修改后的代码目录ceph-deploy-1.5.36打成.tar.bz2文件12[root@node ~]# cd /opt[root@node ~]# tar jcf ceph-deploy-1.5.36.tar.bz2 ceph-deploy-1.5.36 将ceph-deploy-1.5.36.tar.bz2复制到/home/ceph/rpmbuild/SOURCES目录1[root@node ~]# cp /opt/ceph-deploy-1.5.36/ceph-deploy-1.5.36.tar.bz2 /home/ceph/rpmbuild/SOURCES 将ceph-deploy.spec复制到/home/ceph/rpmbuild/SPECS目录1[root@node ~]# cp /opt/ceph-deploy-1.5.36/ceph-deploy.spec /home/ceph/rpmbuild/SPECS 编译打包123[root@node ~]# su ceph[root@node root]$ cd /home/ceph/rpmbuild/SPECS[root@node SPECS]$ rpmbuild -ba ceph-deploy.spec 执行完成之后， 在/home/ceph/rpmbuild/SRPMS/ 生成源码包 在/home/ceph/rpmbuild/RPMS/noarch/ 生成二进制包ceph-deploy-1.5.36-0.noarch.rpm]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[加入Ceph订阅号]]></title>
      <url>%2F2017%2F06%2F09%2Fceph-add-subscribe%2F</url>
      <content type="text"><![CDATA[在Ceph社区新的官网 http://ceph.com/irc/ 的MAILING LISTS部分，加入Ceph订阅号，Email订阅Ceph社区信息。 开发者订阅号 在chep-devel那一行，点击“SUBSCRIBE”，产生一个订阅号申请地址：majordomo@vger.kernel.org 发送一封邮件申请一个校验码，标题为空，内容格式如下： subscribe ceph-devel xxxx@gmail.com其中xxxx@gmail.com是自己的gmail邮箱地址。 收到一封确认邮件，提取里面的校验码。例如：989ada92 再发一封邮件到 majordomo@vger.kernel.org，标题为空，内容格式如下：auth 989ada92 subscribe ceph-devel xxxx@gmail.com其中xxxx@gmail.com是自己的gmail邮箱地址。 收到一封邮件“Welcome to ceph-devel”，欢迎加入ceph开发者订阅号，订阅成功。 其他订阅号ceph-usersceph-commitceph-qaceph-communityceph-cn (chinese language) 在相应订阅号的那一行点击“SUBSCRIBE”按钮后，会在邮箱客户端里产生一个订阅号申请地址，然后发送一个申请邮件到这个地址上。 比如：ceph-usrs订阅的邮件标题可以为please let me join ceph-usrs subscribe，内容也可以为please let me join ceph-usrs subscribe。ceph-cn订阅的邮件标题可以为please let me join ceph-cn subscribe，内容也可以为please let me join ceph-cn subscribe。]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[fio安装使用]]></title>
      <url>%2F2017%2F05%2F22%2Ffio-install-use%2F</url>
      <content type="text"><![CDATA[fio（Flexible I/O Tester）是个非常强大的IO性能测试工具 官网：http://freecode.com/projects/fioGitHub：https://github.com/axboe/fio 安装fio环境CentOS7.2 yum安装fio1[root@node001 ~]# yum install fio 源码安装可以在 http://freecode.com/projects/fio 先下载最新的fio1234[root@node001 ~]# tar -xzvf fio-2.1.10.tar.gz[root@node001 ~]# cd fio-2.1.10/[root@node001 ~]# make[root@node001 ~]# make install 安装libaio引擎1[root@node001 ~]# yum install libaio-devel 查看fio版本 12[root@node001 ~]# fio --versionfio-2.1.10 查看fio支持的engine，最新的版本已支持rbd 12345678910111213141516171819202122[root@node001 ~]# fio --enghelpAvailable IO engines: cpuio mmap sync psync vsync pvsync null net netsplice libaio rdma posixaio falloc e4defrag splice rbd gfapi gfapi_async sg binject fio测试参数 参数名 描述 ioengine 负载引擎，我们一般使用libaio，发起异步IO请求 bs 单次IO的块文件大小，比如4k、8k、4M等 direct 直接写，绕过操作系统Cache。因为我们测试的是硬盘，而不是操作系统的Cache，所以设置为1 rw 读写模式：顺序写write、顺序读read、顺序读写rw、随机写randwrite、随机读randread、随机读写randrw size 寻址空间，IO会落在 [0, size)这个区间的硬盘空间上。一般不设置，默认为测试对象的大小 filename 测试对象（文件名称或者硬盘盘符） iodepth 队列深度，只有使用libaio时才有意义。这是一个可以影响IOPS的参数 numjobs 测试线程数，默认为1 runtime 测试时长，单位为妙 time_based 即使完全写入或者读取，也可以运行runtime指定的时间 rwmixread 在混合读写的模式下，读占用的比例，默认为50 rwmixwrite 在混合读写的模式下，写占用比例，默认为50 group_reporting 如果设置了多任务参数numjobs，用每组报告代替每job报告 thread 使用pthread_create(3)创建的线程，而不是使用fork(2)创建的进程 randrepeat 设置产生的随机数是否可重复的，默认为true stonewall 用在job文件中，启动当前job之前等待job文件中上一个job退出 具体的参数可参考文档 https://linux.die.net/man/1/fio libaio引擎，需要设置-direct=1 块大小必须是扇区的倍数 4K随机读写1fio -filename=/dev/nvme0n1 -iodepth=128 -numjobs=4 -thread -rw=randrw -rwmixread=70 -bs=4k -ioengine=libaio -group_reporting -name=mytest -randrepeat=0 -time_based -runtime=120 -direct=1 4K随机读1fio -filename=/dev/nvme0n1 -iodepth=128 -numjobs=4 -thread -rw=randread -bs=4k -ioengine=libaio -group_reporting -name=mytest -randrepeat=0 -time_based -runtime=120 -direct=1 4K随机写1fio -filename=/dev/nvme0n1 -iodepth=128 -numjobs=4 -thread -rw=randwrite -bs=4k -ioengine=libaio -group_reporting -name=mytest -randrepeat=0 -time_based -runtime=120 -direct=1 Job文件fio可以通过一个job文件来描述IO负载。一个job文件可以控制产生任意数目的线程和文件。典型的job文件有一个global段（定义共享参数），一个或多少job段（描述具体要产生的job）。 运行Job文件时，fio从文件读这些参数，做处理，并根据这些参数描述，启动这些线程/进程。 Job运行方式1fio [jobfile] Job文件格式Job文件格式采用经典的ini文件，[]中的值表示job name，可以采用任意的ASCII字符，‘global’除外，global有特殊的意义。Global section描述了job file中各个job的默认配置值。一个job section可以覆盖global section中的参数，一个job file可以包含几个global section.一个job只会受到它上面的global section的影响。‘;’和‘#’可以用作注释。 例如：Job文件myjob.fio 123456789101112131415161718192021222324[global] #全局参数ioengine=libaioiodepth=128time_baseddirect=1thread=1group_reportingrandrepeat=0norandommapnumjobs=32timeout=60runtime=120[randread-4k] #Job名称，4K随机读rw=randreadbs=4kfilename=/dev/nvme0n1stonewall[randwrite-4k] #Job名称，4K随机写rw=randwritebs=4kfilename=/dev/nvme0n1stonewall 执行Job文件1fio myjob.fio 相当于执行了两条fio命令 4K随机读1fio -filename=/dev/nvme0n1 -iodepth=128 -numjobs=32 -thread -rw=randread -bs=4k -ioengine=libaio -group_reporting -name=randread-4k -randrepeat=0 -time_based -runtime=120 -timeout=60 -direct=1 4K随机写1fio -filename=/dev/nvme0n1 -iodepth=128 -numjobs=32 -thread -rw=randwrite -bs=4k -ioengine=libaio -group_reporting -name=randwrite-4k -randrepeat=0 -time_based -runtime=120 -timeout=60 -direct=1 使用gnuplot画图安装gnuplot安装画图插件gnuplot1[root@node001 ~]# yum install gnuplot 设置fio输出日志fio的输出日志主要包含三种：bw，lat和iops，在Job文件中[randread-4k]下设置这三种参数如下： write_bw_log=randread-4k write_lat_log=randread-4k write_iops_log=randread-4k randread-4k是输出日志文件名的前缀，生成的日志文件如下： randread-4k_bw.log randread-4k_clat.log randread-4k_iops.log randread-4k_lat.log randread-4k_slat.log 画图执行fio_generate_plots命令画图，randread-4k是Job中配置的write_bw_log、write_lat_log、write_iops_log的值，也即日志文件名的前缀1fio_generate_plots randread-4k 生成的图片文件如下： randread-4k-bw.svg randread-4k-clat.svg randread-4k-iops.svg randread-4k-lat.svg randread-4k-slat.svg 硬盘性能指标 指标 IO读写 描述 适用场景 吞吐量(MB/s） 1M顺序读写 文件在硬盘上存储位置是连续的 大文件拷贝（比如视频音乐） IOPS 4K随机读写 在硬盘上随机位置读写数据，每次4KB 操作系统运行、软件运行、数据库]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[V2V转换器（九）——转换后的虚拟机挂载ISO失败]]></title>
      <url>%2F2017%2F04%2F28%2Fv2v-9%2F</url>
      <content type="text"><![CDATA[将 ISO“rhel-server-6.7-x86_64-dvd.iso”加载到 VM rhel 上的 DVD 驱动器中内部错误: Object with type VM and id 9ef237f8-bd9c-6604-1581-949a0a4643e0/vbd.xvdd does not exist in xenopsd 分析vbd的device xvdd在底层不存在。 查看虚拟机的VBD123456789101112131415[root@171 ~]# xe vbd-list vm-uuid=9ef237f8-bd9c-6604-1581-949a0a4643e0uuid ( RO) : 0a766793-c791-34c9-04ba-425ee5eb015c vm-uuid ( RO): 9ef237f8-bd9c-6604-1581-949a0a4643e0 vm-name-label ( RO): rhel vdi-uuid ( RO): 7c350add-6846-4d83-a865-d0c9b5c38f3d empty ( RO): false device ( RO): xvdauuid ( RO) : 8b98a12c-fb3a-0bb2-ed27-c13dcf757bcb vm-uuid ( RO): 9ef237f8-bd9c-6604-1581-949a0a4643e0 vm-name-label ( RO): rhel vdi-uuid ( RO): &lt;not in database&gt; empty ( RO): true device ( RO): UUID为8b98a12c-fb3a-0bb2-ed27-c13dcf757bcb的VBD的vdi-uuid、device都为空,类型为CD1234567891011121314151617181920212223242526[root@171 ~]# xe vbd-param-list uuid=8b98a12c-fb3a-0bb2-ed27-c13dcf757bcbuuid ( RO) : 8b98a12c-fb3a-0bb2-ed27-c13dcf757bcb vm-uuid ( RO): 9ef237f8-bd9c-6604-1581-949a0a4643e0 vm-name-label ( RO): rhel vdi-uuid ( RO): &lt;not in database&gt; vdi-name-label ( RO): &lt;EMPTY&gt; allowed-operations (SRO): attach; insert current-operations (SRO): empty ( RO): true device ( RO): userdevice ( RW): 1 bootable ( RW): false mode ( RW): RO type ( RW): CD unpluggable ( RW): true currently-attached ( RO): false attachable ( RO): true storage-lock ( RO): false status-code ( RO): 0 status-detail ( RO): qos_algorithm_type ( RW): qos_algorithm_params (MRW): qos_supported_algorithms (SRO): other-config (MRW): io_read_kbs ( RO): &lt;unknown&gt; io_write_kbs ( RO): &lt;unknown&gt; 解决方法销毁掉CD类型的VBD，再创建新的CD类型的VBD，重新挂载ISO。 销毁VBD销毁UUID为8b98a12c-fb3a-0bb2-ed27-c13dcf757bcb的VBD1[root@171 ~]# xe vbd-destroy uuid=8b98a12c-fb3a-0bb2-ed27-c13dcf757bcb 创建新的VBD重新创建新的VBD，生成新的VBD UUID（96f9399e-328d-5121-b51e-6d9202f9f742）。其中device为未使用的设备，比如xvdd。12[root@171 ~]# xe vbd-create vm-uuid=9ef237f8-bd9c-6604-1581-949a0a4643e0 type=CD device=xvdd mode=RO96f9399e-328d-5121-b51e-6d9202f9f742 挂载ISO在新创建的VBD上挂载ISO1[root@171 ~]# xe vbd-insert uuid=96f9399e-328d-5121-b51e-6d9202f9f742 vdi-uuid=427d8121-51b0-4439-9237-e9f2b3fe35e0]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[VSM源码分析（四）——创建副本、纠删存储池]]></title>
      <url>%2F2017%2F04%2F07%2Fvsm-create-pool%2F</url>
      <content type="text"><![CDATA[存储池类型，可以是副本存储池或纠删存储池。 副本存储池，保存多份对象副本，以便丢失的数据可以恢复 纠删存储池，获得类似 RAID5 的功能 目前最流行的数据保护方式是副本。相比副本方式，纠删码只需提供更少的空间，但是以牺牲性能为代价的，因为纠删码是将每个object划分为更小的数据块，少量较新的编码块混合到数据中，存储数据会稍慢于副本机制。 下面分析创建副本、纠删类型的存储池的源码。 vsm-dashboard创建副本存储池vsm-dashboard\vsm_dashboard\dashboards\vsm\poolsmanagement\urls.py文件中创建副本存储池的url1url(r'^create_replicated_pool_action/$', create_replicated_pool, name='create_replicated_pool_action'), views处理创建副本存储池请求，调用vsmapi的create_replicated_pool方法1234567891011121314151617def create_replicated_pool(request): status = "" msg = "" body = json.loads(request.body) print body try: rsp, ret = vsmapi.create_storage_pool(request,body=body) res = str(ret['message']).strip() if res.startswith('pool') and res.endswith('created'): status = "OK" msg = "Created storage pool successfully!" except: status = "Failed" msg = "Create Replication Pool Failed!" resp = dict(message=msg, status=status) resp = json.dumps(resp) return HttpResponse(resp) 创建纠删存储池vsm-dashboard\vsm_dashboard\dashboards\vsm\poolsmanagement\urls.py文件中创建纠删存储池的url1url(r'^create_ec_pool_action/$', create_ec_pool, name='create_ec_pool_action'), views处理创建纠删存储池请求，调用vsmapi的create_ec_pool方法12345678910111213141516171819def create_ec_pool(request): status = "" msg = "" body = json.loads(request.body) print body try: rsp, ret = vsmapi.create_storage_pool(request,body=body) res = str(ret['message']).strip() if res.startswith('pool') and res.endswith('created'): status = "OK" msg = "Created storage pool successfully!" except: status = "Failed" msg = "Remove Cache Tier Failed!" resp = dict(message=msg, status=status) resp = json.dumps(resp) return HttpResponse(resp) 创建副本存储池/纠删存储池，都是调用api/vsm.py的create_storage_pool方法 调用vsmclient的create_storage_pool方法12def create_storage_pool(request, body): return vsmclient(request).vsms.create_storage_pool(body) vsmclientvsmclient/v1/vsm.py的create_storage_pool方法123456def create_storage_pool(self, body): """ create a storage pool """ url = '/storage_pool/create' return self.api.client.post(url, body=body) vsmvsm/api/v1/storage_pool.py的create方法 参数校验 db通过name查询storage_group db通过name获取storage_group的sum(choose_num) 如果sum(choose_num)为0，则查询vsm_settings表的osd_pool_default_size配置值作为size，默认是1 判断是否ec_pool,如果是需要重新组装参数 replicated pool校验rule_id replicated pool组装参数 更新参数 调用scheduler api 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120@wsgi.serializers(xml=StoragePoolsTemplate) def create(self, req, body=None): """Create a storage pool.""" LOG.info(body) #&#123;'pool': # &#123;'replicationFactor': 3, # 'name': 'test', # 'enablePoolQuota': False, # 'storageGroupId': '1', # 'u'replicatedStorageGroupId': '1', # 'clusterId': '0', # 'tag': 'abc', # 'createdBy': 'VSM', # 'ecProfileId': '1', # 'ecFailureDomain': 'osd', # 'poolQuota': 0 # &#125; #&#125; if not self.is_valid_body(body, 'pool'): raise exc.HTTPUnprocessableEntity() context = req.environ['vsm.context'] pool_dict = body['pool'] for key in ('name', 'createdBy', 'storageGroupName'): if not key in pool_dict: msg = _("%s is not defined in pool" % key) raise exc.HTTPBadRequest(explanation=msg) name = pool_dict['name'].strip() created_by = pool_dict['createdBy'].strip() storage_group_name = pool_dict['storageGroupName'] tag = pool_dict['tag'].strip() cluster_id = pool_dict['clusterId'] try: cluster_id = int(str(cluster_id)) except ValueError: msg = _('cluster_id must be an interger value') raise exc.HTTPBadRequest(explanation=msg) storage_group = db.storage_group_get_by_name(context, storage_group_name) rule_id = storage_group['rule_id'] storage_group_id = storage_group['id'] size = db.get_size_by_storage_group_name(context,storage_group_name) size = int(size) if size == 0: pool_default_size = db.vsm_settings_get_by_name(context,'osd_pool_default_size') size = int(pool_default_size.value) #LOG.info('size=====%s'%size) #osd_num = 2 #TODO self.scheduler_api.get_osd_num_from_crushmap_by_rule(context, rule_id) is_ec_pool = pool_dict.get('ecProfileId') if is_ec_pool: #erasure code pool body_info = &#123;'name': name, 'cluster_id':cluster_id, 'storage_group_id':storage_group_id, 'storage_group_name':storage_group_name, 'ec_profile_id':pool_dict['ecProfileId'], 'ec_ruleset_root':storage_group['name'], 'ec_failure_domain':pool_dict['ecFailureDomain'], 'created_by':created_by, 'tag':tag&#125; else: #replicated pool crush_ruleset = rule_id#self.conductor_api.get_ruleset_id(context, storage_group_id) if crush_ruleset &lt; 0: msg = _('crush_ruleset must be a non-negative integer value') raise exc.HTTPBadRequest(explanation=msg) #size = pool_dict['replicationFactor'] #replica_storage_group_id = pool_dict['replicatedStorageGroupId'] #try: # size = int(str(size)) # if size &lt; 1: # msg = _('size must be &gt; 1') # raise exc.HTTPBadRequest(explanation=msg) # # host_num = self.conductor_api.count_hosts_by_storage_group_id(context, storage_group_id) # LOG.info("storage_group_id:%s,host_num:%s", storage_group_id, host_num) # if size &gt; host_num: # msg = "The replication factor must be less than or equal to the number of storage nodes in the specific storage group in cluster!" # return &#123;'message': msg&#125; # except ValueError: # msg = _('size must be an interger value') # raise exc.HTTPBadRequest(explanation=msg) #pg_num = self._compute_pg_num(context, osd_num, size) #vsm_id = str(uuid.uuid1()).split('-')[0] pg_num = 64 auto_growth_pg = pool_dict.get("auto_growth_pg",0) if auto_growth_pg and int(auto_growth_pg) &lt; pg_num and int(auto_growth_pg) &gt; 0: pg_num = int(auto_growth_pg) #self._compute_pg_num(context, osd_num, size) body_info = &#123;'name': name, #+ "-vsm" + vsm_id, 'cluster_id':cluster_id, 'storage_group_id':storage_group_id, 'storage_group_name':storage_group_name, 'pool_type':'replicated', 'crush_ruleset':crush_ruleset, 'pg_num':pg_num, 'pgp_num':pg_num, 'size':size, 'min_size':size, 'created_by':created_by, 'tag':tag&#125; body_info.update(&#123; "quota": pool_dict.get("poolQuota"), "enable_quota": pool_dict.get("enablePoolQuota"), "max_pg_num_per_osd": pool_dict.get("max_pg_num_per_osd") or 100, "auto_growth_pg": pool_dict.get("auto_growth_pg") or 0, &#125;) #LOG.info('body_info=====%s'%body_info) return self.scheduler_api.create_storage_pool(context, body_info) vsm/scheduler/api.py的create_storage_pool方法12def create_storage_pool(self, context, body=None): return self.scheduler_rpcapi.create_storage_pool(context, body) vsm/scheduler/rpcapi.py的create_storage_pool方法12def create_storage_pool(self, ctxt, body=None): return self.call(ctxt, self.make_msg('create_storage_pool', body=body)) vsm/scheduler/manager.py的create_storage_pool方法 调用_get_active_monitor方法获取monitor server db通过name查询storage_pool db获取zone的数量，比较replication_factor 如果storage_pool不存在，调用agent rpcapi创建pool 如果storage_pool存在，抛出异常12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152def create_storage_pool(self, context, body=None,cluster_id = None): LOG.info('scheduler/manager.py create_storage_pool') active_monitor = self._get_active_monitor(context, cluster_id=cluster_id) LOG.info('sync call to host = %s' % active_monitor['host']) body['cluster_id'] = active_monitor['cluster_id'] pool_ref = db.pool_get_by_name(context, body['name'], body['cluster_id']) if body.has_key('size'): replication_factor = body['size'] zone_cnt = len(db.zone_get_all(context)) factor_flag = True message = '' if zone_cnt &gt; 1 and replication_factor &gt; zone_cnt: factor_flag = False message = "4004 - replication_factor of %s error:replica count greater than number of zones in Storage Group-loss of zone may result in data loss" % body['name'] LOG.error(message) if zone_cnt &lt;= 1: host_cnt = db.init_node_count_by_status(context,status='Active') if replication_factor &gt; host_cnt: factor_flag = False message = "4005 - replication_factor of %s error:replica count greater than number of host in Storage Group-loss of host may resultin data loss" % body['name'] LOG.error(message) if not factor_flag: res = "pool "+ body['name'] + ":%s"%message return &#123;'message':res&#125; if not pool_ref: self._agent_rpcapi.create_storage_pool(context, body)#TODO need to specify host? #pool_id = self._agent_rpcapi.get_pool_id_by_name(context, # body['name'], active_monitor['host']) #body['pool_id'] = pool_id #body['status'] = 'running' #pool_ref = db.pool_create(context, body) ## If it's created by periodic tasks. ## Update the new ones. #if pool_ref['tag'] == 'SYSTEM': # db.pool_update_by_name(context, # body['name'], # body['cluster_id'], # body) res = "pool "+ body['name'] + "created" else: LOG.error("4003 - Pool %s has alreadey existed." % body['name']) try: raise StorageGroupAddFailed except Exception, e: LOG.error("%s: %s" %(e.code, e.message)) res = "pool "+ body['name'] + "'s name repeated" return &#123;'message':res&#125; vsm/agent/rpcapi.py的create_storage_pool方法，调用rpc call同步方法12def create_storage_pool(self, ctxt, body=None): return self.call(ctxt, self.make_msg('create_storage_pool', body=body)) vsm/agent/manager.py的create_storage_pool方法123def create_storage_pool(self, context, body=None): res = self.ceph_driver.create_storage_pool(context, body) return res vsm/agent/driver.py的create_storage_pool方法 如果是ec pool： db获取ec_profiles 创建新的erasure code profile: ceph osd erasure-code-profile set {name} [{directory=directory}] [{plugin=plugin}] [{ruleset-root=ec_ruleset_root}] [{ruleset-failure-domain=ec_failure_domain}] [{key=value} …] [–force] rule 创建erasure： ceph osd crush rule create-erasure {} 创建ec pool： ceph osd pool create {pool-name} {pg-num} {pgp-num} erasure [erasure-code-profile] [crush-ruleset-name] [expected_num_objects] 如果是replicated pool： 获取CRUSH map： ceph osd getcrushmap -o {compiled-crushmap-filename} 反编译CRUSH map： crushtool -d {compiled-crushmap-filename} -o {decompiled-crushmap-filename} 授权: chown -R vsm:vsm /etc/vsm/ 创建replicated pool： ceph osd pool create {pool-name} {pg-num} [{pgp-num}] [replicated] [crush-ruleset-name] [expected-num-objects] 设置replicated pool的rule、size： ceph osd pool set {pool-name} {key} {value} 其他pool： 创建pool： ceph osd pool create {pool-name} {pg-num} 设置pool的rule、size： ceph osd pool set {pool-name} {key} {value} 如果pool有enable_quota参数： 设置max_bytes：ceph osd pool set-quota {pool-name} [max_bytes {bytes}] 调用get_pool_status方法，执行ceph osd dump命令获取pool 比较name，将新创建的pool保存到db 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293def create_storage_pool(self, context, body): pool_name = body["name"] primary_storage_group = '' if body.get("ec_profile_id"): profile_ref = db.ec_profile_get(context, body['ec_profile_id']) pgp_num = pg_num = profile_ref['pg_num'] plugin = "plugin=" + profile_ref['plugin'] ruleset_root = "ruleset-root=" + body['ec_ruleset_root'] failure_domain = "ruleset-failure-domain=" + body['ec_failure_domain'] rule_name = pool_name kv = eval(profile_ref['plugin_kv_pair']) pair_str = "" for k, v in kv.items(): pair_str += str(k) + "=" + str(v) + " " utils.execute('ceph', 'osd', 'erasure-code-profile','set', profile_ref['name'], \ plugin, ruleset_root, failure_domain, pair_str, '--force', \ run_as_root=True) utils.execute('ceph', 'osd', 'crush', 'rule', 'create-erasure', \ rule_name, profile_ref['name'], run_as_root=True) res = utils.execute('ceph', 'osd', 'pool', 'create', pool_name, pg_num, \ pgp_num, 'erasure', profile_ref['name'], rule_name, \ run_as_root=True) elif body.get('pool_type') == 'replicated': try: utils.execute('ceph', 'osd', 'getcrushmap', '-o', FLAGS.crushmap_bin, run_as_root=True) utils.execute('crushtool', '-d', FLAGS.crushmap_bin, '-o', FLAGS.crushmap_src, run_as_root=True) #ruleset = self._get_new_ruleset() pg_num = str(body['pg_num']) primary_storage_group = body['storage_group_name'] storage_group = db.storage_group_get_by_name(context,primary_storage_group) ruleset = storage_group['rule_id'] utils.execute('chown', '-R', 'vsm:vsm', '/etc/vsm/', run_as_root=True) utils.execute('ceph', 'osd', 'pool', 'create', pool_name, \ pg_num, pg_num, 'replicated', run_as_root=True) utils.execute('ceph', 'osd', 'pool', 'set', pool_name, 'crush_ruleset', ruleset, run_as_root=True) utils.execute('ceph', 'osd', 'pool', 'set', pool_name, 'size', str(body['size']), run_as_root=True) res = True except: LOG.error("create replica storage pool error!") raise return False else: rule = str(body['crush_ruleset']) size = str(body['size']) pg_num = str(body['pg_num']) res = utils.execute('ceph', 'osd', 'pool', 'create', pool_name, \ pg_num, run_as_root=True) utils.execute('ceph', 'osd', 'pool', 'set', pool_name, 'size', size, run_as_root=True) utils.execute('ceph', 'osd', 'pool', 'set', pool_name, 'crush_ruleset', rule, run_as_root=True) #set quota if body.get('enable_quota', False): max_bytes = 1024 * 1024 * 1024 * int(body.get('quota', 0)) utils.execute('ceph', 'osd', 'pool', 'set-quota', pool_name, 'max_bytes', max_bytes,\ run_as_root=True) #update db pool_list = self.get_pool_status() for pool in pool_list: if pool_name == pool['pool_name']: values = &#123; 'pool_id': pool.get('pool'), 'name': pool.get('pool_name'), 'pg_num': pool.get('pg_num'), 'pgp_num': pool.get('pg_placement_num'), 'size': pool.get('size'), 'min_size': pool.get('min_size'), 'crush_ruleset': pool.get('crush_ruleset'), 'crash_replay_interval': pool.get('crash_replay_interval'), 'ec_status': pool.get('erasure_code_profile'), 'replica_storage_group': body.get('pool_type'), 'quota': body.get('quota'), 'max_pg_num_per_osd':body.get('max_pg_num_per_osd'), 'auto_growth_pg':body.get('auto_growth_pg',0), &#125; values['created_by'] = body.get('created_by') values['cluster_id'] = body.get('cluster_id') values['tag'] = body.get('tag') values['status'] = 'running' values['primary_storage_group_id'] = body.get('storage_group_id') db.pool_create(context, values) return res]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[VSM源码分析（三）——移除缓存池]]></title>
      <url>%2F2017%2F04%2F06%2Fvsm-remove-cachepool%2F</url>
      <content type="text"><![CDATA[移除缓存池并不是删除存储池，而只是将缓存池与数据池的缓存关系移除。回写缓存和只读缓存的去除过程不太一样。 回写缓存可能含有更改过的数据，所以在禁用并去除前，必须手工回刷数据到数据池中以免丢失缓存内近期更改的对象。 只读缓存不含变更数据，所以禁用它不会导致任何近期更改的数据丢失。 vsm-dashboardvsm-dashboard\vsm_dashboard\dashboards\vsm\poolsmanagement\urls.py文件中移除缓存池的url1url(r'^remove_cache_tier_action/$', remove_cache_tier, name='remove_cache_tier_action'), views处理移除Cache存储池请求，调用vsmapi的remove_cache_tier方法123456789101112131415def remove_cache_tier(request): status = "" msg = "" body = json.loads(request.body) try: ret = vsmapi.remove_cache_tier(request,body=body) status = "OK" msg = "Remove Cache Tier Successfully!" except: status = "Failed" msg = "Remove Cache Tier Failed!" resp = dict(message=msg, status=status) resp = json.dumps(resp) return HttpResponse(resp) vsmapi的remove_cache_tier方法，调用vsmclient的remove_cache_tier方法12def remove_cache_tier(request, body): return vsmclient(request).storage_pools.remove_cache_tier(body) vsmclientvsmclient/v1/storage_pools.py的remove_cache_tier方法123def remove_cache_tier(self, cache_tier_body): url = '/storage_pools/remove_cache_tier' resp, body = self.api.client.post(url, body=cache_tier_body) vsmvsm/api/v1/storage_pool.py中的remove_cache_tier方法 1234567891011121314151617@wsgi.serializers(xml=StoragePoolsTemplate) def remove_cache_tier(self, req, body=None): """add a cache tier.""" LOG.info(body) #&#123;'cache_tier': # &#123; # 'cache_pool_id': 4, # &#125; #&#125; if not self.is_valid_body(body, 'cache_tier'): raise exc.HTTPUnprocessableEntity() context = req.environ['vsm.context'] cache_tier_body = body['cache_tier'] return self.scheduler_api.remove_cache_tier(context, cache_tier_body) vsm/scheduler/api.py中的remove_cache_tier方法12def remove_cache_tier(self, context, body=None): return self.scheduler_rpcapi.remove_cache_tier(context, body) vsm/scheduler/rpcapi.py中的remove_cache_tier方法 rpc的call同步方法，调用remove_cache_tier12def remove_cache_tier(self, ctxt, body=None): return self.call(ctxt, self.make_msg('remove_cache_tier', body=body)) vsm/scheduler/manager.py中的remove_cache_tier方法，调用agent rpcapi123def remove_cache_tier(self, context, body): active_server = self._get_active_server(context) self._agent_rpcapi.remove_cache_tier(context, body,active_server['host']) vsm/agent/rpc.py中的remove_cache_tier方法 rpc的call同步方法，调用remove_cache_tier1234567def remove_cache_tier(self, context, body, host): topic = rpc.queue_get_for(context, self.topic, host) res = self.call(context, self.make_msg('remove_cache_tier', body=body), topic, version='1.0', timeout=6000) vsm/agent/manager.py中的remove_cache_tier方法12def remove_cache_tier(self, context, body): return self.ceph_driver.remove_cache_tier(context, body) vsm/agent/driver.py中的remove_cache_tier方法 db查询获取cache_pool 如果cache_mode == “writeback”: 修改缓存模式为forward，数据直接写入数据存储池：ceph osd tier cache-mode {cachepool} forward 手动回刷数据：rados -p {cachepool} cache-flush-evict-all 删除覆盖层，数据将不会通过cache：ceph osd tier remove-overlay {storagetier} 如果cache_mode不是writeback： 缓存模式改为none，即为禁用：ceph osd tier cache-mode {cachepool} none 从后端数据池移除cache池：ceph osd tier remove {storagepool} {cachepool} db更新StorgaePool的cache_tier_status123456789101112131415161718192021222324252627282930def remove_cache_tier(self, context, body): LOG.info("Remove Cache Tier") LOG.info(body) cache_pool = db.pool_get(context, body.get("cache_pool_id")) cache_pool_name = cache_pool.get("name") storage_pool_name = cache_pool.get("cache_tier_status").split(":")[1].strip() LOG.info(cache_pool['name']) LOG.info(cache_pool['cache_mode']) cache_mode = cache_pool.get("cache_mode") LOG.info(cache_mode) if cache_mode == "writeback": utils.execute("ceph", "osd", "tier", "cache-mode", cache_pool_name, \ "forward", run_as_root=True) utils.execute("rados", "-p", cache_pool_name, "cache-flush-evict-all", \ run_as_root=True) utils.execute("ceph", "osd", "tier", "remove-overlay", storage_pool_name, \ run_as_root=True) else: utils.execute("ceph", "osd", "tier", "cache-mode", cache_pool_name, \ "none", run_as_root=True) utils.execute("ceph", "osd", "tier", "remove", storage_pool_name, \ cache_pool_name, run_as_root=True) db.pool_update(context, cache_pool.pool_id, &#123;"cache_tier_status": None&#125;) # TODO cluster id if body.has_key('cluster_id') and body['cluster_id']: cluster_id = body['cluster_id'] else: cluster_id = db.cluster_get_all(context)[0]['id'] db.pool_update_by_name(context, storage_pool_name, cluster_id, &#123;"cache_tier_status": None&#125;) return True]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[VSM源码分析（二）——创建缓存池]]></title>
      <url>%2F2017%2F04%2F06%2Fvsm-create-cachepool%2F</url>
      <content type="text"><![CDATA[缓存池可提升后端存储内某些（热点）数据的 I/O 性能。缓存池需创建一个由高速存储设备（例如SSD ）组成的存储池作为缓存层，以及一个相对低速/廉价设备组成的后端存储池作为数据存储层。 创建缓存池的流程图如下： vsm-dashboard vsm-dashboard\vsm_dashboard\dashboards\vsm\poolsmanagement\urls.py文件中添加缓存池的url1url(r'^add_cache_tier_action/$', add_cache_tier, name='add_cache_tier_action'), views处理创建Cache存储池请求，调用vsmapi的add_cache_tier方法12345678910111213141516def add_cache_tier(request): status = "" msg = "" body = json.loads(request.body) try: ret = vsmapi.add_cache_tier(request,body=body) status = "OK" msg = "Add Cache Tier Successfully!" except: status = "Failed" msg = "Add Cache Tier Failed!" resp = dict(message=msg, status=status) resp = json.dumps(resp) return HttpResponse(resp) vsmapi的add_cache_tier方法，调用vsmclient的add_cache_tier方法12def add_cache_tier(request, body): return vsmclient(request).storage_pools.add_cache_tier(body) vsmclientvsmclient/v1/client.py的storage_pools1self.storage_pools = storage_pools.StoragePoolManager(self) vsmclient/v1/storage_pools.py的StoragePoolManager类的add_cache_tier方法，发送POST请求123def add_cache_tier(self, cache_tier_body): url = '/storage_pools/add_cache_tier' resp, body = self.api.client.post(url, body=cache_tier_body) vsmvsm/api/v1/storage_pool.py的add_cache_tier方法，调用scheduler_api的add_cache_tier方法123456789101112131415161718@wsgi.serializers(xml=StoragePoolsTemplate) def add_cache_tier(self, req, body=None): """add a cache tier.""" LOG.info(body) #&#123;'cache_tier': # &#123;'storage_pool_id': 3, # 'cache_pool_id': 4, # 'cache_mode': 'writeback', # &#125; #&#125; if not self.is_valid_body(body, 'cache_tier'): raise exc.HTTPUnprocessableEntity() context = req.environ['vsm.context'] cache_tier_body = body['cache_tier'] return self.scheduler_api.add_cache_tier(context, cache_tier_body) vsm/scheduler/api的add_cache_tier方法12def add_cache_tier(self, context, body=None): return self.scheduler_rpcapi.add_cache_tier(context, body) vsm/scheduler/rpcapi的add_cache_tier方法，发送rpc同步请求12def add_cache_tier(self, ctxt, body=None): return self.call(ctxt, self.make_msg('add_cache_tier', body=body)) vsm/scheduler/manager的add_cache_tier方法 调用_get_active_server获取server 调用agentrpc的add_cache_tier方法123def add_cache_tier(self, context, body): active_server = self._get_active_server(context) self._agent_rpcapi.add_cache_tier(context, body, active_server['host']) _get_active_server方法 查询数据库，获取所有server 在server列表中只获取Active的server 随机获取一台server123456def _get_active_server(self, context): server_list = db.init_node_get_all(context) active_server_list = [x for x in server_list if x['status'] == "Active"] idx = random.randint(0, len(active_server_list)-1) return active_server_list[idx] vsm/agent/rpcapi.py的add_cache_tier方法 获取topic rpc的call同步方法，调用add_cache_tier1234567def add_cache_tier(self, context, body, host): topic = rpc.queue_get_for(context, self.topic, host) res = self.call(context, self.make_msg('add_cache_tier', body=body), topic, version='1.0', timeout=6000) vsm/agent/manager.py的add_cache_tier方法12def add_cache_tier(self, context, body): return self.ceph_driver.add_cache_tier(context, body) vsm/agent/driver.py的add_cache_tier方法 查询数据库，根据ID分别获取storage_pool_name、cache_pool_name 执行添加cache存储池命令 ceph osd tier add {storagepool} {cachepool} --force-nonempty 或 ceph osd tier add {storagepool} {cachepool} 设置cache mode：ceph osd tier cache-mode {cachepool} {cache-mode} 如果cachemode为writeback，设置 ceph osd tier set-overlay {storagepool} {cachepool} 更新StoragePools实体的cache_tier_status、cache_mode字段 调用_configure_cache_tier方法，更新options1234567891011121314151617181920212223242526272829def add_cache_tier(self, context, body): storage_pool_name = db.pool_get(context, body.get("storage_pool_id")).get('name') cache_pool_name = db.pool_get(context, body.get("cache_pool_id")).get('name') cache_mode = body.get("cache_mode") LOG.info("add cache tier start") LOG.info("storage pool %s cache pool %s " % (storage_pool_name, cache_pool_name)) if body.get("force_nonempty"): utils.execute("ceph", "osd", "tier", "add", storage_pool_name, \ cache_pool_name, "--force-nonempty", run_as_root=True) else: utils.execute("ceph", "osd", "tier", "add", storage_pool_name, \ cache_pool_name, run_as_root=True) utils.execute("ceph", "osd", "tier", "cache-mode", cache_pool_name, \ cache_mode, run_as_root=True) if cache_mode == "writeback": utils.execute("ceph", "osd", "tier", "set-overlay", storage_pool_name, \ cache_pool_name, run_as_root=True) db.pool_update(context, body.get("storage_pool_id"), &#123;"cache_tier_status": "Storage pool for:%s" % cache_pool_name&#125;) db.pool_update(context, body.get("cache_pool_id"), &#123; "cache_tier_status": "Cache pool for:%s" % storage_pool_name, "cache_mode": cache_mode&#125;) options = body.get("options") self._configure_cache_tier(cache_pool_name, options) LOG.info("add cache tier end") return True 配置属性设置命令为：ceph osd pool set {cachepool} {key} {value}key分别为： hit_set_type hit_set_count hit_set_period target_max_bytes cache_target_dirty_ratio cache_target_full_ratio target_max_objects cache_min_flush_age cache_min_evict_age 12345678910def _configure_cache_tier(self, cache_pool_name, options): utils.execute("ceph", "osd", "pool", "set", cache_pool_name, "hit_set_type", options["hit_set_type"], run_as_root=True) utils.execute("ceph", "osd", "pool", "set", cache_pool_name, "hit_set_count", options["hit_set_count"], run_as_root=True) utils.execute("ceph", "osd", "pool", "set", cache_pool_name, "hit_set_period", options["hit_set_period_s"], run_as_root=True) utils.execute("ceph", "osd", "pool", "set", cache_pool_name, "target_max_bytes", int(options["target_max_mem_mb"]) * 1000000, run_as_root=True) utils.execute("ceph", "osd", "pool", "set", cache_pool_name, "cache_target_dirty_ratio", options["target_dirty_ratio"], run_as_root=True) utils.execute("ceph", "osd", "pool", "set", cache_pool_name, "cache_target_full_ratio", options["target_full_ratio"], run_as_root=True) utils.execute("ceph", "osd", "pool", "set", cache_pool_name, "target_max_objects", options["target_max_objects"], run_as_root=True) utils.execute("ceph", "osd", "pool", "set", cache_pool_name, "cache_min_flush_age", options["target_min_flush_age_m"], run_as_root=True) utils.execute("ceph", "osd", "pool", "set", cache_pool_name, "cache_min_evict_age", options["target_min_evict_age_m"], run_as_root=True)]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[VSM源码分析（一）——存储池列表]]></title>
      <url>%2F2017%2F04%2F05%2Fvsm-pool-list%2F</url>
      <content type="text"><![CDATA[VSM的代码调用是从vsm-dashboard——&gt;vsmclient——&gt;vsm。 存储池列表查询是从页面dashboard发送请求，获取vsm数据库storage_pools表的存储池数据。 查询存储池的流程图如下： vsm-dashboardvsm-dashboard\vsm_dashboard\dashboards\vsm\dashboard.py所有的面板在dashboard中注册，存储池管理是集群管理的第四个panel 存储池管理的代码结构如下： urls中定义url路径，存储池列表是调用IndexView处理逻辑 views中IndexView类获取存储池列表数据 table_class指定数据表格ListPoolTable template_name 指定模板html文件vsm/poolsmanagement/index.html 重写get_data方法获取data，调用vsmapi.pools_list方法 vsmapi.pools_list方法 vsmclient方法调用vsmclient组件的Client vsmclientvsmclient/v1/client.py vsmclient/v1/vsms.py的VolumeManager类list_storage_pool方法，调用vsm的restapi的url为’/storage_pool/list_storage_pool?all_pools=True’ vsmvsm的api：storage_pool.py，调用conductor api的list_storage_pool方法 调用conductor的rpcapi的list_storage_pool方法 rpc的call同步方法，调用list_storage_pool rpc的同步方法，调用manage.py的list_storage_pool方法 调用db api的pool_get_all方法 db/sqlalchemy/api的pool_get_all方法，查询StoragePool实体映射的storage_pools表的数据]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[VSM消息协议与服务简介]]></title>
      <url>%2F2017%2F04%2F05%2Fvsm-base-core%2F</url>
      <content type="text"><![CDATA[VSM架构与OpenStack相同，使用分布式架构。在分析VSM的源码之前，先简单介绍VSM使用到的规范、协议等，比如WSGI、AMQP。 WSGIPython中的WSGI (Web Server Gateway Interface), 是Python应用程序（或者框架、套件）与服务器之间的一种接口，其定义了两者进行通信的接口规范。服务器端和应用端都必须遵循这套规范。当一个应用程序是按照WSGI规范开发的，那么它可以在任意遵循该规范的服务器上运行。 AMQPAMQP，即Advanced Message Queuing Protocol,一个提供统一消息服务的应用层标准高级消息队列协议。 应用层协议的一个开放标准 面向消息的中间件设计 基于此协议的客户端与消息中间件可传递消息 不受客户端/中间件不同产品、不同开发语言等条件的限制 交换器接受消息并且将消息转发给队列。在每个主机的内部，交换器有唯一对应的名字。交换器可以是持久的，临时的或者自动删除的。持久的交换器会一直存在于Server端直到他被显示的删除；临时交换器在服务器关闭时停止工作；自动删除的交换器在没有应用程序使用它的时候被服务器删除。主要有三种类型的交换器。 广播式交换器类型（fanout）该类交换器不分析所接收到消息中的Routing Key，默认将消息转发到所有与该交换器绑定的队列中去。广播交换器是最简单的一种类型，就像我们从字面上理解到的一样，它把所有接受到的消息广播到所有它所知道的队列中去，不论消息的关键字是什么，消息都会被路由到和该交换器绑定的队列中去。 直接式交换器类型（direct）该类交换器需要精确匹配Routing Key与BindingKey，如消息的Routing Key = Cloud，那么该条消息只能被转发至Binding Key = Cloud的消息队列中去。直接式交换器的转发效率较高，安全性较好，但是缺乏灵活性，系统配置量较大。 主题式交换器（Topic Exchange）该类交换器通过消息的Routing Key与Binding Key的模式匹配，将消息转发至所有符合绑定规则的队列中。Binding Key支持通配符，其中“”匹配一个词组，“#”匹配多个词组（包括零个）。例如，Binding Key=“.Cloud.#”可转发Routing Key=“OpenStack.Cloud.GD.GZ”、“OpenStack.Cloud.Beijing”以及“OpenStack.Cloud”的消息，但是对于Routing Key=“Cloud.GZ”的消息是无法匹配的。 队列“消息队列”，它是一个具名缓冲区，它代表一组消费者应用程序保存消息。这些应用程序在它们的权限范围内可以创建、使用、共享消息队列。类似于交换器，消息队列也可以是持久的，临时的或者自动删除的。临时消息队列在服务器被关闭时停止工作；自动删除队列在没有应用程序使用它的时候被服务器自动删除。消息队列将消息保存在内存、硬盘或两者的组合之中。 绑定可以理解为交换器和消息队列之间的一种关系，绑定之后交换器会知道应该把消息发给那个队列，绑定的关键字称为binding_key。Exchange和Queue的绑定可以是多对多的关系，每个发送给Exchange的消息都会有一个叫做routing_key的关键字，交换器要想把消息发送给某个特定的队列，那么该队列与交换器的binding_key必须和消息的routing_key相匹配才OK。 AMQP原理在AMQP 模型中： 消息的producer 将Message 发送给Exchange Exchange 负责交换/ 路由，将消息正确地转发给相应的Queue 消息的Consumer 从Queue 中读取消息 RabbitMQRabbitMQ是一个在AMQP基础上完整的可复用的企业消息系统 invoker向消息队列中发消息， worker处理消息。invoker端：在vsm中的api会通过rpcapi发出消息到消息队列worker端：manager(endpoint)处理消息 例如：agent rpcapi—&gt;agent managerconductor rpcapi—&gt; conductor managerscheduler rpcapi—&gt; scheduler manager 如下图所示，左边为invoker，右侧为worker RPCvsm在RabbitMQ的基础上实现了RPC。 rpc.callrpc.call：同步，请求/响应模式 rpc调用方初始化Topic Publisher用于发送rpc message, 同时初始化一个Direct Consumer用于接收返回的rpc调用结果, 为了保证收到相应的rpc调用的结果, rpc message中会保存一个唯一标识该消息的messga id(UUID类型), 消息以Routing Key为topic发送至Exchange中 rpc接收方会初始化两个Topic Cosumer, 分别通过topic和topic.host为binding key将Queue与Exchange绑定, rpc 接收方执行rpc调用, 完成后通过Direct Publisher将结果发送到消息队列中, 返回的执行结果message的Routing Key为message id (唯一的UUID), 将被发送至Exchange(名称同样是唯一的message id), 那样调用方的Direct Consumer就可以收到rpc调用的结果了 下图 rpc.call 流程 rpc.castrpc.cast：异步 rpc调用方会初始化Topic Publisher, 将消息以Routing Key为topic发送至配置的Exchange中 rpc接收方会初始化两个Topic Cosumer, 分别通过topic和topic.host为binding key将Queue与Exchange绑定, 这样接收方就可以接收这两种类型的rpc message topic即为消息的Routing Key, 所以接收方可以接受到调用方的rpc message topic.host为特定的host上的接收方获取rpc message的方式 下图 rpc.cast流程]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[修改VSM的Web界面为中文]]></title>
      <url>%2F2017%2F03%2F31%2Fvsm-web-zh%2F</url>
      <content type="text"><![CDATA[安装VSM之后，默认使用的是英文界面。在VSM的控制节点将vsm-dashboard编译为中文就可以将VSM的界面修改成中文。 修改settings.py修改settings.py中的语言编码为中文，将LANGUAGE_CODE = ‘en’改成LANGUAGE_CODE = ‘zh-cn’12[root@vsmcontroller /]# cd /usr/share/vsm-dashboard/vsm_dashboard[root@vsmcontroller vsm_dashboard]# vi settings.py LANGUAGE_CODE = &apos;zh-cn&apos; 重新编译django.mo下载 django.po，放到/usr/share/vsm-dashboard/vsm_dashboard/目录下。 执行 django-admin compilemessages 命令将django.po编译成django.mo123[root@vsmcontroller /]# cd /usr/share/vsm-dashboard/vsm_dashboard/[root@vsmcontroller vsm_dashboard]# django-admin compilemessagesprocessing file django.po in /usr/share/vsm-dashboard/vsm_dashboard/locale/zh_CN/LC_MESSAGES 重启httpd服务12[root@vsmcontroller vsm_dashboard]# service httpd restartRedirecting to /bin/systemctl restart httpd.service 重新登录]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[VSM安装配置]]></title>
      <url>%2F2017%2F03%2F31%2Fvsm-install-deploy%2F</url>
      <content type="text"><![CDATA[VSM的安装配置依赖的包比较多，yum在线安装起来比较麻烦，现收集在CentOS7.2环境下安装VSM2.1的所有包，本地进行安装配置。 部署环境系统环境 CentOS7.2最小版，即 CentOS Linux release 7.2.1511 (Core) VSM版本 VSM v2.1.0 final release Ceph版本 ceph version 0.94.7 节点环境1个控制节点，2个存储节点，管理网络、集群网络、存储网络均使用10.10.111.0/24。 节点类型 节点IP hostname 安装软件 Controller Node 10.10.111.245 vsmcontroller 安装VSM控制平台 Storage Node 10.10.111.246 vsmnode1 安装VSM的Agent Storage Node 10.10.111.247 vsmnode2 安装VSM的Agent 基础安装配置在每台节点上执行以下安装配置 最小化安装centos7.2 关闭防火墙/selinux 123systemctl stop firewalldsystemctl disable firewalldsed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config 安装ntp,并且同步所有节点的时间（可以先略过，在Storage节点安装完成之后再配置NTP） 修改节点上的主机名字，节点的hostname分别为vsmcontroller、vsmnode1、vsmnode2 1hostnamectl set-hostname [hostname] 将所有节点的/etc/hosts同步 123456[root@vsmcontroller ~]# cat /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain610.10.111.245 vsmcontroller10.10.111.246 vsmnode110.10.111.247 vsmnode2 将repo-rpm目录下的3个文件controller.repo、controller.tar.gz、preinstall上传到/opt目录 123456[root@vsmcontroller ~]# cd /opt[root@vsmcontroller opt]# lltotal 187140-rw-r--r-- 1 root root 102 Mar 31 13:26 controller.repo-rw-r--r-- 1 root root 191621120 Mar 31 13:26 controller.tar.gz-rw-r--r-- 1 root root 564 Mar 31 13:26 preinstall 准备vsm repo，解压controller.tar.gz 1234cd /optrm /etc/yum.repos.d/* -rfcp /opt/controller.repo /etc/yum.repos.d/tar -xvf controller.tar.gz controller节点安装安装配置 将controler目录下的2个文件cephcontrollerinstall.sh、cluster.manifest上传到/opt目录中 12345678[root@vsmcontroller opt]# lltotal 187168-rw-r--r-- 1 root root 1291 Mar 31 03:14 cephcontrollerinstall.sh-rw-r--r-- 1 root root 1569 Jul 6 2016 cluster.manifestdrwxr-xr-x. 3 root root 16384 May 16 2016 controller-rw-r--r--. 1 root root 102 May 12 2016 controller.repo-rw-r--r--. 1 root root 191621120 May 16 2016 controller.tar.gz-rw-r--r--. 1 root root 564 May 12 2016 preinstall 执行ssh-keygen -t rsa命令生成默认key 12345678910111213141516171819202122[root@vsmcontroller opt]# ssh-keygen -t rsaGenerating public/private rsa key pair.Enter file in which to save the key (/root/.ssh/id_rsa):Created directory '/root/.ssh'.Enter passphrase (empty for no passphrase):Enter same passphrase again:Your identification has been saved in /root/.ssh/id_rsa.Your public key has been saved in /root/.ssh/id_rsa.pub.The key fingerprint is:9a:8d:26:ef:01:45:83:0e:ef:21:b4:1d:31:fd:0f:1e root@vsmcontrollerThe key's randomart image is:+--[ RSA 2048]----+| o+o || o oo.. || . * ... || o =. E || o.. .S+ || .. =. . || . * . || + . || .o |+-----------------+ Copy key到storage节点，配置controller节点无密码访问storage节点 12ssh-copy-id [hostname]scp /etc/hosts [hostname]:/etc/ 修改cluster.manifest文件中的management_addr/ceph_public_addr/ceph_cluster_addr配置。比如10.10.111.0/24 12345678[management_addr]10.10.111.0/24[ceph_public_addr]10.10.111.0/24[ceph_cluster_addr]10.10.111.0/24 安装VSM控制平台 123cd /optchmod 777 cephcontrollerinstall.sh./cephcontrollerinstall.sh 执行cluster_manifest命令，查看cluster manifest信息，最后输出“Check Success ~~”就是安装成功。 1cluster_manifest 执行以下命令给vsm用户赋予可获取id-rsa权限（可以略过，修改VSM源码免密码添加节点的功能需要这个操作） 123setfacl -m u:vsm:rx /rootsetfacl -m u:vsm:rx /root/.sshsetfacl -m u:vsm:r /root/.ssh/id_rsa 登录Web访问 https://10.10.111.245/dashboard 登录VSM控制平台。 在/etc/vsmdeploy/deployrc文件中可以查看VSM的配置信息，比如：admin登录密码和数据库mysql的root密码1234567891011121314151617181920212223......#---------------------------------------------# MySQL#---------------------------------------------MYSQL_ROOT_PASSWORD=57467dfa1e80e367beb5MYSQL_HOST=10.10.111.245#---------------------------------------------# Keystone#---------------------------------------------MYSQL_KEYSTONE_USER=keystoneMYSQL_KEYSTONE_PASSWORD=95cb683c9614b887d58eKEYSTONE_HOST=10.10.111.245ADMIN_PASSWORD=e607015b3f8b492f68d7ADMIN_TOKEN=d5aa68b1278f1eab2992AGENT_PASSWORD=72ef09df512bab520df5...... admin登录用户密码12[root@vsmcontroller vsm_dashboard]# cat /etc/vsmdeploy/deployrc |grep ADMIN_PASSWORDADMIN_PASSWORD=e607015b3f8b492f68d7 mysql的root密码12[root@vsmcontroller vsm_dashboard]# cat /etc/vsmdeploy/deployrc |grep MYSQL_ROOT_PASSWORDMYSQL_ROOT_PASSWORD=57467dfa1e80e367beb5 生成auth_key安装Storage节点需要Controller生成的auth_key。12[root@vsmcontroller opt]# agent-token186ef3004b3a456995088bc38d5cd972-1236c1f529274de0800d0da9532cafb7 Stroage节点安装 将stroage目录下的3个文件cephstroageinstall.sh、preinstall、server.manifest上传到/opt目录中 123456789[root@vsmnode1 ~]# cd /opt[root@vsmnode1 opt]# lltotal 187172-rw-r--r-- 1 root root 5469 Mar 31 03:16 cephstroageinstall.shdrwxr-xr-x. 3 root root 16384 May 16 2016 controller-rw-r--r--. 1 root root 102 May 12 2016 controller.repo-rw-r--r--. 1 root root 191621120 May 16 2016 controller.tar.gz-rw-r--r--. 1 root root 564 May 12 2016 preinstall-rw-r--r-- 1 root root 733 May 11 2016 server.manifest 修改server.manifest文件中的vsm_controller_ip/ssd/7200_rpm_sata/10krpm_sas配置 控制节点的IP为：10.10.111.245 ssd/7200_rpm_sata/10krpm_sas为不同硬盘的类型，如果只有普通硬盘那么只配置7200_rpm_sata 12[vsm_controller_ip]10.10.111.245 两块普通硬盘sdb、sdc作OSD，sdb已提前创建分区sdb1、sdb2，sdc已提前创建分区sdc1、sdc2 1234[7200_rpm_sata]#format [sata_device] [journal_device]/dev/sdb1 /dev/sdb2/dev/sdc1 /dev/sdc2 安装VSM agent 123cd /optchmod 777 cephstroageinstall.sh./cephstroageinstall.sh [controllerip] [auth_key] controllerip：控制节点的ip地址10.10.111.245 auth_key：控制节点生成的auth_key ，例如前面的186ef3004b3a456995088bc38d5cd972-1236c1f529274de0800d0da9532cafb7 执行server_manifest命令，查看server manifest信息，最后输出“Check Success ~~”就是安装成功。1server_manifest 重新登录Web再次访问 https://10.10.111.245/dashboard 登录VSM控制平台。Storege节点10.10.111.246、10.10.111.247已经添加到服务器列表。 配置中文界面配置中文界面，请参考 《修改VSM的Web界面为中文》 创建集群点击“创建集群”，选择vsmnode1、vsmnode2两个存储服务器来创建Ceph集群，创建完成之后，可显示集群的状态。]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[Ceph开源管理监控平台分析]]></title>
      <url>%2F2017%2F03%2F30%2Fceph-web-manage%2F</url>
      <content type="text"><![CDATA[Ceph的开源管理监控平台有好几个，比如VSM，InkScope，Calamari等。 VSMVSM（Virtual Storage Manager）：Intel的Wang, Yaguang (Yaguang.wang@intel.com)团队开发。 Roadmap VSM架构VSM的设计是完全按照OpenStack的架构设计。遵循一定的原则来设计这个架构： 容易与OpenStack的模块集成 完全与其它的OpenStack模块解耦合 易于合并到其他云平台 具有高可靠性，大规模数据中心的高可用性 易于使用，部署和管理 从上面VSM的架构图，我们可以看出，VSM分为核心的两个部分，VSM控制节点与VSM存储节点。 VSM控制节点 WebUI – 通过访问VSM REST API用于集群的管理、监控 REST API – 供vsm client访问 mariadb, rabbitmq VSM存储节点 使用diamond收集ceph节点的监控信息 vsm-agent工具对ceph节点进行管理 组件 Dashboard（vsm-dashboard）：VSM的webUI界面，用户通过Dashboard来管理与监控ceph集群 vsmclient（python-vsmclient）：VSM restapi 调用的client API（vsm）：VSM的restapi scheduler（vsm）：VSM的调度组件 conductor（vsm）：VSM的数据库操作组件，即所有的数据库操作都是通过conductor来调用mysql RabbitMQ：消息中间件，VSM的各个组件相对独立，都是通过发送消息，通过RPC的方式来相互调用 agent（vsm）：VSM代理服务 组件特点 分布式：分开独立部署 无状态：各个请求独立，可扩展性强 RESTFUL RPC plugin：插件式设计，松耦合 组件代码VSM目前最新的发行版本为2.1，VSM的代码组件分为四个： vsm-dashboard：VSM的管理与监控web界面 python-vsmclient：调用restapi的client vsm：VSM的核心组件（包括api、scheduler、conductor、agent等） vsm-deploy：ceph部署工具 代码基于Python语言，使用了wsgi、django等技术框架 GitHubGit主页：https://github.com/01org/virtual-storage-manager releases版本代码下载：https://github.com/01org/virtual-storage-manager/releases vsm-dependencies：https://github.com/01org/vsm-dependencies 支持功能 仪表盘：查看vsm、cluster、storage group、OSD、MON、MDS、PG的状态统计信息 可以判断OSD是否正常运作，空间是否满 查看IOPS、latency、bandwidth、CPU实时监控信息(通过diamond实现数据的收集) 可以用来发现ntp延迟的问题 所有的宿主节点都需要在安装vsm的时候写在配置文件中 添加删除MON/OSD守护进程 OSD 增删、重启、恢复（N/A） osd pool的管理 – 支持cache tier的增删、replicated/EC pool的创建 StorageGroup的管理 – 添加新的SG，存储资源将以SG为单位进行统计 支持Ceph系统的升级功能，通过github下载源码实现 将通过ssh配置openstack的控制节点把rbd pool present给cinder 管理系统的临界值，将在dashboard中得到体现 vsm 账户管理（keystone管理账户） InkscopeInkscope目前最新版为1.40, http://inkscope.blogspot.fr/ Inkscope架构 组件 inkscope-common：包含inkscope的默认配置文件以及其他进程(cephprobe,sysprobe)启动所需的依赖文件，所有相关节点都需要安装。 inkscope-admviz：包含inkscope的web控制台文件，含接口和界面，仅需要安装一个，该节点（管理节点）上同时需要按安装flask和mongodb inkscope-cephrestapi：用于安装启动 ceph rest api 的脚本，仅需要安装在提供api接口的节点上，即mon节点。 inkscope-cephprobe：用于安装启动 cephprobe 的脚本(整个集群只需一个)，安装在mon节点，脚本主要实现：获取Ceph集群的一些信息，并使用端口（5000）提供服务，将数据存入mongodb数据库中。 inkscope-sysprobe：安装用于所有mon和osd的sysprobe 所需要脚本，即所有节点均安装，实现获取节点设备资源信息如：CPU、内存、磁盘等等。 组件代码 inkscopeViz：Web客户端 inkscopeCtrl：inkscope的服务器端，提供了REST API inkscopeProbe：收集ceph节点的系统信息，收集到的数据将传输到MongoDB cephprobe：用来或者集群的相关信息和操作的 sysprobe：获取节点的磁盘分区等相关信息的 inkscopeMonitor：对接第三方监控框架 代码基于python语言，使用了wsgi、flask等技术框架 GitHubGit主页:https://github.com/inkscope/inkscope inkscope-packaging：https://github.com/inkscope/inkscope-packaging 支持功能 仪表盘：查看cluster、OSD、MON、MDS、PG的状态统计信息可以判断OSD是否正常运作，空间是否满 分模块管理OSD、Pool、PG、RBD、RadosGW、MDS inkscope账户管理 Calamari暂未研究 管理平台比较技术上 对比项 VSM InkScope Calamari license Apache v2 Apache v2 开发语言 Python Python Web框架 Apache/django Apache/flask js_lib 无 AngularJS css bootstrap bootstrap 数据库 MySQL mongoDB Restapi 重量级、封装ceph-rest-api 轻量级，直接用ceph-rest-api 分析 VSM相对比InkScope的代码复杂 多封装一层restapi 依赖openstack的组件，例如keystone、openstack_auth 功能上 对比项 VSM InkScope Calamari 创建集群 vsm-deploy工具 命令行 仪表盘 较多 无带宽等性能 监控状态 state只有列表 较多，有图形展示 MON管理 是 否 OSD管理 有添加删除、无图形展示状态、无更新状态 无添加删除，有图形展示状态及更新状态 Pool管理 是 更多，并支持快照 RBD管理 只有查看state 可以创建image、快照等 RadosGW管理 否 是 MDS 否 只有查看state 纠删码管理 否 是 设置flags 否 是 集成openstack 集合紧密，依赖keystone、openstack_auth等组件 否 分析 VSM有vsm-deploy工具来创建集群，而InkScope只有使用Ceph CLI来创建 InkScope相对比VSM支持的功能更多 VSM支持openstack，可以跟openstack很好地结合]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[Python XlsxWriter操作excel写入中文字符串变量出错]]></title>
      <url>%2F2017%2F03%2F29%2Fxlsxwriter-2%2F</url>
      <content type="text"><![CDATA[XlsxWriter写excel，变量中含有中文字符串出错。 错误信息变量s5ptname中含有中文字符123456Traceback (most recent call last): File "rsq_s5rt.py", line 24, in &lt;module&gt; write_excel.write_excel(jsondata, file_name) File "/opt/rsq_s5rt/write_excel.py", line 38, in write_excel worksheet.write('A1', s5ptname + '成员')UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 0: ordinal not in range(128) 分析python的str默认是ascii编码，和unicode编码冲突，中文字符串变量无法直接写入excel。 解决增加下面几行代码，进行编码转换成utf-8 123import sysreload(sys)sys.setdefaultencoding('utf8')]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[日事清S5跑团打卡统计程序]]></title>
      <url>%2F2017%2F03%2F29%2Fpython-rsq-s5rt%2F</url>
      <content type="text"><![CDATA[日事清S5跑团打卡统计程序，每周一零点1分定时执行统计跑团成员打卡情况，生成excel文件，并发送Email。 需求日事清S5跑团团长在每周日手工统计跑团成员打卡完成情况，制作excel，统计成员的小红花数量，这以人工处理方式比较耗时费神，且可能出错。 使用Python程序每周一0点1分定时执行处理，快速高效且无差错。 处理流程 抓取日事清S5跑团数据，获取成员每周的打卡情况 统计打卡数据，计算成员的小红花数量，生成excel 以附件的方式把excel文件发送Email给团长 代码结构 crontab：定时执行的crontab表达式 date.py：日期、时间工具类 rsq_request.py：日事清请求URL获取打卡数据 rsq_s5rt.py：主程序入口,获取打卡数据，生成excel、发送Email rsq_s5rt_start.sh：定时执行的sh脚本 send_email.py：发送Email translate_number.py：数字转换成中文汉字 write_excel.py：使用XlsxWriter操作excel 代码分析定时执行每周一0点1分定时执行rsq_s5rt_start.sh脚本123[root@node155 rsq_s5rt]# crontab -l#每周一0点1分执行1 0 * * 1 /opt/rsq_s5rt/rsq_s5rt_start.sh 执行的sh脚本在rsq_s5rt_start.sh脚本中执行rsq_s5rt.py主程序123[root@node155 rsq_s5rt]# cat rsq_s5rt_start.shcd /opt/rsq_s5rtpython rsq_s5rt.py 主程序1234567891011if __name__ == '__main__': #获取rishiqing数据 jsondata = rsq_request.get_s5rtdata() file_name = "S5-" + date.get_curdate2() + ".xlsx" #生成excel write_excel.write_excel(jsondata, file_name) #发送email send_email.send_email(file_name) 获取日事清跑团数据登录日事清，获取cookie1234567891011121314151617181920212223# 声明一个CookieJar对象实例来保存cookiecookie = cookielib.CookieJar()# 利用urllib2库的HTTPCookieProcessor对象来创建cookie处理器handler = urllib2.HTTPCookieProcessor(cookie)# 通过handler来构建openeropener = urllib2.build_opener(handler)postdata = urllib.urlencode(&#123; 'j_username': 'xxx@126.com', #用户名 'j_password': 'yyy', #密码 '_spring_security_remember_me': 'true'&#125;)# 登录rishiqing系统的URLloginUrl = 'https://www.rishiqing.com/task/j_spring_security_check'# 此处的open方法同urllib2的urlopen方法，也可以传入request# 模拟登录，并把cookie保存到变量result = opener.open(loginUrl, postdata)for item in cookie: cookie_res += item.name + "=" + item.value + "; "return cookie_res 将获取的cookie_res信息，放入headers中，发送S5跑团请求获取跑团数据12345678headers = &#123;"Accept": "application/json, text/plain, */*", "Content-Type": "application/json;charset=UTF-8", "Cookie": cookie_res&#125;# 查询S5跑团数据r = requests.get("https://www.rishiqing.com/task/v2/kanban/376768?_=1490239997146", headers=headers)jsondata = r.json()return jsondata 生成excel将获取的跑团数据，写入excel文件。 以当前时间所在年度的周，往前推5个星期，作为需要在excel中显示的统计数据的title 比如现在是第13周，那么统计的数据显示8-12周 每周的打卡情况：建卡并打卡+2个小红花，建卡未打卡+1个小红花，未建卡不计小红花 打卡是从第2周开始的，额外奖励第2周建卡并打卡+4个小红花，建卡未打卡+2个小红花 正则表达式匹配卡片名称，含有“周”并且前面有数字或者中文数字的，或者“W”后面有数字的，这些可以识别。 比如：“第12周”、“12周”、“第十二周”、“十二周”、“NO.12周”、“W12” 不能识别数字混搭的：“十2周”、“2十周”这种 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129def write_excel(jsondata, file_name): #获取当前时间在年度内的第几个星期 currweek = date.get_curr_week() currweek_num = int(currweek) s5ptname = jsondata['name'] kanbanCardList = jsondata['kanbanCardList'] workbook = xlsxwriter.Workbook(file_name) # 创建一个excel文件 worksheet = workbook.add_worksheet(s5ptname) # 创建一个excel文件的工作表 括号为空就是默认名 # add_format() 为当前workbook添加一个样式名为titleformat titleformat = workbook.add_format() titleformat.set_bold() # 设置粗体字 titleformat.set_font_size(10) # 设置字体大小为10 titleformat.set_font_name('Microsoft yahei') # 设置字体样式为雅黑 titleformat.set_align('center') # 设置水平居中对齐 titleformat.set_align('vcenter') # 设置垂直居中对齐 # 将titleformat应用在第一行，此行为标题 worksheet.set_row(0, None, titleformat) #加粗、红色字体、自动换行 format = workbook.add_format(&#123;'bold': True, 'font_color': 'red', 'text_wrap': True&#125;) worksheet.write('A1', s5ptname + '成员') worksheet.set_column('A:A', 14) #定义A列宽度为20 #只创建5-6列 num = 5 #从第2周开始创建全部 #num = currweek_num - 2 j = 0 dictA = &#123;&#125; dictB = &#123;&#125; #从字母b开始 for i in range(98,123): chr_str = chr(i).upper() week_name = "第" + str(currweek_num - num) + "周" worksheet.write(chr_str + '1', week_name) dictA.setdefault(chr_str, week_name) dictB.setdefault(week_name, chr_str) num -= 1 if num &lt; 0: j = i break str1 = chr(j).upper() week_name1 = '小红花✿' worksheet.write(str1 + '1', week_name1) worksheet.set_column(str1 + ':'+ str1, 20) #定义列宽度为20 str2 = chr(j+1).upper() week_name2 = '小红花总计' worksheet.write(str2 + '1', week_name2) row_num = 2 for kanbanCard in kanbanCardList: worksheet.write('A' + str(row_num), kanbanCard['name']) safflower_num = 0 safflower_label = "" kanbanItemList = kanbanCard['kanbanItemList'] for kanbanItem in kanbanItemList: week_plan = kanbanItem['name'] #正则表达式匹配'周'前面的数字 SEARCH_PAT = re.compile(r'(\d+)\s*' + u'周') #汉字先转换成unicode pat_search = SEARCH_PAT.search(week_plan) if pat_search != None: week_plan = pat_search.group(1) else: #正则表达式匹配'W'后面的数字 SEARCH_PAT = re.compile(r'W\s*(\d+)') pat_search = SEARCH_PAT.search(week_plan) if pat_search != None: week_plan = pat_search.group(1) #从第2周开始，到当前周 for i in range(2, currweek_num): #数字转换成中文汉字 cnnum = translate_number.translateNumber(i) week_cnnum = '第'+ str(i) + '周' col = dictB.get(week_cnnum) # 汉字前需要加'u'转换成Unicode if week_plan == str(i) or week_plan.find(u"第" + cnnum + u"周") != -1: #有周计划（建卡），小红花加1 safflower_num += 1 safflower_label += "✿" isDone = kanbanItem['isDone'] value = "" if isDone: #value = '✓' value = '✓✿✿' #有完成周计划（打卡），小红花加1 safflower_num += 1 safflower_label += "✿" # 第2周打卡奖励4个小红花 if i == 2: safflower_num += 4 safflower_label += "✿✿✿✿" value += '✿✿✿✿' else: #value = '未打卡' value = '?✿' # 第2周建卡未打卡奖励2个小红花 if i == 2: safflower_num += 2 safflower_label += "✿✿" value += '✿✿' if col: worksheet.write(col + str(row_num), value, format) #找到当前周，退出循环 break #小红花 worksheet.write(str1 + str(row_num), safflower_label, format) worksheet.write(str2 + str(row_num), safflower_num, format) row_num += 1 workbook.close() 发送Email将生成excel文件以附件方式发送Email 1234567891011121314151617181920212223242526272829303132333435363738394041def send_email(file_name): # 第三方 SMTP 服务 mail_host = "smtp.126.com" # 设置服务器 sender = 'xxx@126.com' #用户邮箱 mail_user = "xxx" # 用户名 mail_pass = "yyy" # 密码 #收件人邮箱 receiver = "aaa@qq.com" #收件人 creceiver = "bbb@126.com" #抄送人邮箱 receivers = "aaa@qq.com,bbb@126.com" #所有接收者邮箱 mail_msg = "S5跑团第"+ date.get_curr_week() +"周统计" + date.get_curdate2() #如名字所示Multipart就是分多个部分 message = MIMEMultipart() message['From'] = sender message['To'] = receiver message.add_header('Cc', creceiver) # 邮箱主题 message['Subject'] = Header(mail_msg, 'utf-8') #xlsx类型附件 part = MIMEApplication(open(file_name,'rb').read()) part.add_header('Content-Disposition', 'attachment', filename=file_name) message.attach(part) #发送5次，只要发送成功就退出 for i in range(0, 5): try: print "邮件发送第" + str(i+1) + "次" smtpObj = smtplib.SMTP() smtpObj.connect(mail_host, 25) # 25 为 SMTP 端口号 smtpObj.login(mail_user, mail_pass) smtpObj.sendmail(sender, receivers, message.as_string()) # 发送邮箱、接收邮箱、邮件内容 smtpObj.close() print "邮件发送成功" break except smtplib.SMTPException, e: print e print "Error: 无法发送邮件"]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[XlsxWriter介绍]]></title>
      <url>%2F2017%2F03%2F28%2Fxlsxwriter-1%2F</url>
      <content type="text"><![CDATA[XlsxWriter是一个Python模块，可用于将文本、数字、公式和超链接写入Excel 2007+ XLSX文件中的多个工作表。 它支持格式化等功能，包括： 100％兼容Excel XLSX文件 完整格式化 合并单元格 定义名称 图表 自动过滤器 数据验证和下拉列表 条件格式 工作表PNG / JPEG图像 丰富的多格式字符串 单元评论 文本框 与Pandas集成 用于编写大文件的内存优化模式 XlsxWriter的优缺点优点： 相比其他任何模块，支持更多的excel功能 它拥有一个与excel所产生的高程度的保真度。在大多数情况下的文件产生100%相当于excel文件所产生 它有广泛的文档，例如文件和测试 处理速度很快，即使对于非常大的输出文件也可以被配置非常小的内存 缺点： 无法读取或修改现有的Excel XLSX文件 安装XlsxWriter123456[root@node155 ~]# pip install xlsxwriterCollecting xlsxwriter Downloading XlsxWriter-0.9.6-py2.py3-none-any.whl (137kB) 100% |████████████████████████████████| 143kB 427kB/sInstalling collected packages: xlsxwriterSuccessfully installed xlsxwriter-0.9.6 常用方法说明Workbook类Workbook类创建一个XlsxWriter的Workbook对象,即创建一个excel文件. add_worksheet():用来创建工作表,默认为sheet1 add_format():创建一个新的格式对象来格式化单元格,例如 1bold=workbook.add_format(&#123;'bold':True&#125;) 还可以使用set_bold,例如: 123bold=workbook.add_format() bold.set_bold()#border：边框，align:对齐方式，bg_color：背景颜色，font_size：字体大小，bold：字体加粗top = workbook.add_format(&#123;'border':1,'align':'center','bg_color':'cccccc','font_size':13,'bold':True&#125;) 更多格式化样式的方法见http://xlsxwriter.readthedocs.org/working_with_formats.html add_chart(options):创建一个图表对象,内部是使用insert_chart()方法来实现的,options(dict类型)为图表指定一个字典属性close():关闭文件 Worksheet类worksheet代表一个Excel的工作表,是XlsxWriter的核心,下面是几个核心方法 write(row,col,args):写普通数据到工作表的单元格,row行坐标,col列坐标,起始都是以0开始,args为写入的内容,可以是字符串,文字,公式等,writer方法已经作为其它更具体数据类型方法的别名 write_string():写入字符串类型,worksheet.write_string(0,0,’your text’) write_number():写入数字类型,worksheet.write_number(‘A2’,1.1) write_blank():写入空类型数据,worksheet.write_blank(‘A2’,None) wirte_formula():写入公式类型,worksheet.write_formula(2,0,’=SUM(B1:B5)) write_datetime():写入日期类型数据, worksheet.write_datetime(7,0,datetime.datetime.strptime(‘2014-01-02’,’%Y-%m-%d),workbook.add_format({‘num_format’:’yyyy-mm-dd’})); write_boolean():写入逻辑类数据,worksheet.write_boolean(0,0,True) write_url():写入超链接类型数据,worksheet.write_url(‘A1’,’ftp://www.python.org’) write_column():写入到一列,后面接一个数组 wirte_row():写入到一行,后面接一个数组 set_row(row,height,cell_format,options):此方法设置行单元格的属性,row指定行位置,height指定高度,单位是像素,cell_format指定格式对象,参数options设置hiddeen(隐藏),level(组合分级),collapsed(折叠,例如:cell_format=workbook.add_format({‘bold’:True})worksheet.set_row(0,40,cell_format) 设置第一行高40,加粗 set_column(first_col,last_col,width,cell_format,options):设置列单元格的属性,具体参数说明如上.worksheet.set_column(0,1,10) worksheet.set_column(‘C:D’,20)insert_image(row,col,image[,options]):此方法是插入图片到指定单元格例如插入一个图片的超链接为www.python.orgworksheet.insert_image(‘B5’,’E:\hello.jpg’,{‘url’:’http://www.python.org&#39;}) 样式总结 类别 描述 属性 方法名 字体 字体 font_name set_font_name() 字体大小 font_size set_font_size() 字体颜色 font_color set_font_color() 加粗 bold set_bold() 斜体 italic set_italic() 下划线 underline set_underline() 删除线 font_strikeout set_font_strikeout() 上标/下标 font_script set_font_script() 数字 数字格式 num_format set_num_format() 保护 表格锁定 locked set_locked() 隐藏公式 hidden set_hidden() 对齐 水平对齐 align set_align() 垂直对齐 valign set_align() 旋转 rotation set_rotation() 文本包装 text_wrap set_text_warp() 底端对齐 text_justlast set_text_justlast() 中心对齐 center_across set_center_across 缩进 indent set_indent() 缩小填充 shrink set_shrink() 模式 表格模式 pattern set_pattern() 背景颜色 bg_color set_bg_color() 前景颜色 fg_color set_fg_color() 边框 表格边框 border set_border() 底部边框 bottom set_bottom() 上边框 top set_top() 右边框 right set_right() 边框颜色 border_color set_border_color() 底部颜色 bottom_color set_bottom_color() 顶部颜色 top_color set_top_color() 左边颜色 left_color set_left_color() 右边颜色 right_color set_right_color() 例子demo1新建hello.py文件，代码如下12345678import xlsxwriterworkbook = xlsxwriter.Workbook('hello.xlsx')worksheet = workbook.add_worksheet()worksheet.write('A1', 'Hello world')workbook.close() 1python hello.py demo2123456789101112131415161718workbook = xlsxwriter.Workbook(file_name) # 创建一个excel文件worksheet = workbook.add_worksheet(s5ptname) # 创建一个excel文件的工作表 括号为空就是默认名# add_format() 为当前workbook添加一个样式名为titleformattitleformat = workbook.add_format()titleformat.set_bold() # 设置粗体字titleformat.set_font_size(10) # 设置字体大小为10titleformat.set_font_name('Microsoft yahei') # 设置字体样式为雅黑titleformat.set_align('center') # 设置水平居中对齐titleformat.set_align('vcenter') # 设置垂直居中对齐# 将titleformat应用在第一行，此行为标题worksheet.set_row(0, None, titleformat)#加粗、红色字体、自动换行format = workbook.add_format(&#123;'bold': True, 'font_color': 'red', 'text_wrap': True&#125;)worksheet.write('A1', s5ptname + '成员')worksheet.set_column('A:A', 14) #定义A列宽度为20 文档最新的文档可以参考：https://xlsxwriter.readthedocs.io/contents.html PDF文档：https://github.com/jmcnamara/XlsxWriter/raw/master/docs/XlsxWriter.pdf]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[将整数数字转换成中文]]></title>
      <url>%2F2017%2F03%2F27%2Fpython-translatenum2cn%2F</url>
      <content type="text"><![CDATA[使用Python实现将整数数字转换成中文汉字,参考了JavaScript代码的实现原理。JavaScript源代码参见http://jsfiddle.net/calfzhou/tGEz7/，可以随便输入一些数字试试看。 Python代码1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253def translateNumber(num): CHINESE_NEGATIVE = "负" CHINESE_ZERO = "零" CHINESE_DIGITS = ["", "一", "二", "三", "四", "五", "六", "七", "八", "九"] CHINESE_UNITS = ["", "十", "百", "千"] CHINESE_GROUP_UNITS = ["", "万", "亿", "兆", "京", "垓", "杼", "穰", "溝", "澗", "正", "載", "極"] #数字0 if num == 0: return CHINESE_ZERO groupIsZero = True needZero = False result = "" strNum = str(num) #负数 if strNum[0] == "-": strNum = strNum[1:] result += CHINESE_NEGATIVE len1 = len(strNum) #从左边开始，循环当前数字的每一位 for i in range(0, len1): #当前数字位所在位置 position = len1 - 1 - i #当前数字位的值 digit = int(strNum[i]) #取模，返回余数 unit = position % len(CHINESE_UNITS) #除，返回整数 group = (position - unit) / len(CHINESE_UNITS) if digit != 0: if needZero: result += CHINESE_ZERO if (digit != 1 or unit != 1 or groupIsZero == False or (group == 0 and needZero)): result += CHINESE_DIGITS[digit] result += CHINESE_UNITS[unit] groupIsZero = groupIsZero and (digit == 0) if (unit == 0 and groupIsZero == False): result += CHINESE_GROUP_UNITS[group] needZero = (digit == 0 and (unit != 0 or groupIsZero)) if unit == 0: groupIsZero = True return result 例子例如1：12for num in range(-3, 15): print(translateNumber(num)) 输出123456789101112131415161718负三负二负一零一二三四五六七八九十十一十二十三十四 例如2：1print(translateNumber(203)) 输出1二百零三]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[hexo添加LiveRe评论支持]]></title>
      <url>%2F2017%2F03%2F24%2Fhexo-config-livere%2F</url>
      <content type="text"><![CDATA[多说将于2017年6月1日正式关停服务，不能使用了。所以评论换成LiveRe（来必力）。 注册 LiveRe进入LiveRe，注册账号。 安装City版本获取data-uidLiveRe 有两个版本： City 版：是一款适合所有人使用的免费版本 Premium 版：是一款能够帮助企业实现自动化管理的多功能收费版本 个人技术博客使用City版就可以了，输入网站地址、名称、类型，申请获取代码。 填写完成后，进入到 管理页面 -&gt; 代码管理 -&gt; 一般网站 代码中，data-uid即为next主题所需要的livere_uid。 next主题配置LiveRe在next主题_config.yml文件中，修改livere_uid的配置值为上一步获取到的data-uid。123# Support for LiveRe comments system.# You can get your uid from https://livere.com/insight/myCode (General web site)livere_uid: your uid 部署1hexo g -d]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[招标网站爬虫项目分析]]></title>
      <url>%2F2017%2F03%2F23%2Fscrapy-site-1%2F</url>
      <content type="text"><![CDATA[招标网站爬虫项目采用Scrapy框架，每天定时自动爬取当天通过检索关键字的招标信息。 需求由于市场与销售部门需要及时获知运营商、政府采购等相关行业的招标信息，参与招投标。故提出通过关键字检索，每天以Email的形式发送相关网站的招投标网页链接信息的需求。 Scrapy介绍Scrapy是Python语言开发的一个快速、高层次的屏幕抓取和web抓取框架，用于抓取web站点并从页面中提取结构化的数据。 使用Scrapy可以满足招标网站爬取的需求。 Scrapy主要包括了以下组件： 引擎(Scrapy Engine)用来处理整个系统的数据流处理, 触发事务(框架核心)。 调度器(Scheduler)用来接受引擎发过来的请求, 压入队列中, 并在引擎再次请求的时候返回. 可以想像成一个URL（抓取网页的网址或者说是链接）的优先队列, 由它来决定下一个要抓取的网址是什么, 同时去除重复的网址。 下载器(Downloader)用于下载网页内容, 并将网页内容返回给Spiders(Scrapy下载器是建立在twisted这个高效的异步模型上的)。 爬虫(Spiders)爬虫是主要干活的, 用于从特定的网页中提取自己需要的信息, 即所谓的实体(Item)。用户也可以从中提取出链接,让Scrapy继续抓取下一个页面。 项目管道(Item Pipeline)负责处理爬虫从网页中抽取的实体，主要的功能是持久化实体、验证实体的有效性、清除不需要的信息。当页面被爬虫解析后，将被发送到项目管道，并经过几个特定的次序处理数据。 下载器中间件(Downloader Middlewares)位于Scrapy引擎和下载器之间的框架，主要是处理Scrapy引擎与下载器之间的请求及响应。 爬虫中间件(Spider Middlewares)介于Scrapy引擎和爬虫之间的框架，主要工作是处理爬虫的响应输入和请求输出。 调度中间件(Scheduler Middewares)介于Scrapy引擎和调度之间的中间件，从Scrapy引擎发送到调度的请求和响应。 Scrapy爬取流程Scrapy运行流程大概如下： 引擎从调度器中取出一个链接(URL)用于接下来的抓取 引擎把URL封装成一个请求(Request)传给下载器 下载器把资源下载下来，并封装成应答包(Response) 爬虫解析Response 解析出实体（Item）,则交给实体管道进行进一步的处理 解析出的是链接（URL）,则把URL交给调度器等待抓取 招标网站爬虫实现招标网站爬虫实现过程如下： crontab每天23:00定时执行启动Scrapy 定义10个Spider，分别爬取各个网站当天发布的信息 在Pipeline中通过关键字过滤处理数据，并将数据分别写入到Spider的文本文件中存储 所有Spider爬取完成之后，读取所有的文本文件信息作为Email的内容，发送Email 创建scrapy项目在/opt目录下，创建scrapy_site项目1234567[root@node155 opt]# scrapy startproject scrapy_siteNew Scrapy project 'scrapy_site', using template directory '/usr/lib64/python2.7/site-packages/scrapy/templates/project', created in: /opt/scrapy_siteYou can start your first spider with: cd scrapy_site scrapy genspider example example.com 创建scrapy_site项目，自动生成以下目录文件 scrapy_site/ scrapy.cfg scrapy_site/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ... 这些文件分别是： scrapy.cfg: 项目的配置文件 scrapy_site/: 该项目的python模块 scrapy_site/items.py: 项目中的item文件 scrapy_site/pipelines.py: 项目中的pipelines文件 scrapy_site/settings.py: 项目的设置文件 scrapy_site/spiders/: 放置spider代码的目录 定义配置文件 email.conf:配置Email地址信息 格式为spidername1,spidername2,…===reciveremail1,reciveremail2,…===creciveremail1,creciveremail2,… 例如：b2b10086,bidding===xxx@126.com,yyy@126.com===zzz@qq.com 有10个spider，最多可配置将10个spider的信息发送email keyword.conf：配置检索的关键字信息 格式为spidernamexxx===关键字1,关键字2,… 例如：b2b10086===云数据,云计算 定义Item定义Item的字段：文章标题、链接、描述、发布时间12345class SiteItem(scrapy.Item): title = scrapy.Field() link = scrapy.Field() desc = scrapy.Field() pubtime = scrapy.Field() 定义pipelines定义事件，初始化获取关键字、数据存储文件等，1234567891011121314151617def __init__(self): dispatcher.connect(self.spider_opended, signals.spider_opened) dispatcher.connect(self.spider_closed, signals.spider_closed) dispatcher.connect(self.engine_stopped, signals.engine_stopped) dispatcher.connect(self.engine_started, signals.engine_started) # 获取当前目录，当前目录为scrapy_site项目的根目录 self.curpath = os.getcwd() #爬虫爬取的数据存储文件目录 self.spidername_filepath = self.curpath + "/scrapy_site/msg/" # 从配置文件keyword.conf获取关键字字典值 self.keywordsDict = dict() self.getKeywords() # 爬取信息 self.msgDict = dict() 处理Item数据，将爬取到的数据过滤，放到msgDict字典中存储12345678910111213141516171819202122232425def process_item(self, item, spider): if not item['pubtime'] or not item['title'] : return item # 去除换行与空格及[] pubtime = item['pubtime'].encode(const.ENCODE) title = item['title'].encode(const.ENCODE) if self.checkTilte(self.keywordsDict.get(spider.name), title) and date.get_curdate() == pubtime: msgArr = self.msgDict.get(spider.name) if msgArr is None: msgArr = &#123;&#125; msgArr['id'] = 0 msgArr['msg'] = "" msgArr['id'] += 1 msgArr['msg'] += str(msgArr['id']) msgArr['msg'] += '---' msgArr['msg'] += item['title'].encode(const.ENCODE) msgArr['msg'] += '---' msgArr['msg'] += item['link'].encode(const.ENCODE) msgArr['msg'] += '\n' print("msgArr['msg']=" + msgArr['msg']) self.msgDict.setdefault(spider.name, msgArr) return item 爬取完成之后，将数据存储到该spider文件中123456789def spider_closed(self, spider): file = open(self.spidername_filepath + spider.name, 'w') #将爬取信息写入爬虫文件中 if self.msgDict : msg = self.msgDict.get(spider.name) if msg : message = msg['msg'] file.write(message) file.close() 主程序run.py定义run.py文件作为scrapy_site项目的入口。 通过CrawlerRunner来运行所有的spider12345678910111213141516171819......settings = get_project_settings()configure_logging(settings)runner = CrawlerRunner(settings)# 运行所有的spiderfor spider_name in runner.spider_loader.list(): runner.crawl(spider_name)d = runner.join()d.addBoth(lambda _: reactor.stop())# 阻塞直到所有的spider完成reactor.run() # 发送Email...... 发送Email使用smtplib发送Email123456789101112131415161718192021222324252627282930313233343536def sendMail(receiver, creceiver): receivers = (receiver + ',' + creceiver).split(',') loginfo = "收件人：" + receiver + ",Cc：" + creceiver print loginfo _log.info(loginfo) temp = getMsgByEmail(receiver) mail_msg = temp[0] isHasHref = temp[1] message = MIMEText(mail_msg, 'html', 'utf-8') message['From'] = const.SENDER message['To'] = receiver # 如果邮件中含有链接信息则发送给收件人 if isHasHref: message.add_header('Cc', creceiver) else: message.add_header('Cc', const.SENDER) # 邮箱主题 subject = '招标网站最新信息' + time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(time.time())) message['Subject'] = Header(subject, 'utf-8') try: smtpObj = smtplib.SMTP() smtpObj.connect(const.MAIL_HOST, 25) # 25 为 SMTP 端口号 smtpObj.login(const.MAIL_USER, const.MAIL_PASS) if isHasHref: smtpObj.sendmail(const.SENDER, receivers, message.as_string()) # 发送邮箱、接收邮箱、邮件内容 else: smtpObj.sendmail(const.SENDER, const.SENDER, message.as_string()) # 发送邮箱、接收邮箱、邮件内容 print "邮件发送成功" except smtplib.SMTPException, e: _log.error('无法发送邮件:' + str(e)) print e print "Error: 无法发送邮件" 定义spider前面是基础的框架，不涉及具体的网站的爬取。 定义10个spider分别爬取： b2b10086_spider.py bidding_spider.py chinabidding_spider.py chinaunicombidding_spider.py csbidding_spider.py gdgpogov_spider.py gzsggzyjyzx_spider.py mssportal_spider.py telewiki_spider.py zycggov_spider.py 后续如果需要增加爬取网站，只需增加爬取网站的spider，修改email.conf与keyword.conf配置文件。 定时启动scrapy使用crontab定时启动scrapy_site10 23 * * * /opt/scrapy_site/scrapy_start.sh scrapy_start.sh文件内容123[root@node155 scrapy_site]# cat scrapy_start.shcd /opt/scrapy_site/python scrapy_site/run.py]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[修改dom0的内存]]></title>
      <url>%2F2017%2F03%2F16%2Fxenserver-vm-6%2F</url>
      <content type="text"><![CDATA[增大dom0的内存增大dom0的内存为4G，修改/boot/grub/grub.cfg或者/etc/grub.cfg文件，将“dom0_mem=1752M,max:1752M”修改为“dom0_mem=4096M,max:4096M” 1[root@node0001 ~]# vi /boot/grub/grub.cfg 重启主机1[root@node0001 ~]# reboot]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[虚拟机绑定物理CPU]]></title>
      <url>%2F2017%2F03%2F16%2Fxenserver-vm-5%2F</url>
      <content type="text"><![CDATA[VM绑定物理CPU设置VM使用物理CPU排序中的第1,2个CPU1[root@node200 ~]# xe vm-param-set uuid=247e8ddf-9361-4557-889c-f6c1961c6706 VCPUs-params:mask=1,2 设置VM中CPU的权重设置VCPU的权重为512。1[root@node200 ~]# xe vm-param-set uuid=247e8ddf-9361-4557-889c-f6c1961c6706 VCPUs-params:weight=512 默认dom0权重为256，权重决定VM在CPU时间片中所占比例，权重范围从0到65535。 设置VM最大CPU使用率设置VM最大可以使用的CPU为单个CPU的80%。1[root@node200 ~]# xe vm-param-set uuid=247e8ddf-9361-4557-889c-f6c1961c6706 VCPUs-params:cap=80 设置为100，则使用1个物理CPU；设置为80，则只能使用一个物理CPU的80%性能；设置为400，则表示最大可以使用4个物理CPU。默认为0，不限制。]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[虚拟机随物理主机开机自动启动]]></title>
      <url>%2F2017%2F03%2F16%2Fxenserver-vm-4%2F</url>
      <content type="text"><![CDATA[XenServer主机上的虚拟机默认是不会随物理主机开机自动启动的，是为了防止在HA环境中出现某些异常，所以物理主机异常断电恢复启动后，所有的虚拟机均为关机状态。但如果没配置HA，还是可以设置让虚拟机随物理主机开机自动启动。 设置pool的自动启动查看Pool的UUID123456[root@node200 ~]# xe pool-listuuid ( RO) : 53e9df09-bb99-3014-ffdf-3a781a9a84d2 name-label ( RW): xenserver200-201 name-description ( RW): master ( RO): 1e6bfa01-8785-49a9-b27e-351413463450 default-SR ( RW): 6e336c82-acc1-8858-1060-32cdc7016804 设置pool的自动启动1[root@node200 ~]# xe pool-param-set uuid=53e9df09-bb99-3014-ffdf-3a781a9a84d2 other-config:auto_poweron=true 设置虚拟机自动启动设置所有虚拟机随物理主机开机自动启动1[root@xenserver ~]# for i in `xe vm-list params=uuid --minimal|sed 's/,/ /g'`;do xe vm-param-set uuid=$i other-config:auto_poweron=true;done 如果只需要设置单台虚拟机随物理主机开机自动启动，则根据虚拟机的UUID来指定auto_poweron=true1[root@node200 ~]# xe vm-param-set uuid=107aa1fd-12ad-55b3-6a9e-3db624c45bf6 other-config:auto_poweron=true]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[判断虚拟机是否灵活]]></title>
      <url>%2F2017%2F03%2F16%2Fxenserver-vm-3%2F</url>
      <content type="text"><![CDATA[灵活的虚拟机必须满足下面三个条件： 虚拟机的所有虚拟磁盘必须置于共享存储中 虚拟机一定不能连接到配置的本地 DVD 驱动器 虚拟机的虚拟网络接口应位于池范围内的网络中 通过这三个条件统一来判断，只要有一项不满足，虚拟机就是不灵活的。另外可以直接使用xe diagnostic-vm-status uuid=[vm uuid] 命令判断虚拟机是否灵活。 灵活的虚拟机，最后显示VM is agile1234567891011121314151617181920212223242526272829303132333435[root@xenserver-8 ~]# xe diagnostic-vm-status uuid=7e2bdfa7-3b34-69c2-812e-6a36d3c080eauuid ( RO) : 7e2bdfa7-3b34-69c2-812e-6a36d3c080ea name-label ( RW): cy-windows power-state ( RO): running possible-hosts ( RO): 4ee4bd6c-57fd-45cb-b305-78caa61e4aca; 8f6ecd3d-8474-4208-9593-2e03e687249dChecking to see whether disks are attachableuuid ( RO) : cf507bfe-7369-15b0-3e3c-d4fe3727a7a3 vdi-uuid ( RO): &lt;not in database&gt; empty ( RO): true device ( RO): hdb userdevice ( RW): 1 mode ( RW): RO type ( RW): CD attachable ( RO): true storage-lock ( RO): falseuuid ( RO) : bd56ab95-5ac2-9a3c-c252-888bf06ca2e2 vdi-uuid ( RO): 8b79fae7-e488-4087-b787-b3a99fe282c5 empty ( RO): false device ( RO): hda userdevice ( RW): 0 mode ( RW): RW type ( RW): Disk attachable ( RO): true storage-lock ( RO): falseChecking to see whether VM can boot on each hostxenserver-6 : OK xenserver-8: OKVM is agile. 不灵活的虚拟机，最后显示VM is not agile，并提示不灵活原因是使用了本地存储。123456789101112131415161718192021222324252627282930313233343536[root@xenserver-8 ~]# xe diagnostic-vm-status uuid=e15312b4-6222-46e4-90fe-e0f13bb3b7b1uuid ( RO) : e15312b4-6222-46e4-90fe-e0f13bb3b7b1 name-label ( RW): test power-state ( RO): running possible-hosts ( RO): 4ee4bd6c-57fd-45cb-b305-78caa61e4aca; 8f6ecd3d-8474-4208-9593-2e03e687249dChecking to see whether disks are attachableuuid ( RO) : e73e640a-9d18-195a-9250-a359129d819c vdi-uuid ( RO): 055c23d2-405f-409c-a460-d5c1feced54b empty ( RO): false device ( RO): xvda userdevice ( RW): 0 mode ( RW): RW type ( RW): Disk attachable ( RO): true storage-lock ( RO): falseuuid ( RO) : 20c602f4-b70d-949d-9bda-016851923593 vdi-uuid ( RO): &lt;not in database&gt; empty ( RO): true device ( RO): xvdb userdevice ( RW): 1 mode ( RW): RO type ( RW): CD attachable ( RO): true storage-lock ( RO): falseChecking to see whether VM can boot on each hostxenserver-6 : Cannot start here [VM requires access to SR: cddf84c1-853d-8752-dbdf-ce8b17372b13 (Local storage)] xenserver-8: OKVM is not agile because: VM requires access to non-shared SR: cddf84c1-853d-8752-dbdf-ce8b17372b13 (Local storage). SR must both be marked as shared and a properly configured PBD must be plugged-in on every host]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[虚拟机进不了控制台或者挂住]]></title>
      <url>%2F2017%2F03%2F16%2Fxenserver-vm-2%2F</url>
      <content type="text"><![CDATA[在RHEL6、Centos6、Win2008R2、Win7以上操作系统的全虚VM，安装tools之后重启进不了控制台（半虚的VM正常），是因为VM参数platform:viridian设置为true，由于xapi会判断如果xs-tools是up-to-date，并且没有viridian，就会自动加上并设置true，需要手工修改platform:viridian的值为false。 解决方法查看VM的platform:viridian值为true1234567891011121314151617181920212223242526272829303132333435363738[root@xenserver-169 ~]# xe vm-param-list uuid=eae5ea6d-2b41-f390-626d-bf33c5f8ddcauuid ( RO) : eae5ea6d-2b41-f390-626d-bf33c5f8ddca name-label ( RW): wuyz1 name-description ( RW): Imported by XenServer Conversion Manager. user-version ( RW): 1 is-a-template ( RW): false is-a-snapshot ( RO): false snapshot-of ( RO): &lt;not in database&gt; snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z snapshot-info ( RO): parent ( RO): &lt;not in database&gt; children ( RO): is-control-domain ( RO): false power-state ( RO): running memory-actual ( RO): 2147450880 memory-target ( RO): 0 memory-overhead ( RO): 19922944 memory-static-max ( RW): 2147483648 memory-dynamic-max ( RW): 2147483648 memory-dynamic-min ( RW): 2147483648 memory-static-min ( RW): 16777216 suspend-VDI-uuid ( RW): &lt;not in database&gt; suspend-SR-uuid ( RW): &lt;not in database&gt; VCPUs-params (MRW): VCPUs-max ( RW): 1 VCPUs-at-startup ( RW): 1 actions-after-shutdown ( RW): Destroy actions-after-reboot ( RW): Restart actions-after-crash ( RW): Restart console-uuids (SRO): d5ec329c-d6ab-52be-ddb9-59d6c8496f84 platform (MRW): viridian: true; timeoffset: 0; apic: true; pae: true; acpi: true; stdvga: 0; nx: true allowed-operations (SRO): changing_dynamic_range; migrate_send; pool_migrate; changing_VCPUs_live; suspend; hard_reboot; hard_shutdown; clean_reboot; clean_shutdown; pause; checkpoint; snapshot current-operations (SRO): blocked-operations (MRW): allowed-VBD-devices (SRO): 1; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15 allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6 possible-hosts ( RO): 87e2c0c0-dff2-4770-839b-40b4ff90de32 将VM参数platform:viridian设置为false，重启VM。12[root@xenserver-169 ~]# xe vm-param-set platform:viridian=false uuid=eae5ea6d-2b41-f390-626d-bf33c5f8ddca[root@xenserver-169 ~]# xe vm-reboot uuid=eae5ea6d-2b41-f390-626d-bf33c5f8ddca force=true]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[Linux虚拟机安装、卸载XenServer Tools]]></title>
      <url>%2F2017%2F03%2F16%2Fxenserver-vm-1%2F</url>
      <content type="text"><![CDATA[XenServer下Linux虚拟机安装、卸载XenServer Tools。 安装先选择要安装Tools的虚拟机，在XenCenter上挂载xs-tools.iso进入要安装的虚拟机Console，先后执行下面命令： 123# mkdir /mnt/xs-tools# mount /dev/xvdd /mnt/xs-tools# /mnt/xs-tools/Linux/install.sh 如果是自定义模板安装的虚拟机，则如下：123# mkdir /mnt/xs-tools# mount /dev/cdrom /mnt/xs-tools# /mnt/xs-tools/Linux/install.sh reboot重启虚拟机，生效OK。 卸载12345[root@hvm ~]# rpm -qa | grep xe-guestxe-guest-utilities-6.2.0-1120.x86_64xe-guest-utilities-xenstore-6.2.0-1120.x86_64[root@hvm ~]# rpm -e xe-guest-utilities-6.2.0-1120.x86_64[root@hvm ~]# rpm -e xe-guest-utilities-xenstore-6.2.0-1120.x86_64]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[虚拟机挂载磁盘个数]]></title>
      <url>%2F2017%2F03%2F16%2Fxenserver-vdi-2%2F</url>
      <content type="text"><![CDATA[原来的计算方式虚拟机可以挂载磁盘设备为allowed-VBD-devices中的值，值的个数为还可以挂载的VDI个数（在XenCenter上虚拟机挂载VDI时也是通过这种方式来计算校验） 虚拟机挂载磁盘个数默认最大值为16（除去CD光驱就是15个） 如果虚拟机系统异常，获取不到操作系统、未安装tools等，则挂载磁盘个数最大值为4（除去CD光驱就是3个） 查看allowed-VBD-devices的值，虚拟机还可以挂载的VDI为4个，可用的vbd的userdevice分别为1,3,4,15123456789101112[root@xenserver-155 ~]# xe vm-param-list uuid=85db576d-d694-af0a-6430-267532168458uuid ( RO) : 85db576d-d694-af0a-6430-267532168458 name-label ( RW): lh-src name-description ( RW): user-version ( RW): 1 is-a-template ( RW): false is-a-snapshot ( RO): false snapshot-of ( RO): &lt;not in database&gt; snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z allowed-VBD-devices (SRO): 1,3,4,15 现在的挂载不以allowed-VBD-devices来创建，查找VM的VBD的userdevice，设置一个未被使用的值就可以了。这样就没有数量的限制。 使用CLI方式设置userdevice（对应命令行中的device）为7，给虚拟机与VDI创建vbd12[root@node200 ~]# xe vbd-create bootable=false device=7 mode=RW type=Disk unpluggable=true vdi-uuid=456a16a1-51e3-4cee-9eb4-a3b246078cc0 vm-uuid=107aa1fd-12ad-55b3-6a9e-3db624c45bf68a656533-c3f8-3ba2-6aa8-278eab6ac338 C#代码挂载VDI给虚拟机，只要userdevice未被虚拟机上的其他VBD使用，就可以挂载。12345678910VBD vbd = new VBD ();vbd.bootable = false;vbd.userdevice = &quot;6&quot;; //查找VM的VBD的userdevice，设置一个未被使用的值vbd.mode = vbd_mode.RW;vbd.type = vbd_type.Disk;vbd.unpluggable = true;vbd.VDI = VDI.get_by_uuid(session, &quot;8cc20e68-62eb-40bd-9af4-492f4e8c2ca9&quot; );vbd.VM = VM.get_by_uuid(session, &quot;dc600733-932d-0e77-e9bf-9deb2e4e5813&quot; );VBD.create(session, vbd);]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[虚拟机VDI扩容]]></title>
      <url>%2F2017%2F03%2F16%2Fxenserver-vdi-1%2F</url>
      <content type="text"><![CDATA[虚拟机的磁盘容量不够时，可以进行扩容操作。两种方式： 增加新的虚拟磁盘,比较简单，直接添加数据盘通过vdi-create、vbd-plug就可以。 扩展磁盘的大小，增大磁盘的容量。 下面以Centos7为例，扩展系统盘的大小从10G扩展到50G。 在XenServer主机上扩容虚拟磁盘扩展vdi的大小为50G1[root@xenserver155 ~]# xe vdi-resize uuid=13575f55-e55d-7f49-f22d-62427cb8fa22 disk-size=50GiB disk-size的单位可以为GiB、MiB，不带单位默认为B 在虚拟机上扩展磁盘分区增加磁盘分区增加一个大小为40G的新分区xvda31[root@controller161 ~]# fdisk /dev/xvda 依次输入n、w，其他的默认。创建一个分区xvda312345678910111213141516171819202122Command (m for help): nPartition type: p primary (2 primary, 0 extended, 2 free) e extendedSelect (default p):Using default response pPartition number (3,4, default 3):First sector (20971520-104857599, default 20971520):Using default value 20971520Last sector, +sectors or +size&#123;K,M,G&#125; (20971520-104857599, default 104857599):Using default value 104857599Partition 3 of type Linux and of size 40 GiB is setCommand (m for help): wThe partition table has been altered!Calling ioctl() to re-read partition table.WARNING: Re-reading the partition table failed with error 16: Device or resource busy.The kernel still uses the old table. The new table will be used atthe next reboot or after you run partprobe(8) or kpartx(8)Syncing disks. 使分区生效1[root@controller161 ~]# partprobe 或者1[root@controller161 ~]# reboot 创建pv创建pv12[root@controller161 ~]# pvcreate /dev/xvda3Physical volume "/dev/xvda3" successfully created 查看pv，xvda3的pv已经创建12345678910111213141516171819202122[root@controller161 ~]# pvdisplay --- Physical volume --- PV Name /dev/xvda2 VG Name centos PV Size 9.51 GiB / not usable 2.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 2434 Free PE 10 Allocated PE 2424 PV UUID kkBtLT-uY9U-7ChE-DikQ-PiPT-DNYL-oCvKID --- Physical volume --- PV Name /dev/xvda3 VG Name PV Size 40.00 GiB / not usable 4.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 10239 Free PE 0 Allocated PE 10239 PV UUID h24gB9-Toe2-Pq2J-9dse-S0nK-pJUC-oz5bdc 扩容vg查看扩容之前的vg123456789101112131415161718192021[root@controller161 ~]# vgdisplay --- Volume group --- VG Name centos System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 6 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 2 Max PV 0 Cur PV 2 Act PV 2 VG Size 9.50 GiB PE Size 4.00 MiB Total PE 12673 Alloc PE / Size 12663 / 9.46 GiB Free PE / Size 10 / 40.00 MiB VG UUID A9ge4D-F5u0-8zAS-Jspw-fWou-Q89d-EVOH0I 扩容pv到vg12[root@controller161 ~]# vgextend centos /dev/xvda3Volume group "centos" successfully extended 查看扩容之后的vg123456789101112131415161718192021[root@controller161 ~]# vgdisplay --- Volume group --- VG Name centos System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 6 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 2 Max PV 0 Cur PV 2 Act PV 2 VG Size 49.50 GiB PE Size 4.00 MiB Total PE 12673 Alloc PE / Size 12663 / 49.46 GiB Free PE / Size 10 / 40.00 MiB VG UUID A9ge4D-F5u0-8zAS-Jspw-fWou-Q89d-EVOH0I 扩容lv查看扩容前的lv12345678910111213141516171819202122232425262728293031323334[root@controller161 ~]# lvdisplay --- Logical volume --- LV Path /dev/centos/swap LV Name swap VG Name centos LV UUID o2UgDI-TFjU-WGIP-pZQq-XAD3-5SdQ-WbSYSR LV Write Access read/write LV Creation host, time localhost, 2016-06-28 10:35:23 +0800 LV Status available # open 2 LV Size 1.00 GiB Current LE 256 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:1 --- Logical volume --- LV Path /dev/centos/root LV Name root VG Name centos LV UUID BlUZvD-D8Pf-OC0b-Jrje-CT1g-u0QC-FJf3Eo LV Write Access read/write LV Creation host, time localhost, 2016-06-28 10:35:24 +0800 LV Status available # open 1 LV Size 8.46 GiB Current LE 12407 Segments 2 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:0 扩容lv1[root@controller161 ~]# lvextend /dev/centos/root /dev/xvda3 查看扩容之后的lv12345678910111213141516171819202122232425262728293031323334[root@controller161 ~]# lvdisplay --- Logical volume --- LV Path /dev/centos/swap LV Name swap VG Name centos LV UUID o2UgDI-TFjU-WGIP-pZQq-XAD3-5SdQ-WbSYSR LV Write Access read/write LV Creation host, time localhost, 2016-06-28 10:35:23 +0800 LV Status available # open 2 LV Size 1.00 GiB Current LE 256 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:1 --- Logical volume --- LV Path /dev/centos/root LV Name root VG Name centos LV UUID BlUZvD-D8Pf-OC0b-Jrje-CT1g-u0QC-FJf3Eo LV Write Access read/write LV Creation host, time localhost, 2016-06-28 10:35:24 +0800 LV Status available # open 1 LV Size 48.46 GiB Current LE 12407 Segments 2 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:0 查看分区的文件系统类型123456789101112131415[root@controller161 ~]# parted /dev/xvdaGNU Parted 3.1Using /dev/xvdaWelcome to GNU Parted! Type 'help' to view a list of commands.(parted) pModel: Xen Virtual Block Device (xvd)Disk /dev/xvda: 53.7GBSector size (logical/physical): 512B/512BPartition Table: msdosDisk Flags:Number Start End Size Type File system Flags1 1049kB 525MB 524MB primary xfs boot2 525MB 10.7GB 10.2GB primary lvm3 10.7GB 53.7GB 42.9GB primary lvm 扩展文件系统1[root@controller161 ~]# xfs_growfs /dev/mapper/centos-root centos6及以下的文件系统为ext3、ext4 ，执行resize2fs /dev/mapper/centos-root centos7默认文件系统为xfs，执行 xfs_growfs /dev/mapper/centos-root 查看容量变化原容量10G，扩容之后的大小为50G12345678910[root@controller161 ~]# df -hFilesystem Size Used Avail Use% Mounted on/dev/mapper/centos-root 49G 8.3G 41G 18% /devtmpfs 1.9G 0 1.9G 0% /devtmpfs 1.8G 0 1.8G 0% /dev/shmtmpfs 1.8G 8.5M 1.8G 1% /runtmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup/dev/xvda1 497M 169M 329M 34% /boottmpfs 354M 0 354M 0% /run/user/0tmpfs 354M 0 354M 0% /run/user/990]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[查看SR与VDI路径]]></title>
      <url>%2F2017%2F03%2F15%2Fxenserver-sr-4%2F</url>
      <content type="text"><![CDATA[查看SR与VDI在物理主机上的文件路径。SR类型： NFS与ISO LVM与HBA NFS与ISO查看NFS与ISO类型的SR，使用 df -h 命令，SR目录文件放在/var/run/sr-mount/[sr-uuid]下，子目录放置的是VDI文件123456789[root@xenserver153 sr-mount]# df -hFilesystem Size Used Avail Use% Mounted on/dev/sda1 3.9G 1.9G 1.9G 51% /tmpfs 1.6G 64K 1.6G 1% /dev/shm192.168.217.129:/WYY-NFS/07bec9d7-aa21-ddea-0bbd-517559759cf3 50G 7.1G 40G 16% /var/run/sr-mount/07bec9d7-aa21-ddea-0bbd-517559759cf310.10.152.1:/os_iso 372G 314G 40G 89% /var/run/sr-mount/f11c03b3-13b5-93b0-b2df-0f8c156336a210.0.0.25:/nfs/6225306e-bdc1-f6ab-8dbc-1e8bcf42d39b 197G 185G 2.3G 99% /var/run/sr-mount/6225306e-bdc1-f6ab-8dbc-1e8bcf42d39b 12345[root@xenserver153 sr-mount]# ls /var/run/sr-mount/6225306e-bdc1-f6ab-8dbc-1e8bcf42d39b/387d5bcf-26ac-4316-9600-7bdeacd67e46.vhd536fc728-05e7-4afa-80d6-cde8401dd5c2.vhdf168399e-86b7-47e3-b344-655446ccefd3.vhdfc457d9e-2ef4-4111-bcc4-6b3ab446e464.vhd LVM与HBA查看本地SR与HBA SR，使用 ls /dev/VG_XenStorage-[sr-uuid]下，子目录放置的是VDI文件1234567891011121314151617[root@xenserver-171 /]# ls /dev/VG_XenStorage-1a659635-c40c-95b9-4958-5afd1df4e0ebMGTVHD-17eb0cbb-e5e0-42d8-b566-c108021b95f6VHD-1ff5aa97-d289-4c5d-9168-e27261987098VHD-25296331-af1d-4539-b82b-059da6156978VHD-2b251f67-bc91-4b1d-a278-c07bc26bbd27VHD-4a6c24a5-f724-40a4-bf5b-0171086bf757VHD-636a3fff-3c90-41a8-b448-7e675858df54VHD-7d4fcfcc-96d6-4995-aa47-cd4c9ceae592VHD-860332e4-7d33-4052-bb38-feb0508ed7a5VHD-9164771a-cb81-4a77-a6b9-58f9c0180eafVHD-951ea29e-9407-4b9d-af5e-cdded140e707VHD-bdb7c50c-6167-4a9c-ae71-2cf63e229091VHD-d0e5ece1-d826-48a0-a273-866c084e0f68VHD-ebb9defa-49c3-4c60-b7ac-88efe1518f96VHD-ed9d01ad-ea2e-40f1-a21f-444cff043aedVHD-f9e07921-0a0a-438d-a87f-fb17249a07f5 LVM类型的本地SR，可以使用 lvs 命令来查看全部VDI1234567[root@xenserver-171 /]# lvs LV VG Attr LSize Origin Snap% Move Log Copy% Convert MGT VG_XenStorage-1a659635-c40c-95b9-4958-5afd1df4e0eb -wi-a- 4.00M VHD-0db56453-2039-4df6-bda9-04d387e74015 VG_XenStorage-1a659635-c40c-95b9-4958-5afd1df4e0eb -wi--- 24.05G VHD-16e5938c-7668-4715-bbce-109940da1af5 VG_XenStorage-1a659635-c40c-95b9-4958-5afd1df4e0eb -wi--- 24.05G VHD-17eb0cbb-e5e0-42d8-b566-c108021b95f6 VG_XenStorage-1a659635-c40c-95b9-4958-5afd1df4e0eb -ri-ao 5.78G VHD-1ff5aa97-d289-4c5d-9168-e27261987098 VG_XenStorage-1a659635-c40c-95b9-4958-5afd1df4e0eb -wi-ao 10.03G]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[卸载PBD失败 Error code: SR_BACKEND_FAILURE_202 Error parameters]]></title>
      <url>%2F2017%2F03%2F15%2Fxenserver-sr-3%2F</url>
      <content type="text"><![CDATA[后端存储问题，卸载PBD失败 Error code: SR_BACKEND_FAILURE_202 Error parameters: , General backend error [opterr=Command os.stat(/var/run/sr-mount/f9facf7f-501e-04ad-a136-37d500bd3a3c) failed (5): failed], 操作步骤在主机上probe sr，显示后端存储有问题12345678910111213141516171819202122[root@xenserver-161 ~]# xe sr-probe type=lvmoiscsi device-config:target=192.168.212.220 device-config:targetIQN=iqn.1992-08.com.netapp:sn.1936860767 host-uuid=d1e73513-e1d9-4850-93c9-241c6975e2a4There was an SR backend failure.status: non-zero exitstdout:stderr: Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 578, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 336, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 135, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 168, in load self._LUNprint(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 405, in _LUNprint self.iscsi.print_LUNs() File "/opt/xensource/sm/ISCSISR.py", line 621, in print_LUNs obj._query(vdi_path, LUNid) File "/opt/xensource/sm/LUNperVDI.py", line 41, in _query self.uuid = scsiutil.gen_uuid_from_string(scsiutil.getuniqueserial(path)) File "/opt/xensource/sm/scsiutil.py", line 77, in gen_uuid_from_string raise util.CommandException(1)util.CommandException: 1 找到已挂载的PBD12345678910111213[root@xenserver-161 ~]# xe pbd-list sr-uuid=f9facf7f-501e-04ad-a136-37d500bd3a3cuuid ( RO) : 43ab5ef1-6153-7cc8-5b8e-f632dbab9546 host-uuid ( RO): d1e73513-e1d9-4850-93c9-241c6975e2a4 sr-uuid ( RO): f9facf7f-501e-04ad-a136-37d500bd3a3c device-config (MRO): serverpath: /NFS1; server: 192.168.213.52; chappassword: ; chapuser: currently-attached ( RO): trueuuid ( RO) : 8c05886d-aa7d-5208-7c5c-1a9e04648d0b host-uuid ( RO): 37fba4cd-26bc-4477-880c-0268375da64c sr-uuid ( RO): f9facf7f-501e-04ad-a136-37d500bd3a3c device-config (MRO): serverpath: /NFS1; server: 192.168.213.52; chappassword: ; chapuser: currently-attached ( RO): false 卸载PBD123[root@xenserver-161 ~]# xe pbd-unplug uuid=43ab5ef1-6153-7cc8-5b8e-f632dbab9546Error code: SR_BACKEND_FAILURE_202Error parameters: , General backend error [opterr=Command os.stat(/var/run/sr-mount/f9facf7f-501e-04ad-a136-37d500bd3a3c) failed (5): failed], 解决方法使用umount命令卸载NFS存储池1[root@xenserver-161 ~]# umount -f /var/run/sr-mount/f9facf7f-501e-04ad-a136-37d500bd3a3c 执行pbd-unplug、pbd-destroy、sr-forget123[root@xenserver-161 ~]# xe pbd-unplug uuid=43ab5ef1-6153-7cc8-5b8e-f632dbab9546[root@xenserver-161 ~]# xe pbd-destroy uuid=43ab5ef1-6153-7cc8-5b8e-f632dbab9546[root@xenserver-161 ~]# xe sr-forget uuid=f9facf7f-501e-04ad-a136-37d500bd3a3c]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[XenServer存储池操作]]></title>
      <url>%2F2017%2F03%2F14%2Fxenserver-sr-2%2F</url>
      <content type="text"><![CDATA[存储池有4种状态：活动、未挂载、挂载异常、异常。 活动：存储池正常挂载在物理主机上面。共享存储池同时挂载给资源池里所有的物理主机。活动状态下可以执行卸载操作。 挂载异常：存储池与物理主机有PBD连接信息，但是未挂载到物理主机上，或者未同时挂载给资源池里所有物理主机。挂载异常状态下可以执行挂载、移除、删除操作。 未挂载：存储池与物理主机没有PBD连接信息。活动状态下的存储池进行了分离操作后出现该状态。未挂载状态下可以执行重新连接、移除、删除操作。 异常：存储池与主机有连接信息，并且挂载到主机上，但是存储池本身出现异常（后端存储LUN不存在、网络异常等）。可以通过xe sr-scan超时来判断。 存储池操作挂载存储池当前状态为“挂载异常”，即PBD的currently-attached=false，可以进行挂载操作，将PBD连接挂载到物理主机上。 如果没有PBD，先创建再挂载1xe pbd-create host-uuid=[HOST_UUID] sr-uuid=[SR_UUID] 挂载PBD1xe pbd-plug uuid=[PBD_UUID] 卸载存储池当前状态为“活动”，即PBD的currently-attached=true，可以进行卸载载操作，将PBD连接从物理主机上卸载。1xe pbd-unplug uuid=[PBD_UUID] 删除存储池当前状态为“挂载异常”或者“未挂载”，可以进行删除操作。存储池数据也会删除。 1xe sr-destroy uuid=[SR_UUID] 移除将存储池与物理主机的联系完全删除，在物理主机上无法看到该存储池。移除的存储池里面的数据会保留，但是虚拟磁盘与虚拟机的关联信息将被删除，磁盘数据还在，再次创建时可以重新连接或者格式化。 存储池当前状态为“挂载异常”、“未挂载”或者“异常”，可以进行移除操作。有挂载了PBD的，先卸载PBD。 有挂载了PBD的，先卸载PBD1xe pbd-unplug uuid=[PBD_UUID] 移除存储池1xe sr-forget uuid=[SR_UUID] 分离操作删除存储池与物理主机的PBD连接信息，使存储池无法访问。需要恢复，使用”重新连接”重新输入连接信息，才可以访问。存储池当前状态为“活动”、“挂载异常”，可以进行分离操作。存储池里面的虚拟磁盘如果有挂载给正在使用的虚拟机，则不可以进行分离操作。 卸载PBD1xe pbd-unplug uuid=[PBD_UUID] 删除PBD1xe pbd-destroy uuid=[PBD_UUID] 重新连接根据重新输入的路径信息，将存储池重新与物理主机建立起关联，使物理主机正常的访问存储池。存储池当前状态为“未挂载”，才可以进行重新连接。需要重新输入存储池的相应信息。例如：NFS类型需要重新输入存储路径、HBA类型需要重新输入目标LUN。 如果有PBD，先删除PBD1xe pbd-destroy uuid=[PBD_UUID] 创建PBD1xe pbd-create host-uuid=[HOST_UUID] sr-uuid=[SR_UUID] device-config:[key]=[value] 挂载PBD1xe pbd-plug uuid=[PBD_UUID] 格式化当移除iSCSI、HBA类型存储池之后，创建新存储池输入的路径与移除掉的存储池路径一致时，可以格式化原来的存储池数据。 重新引入存储池1xe sr-introduce shared=true type=[lvmiscsi/lvmohba] uuid=[SR_UUID] 创建PBD1xe pbd-create host-uuid=[HOST_UUID] sr-uuid=[SR_UUID] device-config:[key]=[value] 挂载PBD1xe pbd-plug uuid=[PBD_UUID] 挂载PBD之后，才能检测是否有VDI1xe vdi-list sr-uuid=[SR_UUID] 如果有VDI，删除VDI1xe vdi-destroy uuid=[VDI_UUID] 卸载PBD1xe pbd-unplug uuid=[PBD_UUID] 删除SR1xe sr-destroy uuid=[SR_UUID]]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[ISCSI操作总结]]></title>
      <url>%2F2017%2F03%2F14%2Fxenserver-sr-1%2F</url>
      <content type="text"><![CDATA[ISCSI操作总结。 增加ISCSI存储发现iscsi存储1iscsiadm -m discovery -t st -p &lt;ISCSI_IP&gt; 或者1iscsiadm -m discovery -t sendtargets -p &lt;ISCSI_IP&gt; 查看iscsi发现记录1iscsiadm -m node 登录iscsi存储1iscsiadm -m node -T &lt;TARGET_NAME&gt; -p &lt;ISCSI_IP&gt; -l 开机自动1iscsiadm -m node –T &lt;TARGET_NAME&gt; -p &lt;ISCSI_IP&gt; --op update -n node.startup -v automatic 删除iscsi存储登出iscsi存储1iscsiadm -m node -T &lt;TARGET_NAME&gt; -p &lt;ISCSI_IP&gt; -u 退出iscsi所有登录1iscsiadm -m node --logoutall=all 删除iscsi发现记录1iscsiadm -m node -o delete -T &lt;TARGET_NAME&gt; -p &lt;ISCSI_IP&gt; 登录需验证的iscsi开启认证1iscsiadm -m node -T &lt;TARGET_NAME&gt; -o update --name node.session.auth.authmethod --value=CHAP 添加用户1iscsiadm -m node -T &lt;TARGET_NAME&gt; --op update --name node.session.auth.username --value=[用户名] 添加密码1iscsiadm –m node –T &lt;TARGET_NAME&gt; –op update --name node.session.auth.password –-value=[密码] 查看登录的iscsi session1iscsiadm -m session 或者1iscsiadm --mode session 查看登录发现的所有iscsi1iscsiadm --mode node -l all 退出发现的所有iscsi1iscsiadm --mode node -u all]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[基于亲和与离散策略的XenServer资源调度负载均衡方案]]></title>
      <url>%2F2017%2F03%2F14%2Fxenserver-pool-2%2F</url>
      <content type="text"><![CDATA[弹性计算是根据业务需要在虚拟机中智能地分配合适的资源，保持业务的持续性和稳定性，从而实现资源池内各个物理主机的动态负载均衡，达到资源的充分利用。 弹性计算的实现采用贪心算法（贪婪算法），不从整体最优上加以考虑，而是某种意义上的局部最优解。 方案一设置一个阀值，只要资源池内的物理主机的cpu和mem不超过阀值，就不做调度。 亲和策略：可以把亲和的虚拟机当成一个整体VM组，考虑VM组整体的cpu和mem，放置在同一台主机或者同一个主机组。 离散策略：与亲和算法类似，不同的是放置虚拟机时，判断是否有离散（互斥）的VM，如果有的话，就不能放置在一起，那就选择下一台目标主机判断处理。 方案二设置一个阀值区间（即一个低阀值、一个高阀值），只要资源池内的物理主机cpu和mem在这个阀值区间内，就认为是负载均衡，不做调度。 算法一：局部贪心，局部贪心策略在每一次选择贪心选择时，只着眼于VM当前迁出或迁入能够得到的最优选择。 算法二、全局贪心算法，与局部贪心策略不同，全局贪心策略在选择一台迁出的VM时，也考虑了迁入这台物理主机对整个系统负载均衡情况的影响，根据当前负载情况，做出最优选择。 方案对比 方案一相对简单，相对负载均衡，可能从某种角度来说不是真正负载均衡。 方案二由于要考虑负载均衡，算法复杂，但可能出现来回动荡情况。 算法1：算法比2简单 算法2：整体考虑负载均衡，负载均衡的效果比1好，实现比较复杂 实现方案采用方案一。 规则调度弹性计算调度中的规则调度部分。 VM到VM规则 亲和：虚拟机应当放置在同一台主机或者同一个主机组 离散：虚拟机应当放置在不同的主机或者不同的主机组 VM到HOST规则 亲和：一组虚拟机放置到一组主机上 离散：一组虚拟机不能放置到一组主机上]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[XenServer组成异构资源池]]></title>
      <url>%2F2017%2F03%2F13%2Fxenserver-pool-1%2F</url>
      <content type="text"><![CDATA[由于现有硬件环境限制，或者后续硬件升级需要将不同CPU的物理主机组成XenServer资源池。默认情况下，异构的物理主机是不能组成资源池的，需要修改CPU的特征码，使用相同特征码的主机才可以强制组成资源池。 XenServer组成异构资源池两种情况： 只有两台主机的CPU不同：使用compare-cpu命令操作获取新的特征码 两台以上主机的CPU不同： 先两台主机compare-cpu命令操作获取新并设置的特征码，然后用新的特征码文件加一台主机执行同样的操作，直到获取最终的特征码； 对所有特征码使用“相与”计算 查看CPU信息两台主机，分别使用 xe host-cpu-info 命令查看cpu信息1234567891011121314[root@xenserver-179 ~]# xe host-cpu-infocpu_count : 16 socket_count: 2 vendor: AuthenticAMD speed: 2000.042 modelname: AMD Opteron(tm) Processor 6128 family: 16 model: 9 stepping: 1 flags: fpu de tsc msr pae mce cx8 apic mtrr mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx constant_tsc nonstop_tsc pni popcnt hypervisor features: 00802009-178bfbff-000837ff-efd3fbff features_after_reboot: 00802009-178bfbff-000837ff-efd3fbff physical_features: 00802009-178bfbff-000837ff-efd3fbff maskable: full 1234567891011121314[root@xenserver-167 ~]# xe host-cpu-infocpu_count : 8 socket_count: 2 vendor: GenuineIntel speed: 2500.072 modelname: Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz family: 6 model: 62 stepping: 4 flags: fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush mmx fxsr sse sse2 ht nx constant_tsc nonstop_tsc aperfmperf pni popcnt hypervisor arat features: 77bee3ff-bfebfbff-00000001-2c100800 features_after_reboot: 77bee3ff-bfebfbff-00000001-2c100800 physical_features: 77bee3ff-bfebfbff-00000001-2c100800 maskable: full 参数说明： features 当前 CPU 使用的 feature features_after_reboot 修改后重启使用的 feature physical_features CPU 物理上支持的 feature maskable：Full 表示该 CPU 硬件支持修改 FeatureMask 计算通过两台不同的CPU的features值，两者“相与”可以计算出COMMON MASK。 把前面xe host-cpu-info命令获取输出的所有cpu信息，在一台XenServer主机上分别保存成两个文件AMD6128.txt和E5-2609.txt 将compare-cpu脚本上传到主机上，进入compare-cpu所在的目录，赋权：chmod -R 777 compare-cpu12[root@xenserver-179 ~]# cd /usr/share/[root@xenserver-179 share]# chmod -R 777 compare-cpu 在compare-cpu所在的目录下输入：./compare-cpu AMD6128.txt E5-2609.txt -v123456789101112131415161718192021222324[root@xenserver-179 share]# ./compare-cpu AMD6128.txt E5-2609.txt -v file1:AMD6128.txt file2:E5-2609.txt pool_mask: ffffff7f-ffffffff-ffffffff-ffffffff CPU 1: model name: AMD Opteron(tm) Processor 6128 features: 00802009-178bfbff-000837ff-efd3fbff masking level: full CPU 2: model name: Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz features: 77bee3ff-bfebfbff-00000001-2c100800 masking level: full Result: CPU 1 and CPU 2 are compatible for masking Mask type: 3 - CPU 1 and CPU 2 have a mutually exclusive set of features but support a common mask Mask: 00802009-178BFBFF-00000001-2C100800Traceback (most recent call last): File "./compare-cpu", line 279, in ? exit(0)TypeError: 'str' object is not callable 得出common mask为：00802009-178BFBFF-00000001-2C100800 修改features在每台XenServer上执行一下命令：1[root@xenserver-179 ~]# xe host-set-cpu-features features=00802009-178BFBFF-00000001-2C100800 uuid=【主机UUID】 然后重启每台主机。 强制组成资源池强制入池强制加入资源池，在主机上执行命令：1[root@xenserver-119 ~]# xe pool-join master-address=192.168.212.153 master-username=root master-password=xen123 force=true 如果强制入池，输出一下信息：You attempted an operation that was not allowed.reason: Network backends differ需要修改网络堆栈。 查看主机网络堆栈查看network.conf配置12[root@xenserver-119 ~]# cat /etc/xensource/network.confbridge 或者查看host参数12[root@xenserver-119 ~]# xe host-list params=software-versionsoftware-version (MRO) : product_version: 2.0.0; platform_name: xenserver; platform_version: 2.0.0; product_brand: xenserver; build_number: B00; hostname: localhost.localdomain; date: 2014-11-01; xapi: 1.3; xen: 4.1.6.1; linux: 2.6.32.43-0.4.1.citrix.2.0.0.xen; xencenter_min: 2.0; xencenter_max: 2.0; network_backend: bridge; xenserver:main: Base Pack, version 2.0.0, build B000 查看资源池内主机网络堆栈12[root@xenserver share]# cat /etc/xensource/network.confopenvswitch 1234[root@xenserver share]# xe host-list params=software-versionsoftware-version (MRO) : product_version: 2.0.0; platform_name: xenserver; platform_version: 2.0.0; product_brand: xenserver; build_number: B00; hostname: localhost.localdomain; date: 2014-11-01; xapi: 1.3; xen: 4.1.6.1; linux: 2.6.32.43-0.4.1.citrix.2.0.0.xen; xencenter_min: 2.0; xencenter_max: 2.0; network_backend: openvswitch; xenserver:main: Base Pack, version 2.0.0, build B000software-version (MRO) : product_version: 2.0.0; platform_name: xenserver; platform_version: 2.0.0; product_brand: xenserver; build_number: B00; hostname: localhost.localdomain; date: 2014-11-01; xapi: 1.3; xen: 4.1.6.1; linux: 2.6.32.43-0.4.1.citrix.2.0.0.xen; xencenter_min: 2.0; xencenter_max: 2.0; network_backend: openvswitch; xenserver:main: Base Pack, version 2.0.0, build B000 将当前主机修改成与资源池其他主机相同12345[root@xenserver-119 ~]# xe-switch-network-backend openvswitchCleaning up old ifcfg filesEnabling openvswitch daemonConfigure system for openvswitch networkingYou *MUST* now reboot your system 重启主机1[root@xenserver-119 ~]# reboot compare-cpu脚本获取common mask的compare-cpu脚本（Python代码）123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279#! /usr/bin/python## compare-cpu## Determines if the specified xe host-cpu-info results # are compatible for XenServer CPU masking## Copyright (c) 2010 Citrix Systems, Inc.## This program is free software; you can redistribute it and/or modify# it under the terms of the GNU Lesser General Public License as published# by the Free Software Foundation; version 2.1 only. with the special# exception on linking described in file LICENSE.## This program is distributed in the hope that it will be useful,# but WITHOUT ANY WARRANTY; without even the implied warranty of# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the# GNU Lesser General Public License for more details.#def findValue(filename,value): file_handle = open(filename, 'r') file_line = file_handle.readline() while file_line != "": start = file_line.find(value) if start != -1: # Remove any EOL chars if file_line.find("\n") != -1: file_line = file_line[0:file_line.find("\n")].rstrip() # +2 skips the colon and space return file_line[start+len(value)+2:len(file_line)] file_line = file_handle.readline() return ""def printFmt(output1, output2): pad_max=22 print output1.rjust(pad_max), output2def compareCPUs(a_features, a_mask_level, b_features, b_mask_level, pool_mask): mask = "" compare_results = ["CPU 1 and CPU 2 have identical feature sets, no masking required", "CPU 1 can be masked to CPU 2", "CPU 1 cannot be masked to CPU 2"] # verify if a mask is required at all if a_features == b_features: return 0, compare_results[0] # remove separating dashes a_features2 = a_features.replace("-","") b_features2 = b_features.replace("-","") pool_mask2 = pool_mask.replace("-","") for i in range(0, 32, 16): # Convert desired portion of the features string to an int a_int = int(a_features2[i:i+16],16) b_int = int(b_features2[i:i+16],16) p_int = int(pool_mask2[i:i+16],16) # Apply pool mask # Don't overwrite original a and b feature values - they are needed later because # pool mask only applies in the evaluation of the mask on join and therefore it # should not be included in the actual joint mask to be returned a_p_int = a_int &amp; p_int b_p_int = b_int &amp; p_int # Evaluate the first half if i==0: # If masking needed if a_p_int != b_p_int: # But not supported return not compatible if a_mask_level == "no": return 2, compare_results[2] # Masking needed, supported; verify if a can be masked to b elif a_p_int &amp; b_p_int != b_p_int: # No common base mask return 2, compare_results[2] # Store the first half of the required mask with formatting mask = featureIntToString(a_int &amp; b_int) mask += "-" # Evaluate the second half if i==16: # If masking needed if a_p_int != b_p_int: # But not supported return not compatible if a_mask_level != "full": return 2, compare_results[2] # Masking needed, supported; verify a can be masked to b elif a_p_int &amp; b_p_int == b_p_int: # Both halves can be masked OK mask += featureIntToString(a_int &amp; b_int) return 1, compare_results[1], mask else: # No common extended mask return 2, compare_results[2] else: # Only first half must be masked, mask OK mask += featureIntToString(a_int &amp; b_int) return 1, compare_results[1], maskdef findJointMask(a_features, a_mask_level, b_features, b_mask_level, pool_mask): joint_mask = "" joint_mask_results = ["CPU 1 does not support masking", "CPU 2 does not support masking", "CPU 1 does not support extended masking", "CPU 2 does not support extended masking", "CPU 1 and CPU 2 support a common mask"] # remove separating dashes a_features2 = a_features.replace("-","") b_features2 = b_features.replace("-","") pool_mask2 = pool_mask.replace("-","") for i in range(0, 32, 16): # Convert desired portion of the features string to an int a_int = int(a_features2[i:i+16],16) b_int = int(b_features2[i:i+16],16) p_int = int(pool_mask2[i:i+16],16) # Apply pool mask # Don't overwrite original a and b feature values - they are needed later because # pool mask only applies in the evaluation of the mask on join and therefore it # should not be included in the actual joint mask to be returned a_p_int = a_int &amp; p_int b_p_int = b_int &amp; p_int joint_mask_half_int = (a_p_int &amp; b_p_int) # Evaluate the first half if i==0: # if a needs to be masked, verify it can be if joint_mask_half_int != a_p_int and a_mask_level != "base" and a_mask_level != "full": return 0, joint_mask_results[0] # if b needs to be masked, verify it can be if joint_mask_half_int != b_p_int and b_mask_level != "base" and b_mask_level != "full": return 1, joint_mask_results[1] # Evaluate the second half if i==16: # if a needs to be masked, verify it can be if joint_mask_half_int != a_p_int and a_mask_level != "full": return 2, joint_mask_results[2] # if b needs to be masked, verify it can be if joint_mask_half_int != b_p_int and b_mask_level != "full": return 3, joint_mask_results[3] # Store the half with formatting joint_mask += featureIntToString(a_int &amp; b_int) if i==0: joint_mask += "-" return 4, joint_mask_results[4], joint_maskdef featureIntToString(features): # Removes the 0x, removes the L, pads with 0s to 16 bytes, and adds hyphens featureString = hex(features).lstrip("0x").rstrip("L").rjust(16,"0") featureString = featureString[0:8] + "-" + featureString[8:16] return featureStringimport sysfrom optparse import OptionParser,OptionGroup# Parse cmd line argumentsparser=OptionParser(usage="usage: %prog -v -p pool_mask file1 file2 | -c features1 mask_level1 features2 mask_level2")more_info=OptionGroup(parser, "More information", "The values specified for the file1 and file2 arguments must contain the output from the xe host-cpu-info command on the joining host and existing pool master respectively. If using the -c option, specify the physical_features and maskable values from xe host-cpu-info on the joining host and existing pool master as the features1/mask_level1 and features2/mask_level2 arguments respectively.")parser.add_option_group(more_info) parser.add_option("-p", "--pool_mask", help="An optional 32 byte hex value for use as a general CPU mask, equivalent to pool.other-config:cpuid_feature_mask", default="ffffff7f-ffffffff-ffffffff-ffffffff")parser.add_option("-v", "--verbose", help="Display verbose output", action="store_true", dest="verbose")parser.add_option("-c", "--cmdline", help="Use feature and mask values specified on the command line rather than from a file", action="store_true", dest="cmdline", default=False)(options, args) = parser.parse_args()# Validate argument counts and filenames if specifiedif options.cmdline: if len(args) !=4: parser.error("Incorrect number of arguments. The -c option requires 4 arguments.") exit(1)else: if len(args) != 2: parser.error("Incorrect number of arguments. 2 arguments required.") exit(1) for i in args: try: file_handle = open(i, 'r') except IOError: print print "File %s does not exist" % (i) print exit(1) else: file_handle.close# Parse/retrieve argument valuesif options.cmdline: features = [args[0], args[2]] mask_levels = [args[1], args[3]]else: features = [findValue(args[0], "physical_features"), findValue(args[1], "physical_features")] mask_levels = [findValue(args[0], "maskable"), findValue(args[1], "maskable")] modelnames = [findValue(args[0], "modelname"), findValue(args[1], "modelname")]# Validate and display valuesif options.verbose and not options.cmdline: print printFmt("file1:", args[0]) printFmt("file2:", args[1])if options.verbose: if options.cmdline: print printFmt("pool_mask:", options.pool_mask) if options.verbose: for i in [0,1]: print printFmt("CPU %s:" % (i+1), "") if not options.cmdline: printFmt("model name:", modelnames[i]) printFmt("features:", features[i]) printFmt("masking level:", mask_levels[i])# Evaluate masking compatibilityresults = compareCPUs(features[0], mask_levels[0], features[1], mask_levels[1], options.pool_mask)printif results[0] == 0: # Identical feature sets, no masking required printFmt("Result:", results[1]) printFmt("Mask type:", "n/a") printFmt("Mask:", "n/a")elif results[0] == 1: # CPU 1 can be masked to CPU 2 (use case 1) printFmt("Result:", "CPU 1 and CPU 2 are compatible for masking") printFmt("Mask type:","1 - CPU 1 has a superset of features to CPU 2") printFmt("Mask:","%s" % results[2])else: results = findJointMask(features[0], mask_levels[0], features[1], mask_levels[1], options.pool_mask) # if joint mask is possible if results[0] == 4: # and joint mask is the same as joining features if results[2] == features[0]: # CPU 2 can be masked to CPU 1 (use case 2) printFmt("Result:", "CPU 1 and CPU 2 are compatible for masking") printFmt("Mask type:","2 - CPU 1 has a subset of features to CPU 2") printFmt("Mask:",results[2]) else: # joint mask required on all hosts (use case 3) printFmt("Result:", "CPU 1 and CPU 2 are compatible for masking") printFmt("Mask type:","3 - CPU 1 and CPU 2 have a mutually exclusive set of features but support a common mask") printFmt("Mask:",results[2]) else: # joint mask required but not supported printFmt("Result:", "CPU 1 and CPU 2 are not compatible for masking because %s" % results[1])printexit(0) Java代码获取featuresJava代码获取features的common mask1234567891011121314151617181920212223242526272829303132333435363738394041424344private String getNewFeatures(List&lt;String&gt; featureList) &#123; // 特征码的第一部分 String feature1 = null; // 特征码的第二部分 String feature2 = null; // 特征码的第三部分 String feature3 = null; // 特征码的第四部分 String feature4 = null; for (String feature : featureList) &#123; String[] currentFeatures = feature.split( "-"); if (feature1 == null) &#123; feature1 = currentFeatures[0]; feature2 = currentFeatures[1]; feature3 = currentFeatures[2]; feature4 = currentFeatures[3]; continue; &#125; // 将之前的结果与当前特征码进行"与"计算 feature1 = bitAnd(feature1, currentFeatures[0]); feature2 = bitAnd(feature2, currentFeatures[1]); feature3 = bitAnd(feature3, currentFeatures[2]); feature4 = bitAnd(feature4, currentFeatures[3]); &#125; String newFeature = feature1 + "-" + feature2 + "-" + feature3 + "-" + feature4; return newFeature; &#125; private String bitAnd(String feature1, String feature2) &#123; Long decimalBit1 = Long. valueOf(feature1, 16); Long decimalBit2 = Long. valueOf(feature2, 16); Long bit1AndBit2 = decimalBit1 &amp; decimalBit2; String result = Long. toHexString(bit1AndBit2); // 不足8位的在前面补充0 StringBuffer zeroSb = new StringBuffer(); for (int i = 0; i &lt; 8 - result.length(); i++) &#123; zeroSb.append( "0"); &#125; return zeroSb.toString() + result; &#125;]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[XenServer切换master主机]]></title>
      <url>%2F2017%2F03%2F13%2Fxenserver-host-5%2F</url>
      <content type="text"><![CDATA[XenServer切换master主机。 禁用HA如果资源池启用了HA，先禁用HA，否则跳过此步骤。1[root@xenserver ~]# xe pool-ha-disable 显示主机列出资源池内的物理主机1234567891011121314[root@xenserver ~]# xe host-listuuid ( RO) : 29fb4d42-258f-4ce1-a81b-519479eb1f3e name-label ( RW): xenserver155 name-description ( RW): Default install of xenserveruuid ( RO) : ba2fcb36-6788-4cc4-90bc-dceb7d3dd2d0 name-label ( RW): xenserver-119 name-description ( RW): Default install of xenserveruuid ( RO) : cc6f3970-ddb3-4c87-8487-efc034ce3b6b name-label ( RW): xenserver153 name-description ( RW): Default install of xenserver 切换新的master使用 xe pool-designate-new-master host-uuid=[host-uuid]，切换新的master为xenserver155主机。1[root@xenserver ~]# xe pool-designate-new-master host-uuid=29fb4d42-258f-4ce1-a81b-519479eb1f3e]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[XenServer主机远程开关机]]></title>
      <url>%2F2017%2F03%2F13%2Fxenserver-host-4%2F</url>
      <content type="text"><![CDATA[XenServer远程开关机有以下几种方式： WoL(Wake-on-LAN): 网络唤醒 IPMI(Intelligent Platform Management Interface): 智能型平台管理接口 iLO(HP Integrated Light-Out): To use this option, iLO must be enabled on the host and be connected to the network. For more information, see HP’s iLO documentation. DRAC(Dell Remote Access Controller): To use this option, the Dell supplemental pack must be installed on the host server to get DRAC support. For more information, see Dell’s DRAC documentation. WoL与IPMI的方式使用较多。 WoLWake-on-LAN简称WOL或WoL，我们一般将其称为“网络唤醒”、“远端唤醒”技术。WOL是一种技术，同时也是该技术的规范标准。Wake-on-LAN功能需要有主板和网卡的支持，在主板BIOS中的网卡设置中必须有“Wake On LAN”设置（开启：On），并且相应网卡也得支持该功能。 主板BIOS设置WoL在主板BIOS中的网卡设置“Wake On LAN”（开启：On） 查看网卡是否支持ethtool命令查看网卡是否支持 Wake on Lan12345678910111213141516171819[root@localhost lhd]# ethtool eth0Settings for eth0:Supported ports: [ TP MII ]Supported link modes: 10baseT/Half 10baseT/Full100baseT/Half 100baseT/FullSupports auto-negotiation: YesAdvertised link modes: 10baseT/Half 10baseT/Full100baseT/Half 100baseT/FullAdvertised auto-negotiation: YesSpeed: 100Mb/sDuplex: FullPort: MIIPHYAD: 32Transceiver: internalAuto-negotiation: onSupports Wake-on: pumbgWake-on: dCurrent message level: 0x00000007 (7)Link detected: yes 可以看到，ethtool把网卡的信息全部列出，我们只关心其中的这两项:Supports Wake-on: pumbgWake-on: d如果 wake-on 一项值 值为 d,表示禁用wake on lan 值为 g,表示启用 wake on lan 开启网卡唤醒如果支持，但是没开启可以设置开启1[root@localhost lhd]# ethtool -s eth0 wol g 或者修改配置永久生效1[root@localhost lhd]# echo '/usr/sbin/ethtool -s eth0 wol g' &gt;&gt; /etc/rc.d/rc.local 配置XenServer主机设置启动模式：1xe host-set-power-on-mode host=[host-uuid] power-on-mode=wake-on-lan 启动XenServer主机资源池主节点上，启动其他XenServer主机1xe host-power-on host=[host-uuid] IPMIIPMI的核心是一个专用芯片/控制器(叫做服务器处理器或基板管理控制器(BMC))，其并不依赖于服务器的处理器、BIOS或操作系统来工作，是一个单独在系统内运行的无代理管理子系统，只要有BMC与IPMI固件其便可开始工作，而BMC通常是一个安装在服务器主板上的独立的板卡。 主板BIOS设置IPMI主板BIOS设置”IPMI Over LAN”为On 网络、用户设置两种方式来设置： BIOS设置 进入IPMI Parameters，设置服务器ip/子网掩码 进入LAN User Confuguration，设置用户名、密码 ipmitool工具 配置XenServer主机设置启动模式为IPMI：1xe host-set-power-on-mode host=[host-uuid] power-on-mode=IPMI power-on-config:power_on_ip=[ip] power-on-config:power_on_user=[user] power-on-config:power_on_password=[password] 如果需要加密密码，则指定power_on_password_secret 启动、关闭、重启XenServer主机资源池主节点上，远程启动、关闭、重启其他XenServer主机123ipmitool -l lan -H [ip] -U [user] -P [password] power off （关机）ipmitool -l lan -H [ip] -U [user] -P [password] power on （开机）ipmitool -l lan -H [ip] -U [user] -P [password] power reset（重启）]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[XenServer iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)]]></title>
      <url>%2F2017%2F03%2F13%2Fxenserver-host-3%2F</url>
      <content type="text"><![CDATA[XenServer主机登陆ISCSI服务器出错：1234[root@xenserver119 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.56vg -p 192.168.212.104 --loginLogging in to [iface: default, target: iqn.2006-01.com.openfiler:tsn.56vg, portal: 192.168.212.104,3260]iscsiadm: Could not login to [iface: default, target: iqn.2006-01.com.openfiler:tsn.56vg, portal: 192.168.212.104,3260]:iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure) 是由于使用CHAP方式，设置了用户密码。 解决方法登录ISCSI时，需要开启CHAP认证，添加用户密码（用户名与密码使用配置的用户名与密码） 发现ISCSI1iscsiadm -m discovery -t st -p [ISCSI_IP] ISCSI_IP:ISCSI服务端IP 例如：123[root@xenserver119 ~]# iscsiadm -m discovery -t st -p 192.168.212.104192.168.212.104:3260,1 iqn.2006-01.com.openfiler:tsn.vg2192.168.212.104:3260,1 iqn.2006-01.com.openfiler:tsn.56vg 开启认证1iscsiadm -m node -T [Target_name] -o update --name node.session.auth.authmethod --value=CHAP 例如：1[root@xenserver119 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.56vg -o update --name node.session.auth.authmethod --value=CHAP 添加用户1iscsiadm -m node -T [Target_name] --op update --name node.session.auth.username --value=[用户名] 例如：1[root@xenserver119 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.56vg --op update --name node.session.auth.username --value=root 添加密码1iscsiadm –m node –T [Target_name] –op update –name node.session.auth.password –-value=[密码] 例如：1[root@xenserver119 ~]# iscsiadm –m node –T iqn.2006-01.com.openfiler:tsn.56vg –op update –name node.session.auth.password –-value=root 登陆ISCSI1iscsiadm -m node -T [Target_name] -p [ISCSI_IP] -l 列如：123[root@xenserver119 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.56vg -p 192.168.212.104 -lLogging in to [iface: default, target: iqn.2006-01.com.openfiler:tsn.56vg, portal: 192.168.212.104,3260]Login to [iface: default, target: iqn.2006-01.com.openfiler:tsn.56vg, portal: 192.168.212.104,3260]: successful]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[XenServer主机上设置ISCSI IQN]]></title>
      <url>%2F2017%2F03%2F13%2Fxenserver-host-2%2F</url>
      <content type="text"><![CDATA[XenServer主机上设置ISCSI IQN。 查看IQN查看ISCSI IQN的值。如果资源池内主机的IQN重复了，可以直接修改/etc/iscsi/initiatorname.iscsi的IQN123[root@xenserver-155 ~]# cat /etc/iscsi/initiatorname.iscsiInitiatorName=iqn.2015-02.com.example:a50ca784InitiatorAlias=xenserver-155 设置IQN设置ISCSI IQN的值1[root@xenserver-155 ~]# xe host-param-set other-config:iscsi_iqn=iqn.2015-02.com.example:a50ca784 uuid=3e184f09-0aaf-4c7b-ac70-bb4b10ddcd23]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[XenServer主机执行xe命令出错：Lost connection to the server]]></title>
      <url>%2F2017%2F03%2F10%2Fxenserver-host-1%2F</url>
      <content type="text"><![CDATA[在三台主机组成的资源池中，没有了master节点，执行xe命令出错：Lost connection to the server 三台主机IP： 192.168.222.181 192.168.222.182 192.168.222.183 查看pool配置文件查看pool配置文件，183为主节点123456[root@macro-222-181 ~]# cat /etc/xensource/pool.confslave:192.168.222.183[root@macro-222-182 ~]# cat /etc/xensource/pool.confslave:192.168.222.183[root@macro-222-183 ~]# cat /etc/xensource/pool.confslave:192.168.222.183 解决方法在183上禁用HA1[root@macro-222-183 ~]# xe host-emergency-ha-disable --force 设置183为主节点12[root@macro-222-183 ~]# xe pool-emergency-transition-to-masterHost agent will restart and transition to master in 10.000 seconds... 通知其他从节点123[root@macro-222-183 ~]# xe pool-recover-slavesb8deaa20-cfe9-4a54-86c8-f18aecdc48d4bf2faa7f-6f50-46d9-8043-f24b41a69464 禁用HA1[root@macro-222-183 ~]# xe pool-ha-disable]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[hexo博客绑定域名]]></title>
      <url>%2F2017%2F03%2F09%2Fhexo-domain%2F</url>
      <content type="text"><![CDATA[注册域名并绑定GitHub博客。 域名注册在腾讯云注册hl10502.com域名并认证 DNS解析添加CNAME类型的DNS记录: CNAME: @ hl10502.github.io. CNAME: www hl10502.github.io. 主机记录@： 主机记录www： 添加CNAME文件hexo源码source目录下增加CNAME文件，不带后缀。 文件内容为 www.hl10502.com]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[XenServer HA could not be enabled on the Pool because a liveset could not be formed: check storage and network heartbeat paths]]></title>
      <url>%2F2017%2F03%2F09%2Fxenserver-ha-4%2F</url>
      <content type="text"><![CDATA[XenServer资源池两台主机，启用HA失败： HA could not be enabled on the Pool because a liveset could not be formed: check storage and network heartbeat paths 两台XenServer主机组成一个资源池： 192.168.200.4 192.168.200.5 日志分析查看主机192.168.200.4的xha日志1234567891011121314151617181920212223242526272829[root@xenserver04 ~]# tail -f /var/log/xha.log......Dec 12 09:51:01 CST 2014 [debug] SM: SF domain is updated [sfdomain = (@0)].Dec 12 09:51:40 CST 2014 [debug] SM: heartbeat is readyDec 12 09:51:40 CST 2014 [debug] SM: statefile is readyDec 12 09:51:40 CST 2014 [debug] SM: other hosts are readyDec 12 09:51:40 CST 2014 [info] Start Criteria: pool state=INIT; excluded=FALSE; liveset=EMPTYDec 12 09:52:40 CST 2014 [notice] Start Criteria: Start timeout (0x5)Dec 12 09:52:40 CST 2014 [err] Start timeout (1901) Cannot start HA daemon, because Start Criteria is not met for the local host. - Abort.Dec 12 09:52:40 CST 2014 [debug] SM: SF domain is updated [sfdomain = (D0)].Dec 12 09:52:41 CST 2014 [notice] HA daemon started shutdown process.Dec 12 09:52:41 CST 2014 [info] SM: sm_initialize(-1).Dec 12 09:52:41 CST 2014 [info] SC: hostweight_initialize(-1).Dec 12 09:52:41 CST 2014 [info] SC: script_initialize(-1).Dec 12 09:52:41 CST 2014 [info] Xapimon: xapimon_initialize(-1).Dec 12 09:52:41 CST 2014 [info] HB: hb_initialize(-1).Dec 12 09:52:41 CST 2014 [info] WD: (watchdog_close) label=heartbeat stopping watchdog timer.Dec 12 09:52:41 CST 2014 [info] WD: (watchdog_close) label=heartbeat watchdog timer has been stopped successfully.Dec 12 09:52:41 CST 2014 [debug] WD: watchdog id file /var/run/xhad.wd.id is updated.Dec 12 09:52:41 CST 2014 [info] BM: bm_initialize(-1).Dec 12 09:52:41 CST 2014 [info] LM: lm_initialize(-1).Dec 12 09:52:41 CST 2014 [info] WD: (watchdog_close) label=statefile stopping watchdog timer.Dec 12 09:52:41 CST 2014 [info] WD: (watchdog_close) label=statefile watchdog timer has been stopped successfully.Dec 12 09:52:41 CST 2014 [debug] WD: watchdog id file /var/run/xhad.wd.id is deleted.Dec 12 09:52:41 CST 2014 [info] COM: com_initialize(-1).Dec 12 09:52:41 CST 2014 [notice] HA daemon completed shutdown process....... 查看主机192.168.200.5的xha日志1234567891011121314151617181920212223[root@xenserver05 ~]# tail -f /var/log/xha.log......Dec 12 09:51:06 CST 2014 [notice] HA daemon started - built at Jun 14 09:15:40 EDT 2013 - Dec 12 09:51:06 CST 2014 [info] CONF: my_index=1 num_host=2 t1=7 T1=60 t2=7 T2=60 Wh=60 Ws=75 Tboot=120 Tenable=120 tXapi=60 TXapi=120 RestartXapi=1 TRestartXapi=300 Tlicence=30Dec 12 09:51:06 CST 2014 [info] LOG: logmask = 100000Dec 12 09:51:06 CST 2014 [info] LOG: OFF:18(DUMPPACKET)Dec 12 09:51:06 CST 2014 [info] LOG: OFF:19(TRACE)Dec 12 09:51:06 CST 2014 [info] LOG: ON :20(FH_TRACE)Dec 12 09:51:06 CST 2014 [info] LOG: OFF:21(LM_TRACE)Dec 12 09:51:06 CST 2014 [info] LOG: OFF:22(SCRIPT)Dec 12 09:51:06 CST 2014 [info] LOG: OFF:23(SC_WARNING)Dec 12 09:51:06 CST 2014 [info] COM: com_initialize(0).Dec 12 09:51:06 CST 2014 [info] SF: phase 0 initialization...Dec 12 09:51:06 CST 2014 [info] LM: lm_initialize(0).Dec 12 09:51:06 CST 2014 [info] BM: bm_initialize(0).Dec 12 09:51:06 CST 2014 [info] HB: hb_initialize(0).Dec 12 09:51:06 CST 2014 [err] HB: cannot bind socket address (IP address = 192.168.200.5:694). (sys 98)Dec 12 09:51:06 CST 2014 [info] BM: bm_initialize(-1).Dec 12 09:51:06 CST 2014 [info] LM: lm_initialize(-1).Dec 12 09:51:06 CST 2014 [info] COM: com_initialize(-1)....... 解决方法HA启动的守护进程使用固定端口694。通过日志分析，发现主机192.168.200.5上的端口694被占用。 执行命令 netstat -naup 查看端口，rpc.statd服务占用了694端口。12345678910[root@xenserver05 ~]# netstat -naupActive Internet connections (servers and established)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name udp 0 0 0.0.0.0:111 0.0.0.0:* 17758/portmap udp 0 0 192.168.200.5:123 0.0.0.0:* 26166/ntpd udp 0 0 127.0.0.1:123 0.0.0.0:* 26166/ntpd udp 0 0 0.0.0.0:123 0.0.0.0:* 26166/ntpd udp 0 0 0.0.0.0:58882 0.0.0.0:* - udp 0 0 0.0.0.0:691 0.0.0.0:* 17899/rpc.statd udp 0 0 0.0.0.0:694 0.0.0.0:* 17899/rpc.statd kill进程1[root@xenserver05 ~]# kill -9 17899 启用ha1[root@xenserver05 ~]# xe pool-ha-enable heartbeat-sr-uuids=285aae43-6838-09af-ad3d-afc2d75d2d1a 启用成功，再次查看，xhad使用了694端口123456789[root@xenserver05 ~]# netstat -naupActive Internet connections (servers and established)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name udp 0 0 0.0.0.0:111 0.0.0.0:* 17758/portmap udp 0 0 192.168.200.5:123 0.0.0.0:* 26166/ntpd udp 0 0 127.0.0.1:123 0.0.0.0:* 26166/ntpd udp 0 0 0.0.0.0:123 0.0.0.0:* 26166/ntpd udp 0 0 0.0.0.0:58882 0.0.0.0:* - udp 0 0 192.168.200.5:694 0.0.0.0:* 26346/xhad rpc.statdrpc.statd是NFS服务，这个进程实现了网络状态监控(NSM)RPC协议,通知NFS客户端什么时候一个NFS服务器非正常重启动.这个进程被nfslock服务自动启动，不需要用户的配置。 12rpcinfo -p localhost #查看本机的rpc信息cat /etc/sysconfig/nfs #查看NFS配置信息 要想关掉RHEL/CentOS的rpc.statd服务，我们只需要执行下面的命令就能搞定1service nfslock stop 为了不让下次自动启动，那就执行下面的命令1chkconfig nfslock off 停止rpcbind服务的命令1service portmap stop 禁止下次开机自动启动命令1chkconfig portmap off]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[XenServer HA Internal error: Xapi_ha.Xha_error(4)]]></title>
      <url>%2F2017%2F03%2F09%2Fxenserver-ha-3%2F</url>
      <content type="text"><![CDATA[XenServer资源池启用HA出现内部错误 Internal error: Xapi_ha.Xha_error(4) 这是因为ha的心跳盘有问题：ha_start_daemon: the HA daemon stopped without forming a liveset (4) 日志分析查看日志出错信息1234567891011121314151617[root@xenserver119 ~]# tail -f /var/log/xensource.log...Apr 30 09:36:23 xenserver119 /opt/xensource/bin/xapi: [debug|xenserver119|40339 UNIX /var/xapi/xapi|host.ha_join_liveset R:f424a3fb1e8f|helpers] /opt/xensource/xha/ha_start_daemon exited with code 4 [stdout = ''; stderr = 'Thu Apr 30 09:36:23 CST 2015 ha_start_daemon: the HA daemon stopped without forming a liveset (4) ']Apr 30 09:36:23 xenserver119 /opt/xensource/bin/xapi: [ warn|xenserver119|40339 UNIX /var/xapi/xapi|host.ha_join_liveset R:f424a3fb1e8f|xapi_ha] /opt/xensource/xha/ha_start_daemon returned MTC_EXIT_WATCHDOG_ERROR (Watchdog error)Apr 30 09:36:23 xenserver119 /opt/xensource/bin/xapi: [debug|xenserver119|40339 UNIX /var/xapi/xapi|host.ha_join_liveset R:f424a3fb1e8f|backtrace] Raised at xapi_ha.ml:62.8-24 -&gt; xapi_ha.ml:1040.20-50 -&gt; xapi_host.ml:591.1-36 -&gt; message_forwarding.ml:233.25-44 -&gt; rbac.ml:229.16-23Apr 30 09:36:23 xenserver119 /opt/xensource/bin/xapi: [debug|xenserver119|40339 UNIX /var/xapi/xapi|host.ha_join_liveset R:f424a3fb1e8f|backtrace] Raised at rbac.ml:238.10-15 -&gt; server_helpers.ml:79.11-41Apr 30 09:36:23 xenserver119 /opt/xensource/bin/xapi: [debug|xenserver119|40339 UNIX /var/xapi/xapi|host.ha_join_liveset R:f424a3fb1e8f|dispatcher] Server_helpers.exec exception_handler: Got exception INTERNAL_ERROR: [ Xapi_ha.Xha_error(4) ]Apr 30 09:36:23 xenserver119 /opt/xensource/bin/xapi: [debug|xenserver119|40339 UNIX /var/xapi/xapi|host.ha_join_liveset R:f424a3fb1e8f|dispatcher] Raised at string.ml:150.25-34 -&gt; stringext.ml:108.13-29Apr 30 09:36:23 xenserver119 /opt/xensource/bin/xapi: [debug|xenserver119|40339 UNIX /var/xapi/xapi|host.ha_join_liveset R:f424a3fb1e8f|backtrace] Raised at string.ml:150.25-34 -&gt; stringext.ml:108.13-29Apr 30 09:36:23 xenserver119 /opt/xensource/bin/xapi: [debug|xenserver119|40339 UNIX /var/xapi/xapi|host.ha_join_liveset R:f424a3fb1e8f|xapi] Raised at server_helpers.ml:94.14-15 -&gt; pervasiveext.ml:22.2-9Apr 30 09:36:23 xenserver119 /opt/xensource/bin/xapi: [debug|xenserver119|40339 UNIX /var/xapi/xapi|host.ha_join_liveset R:f424a3fb1e8f|xapi] Raised at pervasiveext.ml:26.22-25 -&gt; pervasiveext.ml:22.2-9Apr 30 09:36:23 xenserver119 /opt/xensource/bin/xapi: [debug|xenserver119|40339 UNIX /var/xapi/xapi|dispatch:host.ha_join_liveset D:646ea3d5749a|xapi] Raised at pervasiveext.ml:26.22-25 -&gt; pervasiveext.ml:22.2-9Apr 30 09:36:23 xenserver119 /opt/xensource/bin/xapi: [debug|xenserver119|40339 UNIX /var/xapi/xapi|dispatch:host.ha_join_liveset D:646ea3d5749a|backtrace] Raised at pervasiveext.ml:26.22-25 -&gt; server_helpers.ml:140.10-106 -&gt; server.ml:12966.23-171 -&gt; server_helpers.ml:119.4-7Apr 30 09:36:23 xenserver119 /opt/xensource/bin/xapi: [debug|xenserver119|40236|Async.pool.enable_ha R:697af26c5213|backtrace] Raised at sExprParser.ml:109.34-89 -&gt; parsing.ml:138.39-75 -&gt; parsing.ml:160.4-28Apr 30 09:36:23 xenserver119 /opt/xensource/bin/xapi: [error|xenserver119|40236|Async.pool.enable_ha R:697af26c5213|xapi_ha] Caught exception while calling Host.ha_join_liveset: 'xenserver119' ('OpaqueRef:4231a807-42ad-0cd9-fc8f-09136c61f60d') INTERNAL_ERROR: [ Xapi_ha.Xha_error(4) ]Apr 30 09:36:23 xenserver119 /opt/xensource/bin/xapi: [debug|xenserver119|40236|Async.pool.enable_ha R:697af26c5213|backtrace] Raised at hashtbl.ml:93.19-28 -&gt; debug.ml:144.36-65... 解决方法找到名称为”Metadata for HA”、”Statefile for HA”的两个HA检测盘123456789101112131415161718[root@xenserver119 ~]# xe vdi-list name-label="Metadata for HA"uuid ( RO) : e40806a1-1958-4583-9f87-8e95ed9650ba name-label ( RW): Metadata for HA name-description ( RW): Used for master failover sr-uuid ( RO): 6e336c82-acc1-8858-1060-32cdc7016804 virtual-size ( RO): 268435456 sharable ( RO): true read-only ( RO): false[root@xenserver119 ~]# xe vdi-list name-label="Statefile for HA"uuid ( RO) : ce50b8fc-1286-4861-bd3d-2bca2404f22d name-label ( RW): Statefile for HA name-description ( RW): Used for storage heartbeating sr-uuid ( RO): 6e336c82-acc1-8858-1060-32cdc7016804 virtual-size ( RO): 4194304 sharable ( RO): true read-only ( RO): false 删除HA检测盘12[root@xenserver119 ~]# xe vdi-destroy uuid=e40806a1-1958-4583-9f87-8e95ed9650ba[root@xenserver119 ~]# xe vdi-destroy uuid=ce50b8fc-1286-4861-bd3d-2bca2404f22d 重新启用HA，同时指定作为信号检测的存储池1[root@xenserver119 ~]# xe pool-ha-enable heartbeat-sr-uuids=6e336c82-acc1-8858-1060-32cdc7016804]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[XenServer HA Internal error: Xapi_ha.Xha_error(15)]]></title>
      <url>%2F2017%2F03%2F09%2Fxenserver-ha-2%2F</url>
      <content type="text"><![CDATA[XenServer资源池启用HA出现内部错误 Internal error: Xapi_ha.Xha_error(15) 这是因为xha进程已经启动了，启用失败。 日志分析查看日志出错信息12345678910[root@node200 ~]# tail -f /var/log/xensource.log...Nov 6 14:24:12 jtt-cloud-node1 xapi: [debug|jtt-cloud-node1|1754404|Async.pool.enable_ha R:d2ea42a29c9d|helpers] /opt/xensource/xha/ha_set_pool_state init exited with code 15 [stdout = ''; stderr = 'Wed Nov 6 14:24:12 CST 2013 Could not set the pool state to INIT because the HA daemon may be present (15) ']Nov 6 14:24:12 jtt-cloud-node1 xapi: [ warn|jtt-cloud-node1|1754404|Async.pool.enable_ha R:d2ea42a29c9d|xapi_ha] /opt/xensource/xha/ha_set_pool_state init returned MTC_EXIT_DAEMON_IS_PRESENT (Daemon is (already) present)Nov 6 14:24:12 jtt-cloud-node1 xapi: [debug|jtt-cloud-node1|1754404|Async.pool.enable_ha R:d2ea42a29c9d|backtrace] Raised at xapi_ha.ml:62.8-24 -&gt; xapi_ha.ml:1450.20-60Nov 6 14:24:12 jtt-cloud-node1 xapi: [debug|jtt-cloud-node1|1754404|Async.pool.enable_ha R:d2ea42a29c9d|xapi_ha] Caught exception while enabling HA: INTERNAL_ERROR: [ Xapi_ha.Xha_error(15) ]Nov 6 14:24:12 jtt-cloud-node1 xapi: [debug|jtt-cloud-node1|1754404|Async.pool.enable_ha R:d2ea42a29c9d|backtrace] Raised at xapi_ha.ml:1539.8-11 -&gt; threadext.ml:20.20-24 -&gt; threadext.ml:20.62-65 -&gt; rbac.ml:229.16-23Nov 6 14:24:12 jtt-cloud-node1 xapi: [debug|jtt-cloud-node1|1754404|Async.pool.enable_ha R:d2ea42a29c9d|backtrace] Raised at rbac.ml:238.10-15 -&gt; server_helpers.ml:79.11-41Nov 6 14:24:12 jtt-cloud-node1 xapi: [debug|jtt-cloud-node1|1754404|Async.pool.enable_ha R:d2ea42a29c9d|dispatcher] Server_helpers.exec exception_handler: Got exception INTERNAL_ERROR: [ Xapi_ha.Xha_error(15) ]... 启用HA过程启用HA时，创建心跳盘，会初始化pool的状态，检测是否启动ha守护进程，如果已经存在则会报错： Could not set the pool state to INIT because the HA daemon may be present 不存在，就会启动xha进程，监控池内主机是否发生心跳信号（存储检测、网络检测） 解决方法查找xhad进程，进程存在会生成一个/etc/xensource/xhad.conf配置文件，记录HA的心跳检测、主机等配置信息123[root@node200 ~]# ps aux |grep xhadroot 8167 0.8 0.4 426616 17100 ? SLl 11:08 0:01 xhad /etc/xensource/xhad.confroot 12219 0.0 0.0 112652 976 pts/3 S+ 11:10 0:00 grep --color=auto xhad kill进程1[root@node200 ~]# kill -9 8167 重新启用HA，同时指定作为信号检测的存储池1[root@node200 ~]# xe pool-ha-enable heartbeat-sr-uuids=6e336c82-acc1-8858-1060-32cdc7016804]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[XenServer HA]]></title>
      <url>%2F2017%2F03%2F09%2Fxenserver-ha-1%2F</url>
      <content type="text"><![CDATA[HA（High Availability）：XenServer高可用是一套自动化功能，旨在针对导致XenServer主机停机或者无法访问的问题（例如因物理原因造成网络连接中断或者主机硬件发生故障）制定计划并进行安全地恢复。 在XenServer资源池配置受保护虚拟机、启用高可用之后，高可用守护进程发送心跳信息，监听池内主机是否出现网络或者存储故障。通过选举算法选择一个或者一组故障主机，故障主机通过底层的主机保护机制（hypervisor-level fencing）自动重启来保护数据安全。在故障主机上受保护的虚拟机自动在其他正常的主机上启动，达到保护虚拟机正常运行的目的。 HA启用条件 XenServer资源池建议资源池内至少包含3个XenServer主机，且每个主机都是使用静态IP地址 共享存储池一个大小至少为356MB或更大的iSCSI或光纤通道LUN作为检测信号的存储池，创建两个磁盘： 4 MB 检测信号卷：用于检测信号 256 MB 元数据卷：存储池主服务器元数据，以便在主服务器故障转移时使用 为最大程度提高可靠性，建议使用专用的iSCSI存储，且不得用于任何其他用途。 灵活的虚拟机受保护的虚拟机必须是灵活的。 虚拟机的虚拟磁盘必须置于共享存储中 虚拟机一定不能连接到配置的本地 DVD 驱动器 虚拟机的虚拟网络接口应位于池范围内的网络中 启用过程设置启用HA的参数，创建两个磁盘，一个4M的“Statefile for HA”，用来检测网络信号，一个256M的“MetaData for HA”用来存储元数据信息，初始化资源池的状态，检测是否启动HA守护进程，如果未启动则启动Xha进程，实时监控池内物理主机。 可用性检测可用性检测分为：心跳盘检测和存储盘检测。 心跳盘检测检测基于socket通信，不是ping。各个物理主机通过管理网络发送UDP数据包。UDP是在传输层，面向无连接的用户数据包协议，ICMP是Internet控制报文协议，面向连接的网络层，ping基于ICMP。心跳盘上记录的是物理主机之间通讯的信息。 存储盘检测存储池主服务器元数据，以便在主服务器故障转移时使用，所有主机可见的和可写的一个共享SR的VDI。存储盘上存放各物理主机的虚拟机列表，它包含着虚拟机的CPU预留及内存开销信息，master主机可以访问资源池内所有的虚拟机的信息。 启用HA查看虚拟机是否灵活的虚拟机，灵活的虚拟机才受高可用保护。1xe diagnostic-vm-status uuid=&lt;VM_UUID&gt; VM_UUID：虚拟机UUID 例如：UUID为0ec2a592-29ad-efe1-6ac7-94cafb50d37e的虚拟机为灵活的虚拟机123456789101112131415161718192021222324252627282930313233343536[root@node200 ~]# xe diagnostic-vm-status uuid=0ec2a592-29ad-efe1-6ac7-94cafb50d37euuid ( RO) : 0ec2a592-29ad-efe1-6ac7-94cafb50d37e name-label ( RW): CentOS 6 power-state ( RO): running possible-hosts ( RO): 1e6bfa01-8785-49a9-b27e-351413463450; 7a7998f3-64d9-45df-8538-589e3f32192fChecking to see whether disks are attachableuuid ( RO) : 8a50eb50-37a2-6c6d-7ac5-be4a9318b40c vdi-uuid ( RO): bb0d3098-aea1-45b6-bcde-dfd7cbc34c63 empty ( RO): false device ( RO): xvda userdevice ( RW): 0 mode ( RW): RW type ( RW): Disk attachable ( RO): true storage-lock ( RO): falseuuid ( RO) : 816665c0-9f19-a16d-7ed5-97dfe377e454 vdi-uuid ( RO): &lt;not in database&gt; empty ( RO): true device ( RO): userdevice ( RW): 3 mode ( RW): RO type ( RW): CD attachable ( RO): true storage-lock ( RO): falseChecking to see whether VM can boot on each hostxenserver-200 : OK xenserver-201: OKVM is agile. 根据虚拟机重启优先级，计算资源池允许最大故障数1xe pool-ha-compute-hypothetical-max-host-failures-to-tolerate vm-uuid=&lt;VM_UUID&gt; restart-priority=&lt;restart/best-effort/""&gt; 例如：12[root@node200 ~]# xe pool-ha-compute-hypothetical-max-host-failures-to-tolerate vm-uuid=0ec2a592-29ad-efe1-6ac7-94cafb50d37e restart-priority=restart1 设置受保护的VM1[root@node200 ~]# xe vm-param-set uuid=&lt;VM_UUID&gt; ha-restart-priority=&lt;restart/best-effort/""&gt; ha-always-run=&lt;true/false&gt; VM_UUID：虚拟机UUID restart/best-effort/&quot;&quot;：重启/资源满足时重启/不重启 true/false：是否总是运行状态 例如：1[root@node200 ~]# xe vm-param-set uuid=0ec2a592-29ad-efe1-6ac7-94cafb50d37e ha-restart-priority=restart ha-always-run=true 获取资源池当前配置下，允许的最大故障主机数故障主机超过这个数目，资源池将不足运行池内受保护的虚拟机12[root@node200 ~]# xe pool-ha-compute-max-host-failures-to-tolerate1 设置资源池允许的故障主机数（不能大于允许的最大故障主机数）1xe pool-param-set ha-host-failures-to-tolerate=&lt;num&gt; uuid=&lt;POOL_UUID&gt; num：数字，不能大于允许的最大故障主机数 POOL_UUID：资源池UUID 例如：1[root@node200 ~]# xe pool-param-set ha-host-failures-to-tolerate=1 uuid=53e9df09-bb99-3014-ffdf-3a781a9a84d2 启用资源池高可用1xe pool-ha-enable heartbeat-sr-uuids=&lt;SR_UUID&gt; SR_UUID为信号检测存储池的UUID ha-config:timeout可以配置超时检测，默认为120s 例如：1[root@node200 ~]# xe pool-ha-enable heartbeat-sr-uuids=6e336c82-acc1-8858-1060-32cdc7016804 ha-config:timeout=180 禁用HA解除VM的高可用保护1[root@node200 ~]# xe vm-param-set uuid=&lt;VM_UUID&gt; ha-always-run=&lt;true/false&gt; VM_UUID：虚拟机UUID true/false：是否总是运行状态 例如：1[root@node200 ~]# xe vm-param-set uuid=0ec2a592-29ad-efe1-6ac7-94cafb50d37e ha-always-run=false 禁用资源池高可用1[root@node200 ~]# xe pool-ha-disable]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[V2V转换器（八）——V2V服务端分析之Converter]]></title>
      <url>%2F2017%2F03%2F08%2Fv2v-8%2F</url>
      <content type="text"><![CDATA[Converter组件在客户端界面操作转换虚拟机时，每个需要转换的VMware虚拟机都会创建一条job数据存储到job文件中。 Converter是转换服务的核心组件。转换过程是： 获取配置文件信息 加载job文件数据 Quartz调度获取job信息，立即执行调度 获取VMware信息，创建XenServer虚拟机，创建VBD、VDI 将之前创建的VDI卸载 VDI挂载给服务器端VM 下载VMware虚拟机磁盘数据 将VDI从服务端VM上卸载 将VDI挂载给导入的虚拟机 整个过程会实时更新job状态与进度，所有的VDI数据下载完成并挂载给导入的XenServer虚拟机之后，转换操作才结束。 ConversionMgrConversionMgr初始化Initialize方法，调用JobManager的Initialize方法。12345678910public void Initialize()&#123; LOG.Info(&quot;Conversion mgr initializing...&quot;); //读取配置信息 this.ReadConfigurations(); //JobManager初始化 this.m_jobMgr.Initialize(); this.m_bInitialize = true; LOG.Info(&quot;Initialization complete.&quot;);&#125; JobManagerJobManager初始化方法，使用了Quartz框架来调度任务。客户端转换VM会生成job数据，保存到JobStore文件。 123456789101112131415161718192021222324252627282930313233 public void Initialize()&#123; LOG.Info(&quot;Bringing up job manager...&quot;); if (this.m_jobScheduler != null) &#123; throw new InvalidOperationException(Messages.EXCEPTION_ALREADY_INITIALIZED); &#125; //读取配置信息 this.ReadConfigurations(); //加载Job数据文件 this.m_jobStore = JobStore.Load(this.m_dbFile); //Quartz调度框架 /** QUARTZ.NET 是一个定时任务框架 概念： 计划者：IScheduler; 工作 IJob; 触发器：Trigger; 我们先得到一个计划者(IScheduler) 然后创建一个任务工作(IJob)，把这个任务扔给计划者，并且告诉它在什么条件下（触发器Trigger）做这件事 将要定时执行的任务的代码写到IJob接口的Execute方法中即可，时间到来的时候Execute方法会被调用 CrondTrigger是通过Crond表达式设置的触发器，还有SimpleTrigger等简单的触发器， 可以通过TriggerUtils的MakeDailyTrigger（每天执行一次），MakeHourlyTrigger（每小时执行一次） 等方法简化调用。 */ ISchedulerFactory factory = new StdSchedulerFactory(); //获取计划者 this.m_jobScheduler = factory.GetScheduler(); //添加调度任务监听 this.m_jobListener.JobCompletedEvent += new EventHandler&lt;JobCompletedEventArgs&gt;(this.JobCompletedEvent); this.m_jobScheduler.AddGlobalJobListener(this.m_jobListener); this.m_autoShutdownTimer = new Timer((double) (this.m_autoShutdownDelay * 0x3e8)); this.m_autoShutdownTimer.Elapsed += new ElapsedEventHandler(JobManager.OnShutdownTimer); //启动调度 this.m_jobScheduler.Start(); LOG.Info(&quot;Job scheduler started.&quot;); //job调度 this.ResubmitJobsAtStartup();&#125; ResubmitJobsAtStartup方法调用RunJob方法。在RunJob方法中，触发ConversionJob类型的job。1234567JobDetail jobDetail = new JobDetail(obj3.Id, &quot;g1&quot;, typeof(ConversionJob));jobDetail.JobDataMap[&quot;JobObject&quot;] = obj3;Interlocked.Increment(ref this.m_concurrentJobs);//立即执行Trigger trigger = TriggerUtils.MakeImmediateTrigger(0, TimeSpan.Zero);trigger.Name = obj3.Id;this.m_jobScheduler.ScheduleJob(jobDetail, trigger); ConversionJobConversionJob继承抽象类JobBase。JobBase实现接口IJob，实现了Execute方法。12345678910111213141516171819202122public void Execute(JobExecutionContext context)&#123; LOG.Debug(&quot;Execute()&quot;); this.m_job = (JobObject) context.JobDetail.JobDataMap[&quot;JobObject&quot;]; if (this.m_job == null) &#123; LOG.Error(&quot;NULL job object!!&quot;); &#125; else &#123; m_jobContext = this.m_job; try &#123; //Run抽象方法，由子类重写实现 this.Run(context); &#125; finally &#123; m_jobContext = null; &#125; &#125;&#125; ConversionJob类重写Run方法。调用ConvertVM方法。1234567891011121314151617public override void Run(JobExecutionContext context)&#123; Thread.CurrentThread.Priority = ThreadPriority.Lowest; if (!base.m_job.IsRemoved) &#123; this.m_vmImportInfo = new VmImportInfo(); this.m_vmImportInfo.SRuuidToUse = base.m_job.SRUuid; JobInstance jobInstanceData = base.m_job.GetJobInstanceData(); try &#123; //更新Job状态为执行中 base.m_job.UpdateJobState(JobState.Running); //开始转换虚拟机 this.ConvertVM(base.m_job); //更新job状态为完成 base.m_job.UpdateJobState(JobState.Completed); &#125; ConvertVM方法开始虚拟机转换。导出VMware虚拟机信息，导入XenServer虚拟机。12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485private void ConvertVM(JobObject job)&#123; &lt;ConvertVM&gt;c__AnonStorey0 storey = new &lt;ConvertVM&gt;c__AnonStorey0(); DateTime now = DateTime.Now; storey.instance = job.GetJobInstanceData(); object[] args = new object[] &#123; JobBase.JobId, storey.instance.JobInfo.SourceVmName, storey.instance.JobInfo.Source.Hostname, base.m_job.XenHost, DateTime.Now.ToString(&quot;r&quot;) &#125;; JobBase.LOG.InfoFormat(&quot;&lt;&#123;0&#125;&gt;: Converting VM &#123;1&#125; from source &#123;2&#125; to dest &#123;3&#125; started at &#123;4&#125; ...&quot;, args); ViVmService service = new ViVmService(); XenVmService service2 = new XenVmService(); //导入状态事件监听 service2.ImportStatusUpdateEvent += new EventHandler&lt;ImportStatusEventArgs&gt;(this.OnImportStatusUpdateEvent); VMExportInfo exportedInfo = null; ViVm vM = null; try &#123; JobBase.LOG.InfoFormat(&quot;&lt;&#123;0&#125;&gt;: Connecting to XenServer at &#123;1&#125; ...&quot;, JobBase.JobId, job.XenHost); //更新job状态为连接XenServer主机 job.UpdateJobStatus(string.Format(Messages.JOB_STATUS_CONNECTING_TO_XS, &quot;XenServer&quot;, job.XenHost), 0L, 0L); //连接XenServer主机 service2.Connect(new Uri(job.XenHost), job.XenUser, job.XenPwd); //更新job进度为5% job.UpdateJobPercentageComplete(5L); JobBase.LOG.InfoFormat(&quot;&lt;&#123;0&#125;&gt;: Connecting to ESX/VirtualCenter at &#123;1&#125;...&quot;, JobBase.JobId, storey.instance.JobInfo.Source.Hostname); //更新job状态为连接VMware主机 job.UpdateJobStatus(string.Format(Messages.JOB_STATUS_CONNECTING_TO_VCENTER, storey.instance.JobInfo.Source.Hostname), 0L, 0L); //连接VMware service.Connect(storey.instance.JobInfo.Source.Hostname, storey.instance.JobInfo.Source.Username, storey.instance.JobInfo.Source.Password); //更新job进度为10% job.UpdateJobPercentageComplete(10L); JobBase.LOG.InfoFormat(&quot;START: find VM object...&quot;, new object[0]); //根据job中的虚拟机uuid获取VMware虚拟机，返回ViVm对象 vM = service.GetVM(storey.instance.JobInfo.SourceVmUUID); if (vM == null) &#123; vM = service.GetVMs().Find(new Predicate&lt;ViVm&gt;(storey.&lt;&gt;m__0)); if (vM == null) &#123; throw new Exception(string.Format(Messages.EXCEPTION_FAILED_TO_FIND_SOURCE_VM, storey.instance.JobInfo.SourceVmName)); &#125; &#125; JobBase.LOG.InfoFormat(&quot;DONE: finding VM object&quot;, new object[0]); //更新job进度为15% job.UpdateJobPercentageComplete(15L); this.m_vmImportInfo.IsTemplate = vM.IsTemplate; //判断是否保留MAC地址 if (job.GetJobInstanceData().JobInfo.PreserveMAC.HasValue) &#123; this.m_vmImportInfo.preserveMAC = job.GetJobInstanceData().JobInfo.PreserveMAC.Value; &#125; else &#123; this.m_vmImportInfo.preserveMAC = false; &#125; this.m_vmImportInfo.networkMappings = job.GetJobInstanceData().JobInfo.NetworkMappings; job.VMTotalBytes = vM.CommitedStorage; if (this.m_vmImportInfo.IsTemplate) &#123; object[] objArray2 = new object[] &#123; JobBase.JobId, storey.instance.JobInfo.SourceVmName, job.VMTotalBytes, now.ToShortTimeString() &#125;; JobBase.LOG.InfoFormat(&quot;&lt;&#123;0&#125;&gt;: Exporting VM Template&apos;&#123;1&#125;&apos; with &#123;2&#125; bytes at &#123;3&#125;&quot;, objArray2); &#125; else &#123; object[] objArray3 = new object[] &#123; JobBase.JobId, storey.instance.JobInfo.SourceVmName, job.VMTotalBytes, now.ToShortTimeString() &#125;; JobBase.LOG.InfoFormat(&quot;&lt;&#123;0&#125;&gt;: Exporting VM &apos;&#123;1&#125;&apos; with &#123;2&#125; bytes at &#123;3&#125;&quot;, objArray3); &#125; //更新job状态为开始导出VMware虚拟机 job.UpdateJobStatus(Messages.JOB_STATUS_EXPORTING_VM, 0L, 0L); //调用ViVm的BeginExport方法，返回导出数据信息（包括虚拟机名称、网络信息、MAC地址、导出设备信息、OVF信息等） exportedInfo = vM.BeginExport(); //XenServer导入虚拟机 service2.ImportVM(exportedInfo, this.m_vmImportInfo); &#125; finally &#123; if ((vM != null) &amp;&amp; (exportedInfo != null)) &#123; vM.EndExport(exportedInfo); &#125; service.Disconnect(); service2.Disconnect(); DateTime time3 = DateTime.Now; object[] objArray4 = new object[] &#123; JobBase.JobId, storey.instance.JobInfo.SourceVmName, time3.ToShortTimeString(), (TimeSpan) (time3 - now) &#125;; JobBase.LOG.InfoFormat(&quot;&lt;&#123;0&#125;&gt;: Converting VM &apos;&#123;1&#125;&apos; stopped at &#123;2&#125; after a duration of &#123;3&#125;&quot;, objArray4); &#125;&#125; XenVmServiceXenVmService的ImportVM方法。 调用Importer的Process方法， 根据VMware虚拟机OVF等信息创建XenServer虚拟机 执行Process方法之后，调用ImportVMDisks方法，导入虚拟磁盘。1234567891011121314151617181920Importer import = new Importer(); XenVm importedVm = null; try &#123; import.Process(this.m_session, ovfEnvelope, vmImportInfo.SRuuidToUse, vmImportInfo.preserveMAC, exportedInfo.vmNetworkMACInfo, vmImportInfo.networkMappings, vmImportInfo.IsTemplate); if (JobBase.IsRemoved) &#123; e.currentTaskDesc = Messages.TASK_DESC_JOB_CANCEL_PROCESSED; this.ImportStatusUpdateEvent(this, e); LOG.WarnFormat(&quot;&lt;&#123;0&#125;&gt;: Cancelling running job during meta import for VM &#123;1&#125;&quot;, JobBase.JobId, exportedInfo.VmName); throw new UserAbortException(string.Format(Messages.USER_ABORT_EXCEPTION_CANCEL_JOB3, exportedInfo.VmName)); &#125; &#125;........//导入虚拟磁盘this.ImportVMDisks(importedVm, import, exportedInfo, vmImportInfo);........ ImporterProcess方法，调用AddResourceSettingData方法添加VBD与VDI。123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657public void Process(Session xenSession, EnvelopeType ovfEnvelope, string srUuid, bool preserveMAC, List&lt;VMNetworkMACInfo&gt; vmNetworkMACInfo, Hashtable networkMappings, bool isTemplate)&#123; if (xenSession == null) &#123; throw new InvalidOperationException(string.Format(Messages.ERROR_NOT_CONNECTED, &quot;XenServer&quot;)); &#125; this.m_ovfEnv = ovfEnvelope; this.m_importedVMInfo = new ImportedVMInfo(); this.m_importedVMInfo.disksInfo = new List&lt;ImportedDiskInfo&gt;(); this.vifDeviceIndex = 0; if (ovfEnvelope.Item is VirtualSystem_Type) &#123; VirtualSystem_Type item = (VirtualSystem_Type) ovfEnvelope.Item; ovfEnvelope.Item = new VirtualSystemCollection_Type(); ((VirtualSystemCollection_Type) ovfEnvelope.Item).Content = new Content_Type[] &#123; item &#125;; &#125; VirtualHardwareSection_Type system = null; foreach (VirtualSystem_Type type3 in ((VirtualSystemCollection_Type) ovfEnvelope.Item).Content) &#123; system = this.ovfImport.FindVirtualHardwareSectionByAffinity(ovfEnvelope, type3.id, &quot;xen&quot;); string ovfName = this.ovfImport.FindSystemName(ovfEnvelope, type3.id); //创建虚拟机 XenRef&lt;VM&gt; vmRef = this.DefineSystem(xenSession, system, ovfName); if (vmRef == null) &#123; throw new ImportException(Messages.ERROR_DEFINE_SYSTEM); &#125; this.m_importedVMInfo.vmInfo = VM.get_record(xenSession, (string) vmRef); this.SetDeviceConnections(ovfEnvelope, system); if (isTemplate) &#123; VM.set_is_a_template(xenSession, (string) vmRef, true); &#125; try &#123; foreach (RASD_Type type4 in system.Item) &#123; if (((type4.ResourceType.Value == 0x11) || (type4.ResourceType.Value == 0x13)) || (type4.ResourceType.Value == 0x15)) &#123; if (OVF.ValidateProperty(&quot;Caption&quot;, type4) &amp;&amp; ((type4.Caption.Value.ToUpper().Contains(&quot;COM&quot;) || type4.Caption.Value.ToUpper().Contains(&quot;FLOPPY&quot;)) || type4.Caption.Value.ToUpper().Contains(&quot;ISO&quot;))) &#123; continue; &#125; this.SetIfDeviceIsBootable(ovfEnvelope, type4); &#125; string compressed = &quot;None&quot;; //添加资源，VBD与VDI this.AddResourceSettingData(xenSession, vmRef, type4, this.ovfImport.FindRasdFileName(ovfEnvelope, type4, out compressed), srUuid, preserveMAC, vmNetworkMACInfo, networkMappings); &#125; &#125; catch (Exception exception) &#123; LOG.ErrorFormat(&quot;Importer: Failed processing device types in OVF file: Error &apos;&#123;0&#125;&apos;.&quot;, exception.Message); throw exception; &#125; &#125;&#125; 调用ImportVMDisks方法 将之前创建的VDI卸载 VDI挂载给服务器端VM 下载VMware虚拟机磁盘数据 将VDI从服务端VM上卸载 将VDI挂载给导入的虚拟机 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657 private void ImportVMDisks(XenVm importedVm, Importer import, VMExportInfo exportedInfo, VmImportInfo importInfo)&#123; ImportStatusEventArgs e = new ImportStatusEventArgs(Messages.IMPORTING_VM_DISKS_STATUS, string.Empty, 0L, 0L, 0L); this.ImportStatusUpdateEvent(this, e); foreach (ImportedDiskInfo info in import.ImportedVMInfo.disksInfo) &#123; VBD vbdInfo = null; try &#123; if (JobBase.IsRemoved) &#123; e.currentTaskDesc = Messages.TASK_DESC_JOB_CANCEL_PROCESSED; this.ImportStatusUpdateEvent(this, e); LOG.WarnFormat(&quot;&lt;&#123;0&#125;&gt;: Cancelling running job during disk import for VM &#123;1&#125;&quot;, JobBase.JobId, importedVm.Name); throw new UserAbortException(string.Format(Messages.USER_ABORT_EXCEPTION_CANCEL_JOB2, importedVm.Name)); &#125; LOG.InfoFormat(&quot;&lt;&#123;0&#125;&gt;: Detaching VDI &#123;1&#125; from imported VM &#123;2&#125;.&quot;, JobBase.JobId, info.vdiInfo.uuid, importedVm.Name); //VBD unplug（之前创建的VDI是没有plug给VM的，这里做一下unplug操作，以防万一被plug） importedVm.DetachDisk(info.vbdInfo); LOG.InfoFormat(&quot;&lt;&#123;0&#125;&gt;: Attaching VDI &#123;1&#125; to this appliance.&quot;, JobBase.JobId, info.vdiInfo.uuid); //将VDI plug给服务端VM vbdInfo = this.m_conversionVM.AttachDisk(info.vdiInfo); LOG.InfoFormat(&quot;&lt;&#123;0&#125;&gt;: Newly-attached VBD disk on this appliance &#123;1&#125;.&quot;, JobBase.JobId, vbdInfo.device); File_Type type = import.Ovf.FindFileReference(import.Envelope, info.OvfDiskInfo.fileRef); LOG.InfoFormat(&quot;&lt;&#123;0&#125;&gt;: Searching for disk key &#123;1&#125;.&quot;, JobBase.JobId, type.href); VMExportDeviceInfo devInfo = this.FindExportDeviceInfo(exportedInfo, type.href); if (devInfo != null) &#123; LOG.InfoFormat(&quot;&lt;&#123;0&#125;&gt;: Imported disk key=&#123;1&#125; match to exported key=&#123;2&#125; &quot;, JobBase.JobId, type.href, devInfo.Key); LOG.InfoFormat(&quot;&lt;&#123;0&#125;&gt;: Streaming from url=&#123;1&#125; to device=&#123;2&#125;...&quot;, JobBase.JobId, devInfo.Url, vbdInfo.device); //下载VMware磁盘数据 this.DownloadDisk(exportedInfo, devInfo, vbdInfo, JobBase.JobId); LOG.InfoFormat(&quot;&lt;&#123;0&#125;&gt;: Streaming done for device=&#123;1&#125;.&quot;, JobBase.JobId, vbdInfo.device); &#125; else &#123; LOG.ErrorFormat(&quot;&lt;&#123;0&#125;&gt;: No match key found for imported disk key=&#123;1&#125;!!&quot;, JobBase.JobId, type.href); throw new InvalidOperationException(string.Format(Messages.EXCEPTION_NO_MATCH_KEY_FOUND, type.href)); &#125; LOG.InfoFormat(&quot;&lt;&#123;0&#125;&gt;: Detaching device=&#123;1&#125; from appliance.&quot;, JobBase.JobId, vbdInfo.device); //将VDI从服务端VM上unlpug this.m_conversionVM.DetachDisk(vbdInfo); LOG.InfoFormat(&quot;&lt;&#123;0&#125;&gt;: Reattaching device=&#123;1&#125; to imported VM.&quot;, JobBase.JobId, info.vbdInfo.userdevice); //将VDI挂载给导入的虚拟机 importedVm.ReattachDisk(info.vbdInfo); &#125; catch &#123; if (vbdInfo != null) &#123; this.m_conversionVM.DestroyDisk(vbdInfo); &#125; throw; &#125; &#125;&#125; DownloadDisk方法下载VMware磁盘数据，使用/opt/vpxxcm/conversion/vdiskget来执行1234567891011121314151617private void DownloadDisk(VMExportInfo exportedInfo, VMExportDeviceInfo devInfo, VBD vbdInfo, string jobId)&#123; string str = XenVmDiskPath.FixupDiskPath(vbdInfo.device); string str2 = string.Format(&quot;/dev/&#123;0&#125;&quot;, str); int num = -1; for (int i = 0; i &lt; Settings.Default.DownloadRetryCount; i++) &#123; string[] arguments = new string[] &#123; Settings.Default.DownloadOptions, devInfo.Url, str2, jobId, this.Uri.ToString() &#125;; num = this.RunCancelableJobProcess(&quot;/opt/vpxxcm/conversion/vdiskget&quot;, Settings.Default.DownloadTimeoutMs, Settings.Default.DownloadPeriodMs, arguments); if (num == 0) &#123; return; &#125; Thread.Sleep(Settings.Default.DownloadRetryDelayMs); &#125; throw new Exception(string.Format(&quot;Failed to download disk from &#123;0&#125; to &#123;1&#125;. (vdiskget result = &#123;2&#125;)&quot;, devInfo.Url, str2, num));&#125;]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[V2V转换器（七）——V2V服务端分析之convsvc]]></title>
      <url>%2F2017%2F03%2F07%2Fv2v-7%2F</url>
      <content type="text"><![CDATA[V2V转换器服务端是运行在XenServer Conversion Manager Virtual Appliance上，主要有以下组件： convsvc：主服务，Windows服务程序 Converter：转换的核心组件 Citrix.ExportImport.CommonTypes：公共类库 VimService：VMware API Quartz：调度组件 XenServer：XAPI convsvc服务convsvc服务是运行在V2V转换器的服务端XenServer Conversion Manager Virtual Appliance上，作为Windows Service在mono中运行。 主要有两个类： ConversionSvc ConversionSvcWrapper ConversionSvcWrapperConversionSvcWrapper类继承ServiceBase，重写了OnStart、OnStop、OnShutdown方法，是ConversionSvc服务的包装类，可以把ConversionSvc作为一个Windows服务程序来运行。 ConversionSvcConversionSvc类定义了获取VMware虚拟机、网络、磁盘空间，以及job、log等操作的一些方法。实际上是对外开放了这些接口。 代码分析ConversionSvcWrapper的Main方法作为Windows服务程序的入口。12345678Console.WriteLine(&quot;Starting conversion service in interactive mode&quot;);//调用OnStart方法wrapper.OnStart(args);Console.WriteLine(&quot;Prese any key and &lt;Enter&gt; to exit&quot;);Console.ReadLine();wrapper.OnStop();Console.ReadLine();Console.WriteLine(&quot;Exiting...&quot;); 重写的OnStart方法。12345678910111213protected override void OnStart(string[] args)&#123; LOG.Info(&quot;Starting service...&quot;); try &#123; //调用ConversionSvc的Start方法 this.m_service.Start(); &#125; catch (Exception exception) &#123; throw new Exception(&quot;Failed to start service.&quot;, exception); &#125;&#125; ConversionSvc的Start方法。ConversionMgr是Converter组件的类。1234567891011public void Start()&#123; LOG.Info(&quot;Starting the conversion service...&quot;); //获取配置文件中的配置信息 this.ReadConfigurations(); //初始化ConversionMgr ConversionMgr.Instance.Initialize(); //HTTP监听线程启动 this.m_listener.Start(); this.m_listenerThread.Start();&#125;]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[使用Git GUI提交代码到GitHub]]></title>
      <url>%2F2017%2F03%2F07%2Fgit-gui-1%2F</url>
      <content type="text"><![CDATA[使用Git GUI提交代码到GitHub，主要有以下步骤： GitHub创建仓库 clone远程仓库 复制代码 Git GUI操作 下面以XenCenter为例，环境： Windows 7 x64 Git for Windows(Git-2.11.0-64-bit.exe) GitHub创建仓库先在github上新建一个仓库XenCenter，仓库的地址为：https://github.com/hl10502/XenCenter.git clone远程仓库在Git GUI上clone远程仓库到本地 复制代码复制XenCenter代码到Target Directory。比如XenCenter的代码目录为E:\visual studio 2010\Projects\xenadmin-master7.0\xenadmin Git GUI操作Git GUI操作提交代码。 Rescan:重新扫描，显示本地有新增、修改的代码文件 Stage Changed：阶段提交，显示需要提交的文件 Commit：提交到本地 Push：上传到GitHub]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[V2V转换器（六）——Unable to Boot VMware SCSI Disk]]></title>
      <url>%2F2017%2F03%2F07%2Fv2v-6%2F</url>
      <content type="text"><![CDATA[Unable to Boot VMware SCSI Disk如果VMware VM从SCSI磁盘引导，但还包含一个或多个IDE硬盘，当做V2V转换成功到XenServer上之后，则VM可能无法启动。 这是因为迁移过程IDE硬盘分配的设备号比SCSI磁盘小，但是XenServer从分配给设备0的硬盘启动VM，导致VM启动失败。要解决此问题，需要修改VM的虚拟磁盘位置，以便正确引导系统盘启动VM。 有两种方法： 使用CLI命令行 使用XenCenter界面 使用CLI根据vmUuid找到VBD 1xe vbd-list vm-uuid=5be02646-d70e-ed83-f68b-286bad3890ba 设置系统盘的VBD为device 0 与可引导 12xe vbd-param-set userdevice=0 uuid=873f1c20-212e-913a-50fd-5352f81478dbxe vbd-param-set bootable=true uuid=873f1c20-212e-913a-50fd-5352f81478db XenCenter操作选中一个存储池，在该存储池的“存储”选项，选择虚拟磁盘，查看属性，修改虚拟磁盘的“设备位置”为0]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[C#反编译.resources资源文件]]></title>
      <url>%2F2017%2F02%2F28%2Fvs2013-decompile%2F</url>
      <content type="text"><![CDATA[C#反编译的.resources资源文件，使用resgen命令生成.resx文件在项目中使用C#的.dll文件反编译出来的.resources资源文件，需要生成.resx才能在项目中使用。 打开VS2013命令行 执行resgen命令生成resx文件将资源文件copy到命令目录下（D:\Program Files (x86)\Microsoft Visual Studio 12.0\VC），执行命令 resgen xx.resources xx.resx。 例如WinAPI.FriendlyErrorNames.resources文件 修改resx文件例如：生成的.resx文件为WinAPI.FriendlyErrorNames.resx。找到项目中的FriendlyErrorNames.cs文件，WinAPI为namespace，FriendlyErrorNames.cs文件中资源为WinAPI.FriendlyErrorNames。 重命名WinAPI.FriendlyErrorNames.resx为FriendlyErrorNames.resx，并将FriendlyErrorNames.resx文件copy到项目的WinAPI目录中（与FriendlyErrorNames.cs同一目录）。 注意如下代码的资源文件的路径为”WinAPI.FriendlyErrorNames”。一般加载不了资源可能是路径写错。12345678910111213[EditorBrowsable(EditorBrowsableState.Advanced)] public static System.Resources.ResourceManager ResourceManager &#123; get &#123; if (object.ReferenceEquals(resourceMan, null)) &#123; System.Resources.ResourceManager manager = new System.Resources.ResourceManager(&quot;WinAPI.FriendlyErrorNames&quot;, typeof(FriendlyErrorNames).Assembly); resourceMan = manager; &#125; return resourceMan; &#125; &#125;]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[V2V转换器（五）——ERROR Credentials for user 'root' are invalid on http://169.254.0.1]]></title>
      <url>%2F2017%2F02%2F28%2Fv2v-5%2F</url>
      <content type="text"><![CDATA[ERROR Credentials for user ‘root’ are invalid on http://169.254.0.1XCM所在的XenServer主机的iptables阻止XCM的eth0通过dhcp获取ip，需要关闭XenServer的防火墙。 错误分析XCM的log。 123456-bash-4.2# cat /var/log/conversion/convsvc.log2017-01-13 08:48:03,042 ERROR Credentials for user 'root' are invalid on http://169.254.0.1.2017-01-13 08:48:03,066 ERROR System.InvalidOperationException: Cannot be changed after headers are sent. at System.Net.HttpListenerResponse.set_StatusCode (Int32 value) [0x00000] in &lt;filename unknown&gt;:0 at CookComputing.XmlRpc.XmlRpcListenerService.ProcessRequest (System.Net.HttpListenerContext RequestContext) [0x00000] in &lt;filename unknown&gt;:0 at convsvc.ConversionSvc+&lt;HandleNewContext&gt;c__AnonStorey0.&lt;&gt;m__0 (System.Object state) [0x00000] in &lt;filename unknown&gt;:0 启动XCM的网络失败，eth0不能动态生成IP地址。 123456789-bash-4.2# service network restartShutting down interface eth0: [ OK ]Shutting down interface eth1: [ OK ]Shutting down loopback interface: [ OK ]Bringing up loopback interface: [ OK ]Bringing up interface eth0:Determining IP information for eth0... failed. [FAILED]Bringing up interface eth1: [ OK ] 查看XCM的日志，dhcp接收不到数据。 123456-bash-4.2# tail -f /var/log/messagesJan 13 09:08:28 xcm dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 10 (xid=0x15d3da80)Jan 13 09:08:38 xcm dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 17 (xid=0x15d3da80)Jan 13 09:08:55 xcm dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 15 (xid=0x15d3da80)Jan 13 09:09:10 xcm dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 13 (xid=0x15d3da80)Jan 13 09:09:23 xcm dhclient: No DHCPOFFERS received. 查看XenServer主机的iptables。 12345678910111213141516171819202122232425262728293031[root@xenserver113 ~]# iptables -LChain INPUT (policy ACCEPT)target prot opt source destinationRH-Firewall-1-INPUT all -- anywhere anywhereChain FORWARD (policy ACCEPT)target prot opt source destinationRH-Firewall-1-INPUT all -- anywhere anywhereChain OUTPUT (policy ACCEPT)target prot opt source destinationChain RH-Firewall-1-INPUT (2 references)target prot opt source destinationACCEPT all -- anywhere anywhereACCEPT icmp -- anywhere anywhere icmp anyACCEPT esp -- anywhere anywhereACCEPT ah -- anywhere anywhereACCEPT udp -- anywhere 224.0.0.251 udp dpt:mdnsACCEPT udp -- anywhere anywhere udp dpt:ippACCEPT tcp -- anywhere anywhere tcp dpt:ippACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHEDACCEPT tcp -- anywhere anywhere ctstate NEW tcp dpt:sshACCEPT udp -- anywhere anywhere ctstate NEW udp dpt:ha-clusterACCEPT tcp -- anywhere anywhere ctstate NEW tcp dpt:httpACCEPT tcp -- anywhere anywhere ctstate NEW tcp dpt:httpsACCEPT tcp -- anywhere anywhere ctstate NEW tcp dpt:4430ACCEPT udp -- anywhere anywhere ctstate NEW udp dpt:ntpACCEPT tcp -- anywhere anywhere ctstate NEW tcp dpt:ndmpACCEPT tcp -- anywhere anywhere ctstate NEW tcp dpt:vceREJECT all -- anywhere anywhere reject-with icmp-host-prohibited 查看XenServer主机的iptables的状态。 1234567891011121314151617181920212223242526272829303132[root@xenserver113 ~]# service iptables statusTable: filterChain INPUT (policy ACCEPT)num target prot opt source destination1 RH-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0Chain FORWARD (policy ACCEPT)num target prot opt source destination1 RH-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0Chain OUTPUT (policy ACCEPT)num target prot opt source destinationChain RH-Firewall-1-INPUT (2 references)num target prot opt source destination1 ACCEPT all -- 0.0.0.0/0 0.0.0.0/02 ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 2553 ACCEPT esp -- 0.0.0.0/0 0.0.0.0/04 ACCEPT ah -- 0.0.0.0/0 0.0.0.0/05 ACCEPT udp -- 0.0.0.0/0 224.0.0.251 udp dpt:53536 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:6317 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:6318 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED9 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:2210 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:69411 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:8012 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:44313 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:443014 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:12315 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:1000016 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:1111117 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited 解決方法关闭XenServer主机的iptables，再次重启XCM的网络。 12345[root@xenserver113 ~]# service iptables stopiptables: Flushing firewall rules: [ OK ]iptables: Setting chains to policy ACCEPT: filter [ OK ]iptables: Unloading modules: [ OK ] 123456789[root@xenserver113 ~]# iptables -LChain INPUT (policy ACCEPT)target prot opt source destinationChain FORWARD (policy ACCEPT)target prot opt source destinationChain OUTPUT (policy ACCEPT)target prot opt source destination 重启XCM的网络。 123456789-bash-4.2# service network restartShutting down interface eth0: [ OK ]Shutting down interface eth1: [ OK ]Shutting down loopback interface: [ OK ]Bringing up loopback interface: [ OK ]Bringing up interface eth0:Determining IP information for eth0... done. [ OK ]Bringing up interface eth1: [ OK ] 查看网络，eth0生成动态的IP。 12345678910-bash-4.2# ip a1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo2: eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 32:d9:a4:db:02:8e brd ff:ff:ff:ff:ff:ff inet 169.254.0.2/16 brd 169.254.255.255 scope global eth03: eth1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 46:71:b8:a5:fd:07 brd ff:ff:ff:ff:ff:ff inet 192.168.217.199/24 brd 192.168.217.255 scope global eth1]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[V2V转换器（四）——XenServer Conversion Manager Virtual Appliance虚拟机网络修改]]></title>
      <url>%2F2017%2F02%2F28%2Fv2v-4%2F</url>
      <content type="text"><![CDATA[XenServer Conversion Manager Virtual Appliance网络配置XenServer Conversion Manager Virtual Appliance上有两张网卡。后期由于网络环境变化或者制作模板导入是需要修改网络信息的。 eth0：dhcp方式动态获取，在虚拟化层面使用XenServer主机的内部管理网络，IP地址动态获取为169.254.0.x eth1：static静态配置 查看虚拟机使用的network在主机上查询虚拟机。 1234[root@xenserver113 ~]# xe vm-list name-label="Citrix XCM Virtual Appliance"uuid ( RO) : 522e1ca7-0c99-6090-b0f1-ec4439254286 name-label ( RW): Citrix XCM Virtual Appliance power-state ( RO): running 查看虚拟机的VIF。 1234567891011[root@xenserver113 ~]# xe vif-list vm-uuid=522e1ca7-0c99-6090-b0f1-ec4439254286uuid ( RO) : f0c33a41-f240-9a9f-37fb-3188e3b27358 vm-uuid ( RO): 522e1ca7-0c99-6090-b0f1-ec4439254286 device ( RO): 1 network-uuid ( RO): c73b7b36-e65f-0cf7-0183-3bfdbfed0df0uuid ( RO) : 39ec0a7a-35e6-21ff-085b-371a6e019ef0 vm-uuid ( RO): 522e1ca7-0c99-6090-b0f1-ec4439254286 device ( RO): 0 network-uuid ( RO): 33d92b86-dfc7-950c-51ee-6dc3f9f27394 虚拟机的虚拟网络eth0使用的是物理主机的内部管理网络，物理主机的内部管理IP为169.254.0.1。 123456789101112[root@xenserver113 ~]# xe network-param-list uuid=33d92b86-dfc7-950c-51ee-6dc3f9f27394uuid ( RO) : 33d92b86-dfc7-950c-51ee-6dc3f9f27394 name-label ( RW): Host internal management network name-description ( RW): Network on which guests will be assigned a private link-local IP address which can be used to talk XenAPI VIF-uuids (SRO): 03f850b1-f7bb-320d-8d54-aa3d3cf3a102; 39ec0a7a-35e6-21ff-085b-371a6e019ef0 PIF-uuids (SRO): MTU ( RW): 1500 bridge ( RO): xenapi other-config (MRW): is_guest_installer_network: true; is_host_internal_management_network: true; ip_begin: 169.254.0.1; ip_end: 169.254.255.254; netmask: 255.255.0.0 blobs ( RO): tags (SRW): default-locking-mode ( RW): unlocked 修改虚拟机eth1静态网络eth0的网络是不能修改的。 12345-bash-4.2# cat /etc/sysconfig/network-scripts/ifcfg-eth0DEVICE=eth0BOOTPROTO=dhcpONBOOT=yesNOZEROCONF=true 可以修改eth1的网络（IP、子网掩码、网关）。 123456789-bash-4.2# cat /etc/sysconfig/network-scripts/ifcfg-eth1DEVICE=eth1ONBOOT=yesNOZEROCONF=trueBOOTPROTO=noneNETMASK=255.255.255.0IPADDR=192.168.217.222PEERDNS=yesDNS1= 12345-bash-4.2# cat /etc/sysconfig/networkNETWORKING=yesNETWORKING_IPV6=noHOSTNAME=xcmGATEWAY=192.168.217.254]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[V2V转换器（三）——XenServer Conversion Manager Virtual Appliance转换服务convsvc配置]]></title>
      <url>%2F2017%2F02%2F28%2Fv2v-3%2F</url>
      <content type="text"><![CDATA[convsvc服务配置修改XenServer Conversion Manager Virtual Appliance上的主要服务是convsvc，它是整个V2V转换的核心。 在XenServer Conversion Manager Virtual Appliance导入到XenServer之后，需要修改/opt/vpxxcm/conversion/convsvc.exe.config文件配置信息。 IP地址不是动态获取，修改ConversionServer.IP将1&lt;add key="ConversionServer.IP" value="[/etc/dhclient-enter-hooks script will fill it in]" /&gt; 修改成1&lt;add key="ConversionServer.IP" value="localhost" /&gt; 增加日志编码支持在12&lt;appender name="RollingLogFileAppender" type="log4net.Appender.RollingFileAppender"&gt;&lt;param name="File" value="/var/log/conversion/convsvc.log" /&gt; 下面增加1&lt;param name="encoding" value="utf-8" /&gt; 修改自动关闭系统时间将1&lt;add key="JobManager.AutoShutdownDelay" value="300" /&gt; &lt;!-- 5分钟 --&gt; 修改成1&lt;add key="JobManager.AutoShutdownDelay" value="86400" /&gt; &lt;!-- 1天 --&gt; 重启服务1-bash-4.2# /etc/init.d/convsvcd restart]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[V2V转换器（二）——XenServer Conversion Manager Virtual Appliance虚拟机配置]]></title>
      <url>%2F2017%2F02%2F28%2Fv2v-2%2F</url>
      <content type="text"><![CDATA[XenServer Conversion Manager Virtual ApplianceXenServer Conversion Manager Virtual Appliance是执行转换的虚拟设备，需要导入到XenServer主机或资源池中的master主机。 将XenServer-7.0.0-vpx-conversion.xva导入到XenServer主机中（可以使用WinCenterClient工具导入），配置好网络以及相关配置就可以作为XenServer Conversion Manager Virtual Appliance。 配置密码、主机名、网络输入yes，接受license。 设置密码。 设置hostname。 设置网络信息，不使用DHCP。 最后确定所有的配置，自动重启网络。 修改其他配置（可以不修改）查找含有xenserver字符的文件。 12345678910111213-bash-4.2# cd /etc/init.d/-bash-4.2# grep -in xenserver *convsvcd:26:# Init file for Citrix XenServer VM Conversion serviceconvsvcd:29:# description: Citrix XenServer VM conversion serviceeula_render.sh:6: echo -e "\033[44;37;1mCitrix XenServer Conversion Manager Virtual Appliance - End User License Agreement \033[0m"vpx_startup_setup.sh:19:product="XenServer"vpx_startup_setup.sh:20:banner="\033[44;37;1mCitrix XenServer Conversion Manager Virtual Appliance \033[0m"vpx_startup_setup.sh:39: echo -e "The Citrix XenServer Conversion Manager Service is running at \033[40;33;1m$ip\033[0m"vpx_startup_setup.sh:187:makecert -r -n "CN=Citrix XenServer Conversion Manager Appliance" -sv root.key root.cer &gt;&gt; $logfile 2&gt;&amp;1vpx_startup_setup.sh:250:echo "Citrix XenServer Conversion Manager Virtual Appliance configuration is complete."vpx_startup_setup.sh:257:echo -e "The Citrix XenServer Conversion Manager Service is running at \033[40;33;1m$ip\033[0m"xcm_self_configure.sh:69:makecert -r -n "CN=Citrix XenServer Conversion Manager Appliance" -sv root.key root.cer &gt;&gt; $logfile 2&gt;&amp;1xcm_self_configure.sh:131:echo "Citrix XenServer Conversion Manager Virtual Appliance configuration is complete." 修改Citrix XenServer，使用其他字符替换。xx使用你想替换成的字符。 123456-bash-4.2# sed -i 's/Citrix XenServer/xx/g' vpx_startup_setup.sh-bash-4.2# sed -i 's/Citrix XenServer/xx/g' convsvcd-bash-4.2# sed -i 's/Citrix XenServer/xx/g' xcm_self_configure.sh-bash-4.2# sed -i 's/Citrix XenServer/xx/g' eula_render.sh-bash-4.2# cd /etc/-bash-4.2# sed -i 's/Citrix XenServer/xx/g' issue 重启虚拟机。 1-bash-4.2# reboot]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[V2V转换器（一）——XenServer Conversion Manager]]></title>
      <url>%2F2017%2F02%2F28%2Fv2v-1%2F</url>
      <content type="text"><![CDATA[XenServer Conversion Manager简介XenServer Conversion Manager：简称XCM，是将VMware的虚拟机转换成XenServer上的虚拟机的工具，即是V2V转换器。 XCM分为两个部分： XenServer Conversion Manager Console XenServer Conversion Manager Virtual Appliance XenServer Conversion Manager ConsoleXenServer Conversion Manager Console是Windows下的用户界面，需要连接到XenServer和XenServer Conversion Manager Virtual Appliance。 XenServer Conversion Manager Virtual ApplianceXenServer Conversion Manager Virtual Appliance是执行转换的虚拟设备，需要导入到XenServer主机或资源池中的master主机。 XenServer Conversion Manager Virtual Appliance需要的资源： XenServer 6.1/XenServer 6.2 SP1/XenServer 6.5 SP1/XenServer 7.0 Disk space: 4GB of disk space Memory: 1GB (will use up to 2GB if available) Virtual CPU allocation: 1 vCPU 通过XenServer Conversion Manager Virtual Appliance这台运行在XenServer主机上的虚拟机，将VMware虚拟机转换为XenServer虚拟机格式，并将这些虚拟机导入到XenServer池或主机来完成虚拟机转换。 转换过程XenServer Conversion Manager Console使用一个简单的向导式操作来连接XenServer主机，转换VMware上的多个虚拟机到XenServer。 连接XenServer主机，启动XenServer Conversion Manager Virtual Appliance 连接VMware Server 获取XenServer的本地或共享存储池作为转换的目标存储 获取VMware虚拟机，选择一个或多个需要转换的VMware虚拟机 在VMware和XenServer之间映射网络设置，以便转换的虚拟机启动,并使用正确的网络设置运行 转换虚拟机 XenServer Conversion Manager不会删除或更改现有的VMware环境。虚拟机将复制到XenServer环境中，而不会从VMware中删除。 VMware与XenServerVMware与XenServer对应的术语 XenServer支持版本 XenServer 6.1 XenServer 6.2 Service Pack 1 XenServer 6.5 Service Pack 1 XenServer 7.0 VMware支持版本 vCenter Server 4.0 and 4.1 vSphere 4.0 and 4.1 ESXi 5.0.0, 5.1.0, 5.5.0 and 6.0.0]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[XenCenter分析（二）]]></title>
      <url>%2F2017%2F02%2F27%2Fxencenter-2%2F</url>
      <content type="text"><![CDATA[要使用XenCenter来管理XenServer主机和资源池，以及部署、监视、管理和迁移虚拟机。 首先需要添加XenServer物理主机。 AddServerDialog添加服务器的代码在AddServerDialog.cs中。 添加服务器的对话框“添加”按钮的事件AddButton_Click，调用ConnectToServer方法连接服务器。 1234567891011121314151617181920212223242526272829303132333435363738394041public virtual void AddButton_Click(object sender, EventArgs e)&#123; /** *ip地址与端口，可以一次连接多个主机，以&quot;;&quot;分隔 *一般我们连接一个，直接使用ip，端口默认使用80 */ string hostnameAndPort = ServerNameComboBox.Text.Trim(); //用户名 string username = UsernameTextBox.Text.Trim(); //密码 string password = PasswordTextBox.Text; //多主机 string[] multipleHosts; if (TryGetMultipleHosts(hostnameAndPort, out multipleHosts)) &#123; //循环依次连接 foreach (string h in multipleHosts) &#123; ConnectToServer(null, h, ConnectionsManager.DEFAULT_XEN_PORT, username, password, comboBoxClouds.SelectedItem != null ? comboBoxClouds.SelectedItem.ToString() : string.Empty); &#125; &#125; else &#123; string hostname; int port; if (!StringUtility.TryParseHostname(hostnameAndPort, ConnectionsManager.DEFAULT_XEN_PORT, out hostname, out port)) &#123; //分隔ip与端口 hostname = hostnameAndPort; port = ConnectionsManager.DEFAULT_XEN_PORT; &#125; //连接主机 ConnectToServer(connection, hostname, port, username, password, comboBoxClouds.SelectedItem != null ? comboBoxClouds.SelectedItem.ToString() : string.Empty); &#125; Close();&#125; ConnectToServer方法，经ip、用户名、密码等参数组装XenConnection对象，调用XenConnectionUI的BeginConnect方法。 123456789101112131415161718192021222324protected void ConnectToServer(IXenConnection conn, string hostname, int port, string username, string password, string version) &#123; if (conn == null) &#123; XenConnection connection = new XenConnection(); connection.CachePopulated += conn_CachePopulated; connection.fromDialog = true; conn = connection; &#125; else if (!_changedPass) &#123; conn.EndConnect(); // in case we&apos;re already connected &#125; conn.Hostname = hostname; conn.Port = port; conn.Username = username; conn.Password = password; conn.ExpectPasswordIsCorrect = false; conn.Version = version; if (!_changedPass) XenConnectionUI.BeginConnect(conn, true, Owner, false); &#125; XenConnectionUI的BeginConnect方法，调用XenConnection的BeginConnect方法。 123456789101112131415161718192021222324public static void BeginConnect(IXenConnection connection, bool interactive, Form owner, bool initiateMasterSearch) &#123; Program.AssertOnEventThread(); //注册连接事件，连接结果与连接状态的处理 RegisterEventHandlers(connection); if (interactive) &#123; // CA-214953 - Focus on this connection&apos;s dialog, if one exists, otherwise create one ConnectingToServerDialog dlg; if (connectionDialogs.TryGetValue(connection, out dlg)) &#123; UnregisterEventHandlers(connection); if (dlg.WindowState == FormWindowState.Minimized) dlg.WindowState = FormWindowState.Normal; dlg.Focus(); return; &#125; dlg = new ConnectingToServerDialog(connection); connectionDialogs.Add(connection, dlg); dlg.BeginConnect(owner, initiateMasterSearch); &#125; else ((XenConnection)connection).BeginConnect(initiateMasterSearch, PromptForNewPassword); &#125; XenConnection的BeginConnect方法,开启新线程处理连接 1234567891011121314//清除事件队列ClearEventQueue();OnBeforeMajorChange(false);Cache.Clear();OnAfterMajorChange(false);//设置连接任务对象connectTask = new ConnectTask(Hostname, Port);StopMonitor();heartbeat = new Heartbeat(this, XenAdminConfigManager.Provider.ConnectionTimeout);//开启连接工作线程Thread t = new Thread(ConnectWorkerThread);t.Name = &quot;Connection to &quot; + Hostname;t.IsBackground = true;t.Start(connectTask); ConnectWorkerThread线程，调用NewSession方法创建连接的会话。12345678910111213private void ConnectWorkerThread(object o) &#123; ConnectTask task = (ConnectTask)o; Exception error = null; Pool pool = null; try &#123; log.DebugFormat(&quot;IXenConnection: trying to connect to &#123;0&#125;&quot;, HostnameWithPort); //创建连接会话 Session session = NewSession(task.Hostname, task.Port, Username, Password, false); // Save the session so we can log it out later task.Session = session; NewSession方法调用GetNewSession方法1234567private Session NewSession(string hostname, int port, string username, string password, bool isElevated) &#123; Password = password; Username = username; return GetNewSession(hostname, port, username, password, isElevated); &#125; GetNewSession方法，创建会话之后，使用用户名密码登录1234567891011121314151617181920212223private Session GetNewSession(string hostname, int port, string username, string password, bool isElevated) &#123; const int DELAY = 250; // unit = ms int attempt = 0; while (true) &#123; attempt++; string uname = isElevated ? username : Username; string pwd = isElevated ? password : Password; // Keep the password that we&apos;re using for this iteration, as it may // be changed by another thread handling an authentication failure. // For elevated session we use the elevated username and password passed into this function, // as the connection&apos;s Username and Password are not updated. //创建会话 Session session = SessionFactory.CreateSession(this, hostname, port); if (isElevated) session.IsElevatedSession = true; try &#123; //使用用户名、密码登录 session.login_with_password(uname, pwd, !string.IsNullOrEmpty(Version) ? Version : Helper.APIVersionString(API_Version.LATEST), Session.UserAgent); return session; &#125;]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[XenCenter分析（一）]]></title>
      <url>%2F2017%2F02%2F27%2Fxencenter-1%2F</url>
      <content type="text"><![CDATA[XenCenter是XenServer的Windows客户端管理工具，可以非常方便地管理和监视XenServer主机和资源池，以及部署、监视、管理和迁移虚拟机。 主要有以下功能： 资源池管理 物理主机管理 虚拟机管理 存储池管理 模板管理 代码放在GitHub上托管，XenCenter 这个是我修改过的XenCenter版本。 项目结构 CFUValidator： CommandLib：命令操作的公共类库，主要是IO、tar、HTTP splash：启动界面的效果，使用C++编写 XenAdmin：XenCenter的主项目，包含所有的page、dialog、view XenAdminTests：XenCenter的测试代码 XenCenterLib：一些工具类库 XenCenterVNC：物理主机、虚拟机的控制台 XenModel：模型类以及XAPI接口 XenOvfApi：OVF相关的API XenOvfTransport：OVF导入导出的工具类库 XenServerHealthCheck：主机健康状态检查 xva_verify：xva文件校验 XenAdminXenAdmin是XenCenter代码的主程序的入口，基于C#的WinForm应用程序开发。研究XenCenter先从XenAdmin项目入手。 XenAdmin主要有以下的代码目录： Actions：界面UI与导入OVF虚拟机的action类 Alerts：告警提示类 Commands：命令模式Commands操作类 ConsoleView：VNC控制台的视图与逻辑操作类 Controls：UI界面 Core：公共类库 Diagnostics：资源（Pool、Host、VM、SR等）检查、状态（HA、PBD等）检查 Dialogs：UI的所有弹出对话框 Help：帮助页面以及文档 HomePage：启动XenCenter之后的主页信息 Images：图片资源 Network：SSL连接与XAPI连接XenServer Plugins：插件管理 Properties：license、配置、资源 RDP： ReportViewer：报表视图 ServerDBs：模拟数据库 SettingsPanels：资源设置相关的page TabPages：菜单的tabpages TestResources：测试资源 Utils：工具类 Wizards：向导基础page与向导式操作page Wlb：负载均衡 XenSearch：搜索相关的类 Program.csWinForm应用的Program类main方法12345678static public void Main(string[] Args) &#123; //Upgrade settings //获取当前操作系统信息 System.Reflection.Assembly a = System.Reflection.Assembly.GetExecutingAssembly(); Version appVersion = a.GetName().Version; string appVersionString = appVersion.ToString(); log.DebugFormat(&quot;Application version of new settings &#123;0&#125;&quot;, appVersionString); 1234567//清空XAPI连接信息ConnectionsManager.XenConnections.Clear();//清空连接的历史信息ConnectionsManager.History.Clear();//初始化搜索Search.InitSearch(Branding.Search);TreeSearch.InitSearch(); 1234567891011121314151617switch (Environment.OSVersion.Version.Major) &#123; case 6: // Vista, 2K8, Win7. if (Application.RenderWithVisualStyles) &#123; // Vista, Win7 with styles. //根据操作系统版本设置样式 TitleBarStartColor = Color.FromArgb(242, 242, 242); TitleBarEndColor = Color.FromArgb(207, 207, 207); TitleBarBorderColor = Color.FromArgb(160, 160, 160); TitleBarForeColor = Color.FromArgb(60, 60, 60); HeaderGradientForeColor = Color.White; HeaderGradientFont = new Font(DefaultFont.FontFamily, 11.25f); HeaderGradientFontSmall = DefaultFont; TabbedDialogHeaderFont = HeaderGradientFont; TabPageRowBorder = Color.Gainsboro; TabPageRowHeader = Color.WhiteSmoke; 1234//运行主窗口MainWindow mainWindow = new MainWindow(argType, args);Application.Run(mainWindow); MainWindow是XenCenter的主窗口。 上面部分是菜单栏与操作栏 左侧是树形菜单栏 右侧是主页与选项]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[WinCenterClient分析（二）]]></title>
      <url>%2F2017%2F02%2F24%2Fwincenterclient-2%2F</url>
      <content type="text"><![CDATA[WinCenterClient是WinCenter-Appliance虚拟化管理系统安装程序，使用C# WinForm 框架Wizard方式来实现导入WinCenter虚拟机的功能。 包含以下项目 SharpSSH：C#版本的连接SSH类库，版本比较旧，一直未更新，在WinCenterClient5.4及以后的版本不再使用 Renci.SshNet：C#版本的连接SSH类库，在WinCenterClient5.4及以后的版本使用 WinCenter：WinForm界面 WinServer：C#版本的API Winzardlib：向导式框架类库 SharpSSHSharpSSH是一个C#开发的实现了SSH2协议的开源组件，支持SSH/SCP/SFTP等协议。 SharpSSH执行sh命令的输入输出都是定向到console。因此不容易从其中取出它的结果。因此需要对源码进行一定的修改，从而得到我们想要的结果。 SSH操作： SharpSSH中修改 SshStream.cs，增加一个方法，把输出定向到流中public void set_OutputStream(Stream stream) WinCenter中定义ShellHelp.cs工具类来连接SSH，执行Linux命令，获取输出结果字符串 由于SharpSSH版本比较旧，一直未更新，而且不支持一些加密算法，在WinCenterClient5.4及以后的版本中使用Renci.SshNet来代替。 Renci.SshNetRenci.SshNet支持多种加密算法，且不需要处理输入输出流，可以代替SharpSSH。 WinServerWinServer是C#版本的API。 WizardLibWizardLib是Wizard向导式WinForm 界面的lib类库。 主要的类： WizardFormWizardForm派生自Form，它负责管理WizardPage集合。另外，它提供了缺省的Wizard界面。创建你自己的向导类的第一件事就是扩展这个类，通常从该类继承，编辑属性或者添加额外的按钮。 WizardPageWizardPage派生自UserControl。提供了向导相关的方法，如 OnSetActive, OnWizardNext等。 WizardLib类图 WizardLib通用组件提供复用的组件给WinCenter界面使用。 IP地址及网关输入控件IPBox IPBox继承UserControl，在组件中放置四个textBox，通过重写OnPaint 、OnResize方法绘制边框等。 Button控件MyButton 默认的Button控件在设置透明背景图片时，鼠标点下会出现黑色的边框。MyButton控件继承Button，MyButton重写ShowFocusCues方法，去掉黑色的边框。 Loading效果控件MyOpaqueLayer 在加载数据时间比较长的情况下，需要使用loading效果提示用户正在加载数据或者操作。 MyOpaqueLayer继承Control，重写OnPaint方法绘制窗体，设置透明的屏蔽层。 OpaqueCommand类用来显示与隐藏屏蔽层。 DataGridView单选框控件 要达到单选效果，需要在业务处理中监听单元格点击事件，触发时，选择当前行，并取消其他所有行的选中，同时更改数据状态。 DataGridView组件默认是不支持单选，要实现单选的功能必须重写DataGridViewCell DataGridViewDisableCheckBoxCell继承DataGridViewCheckBoxCell，重写Paint方法来绘制单选框RadioButton DataGridViewDisableCheckBoxColumn 继承DataGridViewCheckBoxColumn]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[WinCenterClient分析（一）]]></title>
      <url>%2F2017%2F02%2F24%2Fwincenterclient-1%2F</url>
      <content type="text"><![CDATA[WinCenterClient是WinCenter-Appliance虚拟化管理系统安装程序，使用C# WinForm 框架Wizard方式来实现导入WinCenter虚拟机的功能。 实现过程 使用HHTP/HHTPS方式导入xva模板文件到主机上，并生成虚拟机 根据元数据文件信息给虚拟机配置相应资源 配置虚拟机 删除原有的VIF 创建新的VIF，生成新的MAC地址，并设置网络为目标主机的管理网络 设置虚拟机的名称 设置虚拟机所属的物理主机 启动虚拟机 动态配置虚拟机的网络信息（IP、网关、子网掩码） 向导式操作步骤向导式的操作，主要由以下步骤组成： 导入来源：虚拟机模板的文件路径 配置主机：虚拟机导入的目标主机 配置存储：虚拟机磁盘数据导入的目标存储 配置网络：配置虚拟机的IP、子网掩码、网关信息 安装虚拟机：导入、配置并启动虚拟机 导入来源选择提前制作好的xva模板文件作为导入来源。在这个步骤中校验模板文件并获取模板文件的元数据信息，包括虚拟磁盘大小、虚拟CPU数量、内存大小、VIF数量、虚拟化版本信息等 配置主机配置虚拟机导入的目标主机 配置存储配置虚拟机导入的目标存储，目标存储可以是目标主机上的本地存储，也可以是资源池内的共享存储 配置网络配置虚拟机可用的IP、子网掩码、网关信息 安装虚拟机导入、配置并启动虚拟机，这个步骤是整个安装过程的最重要的步骤。]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[hexo使用本地图片]]></title>
      <url>%2F2017%2F02%2F24%2Fhexo-local-pic%2F</url>
      <content type="text"><![CDATA[在文章中使用本地方式引用图片。 安装插件npm install hexo-asset-image --save 修改站点_config.ymlpost_asset_folder:true Hexo提供了一种更方便管理 Asset 的设定：post_asset_folder。当设置post_asset_folder为true参数后，在hexo-hey中发布文章时，Hexo会自动建立一个与文章同名的文件夹，可以把与该文章相关的所有图片资源都放到那个文件夹。 hexo使用本地图片 ├── 图片名称.png ├── 图片名称1.png └── 图片名称2.png hexo使用本地图片.md 在文章中直接引用图片![图片title](图片名称.png)]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[hexo博客中的next主题配置]]></title>
      <url>%2F2017%2F02%2F15%2Fhexo-config-more%2F</url>
      <content type="text"><![CDATA[博客搭建之后，修改next主题，增加以下功能： 首页「阅读全文」 站内搜索 文章阅读次数 文章多说评论 社交链接GitHub 设置「阅读全文」在首页显示文章的摘录并显示“阅读全文”按钮，可以通过以下方法： 在每篇文章内容中添加 &lt;!– more –&gt; 配置站内搜索安装hexo-generator-searchdb 1$ npm install hexo-generator-searchdb --save 在站点的 _config.yml中增加 search: path: search.xml field: post 配置阅读次数注册LeanCloud账号，新建应用和Class，可参考next主题配置阅读次数统计 修改next主题的_config.yml文件，将LeanCloud的App ID与App Key复制到leancloud_visitors部分 leancloud_visitors: enable: true app_id: LE6b1aXadasdaksderewldfgrn-gzGzoHsz app_key: WHQseHdLj7t7hyj5546546EXO7 配置多说评论注册多说，并创建站点，可参考next主题集成第三方服务 修改站点与next主题的_config.yml文件，配置duoshuo_shortname duoshuo_shortname: hl10502 hl10502为多说站点名称 修改next主题的_config.yml文件，配置多说分享 uoshuo_share: true 在每一条多说评论后显示评论者所使用的代理信息（如 操作系统、浏览器），修改next主题_config.yml文件，配置 duoshuo_info字段， 设置如下： duoshuo_info: ua_enable: true admin_enable: false user_id: 0 #admin_nickname: 侧边栏社交链接侧栏社交链接的修改包含两个部分，第一是链接，第二是链接图标。两者配置均在next主题的_config.yml文件中。 配置链接，修改social social: #LinkLabel: Link GitHub: https://github.com/hl10502 #Twitter: https://twitter.com/your-user-name #微博: http://weibo.com/your-user-name #豆瓣: http://douban.com/people/your-user-name #知乎: 配置链接图标，修改 social_icons: enable: true # Icon Mappings. # KeyMapsToSocalItemKey: NameOfTheIconFromFontAwesome GitHub: github #Twitter: twitter #Weibo: weibo]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[hexo配置标签]]></title>
      <url>%2F2017%2F02%2F14%2Fhexo-config-tags%2F</url>
      <content type="text"><![CDATA[创建标签页面在source目录下，自动创建tags/index.md文件1$ hexo new page tags 修改标签内容添加以下内容到source/tags/index.md文件 type: &quot;tags&quot; comments: false 修改_config.yml站点的_config.yml文件配置tag_dir tag_dir: tags next主题的_config.yml文件配置tags menu: tags: /tags menu_icons: tags: tags 博客文章中添加tags比如在博客文章《使用hexo+github搭建个人博客》的title下面添加 tags: [hexo,hexo-hey,github,个人博客]]]></content>
    </entry>

    
    <entry>
      <title><![CDATA[使用hexo+github搭建个人博客]]></title>
      <url>%2F2017%2F02%2F13%2Fhexo-github-setup-blog%2F</url>
      <content type="text"><![CDATA[使用hexo+github在windows7环境下搭建个人技术博客,在git bash下执行以下bash操作命令。 环境准备 windows7 x64 nodejs 6.9.5 git 2.11.0 hexo 3.2.2 配置github创建仓库打开github主页，新建github仓库，仓库名称为”hl10502.github.com” 配置SSH key生成key 1$ ssh-keygen -t rsa -C "xxx@126.com" //邮箱为github注册的邮箱 打开github主页，进入个人设置 -&gt; SSH and GPG keys -&gt; New SSH key： 将生成的key文件（用户目录下的.ssh\id_rsa.pub）内容copy到新建的key中 测试SSH配置 1$ ssh -T git@github.com //固定邮箱 配置全局的用户名、邮箱 1$ git config --global user.name "hl10502" //github用户名 1$ git config --global user.email "xxx@126.com" //邮箱为github注册的邮箱 配置hexo安装hexo（注意：之前安装的nodejs在安装时需要添加环境变量） 1$ npm install -g hexo 在F盘下创建 F:\blog\hexo文件夹，作为blog代码目录，初始化hexo 1$ cd /f/blog/hexo 1$ hexo init 下载next主题 1$ git clone https://github.com/iissnan/hexo-theme-next themes/next 修改站点的_config.yml文件的主题为 theme: next 生成静态文件 1$ hexo g 启动hexo服务，可以通过 http://localhost:4000 访问 1$ hexo s 配置hexo-hey插件安装hexo-hey插件 1$ npm install hexo-hey --save 配置站点的_config.yml文件，添加admin部分 #hexo-hey插件 admin: name: hexo password: hey secret: hey hexo expire: 60*1 # cors: http://localhost:4000 登录 http://localhost:4000/admin，可以新建文章，发布到本地生成.md文件 用户名：hexo密码：hey 发布hexo到github安装hexo-deployer-git插件 1$ cd /f/blog/hexo/ 1$ npm install hexo-deployer-git --save 配置站点的_config.yml文件，修改deploy部分，使用SSH上传文件 deploy: type: git repository: git@github.com:hl10502/hl10502.github.com.git branch: master hl10502为github的用户名，hl10502.github.com为仓库名称 部署hexo，上传到github 1$ hexo d 访问个人博客： https://hl10502.github.io]]></content>
    </entry>

    
  
  
</search>