![DRBD 设备上的 Xen DomU:屏障错误](https://linux22.com/image/585800/DRBD%20%E8%AE%BE%E5%A4%87%E4%B8%8A%E7%9A%84%20Xen%20DomU%EF%BC%9A%E5%B1%8F%E9%9A%9C%E9%94%99%E8%AF%AF.png)
我正在测试设置带有 DRBD 存储的 Xen DomU,以便轻松进行故障转移。大多数情况下,启动 DomU 后,我会立即收到 IO 错误:
[ 3.153370] EXT3-fs (xvda2): using internal journal
[ 3.277115] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 3.336014] nf_conntrack version 0.5.0 (3899 buckets, 15596 max)
[ 3.515604] init: failsafe main process (397) killed by TERM signal
[ 3.801589] blkfront: barrier: write xvda2 op failed
[ 3.801597] blkfront: xvda2: barrier or flush: disabled
[ 3.801611] end_request: I/O error, dev xvda2, sector 52171168
[ 3.801630] end_request: I/O error, dev xvda2, sector 52171168
[ 3.801642] Buffer I/O error on device xvda2, logical block 6521396
[ 3.801652] lost page write due to I/O error on xvda2
[ 3.801755] Aborting journal on device xvda2.
[ 3.804415] EXT3-fs (xvda2): error: ext3_journal_start_sb: Detected aborted journal
[ 3.804434] EXT3-fs (xvda2): error: remounting filesystem read-only
[ 3.814754] journal commit I/O error
[ 6.973831] init: udev-fallback-graphics main process (538) terminated with status 1
[ 6.992267] init: plymouth-splash main process (546) terminated with status 1
drbdsetup 的手册页说 LVM(我使用的)不支持屏障(更好地称为tagged command queuing
或native command queing
),因此我将 drbd 设备配置为不使用屏障。这可以在中看到/proc/drbd
(通过“ wo:f
,表示刷新,drbd 在屏障之后选择的下一个方法):
3: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
ns:2160152 nr:520204 dw:2680344 dr:2678107 al:3549 bm:9183 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
在另一台主机上:
3: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----
ns:0 nr:2160152 dw:2160152 dr:0 al:0 bm:8052 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
我还按照 drbd 文档启用了选项 disable_sendpage:
cat /sys/module/drbd/parameters/disable_sendpage
Y
我还尝试将 barriers=0 添加到 fstab 作为挂载选项。但有时它仍然显示:
[ 58.603896] blkfront: barrier: write xvda2 op failed
[ 58.603903] blkfront: xvda2: barrier or flush: disabled
我甚至不知道 ext3 是否有 nobarrier 选项。而且,由于我的存储系统只有一个是电池供电的,所以它无论如何都不是智能的。
为什么当我禁用它时它仍然会抱怨障碍?
两位主持人均为:
Debian: 6.0.4
uname -a: Linux 2.6.32-5-xen-amd64
drbd: 8.3.7
Xen: 4.0.1
客人:
Ubuntu 12.04 LTS
uname -a: Linux 3.2.0-24-generic pvops
drbd 资源:
resource drbdvm
{
meta-disk internal;
device /dev/drbd3;
startup
{
# The timeout value when the last known state of the other side was available. 0 means infinite.
wfc-timeout 0;
# Timeout value when the last known state was disconnected. 0 means infinite.
degr-wfc-timeout 180;
}
syncer
{
# This is recommended only for low-bandwidth lines, to only send those
# blocks which really have changed.
#csums-alg md5;
# Set to about half your net speed
rate 60M;
# It seems that this option moved to the 'net' section in drbd 8.4. (later release than Debian has currently)
verify-alg md5;
}
net
{
# The manpage says this is recommended only in pre-production (because of its performance), to determine
# if your LAN card has a TCP checksum offloading bug.
#data-integrity-alg md5;
}
disk
{
# Detach causes the device to work over-the-network-only after the
# underlying disk fails. Detach is not default for historical reasons, but is
# recommended by the docs.
# However, the Debian defaults in drbd.conf suggest the machine will reboot in that event...
on-io-error detach;
# LVM doesn't support barriers, so disabling it. It will revert to flush. Check wo: in /proc/drbd. If you don't disable it, you get IO errors.
no-disk-barrier;
}
on host1
{
# universe is a VG
disk /dev/universe/drbdvm-disk;
address 10.0.0.1:7792;
}
on host2
{
# universe is a VG
disk /dev/universe/drbdvm-disk;
address 10.0.0.2:7792;
}
}
DomU 配置:
bootloader = '/usr/lib/xen-default/bin/pygrub'
vcpus = '2'
memory = '512'
#
# Disk device(s).
#
root = '/dev/xvda2 ro'
disk = [
'phy:/dev/drbd3,xvda2,w',
'phy:/dev/universe/drbdvm-swap,xvda1,w',
]
#
# Hostname
#
name = 'drbdvm'
#
# Networking
#
# fake IP for posting
vif = [ 'ip=1.2.3.4,mac=00:16:3E:22:A8:A7' ]
#
# Behaviour
#
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
在我的测试设置中:主主机的存储是带电池的 9650SE SATA-II RAID PCIe。辅助存储是软件 RAID1。
DRBD+Xen 不是用得比较多吗?有这些问题,就没法用了。
答案1
我不知道这是否会改变什么,但您也可以在 DomU 配置中指定 DRBD 卷,如下所示:
disk = [ 'drbd:drbdvm,xvda2,w' ... ]
这样,在创建 DomU 时,Xen 会自动将当前节点设为指定资源的主节点(除非该资源已被第二台机器使用)。此外,当 DomU 被销毁时,资源将被释放。
我有许多像这样运行的 DRBD 对,但从未见过您发布的错误。
答案2
添加extra = " barrier=off"
到您的 DomU 配置。请注意障碍。
还要在 DomU 的 /etc/fstab 中添加相应的 barrier/off 选项(根据文件系统的挂载选项)。
更新:
屏障/关闭选项是确保屏障关闭的第二个措施。
至于屏障操作:正如您在启动期间看到的,这些操作会失败。因此关闭它不会让事情变得更糟。除此之外,这些屏障只有在启用了写入缓存的硬盘上才真正有意义,因为在电源故障时不会将数据写回磁盘。
服务器应该有电池供电的 UPS 以及电池供电的 RAID 控制器。因此,开启屏障只会降低性能(即使它可以工作)。