XenServer 6.2 只读且磁盘 I/O 缓慢

XenServer 6.2 只读且磁盘 I/O 缓慢

几个月前,我虚拟化了一个数据中心,我们有 3 台 HP DL360 G5 服务器,每台都有 32GB 内存和双 Intel Xeon。最近我们遇到了 2 个问题,第一个是磁盘读取速度变得非常慢。在只有几个文件的 Linux VM 上键入“ls”需要花费数秒才能返回文件列表。此外,集群上的 VM 有时会自行重新安装为只读文件系统。主机上的 Dmesg 会产生大量“DRDY ERR”错误。我们使用的主要存储库位于 Drobo B800i 上,通过 isci 共享。我在下面发布了 iostat 和来自 dmesg 的 DRDY 错误的 grep,这些是企业服务器,它们会间歇性地停机,这从来都不是好事:

以下是其中一台服务器的 Iostat:[root@XenServer-1 tmp]# iostat Linux 2.6.32.43-0.4.1.xs1.8.0.835.170778xen (XenServer-1.ethoplex.com) 07/31/2014

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.42    0.00    0.46    3.51    0.40   95.21

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
cciss/c0d0       17.30        76.54       304.24  893755376 3552874247
cciss/c0d0p1      1.04         0.27        22.82    3169526  266433488
cciss/c0d0p2      0.00         0.01         0.00      73890          0
cciss/c0d0p3     16.25        76.24       281.43  890365720 3286440759
sda              76.84        59.78        87.32  698047689 1019733585
dm-0              0.68         0.95         0.28   11071656    3217737
sdb               3.44       177.64        37.74 2074378210  440737634
dm-2              0.00         0.01         0.00     135808       2216
dm-3             12.23       361.61       131.55 4222728781 1536204287
sdc               4.05        27.93       328.02  326147810 3830552980
sdd               6.23       101.72       113.03 1187808537 1319897350
tda               1.61         9.74        40.01  113749658  467248640
dm-28             0.84        36.78        23.11  429521222  269838659
dm-14             0.24        56.24         0.00  656723598          0
dm-21             0.08        18.17         0.00  212172507          0
tdb               0.08         0.12         1.44    1384368   16853616
dm-5              0.38         4.03        36.17   47063052  422416430
tdc               0.61         4.03        36.10   47062722  421602000
dm-7              1.26        17.74         5.51  207110960   64292628
tde               1.22        17.64         5.49  206019946   64129696
dm-30             0.03         0.01         0.60      61956    6979438
dm-4              0.02         0.00         8.85       1014  103326613
tdd               0.11         0.00         8.82       1264  103049216
dm-9              0.00         0.02         0.05     175978     591472
tdg               0.00         0.02         0.05     175950     590704
dm-10             0.01         0.09         0.21    1104226    2488947
tdf               0.01         0.09         0.21    1105562    2472346
dm-6              0.00         0.00         0.04       1568     419135
dm-16             0.00         0.01         0.00     132105          0
dm-17             0.03         0.05         0.76     625890    8867990
dm-8              0.00         0.06         0.10     752923    1226072
tdh               0.00         0.07         0.10     788356    1218922
tdi               0.00         0.00         0.00        884          0

Dmesg Grep DRDY:

[11645348.631020] ata1.00: status: { DRDY ERR }
[11646434.714902] ata1.00: status: { DRDY ERR }
[11648427.773389] ata1.00: status: { DRDY ERR }
[11648950.139954] ata1.00: status: { DRDY ERR }
[11649612.475350] ata1.00: status: { DRDY ERR }
[11650177.522603] ata1.00: status: { DRDY ERR }
[11650649.818020] ata1.00: status: { DRDY }
[11651837.989833] ata1.00: status: { DRDY ERR }
[11654729.414605] ata1.00: status: { DRDY ERR }
[11655685.782290] ata1.00: status: { DRDY ERR }
[11657120.774143] ata1.00: status: { DRDY ERR }
[11659704.724995] ata1.00: status: { DRDY }
[11661322.210812] ata1.00: status: { DRDY ERR }
[11662029.088563] ata1.00: status: { DRDY ERR }
[11663314.187972] ata1.00: status: { DRDY ERR }
[11667978.796829] ata1.00: status: { DRDY ERR }
[11670487.088008] ata1.00: status: { DRDY ERR }
[11671800.577054] ata1.00: status: { DRDY ERR }

消息:

[11464689.083861] sr 1:0:0:0: CDB: Get event status notification: 4a 01 00 00 10 00 00 00 08 00
[11464689.083875] ata1.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio 16392 in
[11464689.083876]res 40/00:03:00:00:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
[11464689.083896] ata1.00: status: { DRDY }
[11464694.133755] ata1: link is slow to respond, please be patient (ready=0)
[11464699.123711] ata1: device not ready (errno=-16), forcing hardreset
[11464699.123727] ata1: soft resetting link
[11464699.344063] ata1.00: configured for PIO0
[11464699.348375] ata1: EH complete
[11464706.383733] ata1.00: qc timeout (cmd 0xa0)
[11464706.383766] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[11464706.383782] sr 1:0:0:0: CDB: Test Unit Ready: 00 00 00 00 00 00
[11464706.383794] ata1.00: cmd a0/00:00:00:00:00/00:00:00:00:00/a0 tag 0
[11464706.383795]res 51/20:03:00:00:00/00:00:00:00:00/a0 Emask 0x5 (timeout)
[11464706.383806] ata1.00: status: { DRDY ERR }
[11464711.433625] ata1: link is slow to respond, please be patient (ready=0)
[11464716.433591] ata1: device not ready (errno=-16), forcing hardreset

相关内容