我有 2 个大型 mdadm 阵列，并将它们组合在一个卷组中。我最近在向第二个阵列添加新驱动器时遇到了断电。通常，在中断后让阵列重新启动或恢复重塑不会遇到重大问题，但这次我遇到了很多麻烦。
系统详细信息：CentOS 6.8 x64
Linux myserver 2.6.32-642.4.2.el6.x86_64 #1 SMP Tue Aug 23 19:58:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
当我第一次启动时，阵列会组装但不会启动，因为它已降级。我无法开始：
mdadm -R /dev/md126
然后我停止它：
mdadm -S /dev/md126
如果我正常组装，它不会启动：
mdadm --assemble --scan
mdadm: device 26 in /dev/md/128 has wrong state in superblock, but /dev/sdn seems ok
mdadm: /dev/md/128 assembled from 12 drives and 2 spares - not enough to start the array while not clean - consider --force.
mdadm: No arrays found in config file or automatically
所以我组装：
mdadm --assemble --scan --force
mdadm: clearing FAULTY flag for device 1 in /dev/md/128 for /dev/sdn
mdadm: Marking array /dev/md/128 as 'clean'
mdadm: /dev/md/128 has been started with 12 drives (out of 13) and 2 spares.
此时，我的会话已挂起。但是，如果我 ssh 回到设备，我可以在其他会话继续挂起时运行命令。
猫 /proc/mdstat
md128 : active raid6 sdf[0] sdl[9](S) sdn[14](S) sdo[6] sdm[7] sdc[10] sdb[11] sdd[12] sde[13] sdk[5] sdj[4] sdi[3] sdh[2] sdg[1]
23441323008 blocks super 1.2 level 6, 512k chunk, algorithm 2 [13/12] [UUUUUUUUUUUU_]
[>....................] reshape = 0.0% (128496/3906887168) finish=133658963.6min speed=0K/sec
bitmap: 1/30 pages [4KB], 65536KB chunk
md127 : active (auto-read-only) raid6 sdaa[2] sdab[13] sdac[0] sdad[15] sdy[4] sdx[5] sdz[14] sdw[6] sdr[9] sds[8] sdq[10] sdp[16] sdu[11] sdv[7] sdt[12]
50789533184 blocks super 1.2 level 6, 512k chunk, algorithm 2 [15/15] [UUUUUUUUUUUUUUU]
unused devices: <none>
重塑永远不会超过这一点。它总是卡在那里。
在 dmesg 中我收到以下错误
created bitmap (30 pages) for device md128
md128: bitmap initialized from disk: read 2 pages, set 1 of 59615 bits
md128: detected capacity change from 0 to 24003914760192
md: reshape of RAID array md128
md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
md128:
md: using 128k window, over a total of 3906887168k.
unknown partition table
INFO: task mdadm:2357 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mdadm D 0000000000000002 0 2357 2179 0x00000004
ffff880821773688 0000000000000082 0000000000000000 0000000000000086
ffff880821773618 ffffffff810633a3 0000013e9de36948 ffff88081f96d440
ffff88081dc65ed0 000000010010516a ffff88081d713068 ffff880821773fd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff81130d00>] ? mempool_alloc+0x20/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
[<ffffffff812ab7de>] ? __sg_alloc_table+0x7e/0x130
[<ffffffff81056cc5>] ? gup_pte_range+0xe5/0x130
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8143066f>] md_make_request+0xdf/0x240
[<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
[<ffffffff811d941c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff811d60cb>] ? bio_alloc_bioset+0x5b/0xf0
[<ffffffff8127e180>] submit_bio+0x70/0x120
[<ffffffff811daabd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811db127>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811d73a7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff8113028b>] generic_file_aio_read+0x6bb/0x700
[<ffffffff8143877f>] ? md_ioctl+0x31f/0x1ac0
[<ffffffff811d7f00>] ? blkdev_open+0x0/0xc0
[<ffffffff81196d47>] ? __dentry_open+0x257/0x380
[<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
[<ffffffff81199a6a>] do_sync_read+0xfa/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811d66bc>] ? block_ioctl+0x3c/0x40
[<ffffffff811af522>] ? vfs_ioctl+0x22/0xa0
[<ffffffff811af6c4>] ? do_vfs_ioctl+0x84/0x580
[<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
[<ffffffff8119a365>] vfs_read+0xb5/0x1a0
[<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
[<ffffffff8119a6b1>] sys_read+0x51/0xb0
[<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task md128_reshape:2428 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
md128_reshape D 0000000000000001 0 2428 2 0x00000000
ffff88081fdcfa60 0000000000000046 0000000000000000 0000000000000086
ffff88081fdcf9f0 ffffffff810633a3 0000013d983220e0 ffff88081f96d440
ffff88081dc65ed0 0000000100103f9f ffff88081ea1e5f8 ffff88081fdcffd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa02447c0>] reshape_request+0x440/0xa20 [raid456]
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa02450b2>] sync_request+0x312/0x3a0 [raid456]
[<ffffffff81430ff7>] md_do_sync+0x6c7/0xd60
[<ffffffff81431ae5>] md_thread+0x115/0x150
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff814319d0>] ? md_thread+0x0/0x150
[<ffffffff810a640e>] kthread+0x9e/0xc0
[<ffffffff8100c28a>] child_rip+0xa/0x20
[<ffffffff810a6370>] ? kthread+0x0/0xc0
[<ffffffff8100c280>] ? child_rip+0x0/0x20
INFO: task blkid:2430 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
blkid D 0000000000000002 0 2430 1 0x00000000
ffff8808211d36f8 0000000000000082 0000000000000000 0000000000000082
ffff8808211d3688 ffffffff810633a3 0000013d9831af63 ffff88081f96d440
ffff88081dc65ed0 0000000100104045 ffff88081e615ad8 ffff8808211d3fd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff8142af00>] ? try_module_get+0x30/0xb0
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
[<ffffffff81340b4b>] ? mix_pool_bytes_extract+0x16b/0x180
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff81151319>] ? zone_statistics+0x99/0xc0
[<ffffffff8143066f>] md_make_request+0xdf/0x240
[<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
[<ffffffff81130ba5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81130d43>] ? mempool_alloc+0x63/0x140
[<ffffffff8127e180>] submit_bio+0x70/0x120
[<ffffffff811cfc8d>] submit_bh+0x11d/0x1f0
[<ffffffff811d2a9c>] block_read_full_page+0x27c/0x3a0
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811513ee>] ? __inc_zone_page_state+0x2e/0x30
[<ffffffff81145480>] ? __lru_cache_add+0x40/0x90
[<ffffffff811d7498>] blkdev_readpage+0x18/0x20
[<ffffffff811441aa>] __do_page_cache_readahead+0x20a/0x210
[<ffffffff81144251>] force_page_cache_readahead+0x71/0xa0
[<ffffffff81144773>] page_cache_sync_readahead+0x43/0x50
[<ffffffff81130128>] generic_file_aio_read+0x558/0x700
[<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
[<ffffffff81199a6a>] do_sync_read+0xfa/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8115f3d0>] ? unmap_region+0x110/0x130
[<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
[<ffffffff8119a365>] vfs_read+0xb5/0x1a0
[<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
[<ffffffff8119a6b1>] sys_read+0x51/0xb0
[<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task mdadm:2357 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mdadm D 0000000000000002 0 2357 2179 0x00000004
ffff880821773688 0000000000000082 0000000000000000 0000000000000086
ffff880821773618 ffffffff810633a3 0000013e9de36948 ffff88081f96d440
ffff88081dc65ed0 000000010010516a ffff88081d713068 ffff880821773fd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff81130d00>] ? mempool_alloc+0x20/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
[<ffffffff812ab7de>] ? __sg_alloc_table+0x7e/0x130
[<ffffffff81056cc5>] ? gup_pte_range+0xe5/0x130
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8143066f>] md_make_request+0xdf/0x240
[<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
[<ffffffff811d941c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff811d60cb>] ? bio_alloc_bioset+0x5b/0xf0
[<ffffffff8127e180>] submit_bio+0x70/0x120
[<ffffffff811daabd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811db127>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811d73a7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff8113028b>] generic_file_aio_read+0x6bb/0x700
[<ffffffff8143877f>] ? md_ioctl+0x31f/0x1ac0
[<ffffffff811d7f00>] ? blkdev_open+0x0/0xc0
[<ffffffff81196d47>] ? __dentry_open+0x257/0x380
[<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
[<ffffffff81199a6a>] do_sync_read+0xfa/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811d66bc>] ? block_ioctl+0x3c/0x40
[<ffffffff811af522>] ? vfs_ioctl+0x22/0xa0
[<ffffffff811af6c4>] ? do_vfs_ioctl+0x84/0x580
[<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
[<ffffffff8119a365>] vfs_read+0xb5/0x1a0
[<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
[<ffffffff8119a6b1>] sys_read+0x51/0xb0
[<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task md128_reshape:2428 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
md128_reshape D 0000000000000001 0 2428 2 0x00000000
ffff88081fdcfa60 0000000000000046 0000000000000000 0000000000000086
ffff88081fdcf9f0 ffffffff810633a3 0000013d983220e0 ffff88081f96d440
ffff88081dc65ed0 0000000100103f9f ffff88081ea1e5f8 ffff88081fdcffd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa02447c0>] reshape_request+0x440/0xa20 [raid456]
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa02450b2>] sync_request+0x312/0x3a0 [raid456]
[<ffffffff81430ff7>] md_do_sync+0x6c7/0xd60
[<ffffffff81431ae5>] md_thread+0x115/0x150
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff814319d0>] ? md_thread+0x0/0x150
[<ffffffff810a640e>] kthread+0x9e/0xc0
[<ffffffff8100c28a>] child_rip+0xa/0x20
[<ffffffff810a6370>] ? kthread+0x0/0xc0
[<ffffffff8100c280>] ? child_rip+0x0/0x20
INFO: task blkid:2430 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
blkid D 0000000000000002 0 2430 1 0x00000000
ffff8808211d36f8 0000000000000082 0000000000000000 0000000000000082
ffff8808211d3688 ffffffff810633a3 0000013d9831af63 ffff88081f96d440
ffff88081dc65ed0 0000000100104045 ffff88081e615ad8 ffff8808211d3fd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff8142af00>] ? try_module_get+0x30/0xb0
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
[<ffffffff81340b4b>] ? mix_pool_bytes_extract+0x16b/0x180
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff81151319>] ? zone_statistics+0x99/0xc0
[<ffffffff8143066f>] md_make_request+0xdf/0x240
[<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
[<ffffffff81130ba5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81130d43>] ? mempool_alloc+0x63/0x140
[<ffffffff8127e180>] submit_bio+0x70/0x120
[<ffffffff811cfc8d>] submit_bh+0x11d/0x1f0
[<ffffffff811d2a9c>] block_read_full_page+0x27c/0x3a0
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811513ee>] ? __inc_zone_page_state+0x2e/0x30
[<ffffffff81145480>] ? __lru_cache_add+0x40/0x90
[<ffffffff811d7498>] blkdev_readpage+0x18/0x20
[<ffffffff811441aa>] __do_page_cache_readahead+0x20a/0x210
[<ffffffff81144251>] force_page_cache_readahead+0x71/0xa0
[<ffffffff81144773>] page_cache_sync_readahead+0x43/0x50
[<ffffffff81130128>] generic_file_aio_read+0x558/0x700
[<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
[<ffffffff81199a6a>] do_sync_read+0xfa/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8115f3d0>] ? unmap_region+0x110/0x130
[<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
[<ffffffff8119a365>] vfs_read+0xb5/0x1a0
[<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
[<ffffffff8119a6b1>] sys_read+0x51/0xb0
[<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task mdadm:2481 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mdadm D 0000000000000003 0 2481 2459 0x00000004
ffff8808226ef688 0000000000000086 0000000000000000 0000000000000086
ffff8808226ef618 ffffffff810633a3 0000014fb75098d5 ffff88081f96d440
ffff88081dc65ed0 0000000100117098 ffff88081e7f7068 ffff8808226effd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
[<ffffffff8117ec7c>] ? transfer_objects+0x5c/0x80
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8117f66e>] ? cache_alloc_refill+0x9e/0x240
[<ffffffff8143066f>] md_make_request+0xdf/0x240
[<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
[<ffffffff811d941c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81151319>] ? zone_statistics+0x99/0xc0
[<ffffffff811d60cb>] ? bio_alloc_bioset+0x5b/0xf0
[<ffffffff8127e180>] submit_bio+0x70/0x120
[<ffffffff811daabd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811db127>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811d73a7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff8113028b>] generic_file_aio_read+0x6bb/0x700
[<ffffffff8143877f>] ? md_ioctl+0x31f/0x1ac0
[<ffffffff811d7f00>] ? blkdev_open+0x0/0xc0
[<ffffffff81196d47>] ? __dentry_open+0x257/0x380
[<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
[<ffffffff81199a6a>] do_sync_read+0xfa/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811d66bc>] ? block_ioctl+0x3c/0x40
[<ffffffff811af522>] ? vfs_ioctl+0x22/0xa0
[<ffffffff811af6c4>] ? do_vfs_ioctl+0x84/0x580
[<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
[<ffffffff8119a365>] vfs_read+0xb5/0x1a0
[<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
[<ffffffff8119a6b1>] sys_read+0x51/0xb0
[<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task pvscan:2518 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
pvscan D 0000000000000003 0 2518 2505 0x00000000
ffff8808205e3688 0000000000000086 0000000000000000 0000000000000086
ffff8808205e3618 ffffffff810633a3 000001550b5d4884 ffff88081f96d440
ffff88081dc65ed0 000000010011ca13 ffff88081ddf5ad8 ffff8808205e3fd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
[<ffffffff812ab7de>] ? __sg_alloc_table+0x7e/0x130
[<ffffffff81056cc5>] ? gup_pte_range+0xe5/0x130
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8143066f>] md_make_request+0xdf/0x240
[<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
[<ffffffff811d941c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff811d60cb>] ? bio_alloc_bioset+0x5b/0xf0
[<ffffffff8127e180>] submit_bio+0x70/0x120
[<ffffffff811daabd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811db127>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811d73a7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff8113028b>] generic_file_aio_read+0x6bb/0x700
[<ffffffff811d7ef0>] ? blkdev_get+0x10/0x20
[<ffffffff811d7f00>] ? blkdev_open+0x0/0xc0
[<ffffffff81196d47>] ? __dentry_open+0x257/0x380
[<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
[<ffffffff81199a6a>] do_sync_read+0xfa/0x140
[<ffffffff81397a3f>] ? scsi_device_put+0x2f/0x40
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811d66bc>] ? block_ioctl+0x3c/0x40
[<ffffffff811af522>] ? vfs_ioctl+0x22/0xa0
[<ffffffff811af6c4>] ? do_vfs_ioctl+0x84/0x580
[<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
[<ffffffff8119a365>] vfs_read+0xb5/0x1a0
[<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
[<ffffffff8119a6b1>] sys_read+0x51/0xb0
[<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task mdadm:2357 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mdadm D 0000000000000002 0 2357 2179 0x00000004
ffff880821773688 0000000000000082 0000000000000000 0000000000000086
ffff880821773618 ffffffff810633a3 0000013e9de36948 ffff88081f96d440
ffff88081dc65ed0 000000010010516a ffff88081d713068 ffff880821773fd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff81130d00>] ? mempool_alloc+0x20/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
[<ffffffff812ab7de>] ? __sg_alloc_table+0x7e/0x130
[<ffffffff81056cc5>] ? gup_pte_range+0xe5/0x130
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8143066f>] md_make_request+0xdf/0x240
[<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
[<ffffffff811d941c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff811d60cb>] ? bio_alloc_bioset+0x5b/0xf0
[<ffffffff8127e180>] submit_bio+0x70/0x120
[<ffffffff811daabd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811db127>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811d73a7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff8113028b>] generic_file_aio_read+0x6bb/0x700
[<ffffffff8143877f>] ? md_ioctl+0x31f/0x1ac0
[<ffffffff811d7f00>] ? blkdev_open+0x0/0xc0
[<ffffffff81196d47>] ? __dentry_open+0x257/0x380
[<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
[<ffffffff81199a6a>] do_sync_read+0xfa/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811d66bc>] ? block_ioctl+0x3c/0x40
[<ffffffff811af522>] ? vfs_ioctl+0x22/0xa0
[<ffffffff811af6c4>] ? do_vfs_ioctl+0x84/0x580
[<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
[<ffffffff8119a365>] vfs_read+0xb5/0x1a0
[<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
[<ffffffff8119a6b1>] sys_read+0x51/0xb0
[<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task md128_reshape:2428 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
md128_reshape D 0000000000000001 0 2428 2 0x00000000
ffff88081fdcfa60 0000000000000046 0000000000000000 0000000000000086
ffff88081fdcf9f0 ffffffff810633a3 0000013d983220e0 ffff88081f96d440
ffff88081dc65ed0 0000000100103f9f ffff88081ea1e5f8 ffff88081fdcffd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa02447c0>] reshape_request+0x440/0xa20 [raid456]
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa02450b2>] sync_request+0x312/0x3a0 [raid456]
[<ffffffff81430ff7>] md_do_sync+0x6c7/0xd60
[<ffffffff81431ae5>] md_thread+0x115/0x150
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff814319d0>] ? md_thread+0x0/0x150
[<ffffffff810a640e>] kthread+0x9e/0xc0
[<ffffffff8100c28a>] child_rip+0xa/0x20
[<ffffffff810a6370>] ? kthread+0x0/0xc0
[<ffffffff8100c280>] ? child_rip+0x0/0x20
现在，pvscan 等命令和其他控制 md 阵列的命令挂起。本质上我唯一能做的就是重新启动以回到可以控制阵列的状态。即使重新启动也会在最后挂起，此时我必须进行硬重置。
有人对如何解决这个问题有任何想法吗？我确实尝试启动到 ubunutu live CD，但是当我想要组装阵列时，整个事情都冻结了。
答案1
所以最后，我能够通过执行以下命令来恢复该数组：
mdadm --create /dev/md126 --level=6 --raid-devices=14 --name=gigantor:128 /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdo /dev/sdm missing missing /dev/sdc /dev/sdb /dev/sdd /dev/sde --assume-clean
然后重新添加这两个设备。
mdadm --add /dev/md126 /dev/sdn
mdadm --add /dev/md126 /dev/sdl
后来，由于我的 LVM 信息也损坏了，我必须恢复它：
pvcreate --uuid "kvEA4X-vobg-2Ipz-ITF1-ZhtW-Ewej-6liKVx" --restorefile /etc/lvm/backup/vg_data /dev/md126
vgcfgrestore vg_data
lvchange -ay /dev/vg_data/lvm0
然后我尝试 xfs_check LVM，它说我需要运行修复并销毁日志：
xfs_repair -L /dev/mapper/vg_data-lvm0
修复后，我现在可以挂载我的 LVM 并且我的数据完好无损。
我的 LVM 正在修复。
Personalities : [raid6] [raid5] [raid4]
md126 : active raid6 sdm[14] sdl[7] sdk[15] sdn[6] sdj[5] sdi[4] sdg[2] sde[0] sdf[1] sdh[3] sdc[12] sda[11] sdd[13] sdb[10]
46882646016 blocks super 1.2 level 6, 512k chunk, algorithm 2 [14/12] [UUUUUUUU__UUUU]
[==========>..........] recovery = 53.6% (2097061864/3906887168) finish=2448.0min speed=12321K/sec
bitmap: 0/30 pages [0KB], 65536KB chunk
md127 : active raid6 sdac[15] sdaa[13] sdz[2] sdab[0] sdq[9] sdy[14] sdx[4] sdr[8] sdp[10] sdo[16] sdv[6] sdu[7] sdw[5] sds[12] sdt[11]
50789533184 blocks super 1.2 level 6, 512k chunk, algorithm 2 [15/15] [UUUUUUUUUUUUUUU]
unused devices: <none>
多么痛苦啊……