我制作了一个带有 8x8TB 驱动器的 raid10,其上的文件系统是 XFS。我从未遇到过任何问题。该 raid 配置为 near2 和内部位图。驱动器以 HBA 模式连接到 LSI-9211-8i。
上周,我终于买了另一套 8x8TB 硬盘,因为磁盘阵列已满。以下是我将其添加到现有 RAID 的方法:
- 使用 smartctl 测试驱动器(没有问题)。
- 在每个驱动器上创建了一个新的 GPT 分区选项卡和一个 XFS 分区。
- 使用命令
sudo mdadm /dev/md0 --add /dev/...
添加驱动器。 - 通过执行来扩大阵列
sudo mdadm --grow /dev/md0 --size=max
。首先花了 2 天时间进行重塑,然后花了一天时间进行重新同步。一切顺利,没有任何错误或中断。之后,突袭变得干净且活跃。 - 使用 调整文件系统大小
sudo xfs_grow /dev/md0
。它顺利完成。 - 安装 RAID 阵列并进行测试。它运行良好。我重新启动了服务器以确保阵列仍然干净。
那么发生了什么?!我使用阵列通过 Radarr、Sonarr、Deluge(全部在 docker 中)下载新内容。我不得不重启服务器(我每周重启一次以释放资源),然后发生了一些事情。
raid 阵列现在处于非活动状态(请查看下面的 mdadm 详细信息)。16 个驱动器中有 4 个的事件计数错误。经过多次尝试,我通过执行 成功恢复sudo mdadm --assemble /dev/md0 /dev/sd[...]
。我使用 测试了 XFS 文件系统sudo xfs_repair -n /dev/md0
,pfiou... 一切似乎都正常。
出于好奇,我今天尝试再次重启,确保 Docker 容器已关闭。我检查了阵列是否干净,因此我等待了 5 分钟,然后我做到了sudo reboot
。同样的事情再次发生!!这次 6 个驱动器的事件计数不同步...所以我再次执行了与上述相同的操作,看起来不错,但现在我非常担心,因为我目前没有任何证据可以支持这一点(我知道这很糟糕)。
你们知道发生了什么吗?提前感谢你们的帮助!:)
sudo mdadm --examine...
在我重新组装之前:
/dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260976 sectors, after=1024 sectors
State : clean
Device UUID : 17c3e75a:d0360875:69201f88:cababd94
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:04 2020
Bad Block Log : 512 entries available at offset 128 sectors
Checksum : b8d092ad - correct
Events : 63706
Layout : near=2
Chunk Size : 512K
Device Role : Active device 12
Array State : AAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260976 sectors, after=1024 sectors
State : clean
Device UUID : c0f6cf6d:3c5687e3:8a745314:9fcd0ea6
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 128 sectors
Checksum : d79a4cca - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 11
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260976 sectors, after=1024 sectors
State : clean
Device UUID : 1c3c8784:6b6bcbef:4ce604b6:5734fed2
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:04 2020
Bad Block Log : 512 entries available at offset 128 sectors
Checksum : c9597fc2 - correct
Events : 63706
Layout : near=2
Chunk Size : 512K
Device Role : Active device 8
Array State : AAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260976 sectors, after=1024 sectors
State : clean
Device UUID : f650167a:562c413d:22d8ba5f:f6f4e486
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 128 sectors
Checksum : 69d80807 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 13
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260976 sectors, after=1024 sectors
State : clean
Device UUID : 8397c880:8807eb34:75425522:4f1c2f37
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 128 sectors
Checksum : db18bb75 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 9
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260976 sectors, after=1024 sectors
State : clean
Device UUID : cb43d48b:862f335a:90bddcd7:9060894b
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 128 sectors
Checksum : d54e4f12 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 15
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260976 sectors, after=1024 sectors
State : clean
Device UUID : 602c75b6:00d68a1f:c04c404f:514bffed
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 128 sectors
Checksum : df205817 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 10
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260976 sectors, after=1024 sectors
State : clean
Device UUID : b8db9412:ae32feaa:4d330c95:5cdcd294
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 128 sectors
Checksum : b352dbb1 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 14
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260992 sectors, after=1024 sectors
State : clean
Device UUID : 4af973aa:e4ae2676:67017e23:343e8114
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:04 2020
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 249da51a - correct
Events : 63706
Layout : near=2
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260992 sectors, after=1024 sectors
State : clean
Device UUID : 35255445:73245125:1e05a845:0b213eca
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:04 2020
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 468f2d21 - correct
Events : 63706
Layout : near=2
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdl1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260992 sectors, after=1024 sectors
State : clean
Device UUID : f1af8db7:4e256b3b:bdc65518:b26bb193
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 6ae0c513 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 4
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdm1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260992 sectors, after=1024 sectors
State : clean
Device UUID : e2b66811:4f48ebc6:245f75c2:c64db0c6
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 2d5a6983 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 6
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdn1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260992 sectors, after=1024 sectors
State : clean
Device UUID : dd3f61c8:1ca69a95:d58a4ac2:c46365d0
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:04 2020
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : bcaf91e8 - correct
Events : 63706
Layout : near=2
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdo1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260992 sectors, after=1024 sectors
State : clean
Device UUID : 7b867d58:c5b0551a:753ebe4f:9502faa5
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:04 2020
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 348f359d - correct
Events : 63706
Layout : near=2
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdp1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260992 sectors, after=1024 sectors
State : clean
Device UUID : 72a44102:13e98e3a:844d9e0b:e58637cc
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : e0871f55 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 7
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdq1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Name : Server:0
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Raid Devices : 16
Avail Dev Size : 15627789312 (7451.91 GiB 8001.43 GB)
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
Data Offset : 261120 sectors
Super Offset : 8 sectors
Unused Space : before=260992 sectors, after=1024 sectors
State : clean
Device UUID : 6b0152f0:ca1be94f:1af1ee36:e735cf70
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Mar 25 00:27:16 2020
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : b3da0199 - correct
Events : 63709
Layout : near=2
Chunk Size : 512K
Device Role : Active device 2
Array State : .AA.A.AA.AAA.AAA ('A' == active, '.' == missing, 'R' == replacing)
sudo mdadm -D /dev/md0
在我重新组装之前:
/dev/md0:
Version : 1.2
Raid Level : raid0
Total Devices : 16
Persistence : Superblock is persistent
State : inactive
Working Devices : 16
Name : Server:0
UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Events : 63709
Number Major Minor RaidDevice
- 65 1 - /dev/sdq1
- 8 1 - /dev/sda1
- 8 241 - /dev/sdp1
- 8 225 - /dev/sdo1
- 8 209 - /dev/sdn1
- 8 193 - /dev/sdm1
- 8 177 - /dev/sdl1
- 8 145 - /dev/sdj1
- 8 129 - /dev/sdi1
- 8 113 - /dev/sdh1
- 8 97 - /dev/sdg1
- 8 81 - /dev/sdf1
- 8 65 - /dev/sde1
- 8 49 - /dev/sdd1
- 8 33 - /dev/sdc1
- 8 17 - /dev/sdb1
cat /proc/mdstat
在我重新组装之前:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : inactive sdc1[15](S) sdi1[1](S) sdo1[3](S) sdm1[6](S) sdl1[4](S) sdq1[2](S) sde1[14](S) sdh1[9](S) sdd1[10](S) sdj1[0](S) sdb1[12](S) sdg1[13](S) sdp1[7](S) sdf1[8](S) sdn1[5](S) sda1[11](S)
125022314496 blocks super 1.2
unused devices: <none>
sudo mdadm -D /dev/md0
在我重新组装之后(是的,我启动了 mdadm 检查):
/dev/md0:
Version : 1.2
Creation Time : Fri Mar 29 16:19:52 2019
Raid Level : raid10
Array Size : 62511153152 (59615.28 GiB 64011.42 GB)
Used Dev Size : 7813894144 (7451.91 GiB 8001.43 GB)
Raid Devices : 16
Total Devices : 16
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Mar 25 04:28:53 2020
State : clean, checking
Active Devices : 16
Working Devices : 16
Failed Devices : 0
Spare Devices : 0
Layout : near=2
Chunk Size : 512K
Consistency Policy : bitmap
Check Status : 0% complete
Name : Server:0
UUID : d9b2f19f:01d7ce88:c46d2dd3:c1ab05f0
Events : 63711
Number Major Minor RaidDevice State
0 8 145 0 active sync set-A /dev/sdj1
1 8 129 1 active sync set-B /dev/sdi1
2 65 1 2 active sync set-A /dev/sdq1
3 8 225 3 active sync set-B /dev/sdo1
4 8 177 4 active sync set-A /dev/sdl1
5 8 209 5 active sync set-B /dev/sdn1
6 8 193 6 active sync set-A /dev/sdm1
7 8 241 7 active sync set-B /dev/sdp1
15 8 33 8 active sync set-A /dev/sdc1
14 8 65 9 active sync set-B /dev/sde1
13 8 97 10 active sync set-A /dev/sdg1
12 8 17 11 active sync set-B /dev/sdb1
11 8 1 12 active sync set-A /dev/sda1
10 8 49 13 active sync set-B /dev/sdd1
9 8 113 14 active sync set-A /dev/sdh1