我的服务器昨天断电了。当电源恢复时它自动启动了，但是我的 RAID 现在无法使用。
RAID 阵列是使用 mdadm 构建的软件 RAID6。最初，阵列使用了 8 个驱动器。几个月前，我注意到一个驱动器未通过 SMART 测试，因此我更换了它。在更换驱动器时，我添加了另外 4 个驱动器，这些驱动器最初位于我之前的服务器中并格式化为 ZFS（作为 FreeNAS RAIDZ 的一部分），使驱动器总数达到 12 个。
不幸的是，我不记得我用来扩展阵列的确切命令，而且该命令不再存在于我的 bash_history 中。我不知道为什么今天 mdadm 报告 8 个磁盘，而 fdisk 报告这 4 个驱动器是 ZFS；我相信在过去的几个月里我一直在运行 12 磁盘 RAID 设置。
据我所知，自从更换了坏驱动器并添加了 4 个额外驱动器之后，直到昨天，我的服务器就没有关闭或重新启动过。
自断电以来，我运行过的唯一与 mdadm 相关的命令是mdadm --detail /dev/md0、mdadm --stop /dev/md0和mdadm --assemble --scan -v。我没有重新启动服务器。
我意识到我需要备份，并将尽快实施备份。目前，我只是希望有办法恢复我的数据。
我正在运行 Debian 9 和 mdadm 3.4。
我很乐意提供任何其他信息。（请注意，我绝不认为自己是 RAID 或 mdadm 专家。）
更新：我添加了 mdadm 检查“好”/“坏”驱动器的输出。“好”驱动器正确报告了 raid6 和 12 驱动器。
mdadm --detail /dev/md0
（注意：这是在启动时，运行之前mdadm --stop /dev/md0。还请注意，它报告了 raid0。）
/dev/md0:
Version : 1.2
Raid Level : raid0
Total Devices : 8
Persistence : Superblock is persistent
State : inactive
Name : blackcanary:0 (local to host blackcanary)
UUID : 7bd22c29:465d927f:d0c1d08f:ba3694c9
Events : 72036
Number Major Minor RaidDevice
- 8 64 - /dev/sde
- 8 32 - /dev/sdc
- 8 0 - /dev/sda
- 8 112 - /dev/sdh
- 8 80 - /dev/sdf
- 8 48 - /dev/sdd
- 8 128 - /dev/sdi
- 8 96 - /dev/sdg
fdisk -l
（注意：/dev/sdb 和 /dev/sdj 是仅有的两个不应位于阵列中的驱动器。）
Disk /dev/sdd: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sde: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdc: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sda: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdf: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdg: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdh: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdi: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdb: 57.9 GiB, 62109253632 bytes, 121307136 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 764C7B0B-574D-40DF-95C0-06D765AEB2D2
Device Start End Sectors Size Type
/dev/sdb1 2048 1050623 1048576 512M EFI System
/dev/sdb2 1050624 54255615 53204992 25.4G Linux filesystem
/dev/sdb3 54255616 121305087 67049472 32G Linux swap
Disk /dev/sdj: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sdk: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 41DFFB5A-7F8C-11E5-9D78-7824AF43D5FA
Device Start End Sectors Size Type
/dev/sdk1 128 4194431 4194304 2G FreeBSD swap
/dev/sdk2 4194432 7814037127 7809842696 3.7T FreeBSD ZFS
Disk /dev/sdl: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 40C2CEAE-7F8C-11E5-9D78-7824AF43D5FA
Device Start End Sectors Size Type
/dev/sdl1 128 4194431 4194304 2G FreeBSD swap
/dev/sdl2 4194432 7814037127 7809842696 3.7T FreeBSD ZFS
Disk /dev/sdm: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 402A1394-7F8C-11E5-9D78-7824AF43D5FA
Device Start End Sectors Size Type
/dev/sdm1 128 4194431 4194304 2G FreeBSD swap
/dev/sdm2 4194432 7814037127 7809842696 3.7T FreeBSD ZFS
Disk /dev/sdn: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 415081DC-7F8C-11E5-9D78-7824AF43D5FA
Device Start End Sectors Size Type
/dev/sdn1 128 4194431 4194304 2G FreeBSD swap
/dev/sdn2 4194432 7814037127 7809842696 3.7T FreeBSD ZFS
块
/dev/sdd: UUID="7bd22c29-465d-927f-d0c1-d08fba3694c9" UUID_SUB="91d8e433-aeb9-a4f2-75ca-10cf3ceb0a83" LABEL="blackcanary:0" TYPE="linux_raid_member"
/dev/sde: UUID="7bd22c29-465d-927f-d0c1-d08fba3694c9" UUID_SUB="6ccd4cee-9fdd-872c-add0-209a9d074eb5" LABEL="blackcanary:0" TYPE="linux_raid_member"
/dev/sdc: UUID="7bd22c29-465d-927f-d0c1-d08fba3694c9" UUID_SUB="11bf8212-7c04-0c8c-a061-168bda9c34a5" LABEL="blackcanary:0" TYPE="linux_raid_member"
/dev/sda: UUID="7bd22c29-465d-927f-d0c1-d08fba3694c9" UUID_SUB="6cee76b0-031c-bb6a-6c13-7d15d0b2feee" LABEL="blackcanary:0" TYPE="linux_raid_member"
/dev/sdf: UUID="7bd22c29-465d-927f-d0c1-d08fba3694c9" UUID_SUB="508cd116-e0ac-131c-cd0d-3c5c41c1cbba" LABEL="blackcanary:0" TYPE="linux_raid_member"
/dev/sdg: UUID="7bd22c29-465d-927f-d0c1-d08fba3694c9" UUID_SUB="0080d32e-7fea-037f-75a8-6e40dfa8fdfa" LABEL="blackcanary:0" TYPE="linux_raid_member"
/dev/sdh: UUID="7bd22c29-465d-927f-d0c1-d08fba3694c9" UUID_SUB="f1aa862e-6c4c-0239-5fb4-29d7a9dd7497" LABEL="blackcanary:0" TYPE="linux_raid_member"
/dev/sdi: UUID="7bd22c29-465d-927f-d0c1-d08fba3694c9" UUID_SUB="32472d1f-25c1-dfb3-fa9a-57ab3df17986" LABEL="blackcanary:0" TYPE="linux_raid_member"
/dev/sdb1: UUID="E15C-D029" TYPE="vfat" PARTUUID="dff9fca6-2eef-4084-845c-d780ca7b6cb8"
/dev/sdb2: UUID="bfd3d30c-de34-4e00-89b4-384bcbb7922d" TYPE="ext4" PARTUUID="7696f182-551e-447c-9549-155605cc48a3"
/dev/sdb3: UUID="7f65b504-ffe2-48e4-868f-a9db95865505" TYPE="swap" PARTUUID="48ecdc7c-fb0a-404e-bc02-178063515b6c"
/dev/sdj: UUID="b59275af-0825-4f25-96d2-aff0c3fef5e5" TYPE="ext4"
/dev/sdk1: PARTUUID="41f3ae62-7f8c-11e5-9d78-7824af43d5fa"
/dev/sdk2: LABEL="main" UUID="2939752790805872810" UUID_SUB="11544886583175140825" TYPE="zfs_member" PARTUUID="420ad019-7f8c-11e5-9d78-7824af43d5fa"
/dev/sdl1: PARTUUID="40d655e5-7f8c-11e5-9d78-7824af43d5fa"
/dev/sdl2: LABEL="main" UUID="2939752790805872810" UUID_SUB="15587016790013084755" TYPE="zfs_member" PARTUUID="40ef0474-7f8c-11e5-9d78-7824af43d5fa"
/dev/sdm1: PARTUUID="403ea9c7-7f8c-11e5-9d78-7824af43d5fa"
/dev/sdm2: LABEL="main" UUID="2939752790805872810" UUID_SUB="12390459856885165202" TYPE="zfs_member" PARTUUID="4057cbaa-7f8c-11e5-9d78-7824af43d5fa"
/dev/sdn1: PARTUUID="416507e6-7f8c-11e5-9d78-7824af43d5fa"
/dev/sdn2: LABEL="main" UUID="2939752790805872810" UUID_SUB="16896032374271545514" TYPE="zfs_member" PARTUUID="417ef281-7f8c-11e5-9d78-7824af43d5fa"
mdadm --asemble --scan -v
mdadm: looking for devices for /dev/md0
mdadm: No super block found on /dev/sdn2 (Expected magic a92b4efc, got b1f5ae15)
mdadm: no RAID superblock on /dev/sdn2
mdadm: No super block found on /dev/sdn1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdn1
mdadm: No super block found on /dev/sdn (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdn
mdadm: No super block found on /dev/sdm2 (Expected magic a92b4efc, got fd87a884)
mdadm: no RAID superblock on /dev/sdm2
mdadm: No super block found on /dev/sdm1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdm1
mdadm: No super block found on /dev/sdm (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdm
mdadm: No super block found on /dev/sdl2 (Expected magic a92b4efc, got 946313cc)
mdadm: no RAID superblock on /dev/sdl2
mdadm: No super block found on /dev/sdl1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdl1
mdadm: No super block found on /dev/sdl (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdl
mdadm: No super block found on /dev/sdk2 (Expected magic a92b4efc, got 4de36623)
mdadm: no RAID superblock on /dev/sdk2
mdadm: No super block found on /dev/sdk1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdk1
mdadm: No super block found on /dev/sdk (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdk
mdadm: No super block found on /dev/sdj (Expected magic a92b4efc, got 0000043c)
mdadm: no RAID superblock on /dev/sdj
mdadm: No super block found on /dev/sdb3 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb3
mdadm: No super block found on /dev/sdb2 (Expected magic a92b4efc, got 00000405)
mdadm: no RAID superblock on /dev/sdb2
mdadm: No super block found on /dev/sdb1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb1
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb
mdadm: /dev/sdi is identified as a member of /dev/md0, slot 7.
mdadm: /dev/sdh is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sdg is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sdf is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sda is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 1.
mdadm: added /dev/sdd to /dev/md0 as 1
mdadm: added /dev/sda to /dev/md0 as 2
mdadm: added /dev/sde to /dev/md0 as 3
mdadm: added /dev/sdf to /dev/md0 as 4
mdadm: added /dev/sdg to /dev/md0 as 5
mdadm: added /dev/sdh to /dev/md0 as 6
mdadm: added /dev/sdi to /dev/md0 as 7
mdadm: no uptodate device for slot 8 of /dev/md0
mdadm: no uptodate device for slot 9 of /dev/md0
mdadm: no uptodate device for slot 10 of /dev/md0
mdadm: no uptodate device for slot 11 of /dev/md0
mdadm: added /dev/sdc to /dev/md0 as 0
mdadm: /dev/md0 assembled from 8 drives - not enough to start the array.
配置文件
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#
# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers
# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR root
# definitions of existing MD arrays
# This configuration was auto-generated on Sun, 25 Jun 2017 22:28:24 -0500 by mkconf
ARRAY /dev/md0 metadata=1.2 name=blackcanary:0 UUID=7bd22c29:465d927f:d0c1d08f:ba3694c9
mdadm --检查 /dev/sdd
（注意：对于任何“良好”的 8 个驱动器，都会返回非常相似的信息）
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 7bd22c29:465d927f:d0c1d08f:ba3694c9
Name : blackcanary:0 (local to host blackcanary)
Creation Time : Sun Jun 25 22:53:12 2017
Raid Level : raid6
Raid Devices : 12
Avail Dev Size : 7813780144 (3725.90 GiB 4000.66 GB)
Array Size : 39068897280 (37259.00 GiB 40006.55 GB)
Used Dev Size : 7813779456 (3725.90 GiB 4000.66 GB)
Data Offset : 257024 sectors
Super Offset : 8 sectors
Unused Space : before=256936 sectors, after=688 sectors
State : clean
Device UUID : 91d8e433:aeb9a4f2:75ca10cf:3ceb0a83
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Oct 7 14:23:36 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : b960e905 - correct
Events : 72036
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
mdadm——检查/dev/sdk
（注意：对于任何“坏的” 4 个驱动器都会返回相同的信息）
/dev/sdk:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
fdisk -b 512 -t gpt -l /dev/sdn
Disk /dev/sdn: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 415081DC-7F8C-11E5-9D78-7824AF43D5FA
Device Start End Sectors Size Type
/dev/sdn1 128 4194431 4194304 2G FreeBSD swap
/dev/sdn2 4194432 7814037127 7809842696 3.7T FreeBSD ZFS
fdisk -b 4096 -t gpt -l /dev/sdn
Disk /dev/sdn: 3.7 TiB, 4000787030016 bytes, 976754646 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 415081DC-7F8C-11E5-9D78-7824AF43D5FA
Device Start End Sectors Size Type
/dev/sdn1 128 4194431 4194304 16G FreeBSD swap
/dev/sdn2 4194432 7814037127 7809842696 29.1T FreeBSD ZFS
答案1
经过大量研究，我能够恢复我的 RAID 阵列并且没有明显的数据丢失。
我最终不得不使用mdadm --create --assume-clean。我选择使用覆盖文件这样我就可以非破坏性地测试各种配置，直到找到正确的配置。
我使用mdadm --examine /dev/sd*“好”驱动器来确定它们的顺序。然后，我生成“坏”驱动器的可能排列，并运行它们，直到我有一个可安装的文件系统。幸运的是，我只需要检查 24 种可能的组合。
我尝试了所有 24 种排列，但未能成功安装。我检查了其中一个映射器驱动器 ( mdadm --examine /dev/mapper/sd*)，并将其与原始驱动器进行比较，发现数据偏移量不同。我将data-offset参数添加到配置中，重新测试了排列，并在第 12 次尝试后成功。
这些是我在测试时运行的命令：
mdadm --stop /dev/md0
mdadm --create --assume-clean --run --level=6 --data-offset=128512 --raid-devices=12 /dev/md0 /dev/mapper/sdb /dev/mapper/sdc /dev/mapper/sda /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf /dev/mapper/sdg /dev/mapper/sdh /dev/mapper/sdn /dev/mapper/sdm /dev/mapper/sdl /dev/mapper/sdk
dumpe2fs /dev/md0
fsck.ext4 -v /dev/md0
我曾经dd备份“坏”驱动器的 MBR 和分区表，然后用零覆盖 MBR 和分区表以匹配“好”驱动器，这样这在将来的重启时就不会成为问题。
成功重建后，我生成了一个新mdadm.conf文件并记录了有关该阵列的尽可能多的数据，以备将来需要。现在，我正在寻找一个真正的备份解决方案。