使用 SSD 缓存改进 mdadm RAID5

Question

我有以下设置：3x 3 TB SATA HDD，带有连接到 RAID5 的软件 raid，在此 LVM 之上用于快照（备份），并且在 LVM EXT4 之上格式化。

实际上使用了大约 400GB。我的服务器中有一个备用插槽。

对于较小的文件，Web 服务器的写入性能相当慢。但对于较大的文件，写入性能还不错。

root@srv1:/home/srvadmin# dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 8.93049 s, 120 MB/s
root@srv1:/home/srvadmin# dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 25.2487 s, 42.5 MB/s
root@srv1:/home/srvadmin# dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.0472 s, 97.2 MB/s
root@srv1:/home/srvadmin# dd if=/dev/zero of=/root/testfile bs=100M count=1 oflag=direct
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 2.97071 s, 35.3 MB/s
root@srv1:/home/srvadmin# dd if=/dev/zero of=/root/testfile bs=100M count=10 oflag=direct
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 15.7935 s, 66.4 MB/s
root@srv1:/home/srvadmin# dd if=/dev/zero of=/root/testfile bs=100M count=20 oflag=direct

现在我看到了不同的选项：A：添加 bcache（不知道如何添加）并使用 SSD 作为缓存 B：购买额外的磁盘并创建 raid10 C：购买 SSD，将所有文件复制到其中并单独使用 SSD 而不使用任何 raid D：购买两个 SSD 并使用它们的镜像

我认为解决方案 A 是最便宜的。我已经有一个 120GB 的 SSD 闲置着，文件系统也已就位。但我不知道如何添加 SSD 进行缓存。你做过这样的事情吗？

该服务器是私人使用的，运行 Ubuntu LTS，服务器也定期备份。

谢谢

编辑：

我找到了这个：https://ahelpme.com/linux/lvm/ssd-cache-device-to-a-software-raid5-using-lvm2/

看起来 lvm 提供了缓存。但我不确定目前我是否可以添加缓存。

编辑2：

我想，我做到了！我遵循了以下两个教程： https://www.nocser.net/clients/index.php/knowledgebase/474/Adding-SSD-Cache-to-Existing-LVM.html

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_logical_volumes/enabling-caching-to-improve-logical-volume-performance_configuring-and-managing-logical-volumes

最后得到了这些命令：请检查输出是否可行：我使用了缓存类型 writethrough 而不是 writeback，因为我读到 writethrough 是更安全的断电保护类型。

root@srv1:~# fdisk /dev/sdd

Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/sdd: 111.81 GiB, 120034123776 bytes, 234441648 sectors
Disk model: KINGSTON SA400S3
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2BAB7835-0A17-43DE-83E1-57A2733E4CFD

Command (m for help): n
Partition number (1-128, default 1): 1
First sector (34-234441614, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-234441614, default 234441614):

Created a new partition 1 of type 'Linux filesystem' and of size 111.8 GiB.
Partition #1 contains a vfat signature.

Do you want to remove the signature? [Y]es/[N]o: y

The signature will be removed by a write command.

Command (m for help): p
Disk /dev/sdd: 111.81 GiB, 120034123776 bytes, 234441648 sectors
Disk model: KINGSTON SA400S3
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2BAB7835-0A17-43DE-83E1-57A2733E4CFD

Device     Start       End   Sectors   Size Type
/dev/sdd1   2048 234441614 234439567 111.8G Linux filesystem

Filesystem/RAID signature on partition 1 will be wiped.

Command (m for help): t
Selected partition 1
Partition type (type L to list all types): 8e
Type of partition 1 is unchanged: Linux filesystem.

Command (m for help): p
Disk /dev/sdd: 111.81 GiB, 120034123776 bytes, 234441648 sectors
Disk model: KINGSTON SA400S3
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2BAB7835-0A17-43DE-83E1-57A2733E4CFD

Device     Start       End   Sectors   Size Type
/dev/sdd1   2048 234441614 234439567 111.8G Linux filesystem

Filesystem/RAID signature on partition 1 will be wiped.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

root@srv1:~# pvcreate /dev/sdd1
  Physical volume "/dev/sdd1" successfully created.
root@srv1:~# vgextend vg /dev/sdd1
  Volume group "vg" successfully extended
root@srv1:~# lvcreate -n DataLVcache -L100G vg /dev/sdd1
  Logical volume "DataLVcache" created.
root@srv1:~# lvcreate -n DataLVcacheMeta -L100M vg /dev/sdd1
  Logical volume "DataLVcacheMeta" created.
root@srv1:~# lvconvert --type cache-pool --cachemode writethrough --poolmetadata vg/DataLVcacheMeta vg/DataLVcache
  Using 128.00 KiB chunk size instead of default 64.00 KiB, so cache pool has less than 1000000 chunks.
  WARNING: Converting vg/DataLVcache and vg/DataLVcacheMeta to cache pool's data and metadata volumes with metadata wiping.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert vg/DataLVcache and vg/DataLVcacheMeta? [y/n]: y
  Converted vg/DataLVcache and vg/DataLVcacheMeta to cache pool.
root@srv1:~# lvconvert --type cache --cachepool vg/DataLVcache vg/md128
  Failed to find logical volume "vg/md128"
root@srv1:~# lvconvert --type cache --cachepool vg/DataLVcache vg/root
Do you want wipe existing metadata of cache pool vg/DataLVcache? [y/n]: y
  Logical volume vg/root is now cached.



root@srv1:~# lvs --all --options +devices vg
  LV                        VG Attr       LSize   Pool                Origin       Data%  Meta%  Move Log Cpy%Sync Convert
  [DataLVcache_cpool]       vg Cwi---C--- 100.00g                                  0.61   6.47            0.00
  [DataLVcache_cpool_cdata] vg Cwi-ao---- 100.00g
  [DataLVcache_cpool_cmeta] vg ewi-ao---- 100.00m
  data_snap_backblaze       vg swi-a-s---   2.28t                     root         2.99
  [lvol0_pmspare]           vg ewi------- 100.00m
  root                      vg owi-aoC---   2.27t [DataLVcache_cpool] [root_corig] 0.61   6.47            0.00
  [root_corig]              vg owi-aoC---   2.27t

如果我现在尝试写入磁盘，速度将达到 2GB/s

root@srv1:~# dd if=/dev/zero of=/home/user12/testfile2 bs=100M count=2
2+0 records in
2+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.115231 s, 1.8 GB/s

我猜缓存正在工作？如果我再次执行完全相同的命令，性能会非常差，大约 50MB/s。我猜这是因为文件被写入 RAID5 阵列，而我尝试写入文件，对吗？

编辑3：

好吧，有些事情变得奇怪了...每次我写入同一个文件时它都会变得更慢......

104857600 bytes (105 MB, 100 MiB) copied, 0.0622744 s, 1.7 GB/s
root@srv1:~# dd if=/dev/zero of=/home/user12/testfile1 bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.871399 s, 120 MB/s
root@srv1:~# dd if=/dev/zero of=/home/user12/testfile1 bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 4.3185 s, 24.3 MB/s
root@srv1:~# dd if=/dev/zero of=/home/user12/testfile1 bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 5.31411 s, 19.7 MB/s
root@srv1:~# dd if=/dev/zero of=/home/user12/testfile1 bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 8.95197 s, 11.7 MB/s
root@srv1:~#

等待约 2 分钟后，最后一个

root@srv1:~# dd if=/dev/zero of=/home/user12/testfile1 bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 5.96534 s, 17.6 MB/s

有什么线索吗？我在执行命令之间等待了大约 20-50 秒。

没有缓存：

root@srv1:~# dd if=/dev/zero of=/home/user12/testfile2 bs=100M count=1 oflag=direct
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 11.1603 s, 9.4 MB/s