我们在新的 Dell PowerEdge T330 上安装了 Debian 8，使用 mdadm 在 RAID1 阵列中有两个分区 / 和 /var。在测试主要应用程序期间：mysql 和 tomcat 被停止。尽管读取性能足够，但两个分区的写入性能都很糟糕。这是以相同方式设置的两台相同服务器之一的观察结果。任何帮助，将不胜感激。
表现
root@bcmdit-519:/home/bcmdit# FILE=/tmp/test_data && dd bs=16k \ count=102400 oflag=direct if=/dev/zero of=$FILE && \ rm $FILE && FILE=/var/ tmp/test_data && dd bs=16k \ count=102400 oflag=direct if=/dev/zero of=$FILE && rm $FILE
102400+0 records in
102400+0 records out
1677721600 bytes (1.7 GB) copied, 886.418 s, 1.9 MB/s
102400+0 records in
102400+0 records out
1677721600 bytes (1.7 GB) copied, 894.832 s, 1.9 MB/s
root@bcmdit-519:/home/bcmdit# hdparm -t /dev/sda ; hdparm -t /dev/sdb ； hdparm -t /dev/md0 ; hdparm -t /dev/md1
/dev/sda:
Timing buffered disk reads: 394 MB in 3.00 seconds = 131.15 MB/sec
/dev/sdb:
Timing buffered disk reads: 394 MB in 3.01 seconds = 131.05 MB/sec
/dev/md0:
Timing buffered disk reads: 398 MB in 3.00 seconds = 132.45 MB/sec
/dev/md1:
Timing buffered disk reads: 318 MB in 3.00 seconds = 106.00 MB/sec
参考
https://severfault.com/questions/832117/how-increase-write-speed-of-raid1-mdadm https://wiki.mikejung.biz/Software_RAID RAID1 上的写访问时间很慢 https://bbs.archlinux.org/viewtopic.php?id=173791 等...
配置
加密设置使用：
root@bcmdit-519:/home/bcmdit# cryptsetup luksDump UUID=1e7b64ac-f187-4fac-9712-8e0dacadfca7|grep -E '密码|哈希'
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha1
配置片段
root@bcmdit-519:/home/bcmdit# facter virtual productname lsbdistid \
lsbdistrelease processor0 blockdevice_sda_model \
blockdevice_sdb_model bios_version && uname -a && uptime
----------
bios_version => 2.4.3
blockdevice_sda_model => ST1000NX0423
blockdevice_sdb_model => ST1000NX0423
lsbdistid => Debian
lsbdistrelease => 8.10
processor0 => Intel(R) Xeon(R) CPU E3-1230 v6 @ 3.50GHz
productname => PowerEdge T330
virtual => physical
Linux bcmdit-519 3.16.0-4-amd64 #1 SMP Debian 3.16.51-3 (2017-12-13) x86_64 GNU/Linux
14:45:58 up 2:49, 2 users, load average: 0.06, 0.17, 0.44
root@bcmdit-519:/home/bcmdit# grep GRUB_CMDLINE_LINUX_DEFAULT /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet erst_disable elevator=deadline"
root@bcmdit-519:/home/bcmdit# free -m
total used free shared buffers cached
Mem: 32202 1532 30670 9 17 369
-/+ buffers/cache: 1145 31056
Swap: 0 0 0
root@bcmdit-519:/home/bcmdit# 分开 /dev/sda 打印
Model: ATA ST1000NX0423 (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 500GB 500GB primary boot, raid
2 500GB 1000GB 500GB extended
5 500GB 1000GB 500GB logical raid
root@bcmdit-519:/home/bcmdit# 分开 /dev/sdb 打印
Model: ATA ST1000NX0423 (scsi)
Disk /dev/sdb: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 500GB 500GB primary raid
2 500GB 1000GB 500GB extended
5 500GB 1000GB 500GB logical raid
----------
root@bcmdit-519:/home/bcmdit# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sda5[0] sdb5[1]
488249344 blocks super 1.2 [2/2] [UU]
bitmap: 3/4 pages [12KB], 65536KB chunk
md0 : active raid1 sda1[0] sdb1[1]
488248320 blocks super 1.2 [2/2] [UU]
bitmap: 2/4 pages [8KB], 65536KB chunk
unused devices: <none>
root@bcmdit-519:/home/bcmdit# mdadm --query --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Mon Apr 16 13:46:51 2018
Raid Level : raid1
Array Size : 488248320 (465.63 GiB 499.97 GB)
Used Dev Size : 488248320 (465.63 GiB 499.97 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Tue May 15 14:26:47 2018
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : bcmdit-519:0 (local to host bcmdit-519)
UUID : afd3968c:2e8b191d:4504f21e:255b6470
Events : 1703
Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1
root@bcmdit-519:/home/bcmdit# mdadm --query --detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Mon Apr 16 13:47:06 2018
Raid Level : raid1
Array Size : 488249344 (465.63 GiB 499.97 GB)
Used Dev Size : 488249344 (465.63 GiB 499.97 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Tue May 15 14:15:11 2018
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : bcmdit-519:1 (local to host bcmdit-519)
UUID : e46f968a:e8fff775:ecee9cfb:4ad88574
Events : 2659
Number Major Minor RaidDevice State
0 8 5 0 active sync /dev/sda5
1 8 21 1 active sync /dev/sdb5
root@bcmdit-519:/home/bcmdit# cat /etc/crypttab
crypt1 UUID=1e7b64ac-f187-4fac-9712-8e0dacadfca7 /root/.crypt1 luks
root@bcmdit-519:/home/bcmdit# grep -v '^#' /etc/fstab
UUID=c6baa173-8ea6-4598-a965-eee728a93d69 / ext4 defaults,errors=remount-ro 0 1
/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0
/dev/mapper/crypt1 /var ext4 defaults,errors=remount-ro 0 2
/var/swapfile1 none swap sw,nofail 0 0
root@bcmdit-519:/home/bcmdit# smartctl -a /dev/sda|head -n 20
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.16.0-4-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: ST1000NX0423
Serial Number: W4713QXE
LU WWN Device Id: 5 000c50 0abb06247
Add. Product Id: DELL(tm)
Firmware Version: NA07
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 2.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 (minor revision not indicated)
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue May 15 14:29:03 2018 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
root@bcmdit-519:/home/bcmdit# smartctl -a /dev/sdb|head -n 20
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.16.0-4-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: ST1000NX0423
Serial Number: W4714VDQ
LU WWN Device Id: 5 000c50 0abf99927
Add. Product Id: DELL(tm)
Firmware Version: NA07
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 2.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 (minor revision not indicated)
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue May 15 14:29:11 2018 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
更新1
带16M
root@bcmdit-519:/tmp# FILE=/tmp/test_data \
&& dd bs=16M count=102 oflag=direct if=/dev/zero of=$FILE \
&& rm $FILE \
&& FILE=/var/tmp/test_data \
&& dd bs=16M count=102 oflag=direct if=/dev/zero of=$FILE \
&& rm $FILE
102+0 records in
102+0 records out
1711276032 bytes (1.7 GB) copied, 16.6394 s, 103 MB/s
102+0 records in
102+0 records out
1711276032 bytes (1.7 GB) copied, 17.8649 s, 95.8 MB/s
更新2 发现 SMART 的 Seagate 硬盘序列号表示企业级硬盘： https://www.cnet.com/products/seagate-enterprise-capacity-2-5-hdd-v-3-1tb-sata-512n/specs/
更新3 我发现驱动器写入缓存已关闭，但将其设置为打开：
hdparm -W1 /dev/sd*
我现在使用 bs=16k 得到了更好的结果
root@bcmdit-519:/home/bcmdit# FILE=/tmp/test_data && dd bs=16k count=102400 oflag=direct if=/dev/zero of=$FILE && rm $FILE
102400+0 records in
102400+0 records out
1677721600 bytes (1.7 GB) copied, 14.0708 s, 119 MB/s
更新4
root@ecm-oscar-519:/home/bcmdit# cryptsetup 基准测试
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1 1394382 iterations per second
PBKDF2-sha256 923042 iterations per second
PBKDF2-sha512 728177 iterations per second
PBKDF2-ripemd160 804122 iterations per second
PBKDF2-whirlpool 313569 iterations per second
# Algorithm | Key | Encryption | Decryption
aes-cbc 128b 1149.9 MiB/s 3655.8 MiB/s
serpent-cbc 128b 99.6 MiB/s 743.4 MiB/s
twofish-cbc 128b 219.0 MiB/s 400.0 MiB/s
aes-cbc 256b 867.5 MiB/s 2904.5 MiB/s
serpent-cbc 256b 99.6 MiB/s 742.6 MiB/s
twofish-cbc 256b 218.9 MiB/s 399.8 MiB/s
aes-xts 256b 3615.1 MiB/s 3617.3 MiB/s
serpent-xts 256b 710.8 MiB/s 705.0 MiB/s
twofish-xts 256b 388.1 MiB/s 394.5 MiB/s
aes-xts 512b 2884.9 MiB/s 2888.1 MiB/s
serpent-xts 512b 710.7 MiB/s 704.7 MiB/s
twofish-xts 512b 388.0 MiB/s 394.3 MiB/s
答案1
当您请求 ddbs=16K并且oflag=direct请求大量小写入时，这就是 HDD 不擅长的事情，而 SSD 擅长的事情。
您可以使用 LVMCache 来获得两者的好处（取决于 SSD 大小）
如果您使用bs=16M或不使用，oflag那么写入将被拆分/组合/缓存在 RAM 中并以最佳大小写入。
例如;
> dd if=/dev/zero of=test.bin bs=16k count=1000 oflag=direct
1000+0 records in
1000+0 records out
16384000 bytes (16 MB, 16 MiB) copied, 3.19453 s, 5.1 MB/s
> dd if=/dev/zero of=test.bin bs=16M count=1 oflag=direct
1+0 records in
1+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.291366 s, 57.6 MB/s
> dd if=/dev/zero of=test.bin bs=16k count=1000
1000+0 records in
1000+0 records out
16384000 bytes (16 MB, 16 MiB) copied, 0.0815558 s, 201 MB/s
> uname -r
4.14.41-130