软件 RAID 在清空缓冲区时速度变慢

Question

我正在mdraidCentOS 7.6 中进行实验，试图了解fio.我正在对 4 个条带阵列进行顺序写入，写入 RAM 缓冲区时速度一开始很好，然后在 fsync 期间速度减慢。我用来end_fsync确保数据写入磁盘。在此测试期间，我iostat在另一个窗口中 监视 4 个磁盘。

一开始，iostat每个磁盘上显示165MBps，并且fio显示写入速度超过3000MBps。所以我们正在写入 RAM，并且内核同时写入磁盘。一旦 fio 完成写入，它将缓冲区刷新到磁盘。在此期间，写入速度为 0，并iostat下降至约 120MBps。我试图找出为什么 fsync 不能更快，因为我认为它应该能够与初始写入一样快。

$ fio --name=seqwrite --rw=write --bs=1M --size=3G --directory=/mnt/raid --end_fsync=1 --numjobs=4 --group_reporting
seqwrite: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
...
fio-3.1
Starting 4 processes
Jobs: 2 (f=2): [_(1),F(1),_(1),F(1)][-.-%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=4): err= 0: pid=9959: Thu Sep 12 16:47:50 2019
  write: IOPS=534, BW=534MiB/s (560MB/s)(12.0GiB/23009msec)
    clat (usec): min=315, max=21855, avg=1356.25, stdev=3850.63
     lat (usec): min=324, max=21881, avg=1373.31, stdev=3852.25
    clat percentiles (usec):
     |  1.00th=[  322],  5.00th=[  326], 10.00th=[  330], 20.00th=[  343],
     | 30.00th=[  347], 40.00th=[  359], 50.00th=[  367], 60.00th=[  371],
     | 70.00th=[  383], 80.00th=[  445], 90.00th=[  586], 95.00th=[14746],
     | 99.00th=[19268], 99.50th=[19530], 99.90th=[21627], 99.95th=[21890],
     | 99.99th=[21890]
   bw (  KiB/s): min=14364, max=2881154, per=100.00%, avg=704062.00, stdev=949759.73, samples=35
   iops        : min=   14, max= 2813, avg=687.29, stdev=927.47, samples=35
  lat (usec)   : 500=85.28%, 750=8.30%, 1000=0.20%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.15%, 20=5.57%, 50=0.46%
  cpu          : usr=0.30%, sys=17.58%, ctx=100619, majf=0, minf=106
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,12288,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=534MiB/s (560MB/s), 534MiB/s-534MiB/s (560MB/s-560MB/s), io=12.0GiB (12.9GB), run=23009-23009msec

Disk stats (read/write):
    md0: ios=0/3145734, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/6655, aggrmerge=0/778369, aggrticks=0/2292548, aggrin_queue=2292870, aggrutil=95.06%
  sda: ios=0/6502, merge=0/778634, ticks=0/3019328, in_queue=3019466, util=95.06%
  sdb: ios=0/6465, merge=0/778498, ticks=0/2554715, in_queue=2555016, util=92.24%
  sdc: ios=0/6780, merge=0/778266, ticks=0/1745808, in_queue=1746414, util=92.00%
  sdd: ios=0/6875, merge=0/778080, ticks=0/1850343, in_queue=1850585, util=91.34%

该阵列是 4 个旋转磁盘，由 4K 块组成条带，并且具有 XFS 文件系统。如果我对没有 RAID 的单个 XFS 磁盘进行相同的测试，则不会出现此问题；写入速度始终稳定在160MBps。