什么原因导致 SSD 写入超时超过 120 秒?

什么原因导致 SSD 写入超时超过 120 秒?

系统:CentOS Linux 版本 7.6.1810(核心)

内核:Linux l-db11.tj.cn5 3.10.0-957.10.1.el7.x86_64 #1 SMP 2019 年 3 月 18 日星期一 15:06:45 UTC x86_64 x86_64 x86_64 GNU/Linux

SSD:德拉

CPU:24核

内存:128G

操作系统只能以很小的速率写入。

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme0n1           0.00     0.00    2.00  130.00     0.03     0.91    14.67     0.00    0.03    0.00    0.03   0.03   0.40
nvme0n1           0.00     0.00    0.00  122.00     0.00     0.84    14.05     0.00    0.03    0.00    0.03   0.03   0.40
nvme0n1           0.00     0.00   16.00  201.00     0.25     3.83    38.53     0.01    0.03    0.06    0.03   0.03   0.60
nvme0n1           0.00     0.00    8.00  204.00     0.12     1.12    12.03     0.01    0.03    0.12    0.03   0.03   0.70
nvme0n1           0.00     0.00    0.00  154.00     0.00     1.05    13.95     0.00    0.01    0.00    0.01   0.01   0.20
nvme0n1           0.00     0.00    1.00  162.00     0.02     1.07    13.65     0.01    0.04    0.00    0.04   0.04   0.70

记忆:

free -g
              total        used        free      shared  buff/cache   available
Mem:            125          58          50           3          16          62
Swap:            63           0          63

dmesg:

[Mon Mar 25 14:40:24 2024] INFO: task mysqld:25801 blocked for more than 120 seconds.
[Mon Mar 25 14:40:24 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 25 14:40:24 2024] mysqld          D ffff8d94ebbdb0c0     0 25801  23578 0x00000080
[Mon Mar 25 14:40:24 2024] Call Trace:
[Mon Mar 25 14:40:24 2024]  [<ffffffffa4e1c352>] ? kmem_cache_alloc+0x1c2/0x1f0
[Mon Mar 25 14:40:24 2024]  [<ffffffffc0577227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
[Mon Mar 25 14:40:24 2024]  [<ffffffffa5368c49>] schedule+0x29/0x70
[Mon Mar 25 14:40:24 2024]  [<ffffffffa536a535>] rwsem_down_write_failed+0x225/0x3a0
[Mon Mar 25 14:40:24 2024]  [<ffffffffc05778e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
[Mon Mar 25 14:40:24 2024]  [<ffffffffc05667dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:24 2024]  [<ffffffffa4f870a7>] call_rwsem_down_write_failed+0x17/0x30
[Mon Mar 25 14:40:25 2024]  [<ffffffffa5367f4d>] down_write+0x2d/0x3d
[Mon Mar 25 14:40:25 2024]  [<ffffffffc0569544>] xfs_ilock+0xc4/0x120 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffc05667dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4e5e5c8>] update_time+0x28/0xd0
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4e5e710>] file_update_time+0xa0/0xf0
[Mon Mar 25 14:40:25 2024]  [<ffffffffc055c3bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffc055c6a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4cd6530>] ? try_to_wake_up+0x190/0x390
[Mon Mar 25 14:40:25 2024]  [<ffffffffc055cce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffc055cbe0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4e97b33>] do_io_submit+0x3e3/0x8a0
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4e98000>] SyS_io_submit+0x10/0x20
[Mon Mar 25 14:40:25 2024]  [<ffffffffa5375ddb>] system_call_fastpath+0x22/0x27
[Mon Mar 25 14:40:25 2024] INFO: task mysqld:25802 blocked for more than 120 seconds.
[Mon Mar 25 14:40:25 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 25 14:40:25 2024] mysqld          D ffff8d94f0b85140     0 25802  23578 0x00000080
[Mon Mar 25 14:40:25 2024] Call Trace:
[Mon Mar 25 14:40:25 2024]  [<ffffffffa5368c49>] schedule+0x29/0x70
[Mon Mar 25 14:40:25 2024]  [<ffffffffa536a27d>] rwsem_down_read_failed+0x10d/0x1a0
[Mon Mar 25 14:40:25 2024]  [<ffffffffc05695d0>] ? xfs_ilock_data_map_shared+0x30/0x40 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4f87078>] call_rwsem_down_read_failed+0x18/0x30
[Mon Mar 25 14:40:25 2024]  [<ffffffffa5367f00>] down_read+0x20/0x40
[Mon Mar 25 14:40:25 2024]  [<ffffffffc056955c>] xfs_ilock+0xdc/0x120 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffc05695d0>] xfs_ilock_data_map_shared+0x30/0x40 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffc054f9ca>] __xfs_get_blocks+0x9a/0x6d0 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4c79912>] ? get_user_pages_fast+0x122/0x1a0
[Mon Mar 25 14:40:25 2024]  [<ffffffffc0550e24>] xfs_get_blocks_direct+0x14/0x20 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4e848f6>] do_blockdev_direct_IO+0xf96/0x20a0
[Mon Mar 25 14:40:25 2024]  [<ffffffffc0550e10>] ? xfs_setfilesize+0x80/0x80 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffa4e85a55>] __blockdev_direct_IO+0x55/0x60
[Mon Mar 25 14:40:25 2024]  [<ffffffffc0550e10>] ? xfs_setfilesize+0x80/0x80 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffc0550e30>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
[Mon Mar 25 14:40:25 2024]  [<ffffffffc055c759>] xfs_file_dio_aio_write+0x179/0x340 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffc0550e10>] ? xfs_setfilesize+0x80/0x80 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffc0550e30>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffc055cce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffc055cbe0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4e97b33>] do_io_submit+0x3e3/0x8a0
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4e98000>] SyS_io_submit+0x10/0x20
[Mon Mar 25 14:40:26 2024]  [<ffffffffa5375ddb>] system_call_fastpath+0x22/0x27
[Mon Mar 25 14:40:26 2024] INFO: task mysqld:25803 blocked for more than 120 seconds.
[Mon Mar 25 14:40:26 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 25 14:40:26 2024] mysqld          D ffff8d94f0b830c0     0 25803  23578 0x00000080
[Mon Mar 25 14:40:26 2024] Call Trace:
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4e1c352>] ? kmem_cache_alloc+0x1c2/0x1f0
[Mon Mar 25 14:40:26 2024]  [<ffffffffc0577227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffa5368c49>] schedule+0x29/0x70
[Mon Mar 25 14:40:26 2024]  [<ffffffffa536a535>] rwsem_down_write_failed+0x225/0x3a0
[Mon Mar 25 14:40:26 2024]  [<ffffffffc05667dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4f870a7>] call_rwsem_down_write_failed+0x17/0x30
[Mon Mar 25 14:40:26 2024]  [<ffffffffa5367f4d>] down_write+0x2d/0x3d
[Mon Mar 25 14:40:26 2024]  [<ffffffffc0569544>] xfs_ilock+0xc4/0x120 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffc05667dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4e5e5c8>] update_time+0x28/0xd0
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4e5e710>] file_update_time+0xa0/0xf0
[Mon Mar 25 14:40:26 2024]  [<ffffffffc055c3bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffc055c6a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4cd6530>] ? try_to_wake_up+0x190/0x390
[Mon Mar 25 14:40:26 2024]  [<ffffffffc055cce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffc055cbe0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4e97b33>] do_io_submit+0x3e3/0x8a0
[Mon Mar 25 14:40:26 2024]  [<ffffffffa4e98000>] SyS_io_submit+0x10/0x20
[Mon Mar 25 14:40:26 2024]  [<ffffffffa5375ddb>] system_call_fastpath+0x22/0x27
[Mon Mar 25 14:40:26 2024] INFO: task mysqld:25804 blocked for more than 120 seconds.
[Mon Mar 25 14:40:26 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 25 14:40:26 2024] mysqld          D ffff8d94ea39e180     0 25804  23578 0x00000080
[Mon Mar 25 14:40:27 2024] Call Trace:
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4e1c352>] ? kmem_cache_alloc+0x1c2/0x1f0
[Mon Mar 25 14:40:27 2024]  [<ffffffffc0577227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffa5368c49>] schedule+0x29/0x70
[Mon Mar 25 14:40:27 2024]  [<ffffffffa536a535>] rwsem_down_write_failed+0x225/0x3a0
[Mon Mar 25 14:40:27 2024]  [<ffffffffc05667dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4f870a7>] call_rwsem_down_write_failed+0x17/0x30
[Mon Mar 25 14:40:27 2024]  [<ffffffffa5367f4d>] down_write+0x2d/0x3d
[Mon Mar 25 14:40:27 2024]  [<ffffffffc0569544>] xfs_ilock+0xc4/0x120 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffc05667dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4e5e5c8>] update_time+0x28/0xd0
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4e5e710>] file_update_time+0xa0/0xf0
[Mon Mar 25 14:40:27 2024]  [<ffffffffc055c3bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffc055c6a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4cd6530>] ? try_to_wake_up+0x190/0x390
[Mon Mar 25 14:40:27 2024]  [<ffffffffc055cce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffc055cbe0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4e97b33>] do_io_submit+0x3e3/0x8a0
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4e98000>] SyS_io_submit+0x10/0x20
[Mon Mar 25 14:40:27 2024]  [<ffffffffa5375ddb>] system_call_fastpath+0x22/0x27
[Mon Mar 25 14:40:27 2024] INFO: task mysqld:25809 blocked for more than 120 seconds.
[Mon Mar 25 14:40:27 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 25 14:40:27 2024] mysqld          D ffff8d94b53b0000     0 25809  23578 0x00000080
[Mon Mar 25 14:40:27 2024] Call Trace:
[Mon Mar 25 14:40:27 2024]  [<ffffffffa5368c49>] schedule+0x29/0x70
[Mon Mar 25 14:40:27 2024]  [<ffffffffa536a27d>] rwsem_down_read_failed+0x10d/0x1a0
[Mon Mar 25 14:40:27 2024]  [<ffffffffc055b1b4>] ? xfs_file_fsync+0xc4/0x1c0 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4f87078>] call_rwsem_down_read_failed+0x18/0x30
[Mon Mar 25 14:40:27 2024]  [<ffffffffc055b0f0>] ? xfs_file_readdir+0x40/0x40 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffa5367f00>] down_read+0x20/0x40
[Mon Mar 25 14:40:27 2024]  [<ffffffffc056955c>] xfs_ilock+0xdc/0x120 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffc055b1b4>] xfs_file_fsync+0xc4/0x1c0 [xfs]
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4e762e7>] do_fsync+0x67/0xb0
[Mon Mar 25 14:40:27 2024]  [<ffffffffa4e765d0>] SyS_fsync+0x10/0x20
[Mon Mar 25 14:40:28 2024]  [<ffffffffa5375ddb>] system_call_fastpath+0x22/0x27

操作系统或 XFS 有问题吗?

相关内容