系统:CentOS Linux 版本 7.6.1810(核心)
内核:Linux l-db11.tj.cn5 3.10.0-957.10.1.el7.x86_64 #1 SMP 2019 年 3 月 18 日星期一 15:06:45 UTC x86_64 x86_64 x86_64 GNU/Linux
SSD:德拉
CPU:24核
内存:128G
操作系统只能以很小的速率写入。
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 0.00 2.00 130.00 0.03 0.91 14.67 0.00 0.03 0.00 0.03 0.03 0.40
nvme0n1 0.00 0.00 0.00 122.00 0.00 0.84 14.05 0.00 0.03 0.00 0.03 0.03 0.40
nvme0n1 0.00 0.00 16.00 201.00 0.25 3.83 38.53 0.01 0.03 0.06 0.03 0.03 0.60
nvme0n1 0.00 0.00 8.00 204.00 0.12 1.12 12.03 0.01 0.03 0.12 0.03 0.03 0.70
nvme0n1 0.00 0.00 0.00 154.00 0.00 1.05 13.95 0.00 0.01 0.00 0.01 0.01 0.20
nvme0n1 0.00 0.00 1.00 162.00 0.02 1.07 13.65 0.01 0.04 0.00 0.04 0.04 0.70
记忆:
free -g
total used free shared buff/cache available
Mem: 125 58 50 3 16 62
Swap: 63 0 63
dmesg:
[Mon Mar 25 14:40:24 2024] INFO: task mysqld:25801 blocked for more than 120 seconds.
[Mon Mar 25 14:40:24 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 25 14:40:24 2024] mysqld D ffff8d94ebbdb0c0 0 25801 23578 0x00000080
[Mon Mar 25 14:40:24 2024] Call Trace:
[Mon Mar 25 14:40:24 2024] [<ffffffffa4e1c352>] ? kmem_cache_alloc+0x1c2/0x1f0
[Mon Mar 25 14:40:24 2024] [<ffffffffc0577227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
[Mon Mar 25 14:40:24 2024] [<ffffffffa5368c49>] schedule+0x29/0x70
[Mon Mar 25 14:40:24 2024] [<ffffffffa536a535>] rwsem_down_write_failed+0x225/0x3a0
[Mon Mar 25 14:40:24 2024] [<ffffffffc05778e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
[Mon Mar 25 14:40:24 2024] [<ffffffffc05667dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:24 2024] [<ffffffffa4f870a7>] call_rwsem_down_write_failed+0x17/0x30
[Mon Mar 25 14:40:25 2024] [<ffffffffa5367f4d>] down_write+0x2d/0x3d
[Mon Mar 25 14:40:25 2024] [<ffffffffc0569544>] xfs_ilock+0xc4/0x120 [xfs]
[Mon Mar 25 14:40:25 2024] [<ffffffffc05667dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:25 2024] [<ffffffffa4e5e5c8>] update_time+0x28/0xd0
[Mon Mar 25 14:40:25 2024] [<ffffffffa4e5e710>] file_update_time+0xa0/0xf0
[Mon Mar 25 14:40:25 2024] [<ffffffffc055c3bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
[Mon Mar 25 14:40:25 2024] [<ffffffffc055c6a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
[Mon Mar 25 14:40:25 2024] [<ffffffffa4cd6530>] ? try_to_wake_up+0x190/0x390
[Mon Mar 25 14:40:25 2024] [<ffffffffc055cce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[Mon Mar 25 14:40:25 2024] [<ffffffffc055cbe0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
[Mon Mar 25 14:40:25 2024] [<ffffffffa4e97b33>] do_io_submit+0x3e3/0x8a0
[Mon Mar 25 14:40:25 2024] [<ffffffffa4e98000>] SyS_io_submit+0x10/0x20
[Mon Mar 25 14:40:25 2024] [<ffffffffa5375ddb>] system_call_fastpath+0x22/0x27
[Mon Mar 25 14:40:25 2024] INFO: task mysqld:25802 blocked for more than 120 seconds.
[Mon Mar 25 14:40:25 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 25 14:40:25 2024] mysqld D ffff8d94f0b85140 0 25802 23578 0x00000080
[Mon Mar 25 14:40:25 2024] Call Trace:
[Mon Mar 25 14:40:25 2024] [<ffffffffa5368c49>] schedule+0x29/0x70
[Mon Mar 25 14:40:25 2024] [<ffffffffa536a27d>] rwsem_down_read_failed+0x10d/0x1a0
[Mon Mar 25 14:40:25 2024] [<ffffffffc05695d0>] ? xfs_ilock_data_map_shared+0x30/0x40 [xfs]
[Mon Mar 25 14:40:25 2024] [<ffffffffa4f87078>] call_rwsem_down_read_failed+0x18/0x30
[Mon Mar 25 14:40:25 2024] [<ffffffffa5367f00>] down_read+0x20/0x40
[Mon Mar 25 14:40:25 2024] [<ffffffffc056955c>] xfs_ilock+0xdc/0x120 [xfs]
[Mon Mar 25 14:40:25 2024] [<ffffffffc05695d0>] xfs_ilock_data_map_shared+0x30/0x40 [xfs]
[Mon Mar 25 14:40:25 2024] [<ffffffffc054f9ca>] __xfs_get_blocks+0x9a/0x6d0 [xfs]
[Mon Mar 25 14:40:25 2024] [<ffffffffa4c79912>] ? get_user_pages_fast+0x122/0x1a0
[Mon Mar 25 14:40:25 2024] [<ffffffffc0550e24>] xfs_get_blocks_direct+0x14/0x20 [xfs]
[Mon Mar 25 14:40:25 2024] [<ffffffffa4e848f6>] do_blockdev_direct_IO+0xf96/0x20a0
[Mon Mar 25 14:40:25 2024] [<ffffffffc0550e10>] ? xfs_setfilesize+0x80/0x80 [xfs]
[Mon Mar 25 14:40:25 2024] [<ffffffffa4e85a55>] __blockdev_direct_IO+0x55/0x60
[Mon Mar 25 14:40:25 2024] [<ffffffffc0550e10>] ? xfs_setfilesize+0x80/0x80 [xfs]
[Mon Mar 25 14:40:25 2024] [<ffffffffc0550e30>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
[Mon Mar 25 14:40:25 2024] [<ffffffffc055c759>] xfs_file_dio_aio_write+0x179/0x340 [xfs]
[Mon Mar 25 14:40:26 2024] [<ffffffffc0550e10>] ? xfs_setfilesize+0x80/0x80 [xfs]
[Mon Mar 25 14:40:26 2024] [<ffffffffc0550e30>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
[Mon Mar 25 14:40:26 2024] [<ffffffffc055cce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[Mon Mar 25 14:40:26 2024] [<ffffffffc055cbe0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
[Mon Mar 25 14:40:26 2024] [<ffffffffa4e97b33>] do_io_submit+0x3e3/0x8a0
[Mon Mar 25 14:40:26 2024] [<ffffffffa4e98000>] SyS_io_submit+0x10/0x20
[Mon Mar 25 14:40:26 2024] [<ffffffffa5375ddb>] system_call_fastpath+0x22/0x27
[Mon Mar 25 14:40:26 2024] INFO: task mysqld:25803 blocked for more than 120 seconds.
[Mon Mar 25 14:40:26 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 25 14:40:26 2024] mysqld D ffff8d94f0b830c0 0 25803 23578 0x00000080
[Mon Mar 25 14:40:26 2024] Call Trace:
[Mon Mar 25 14:40:26 2024] [<ffffffffa4e1c352>] ? kmem_cache_alloc+0x1c2/0x1f0
[Mon Mar 25 14:40:26 2024] [<ffffffffc0577227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
[Mon Mar 25 14:40:26 2024] [<ffffffffa5368c49>] schedule+0x29/0x70
[Mon Mar 25 14:40:26 2024] [<ffffffffa536a535>] rwsem_down_write_failed+0x225/0x3a0
[Mon Mar 25 14:40:26 2024] [<ffffffffc05667dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:26 2024] [<ffffffffa4f870a7>] call_rwsem_down_write_failed+0x17/0x30
[Mon Mar 25 14:40:26 2024] [<ffffffffa5367f4d>] down_write+0x2d/0x3d
[Mon Mar 25 14:40:26 2024] [<ffffffffc0569544>] xfs_ilock+0xc4/0x120 [xfs]
[Mon Mar 25 14:40:26 2024] [<ffffffffc05667dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:26 2024] [<ffffffffa4e5e5c8>] update_time+0x28/0xd0
[Mon Mar 25 14:40:26 2024] [<ffffffffa4e5e710>] file_update_time+0xa0/0xf0
[Mon Mar 25 14:40:26 2024] [<ffffffffc055c3bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
[Mon Mar 25 14:40:26 2024] [<ffffffffc055c6a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
[Mon Mar 25 14:40:26 2024] [<ffffffffa4cd6530>] ? try_to_wake_up+0x190/0x390
[Mon Mar 25 14:40:26 2024] [<ffffffffc055cce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[Mon Mar 25 14:40:26 2024] [<ffffffffc055cbe0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
[Mon Mar 25 14:40:26 2024] [<ffffffffa4e97b33>] do_io_submit+0x3e3/0x8a0
[Mon Mar 25 14:40:26 2024] [<ffffffffa4e98000>] SyS_io_submit+0x10/0x20
[Mon Mar 25 14:40:26 2024] [<ffffffffa5375ddb>] system_call_fastpath+0x22/0x27
[Mon Mar 25 14:40:26 2024] INFO: task mysqld:25804 blocked for more than 120 seconds.
[Mon Mar 25 14:40:26 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 25 14:40:26 2024] mysqld D ffff8d94ea39e180 0 25804 23578 0x00000080
[Mon Mar 25 14:40:27 2024] Call Trace:
[Mon Mar 25 14:40:27 2024] [<ffffffffa4e1c352>] ? kmem_cache_alloc+0x1c2/0x1f0
[Mon Mar 25 14:40:27 2024] [<ffffffffc0577227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
[Mon Mar 25 14:40:27 2024] [<ffffffffa5368c49>] schedule+0x29/0x70
[Mon Mar 25 14:40:27 2024] [<ffffffffa536a535>] rwsem_down_write_failed+0x225/0x3a0
[Mon Mar 25 14:40:27 2024] [<ffffffffc05667dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:27 2024] [<ffffffffa4f870a7>] call_rwsem_down_write_failed+0x17/0x30
[Mon Mar 25 14:40:27 2024] [<ffffffffa5367f4d>] down_write+0x2d/0x3d
[Mon Mar 25 14:40:27 2024] [<ffffffffc0569544>] xfs_ilock+0xc4/0x120 [xfs]
[Mon Mar 25 14:40:27 2024] [<ffffffffc05667dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
[Mon Mar 25 14:40:27 2024] [<ffffffffa4e5e5c8>] update_time+0x28/0xd0
[Mon Mar 25 14:40:27 2024] [<ffffffffa4e5e710>] file_update_time+0xa0/0xf0
[Mon Mar 25 14:40:27 2024] [<ffffffffc055c3bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
[Mon Mar 25 14:40:27 2024] [<ffffffffc055c6a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
[Mon Mar 25 14:40:27 2024] [<ffffffffa4cd6530>] ? try_to_wake_up+0x190/0x390
[Mon Mar 25 14:40:27 2024] [<ffffffffc055cce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[Mon Mar 25 14:40:27 2024] [<ffffffffc055cbe0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
[Mon Mar 25 14:40:27 2024] [<ffffffffa4e97b33>] do_io_submit+0x3e3/0x8a0
[Mon Mar 25 14:40:27 2024] [<ffffffffa4e98000>] SyS_io_submit+0x10/0x20
[Mon Mar 25 14:40:27 2024] [<ffffffffa5375ddb>] system_call_fastpath+0x22/0x27
[Mon Mar 25 14:40:27 2024] INFO: task mysqld:25809 blocked for more than 120 seconds.
[Mon Mar 25 14:40:27 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 25 14:40:27 2024] mysqld D ffff8d94b53b0000 0 25809 23578 0x00000080
[Mon Mar 25 14:40:27 2024] Call Trace:
[Mon Mar 25 14:40:27 2024] [<ffffffffa5368c49>] schedule+0x29/0x70
[Mon Mar 25 14:40:27 2024] [<ffffffffa536a27d>] rwsem_down_read_failed+0x10d/0x1a0
[Mon Mar 25 14:40:27 2024] [<ffffffffc055b1b4>] ? xfs_file_fsync+0xc4/0x1c0 [xfs]
[Mon Mar 25 14:40:27 2024] [<ffffffffa4f87078>] call_rwsem_down_read_failed+0x18/0x30
[Mon Mar 25 14:40:27 2024] [<ffffffffc055b0f0>] ? xfs_file_readdir+0x40/0x40 [xfs]
[Mon Mar 25 14:40:27 2024] [<ffffffffa5367f00>] down_read+0x20/0x40
[Mon Mar 25 14:40:27 2024] [<ffffffffc056955c>] xfs_ilock+0xdc/0x120 [xfs]
[Mon Mar 25 14:40:27 2024] [<ffffffffc055b1b4>] xfs_file_fsync+0xc4/0x1c0 [xfs]
[Mon Mar 25 14:40:27 2024] [<ffffffffa4e762e7>] do_fsync+0x67/0xb0
[Mon Mar 25 14:40:27 2024] [<ffffffffa4e765d0>] SyS_fsync+0x10/0x20
[Mon Mar 25 14:40:28 2024] [<ffffffffa5375ddb>] system_call_fastpath+0x22/0x27
操作系统或 XFS 有问题吗?