MariaDB 崩溃后 CPU 负载过高

MariaDB 崩溃后 CPU 负载过高

我在使用 MariaDB 10.3.16 时遇到了问题。我试图将一个相当大的表 (26GB) 从 MyISAM 转换为 InnoDB,但不幸的是,在此过程中磁盘空间用完了。当磁盘空间满了时,它会记录大量如下消息:

2022-03-25 0:03:54 18688637 [ERROR] InnoDB: preallocating 26935820288 bytes for file ./mydatabase/#sql-22ba_11d2a7d.ibd failed with error 28

...然后 MariaDB 崩溃了。我释放了一些空间并再次启动了 MariaDB -> 日志显示两个表崩溃了并且它们被检查了。它运行了一段时间,然后检查完​​成,一切似乎都正常了。表是最新的并且可用。(当然那个表仍然是 MyISAM)

但奇怪的是,从那时起,mysqld该进程的 CPU 使用率一直很高(大约 75%),而且似乎没有恢复。过去 24 小时内,服务器的 CPU 使用率一直很高。没有运行任何异常查询,我只是不知道为什么它会占用所有 CPU。

我跑了,SHOW ENGINE INNODB STATUS结果给了我这个:

=====================================
2022-03-26 00:09:30 0x7ff7d80ab700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 16 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 51 srv_active, 0 srv_shutdown, 989 srv_idle
srv_master_thread log flush and writes: 1040
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 5965
OS WAIT ARRAY INFO: signal count 1779
RW-shared spins 0, rounds 719, OS waits 286
RW-excl spins 0, rounds 61, OS waits 2
RW-sx spins 0, rounds 0, OS waits 0
Spin rounds per wait: 719.00 RW-shared, 61.00 RW-excl, 0.00 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 13314188
Purge done for trx's n:o < 13314188 undo n:o < 0 state: running but idle
History list length 2
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 422177494315392, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422177494311176, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422177494306960, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (read thread)
I/O thread 7 state: waiting for completed aio requests (read thread)
I/O thread 8 state: waiting for completed aio requests (read thread)
I/O thread 9 state: waiting for completed aio requests (read thread)
I/O thread 10 state: waiting for completed aio requests (read thread)
I/O thread 11 state: waiting for completed aio requests (read thread)
I/O thread 12 state: waiting for completed aio requests (read thread)
I/O thread 13 state: waiting for completed aio requests (read thread)
I/O thread 14 state: waiting for completed aio requests (read thread)
I/O thread 15 state: waiting for completed aio requests (read thread)
I/O thread 16 state: waiting for completed aio requests (read thread)
I/O thread 17 state: waiting for completed aio requests (read thread)
I/O thread 18 state: waiting for completed aio requests (read thread)
I/O thread 19 state: waiting for completed aio requests (read thread)
I/O thread 20 state: waiting for completed aio requests (read thread)
I/O thread 21 state: waiting for completed aio requests (read thread)
I/O thread 22 state: waiting for completed aio requests (read thread)
I/O thread 23 state: waiting for completed aio requests (read thread)
I/O thread 24 state: waiting for completed aio requests (read thread)
I/O thread 25 state: waiting for completed aio requests (read thread)
I/O thread 26 state: waiting for completed aio requests (read thread)
I/O thread 27 state: waiting for completed aio requests (read thread)
I/O thread 28 state: waiting for completed aio requests (read thread)
I/O thread 29 state: waiting for completed aio requests (read thread)
I/O thread 30 state: waiting for completed aio requests (read thread)
I/O thread 31 state: waiting for completed aio requests (read thread)
I/O thread 32 state: waiting for completed aio requests (read thread)
I/O thread 33 state: waiting for completed aio requests (read thread)
I/O thread 34 state: waiting for completed aio requests (read thread)
I/O thread 35 state: waiting for completed aio requests (read thread)
I/O thread 36 state: waiting for completed aio requests (read thread)
I/O thread 37 state: waiting for completed aio requests (read thread)
I/O thread 38 state: waiting for completed aio requests (read thread)
I/O thread 39 state: waiting for completed aio requests (read thread)
I/O thread 40 state: waiting for completed aio requests (read thread)
I/O thread 41 state: waiting for completed aio requests (read thread)
I/O thread 42 state: waiting for completed aio requests (read thread)
I/O thread 43 state: waiting for completed aio requests (read thread)
I/O thread 44 state: waiting for completed aio requests (read thread)
I/O thread 45 state: waiting for completed aio requests (read thread)
I/O thread 46 state: waiting for completed aio requests (read thread)
I/O thread 47 state: waiting for completed aio requests (read thread)
I/O thread 48 state: waiting for completed aio requests (read thread)
I/O thread 49 state: waiting for completed aio requests (read thread)
I/O thread 50 state: waiting for completed aio requests (read thread)
I/O thread 51 state: waiting for completed aio requests (read thread)
I/O thread 52 state: waiting for completed aio requests (read thread)
I/O thread 53 state: waiting for completed aio requests (read thread)
I/O thread 54 state: waiting for completed aio requests (read thread)
I/O thread 55 state: waiting for completed aio requests (read thread)
I/O thread 56 state: waiting for completed aio requests (read thread)
I/O thread 57 state: waiting for completed aio requests (read thread)
I/O thread 58 state: waiting for completed aio requests (read thread)
I/O thread 59 state: waiting for completed aio requests (read thread)
I/O thread 60 state: waiting for completed aio requests (read thread)
I/O thread 61 state: waiting for completed aio requests (read thread)
I/O thread 62 state: waiting for completed aio requests (read thread)
I/O thread 63 state: waiting for completed aio requests (read thread)
I/O thread 64 state: waiting for completed aio requests (read thread)
I/O thread 65 state: waiting for completed aio requests (read thread)
I/O thread 66 state: waiting for completed aio requests (write thread)
I/O thread 67 state: waiting for completed aio requests (write thread)
I/O thread 68 state: waiting for completed aio requests (write thread)
I/O thread 69 state: waiting for completed aio requests (write thread)
I/O thread 70 state: waiting for completed aio requests (write thread)
I/O thread 71 state: waiting for completed aio requests (write thread)
I/O thread 72 state: waiting for completed aio requests (write thread)
I/O thread 73 state: waiting for completed aio requests (write thread)
I/O thread 74 state: waiting for completed aio requests (write thread)
I/O thread 75 state: waiting for completed aio requests (write thread)
I/O thread 76 state: waiting for completed aio requests (write thread)
I/O thread 77 state: waiting for completed aio requests (write thread)
I/O thread 78 state: waiting for completed aio requests (write thread)
I/O thread 79 state: waiting for completed aio requests (write thread)
I/O thread 80 state: waiting for completed aio requests (write thread)
I/O thread 81 state: waiting for completed aio requests (write thread)
I/O thread 82 state: waiting for completed aio requests (write thread)
I/O thread 83 state: waiting for completed aio requests (write thread)
I/O thread 84 state: waiting for completed aio requests (write thread)
I/O thread 85 state: waiting for completed aio requests (write thread)
I/O thread 86 state: waiting for completed aio requests (write thread)
I/O thread 87 state: waiting for completed aio requests (write thread)
I/O thread 88 state: waiting for completed aio requests (write thread)
I/O thread 89 state: waiting for completed aio requests (write thread)
I/O thread 90 state: waiting for completed aio requests (write thread)
I/O thread 91 state: waiting for completed aio requests (write thread)
I/O thread 92 state: waiting for completed aio requests (write thread)
I/O thread 93 state: waiting for completed aio requests (write thread)
I/O thread 94 state: waiting for completed aio requests (write thread)
I/O thread 95 state: waiting for completed aio requests (write thread)
I/O thread 96 state: waiting for completed aio requests (write thread)
I/O thread 97 state: waiting for completed aio requests (write thread)
I/O thread 98 state: waiting for completed aio requests (write thread)
I/O thread 99 state: waiting for completed aio requests (write thread)
I/O thread 100 state: waiting for completed aio requests (write thread)
I/O thread 101 state: waiting for completed aio requests (write thread)
I/O thread 102 state: waiting for completed aio requests (write thread)
I/O thread 103 state: waiting for completed aio requests (write thread)
I/O thread 104 state: waiting for completed aio requests (write thread)
I/O thread 105 state: waiting for completed aio requests (write thread)
I/O thread 106 state: waiting for completed aio requests (write thread)
I/O thread 107 state: waiting for completed aio requests (write thread)
I/O thread 108 state: waiting for completed aio requests (write thread)
I/O thread 109 state: waiting for completed aio requests (write thread)
I/O thread 110 state: waiting for completed aio requests (write thread)
I/O thread 111 state: waiting for completed aio requests (write thread)
I/O thread 112 state: waiting for completed aio requests (write thread)
I/O thread 113 state: waiting for completed aio requests (write thread)
I/O thread 114 state: waiting for completed aio requests (write thread)
I/O thread 115 state: waiting for completed aio requests (write thread)
I/O thread 116 state: waiting for completed aio requests (write thread)
I/O thread 117 state: waiting for completed aio requests (write thread)
I/O thread 118 state: waiting for completed aio requests (write thread)
I/O thread 119 state: waiting for completed aio requests (write thread)
I/O thread 120 state: waiting for completed aio requests (write thread)
I/O thread 121 state: waiting for completed aio requests (write thread)
I/O thread 122 state: waiting for completed aio requests (write thread)
I/O thread 123 state: waiting for completed aio requests (write thread)
I/O thread 124 state: waiting for completed aio requests (write thread)
I/O thread 125 state: waiting for completed aio requests (write thread)
I/O thread 126 state: waiting for completed aio requests (write thread)
I/O thread 127 state: waiting for completed aio requests (write thread)
I/O thread 128 state: waiting for completed aio requests (write thread)
I/O thread 129 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] , aio writes: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
8105 OS file reads, 503 OS file writes, 335 OS fsyncs
0.44 reads/s, 16384 avg bytes/read, 1.25 writes/s, 1.19 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 56, seg size 58, 7 merges
merged operations:
insert 7, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 1593833, node heap has 21 buffer(s)
Hash table size 1593833, node heap has 0 buffer(s)
Hash table size 1593833, node heap has 0 buffer(s)
Hash table size 1593833, node heap has 6 buffer(s)
Hash table size 1593833, node heap has 1 buffer(s)
Hash table size 1593833, node heap has 1 buffer(s)
Hash table size 1593833, node heap has 32 buffer(s)
Hash table size 1593833, node heap has 3 buffer(s)
18.56 hash searches/s, 10.12 non-hash searches/s
---
LOG
---
Log sequence number 57880104793
Log flushed up to   57880104793
Pages flushed up to 57880104793
Last checkpoint at  57880104784
0 pending log flushes, 0 pending chkp writes
100 log i/o's done, 0.31 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 7127171072
Dictionary memory allocated 176256
Buffer pool size   393216
Free buffers       385734
Database pages     7418
Old database pages 2895
Modified db pages  0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 7287, created 131, written 293
0.44 reads/s, 0.00 creates/s, 0.50 writes/s
Buffer pool hit rate 999 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 7418, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size   49152
Free buffers       48320
Database pages     824
Old database pages 324
Modified db pages  0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 820, created 4, written 34
0.06 reads/s, 0.00 creates/s, 0.06 writes/s
Buffer pool hit rate 999 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 824, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size   49152
Free buffers       48319
Database pages     825
Old database pages 324
Modified db pages  0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 825, created 0, written 20
0.00 reads/s, 0.00 creates/s, 0.06 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 825, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size   49152
Free buffers       48199
Database pages     945
Old database pages 368
Modified db pages  0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 945, created 0, written 16
0.00 reads/s, 0.00 creates/s, 0.06 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 945, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 3
Buffer pool size   49152
Free buffers       48081
Database pages     1063
Old database pages 412
Modified db pages  0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1063, created 0, written 28
0.06 reads/s, 0.00 creates/s, 0.06 writes/s
Buffer pool hit rate 999 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 1063, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 4
Buffer pool size   49152
Free buffers       48163
Database pages     981
Old database pages 382
Modified db pages  0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 981, created 0, written 31
0.06 reads/s, 0.00 creates/s, 0.12 writes/s
Buffer pool hit rate 998 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 981, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 5
Buffer pool size   49152
Free buffers       48201
Database pages     943
Old database pages 368
Modified db pages  0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 943, created 0, written 18
0.06 reads/s, 0.00 creates/s, 0.06 writes/s
Buffer pool hit rate 999 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 943, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 6
Buffer pool size   49152
Free buffers       48130
Database pages     1014
Old database pages 394
Modified db pages  0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 951, created 63, written 73
0.06 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 998 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 1014, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 7
Buffer pool size   49152
Free buffers       48321
Database pages     823
Old database pages 323
Modified db pages  0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 759, created 64, written 73
0.12 reads/s, 0.00 creates/s, 0.06 writes/s
Buffer pool hit rate 991 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 823, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=1957, Main thread ID=140695161886464, state: sleeping
Number of rows inserted 15, updated 1, deleted 0, read 1046108
0.06 inserts/s, 0.00 updates/s, 0.00 deletes/s, 1589.46 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================

FILE I/O部分内容看起来相当可疑I/O thread 0 state: waiting for completed aio requests,但我真的不知道如何处理这些信息?

重启服务器没有帮助,CPU 仍然很高,并且不会下降:

Tasks: 117 total,   1 running, 116 sleeping,   0 stopped,   0 zombie
%Cpu(s): 72.2 us,  1.9 sy,  0.0 ni, 25.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 16266120 total, 13867856 free,  1429552 used,   968712 buff/cache
KiB Swap:  2047996 total,  2047996 free,        0 used. 14562068 avail Mem

任何帮助,将不胜感激。

编辑1: 我刚刚发现确实存在一些值得注意的进程。我以为这些 InnoDB 进程是正常的,但我猜它们不知何故卡住了?(关于上面的 FILE I/O)

| Id    | User        | Host      | db   | Command     | Time  | State                                                                 | Info                  | Progress |
+-------+-------------+-----------+------+-------------+-------+-----------------------------------------------------------------------+-----------------------+----------+
|     3 | system user |           | NULL | Daemon      |  NULL | InnoDB purge worker                                                   | NULL                  |    0.000 |
|     2 | system user |           | NULL | Daemon      |  NULL | InnoDB purge worker                                                   | NULL                  |    0.000 |
|     1 | system user |           | NULL | Daemon      |  NULL | InnoDB purge coordinator                                              | NULL                  |    0.000 |
|     4 | system user |           | NULL | Daemon      |  NULL | InnoDB purge worker                                                   | NULL                  |    0.000 |
|     5 | system user |           | NULL | Daemon      |  NULL | InnoDB shutdown handler                                               | NULL                  |    0.000 |
|    10 | slave-db2   | db2:39696 | NULL | Binlog Dump | 41840 | Master has sent all binlog to slave; waiting for binlog to be updated | NULL                  |    0.000 |
| 46369 | root        | localhost | NULL | Query       |     0 | Init                                                                  | show full processlist |    0.000 |
+-------+-------------+-----------+------+-------------+-------+-----------------------------------------------------------------------+-----------------------+----------+

编辑2: 我想让你知道,CPU 使用率恢复正常,没有明显的原因。似乎服务器在过去 3 天内一直处于繁忙状态,计算着谁知道什么,但我很高兴它完成了,一切似乎又恢复正常了。我会再等一段时间,释放更多空间,然后再运行一次 InnoDB 转换。

感谢@Rick James 的努力。

答案1

转换

InnoDB 使用的磁盘空间是 MyISAM 的 2 到 3 倍。这是 InnoDB 的极少数缺点之一(相对于 MyISAM)。

如果您可以使用更大的磁盘重新开始,那将是最好的。

如果您正在“就地”转换过程中遇到困难,请终止该过程。(除非您能奇迹般地释放一些磁盘空间,否则转换过程不可能完成。)然后我们可以讨论如何处理半完成的转换。

更多讨论: http://mysql.rjweb.org/doc.php/myisam2innodb

CPU 占用高

连接到数据库并SHOW FULL PROCESSLIST; This _should_ tell us what is running (inside MySQL); that may give a clue of the CPU hog. Assuming it is referring to some table, please provide对其执行 SHOW CREATE TABLE`。

相关内容