我们的组织拥有一个相当先进的（就像许多活动部件一样）网络应用程序，直到最近它一直运行良好，但没有发生明显的变化。
有 Apache Web 服务器、mySQL 服务器（数据处理），然后还有另一个处理公共重复请求的 mySQL 服务器。主 Web 服务器的流量涌入几乎只会影响辅助 SQL 服务器。重复请求服务器是数据处理服务器的从属服务器，但没有写入查询发送到重复请求服务器 - 只读。
我面临的问题是，数据处理 mySQL 服务器似乎会随机地将 CPU 使用率飙升至 100%，而通常运行在 15-20% 的水平，即使在高负载下也是如此。100% 的 CPU 使用率持续约 8 秒，有时每 2-3 分钟就会突然出现一次，直到它自行随机消失。
慢速查询日志中除了在 100% 时尝试处理的查询外，没有任何查询。日志中记录的查询与随机峰值之前通常运行正常的查询相同。
HTOP 显示的唯一活动是 mySQL，此服务器上也没有安排 cron，并且在此期间其他查询活动没有增加。打开的连接线程帐户保持稳定在 3-5 左右，并且进程列表在之前、期间或之后也只有 3-5 个查询。
SELECT * FROM audio
WHERE associated_incident IS null
AND archive IS NULL
AND temp_skip IS null
AND length >= 3
AND (locked <> 1 or locked IS NULL)
AND timestamp > (NOW() - INTERVAL 4 HOUR)
ORDER BY `audio`.`id` DESC
LIMIT 1;
UPDATE audio SET locked=1, lockexpr = '$ulockexpr', lockuser = '$ulockuser' WHERE id = $audio_id;
上述查询在我们员工使用的软件中重复运行，但在给定时间内每秒不超过 1-2 次，但这是我能缩小问题范围的最接近的一次。此查询在高峰期也会显示在 slow_query_log 中。但需要再次澄清的是，此查询全天都在使用，我们并不总是遇到这些问题。
我可以使用此查询对服务器进行压力测试，在 5 秒内运行 100 次，并且仍然可以获得平均返回时间为 0.002 到 0.005 秒的结果。
在 /var/log/mysql 中我有 8.2Gb 的 mysql-bin 文件。
仅可能对 my.cnf 进行修改
key_buffer_size = 16M
myisam-recover-options = BACKUP
log_error = /var/log/mysql/error.log
slow_query_log = 1
slow_query_log_file = /var/log/mysql/mysql-slow.log
long_query_time = 2
server-id = 1
log_bin = /var/log/mysql/mysql-bin.log
max_binlog_size = 100M
binlog_do_db = incident_log
其余所有内容均未指定或被注释掉。
Mysql 调谐器
>> MySQLTuner 2.1.1
* Jean-Marie Renouard <[email protected]>
* Major Hayden <[email protected]>
>> Bug reports, feature requests, and downloads at http://mysqltuner.pl/
>> Run with '--help' for additional options and output filtering
[--] Skipped version check for MySQLTuner script
[OK] Logged in using credentials from Debian maintenance account.
[OK] Currently running supported MySQL version 8.0.32-0ubuntu0.20.04.2
[OK] Operating on 64-bit architecture
-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/log/mysql/error.log exists
[--] Log file: /var/log/mysql/error.log (0B)
[--] Log file /var/log/mysql/error.log is empty. Assuming log-rotation. Use --server-log={file} for explicit file
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA
[--] Data in InnoDB tables: 1.5G (Tables: 107)
[OK] Total fragmented tables: 0
-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.
-------- Views Metrics -----------------------------------------------------------------------------
-------- Triggers Metrics --------------------------------------------------------------------------
-------- Routines Metrics --------------------------------------------------------------------------
-------- Security Recommendations ------------------------------------------------------------------
[--] Skipped due to unsupported feature for MySQL 8.0+
-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 12h 41m 34s (3M q [73.289 qps], 1M conn, TX: 5G, RX: 445M)
[--] Reads / Writes: 81% / 19%
[--] Binary logging is enabled (GTID MODE: OFF)
[--] Physical Memory : 1.9G
[--] Max MySQL memory : 10.1G
[--] Other process memory: 0B
[--] Total buffers: 176.0M global + 65.9M per thread (151 max threads)
[--] Performance_schema Max memory usage: 239M
[--] Galera GCache Max memory usage: 0B
[!!] Maximum reached memory usage: 10.2G (528.62% of installed RAM)
[!!] Maximum possible memory usage: 10.1G (525.28% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (3/3M)
[!!] Highest connection usage: 100% (152/151)
[OK] Aborted connections: 0.09% (1203/1391547)
[!!] Name resolution is active: a reverse name resolution is made for each new connection which can reduce performance
[--] Query cache has been removed since MySQL 8.0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 10K sorts)
[!!] Joins performed without indexes: 5915
[OK] Temporary tables created on disk: 0% (0 on disk / 8K total)
[OK] Thread cache hit rate: 94% (75K created / 1M connections)
[OK] Table cache hit rate: 99% (1M hits / 1M requests)
[OK] table_definition_cache (2000) is greater than number of tables (434)
[OK] Open file limit used: 0% (3/10K)
[OK] Table locks acquired immediately: 100% (1K immediate / 1K locks)
[OK] Binlog cache memory access: 100.00% (346520 Memory / 346520 Total)
-------- Performance schema ------------------------------------------------------------------------
[--] Performance_schema is activated.
[--] Memory used by Performance_schema: 239.2M
[--] Sys schema is installed.
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.
-------- MyISAM Metrics ----------------------------------------------------------------------------
[--] MyISAM Metrics are disabled since MySQL 8.0.
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[!!] InnoDB buffer pool / data size: 128.0M / 1.5G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (75%): 48.0M * 2 / 128.0M should be equal to 25%
[OK] InnoDB buffer pool instances: 1
[--] Number of InnoDB Buffer Pool Chunk: 1 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 99.87% (403976971 hits / 404504329 total)
[!!] InnoDB Write Log efficiency: 59.62% (1713194 hits / 2873563 total)
[OK] InnoDB log waits: 0.00% (0 waits / 1160369 writes)
-------- Aria Metrics ------------------------------------------------------------------------------
[--] Aria Storage Engine not available.
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: ROW
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
MySQL was started within the last 24 hours: recommendations may be inaccurate
Reduce your overall MySQL memory footprint for system stability
Dedicate this server to your database for highest performance.
Reduce or eliminate persistent connections to reduce connection usage
Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
(specially the conclusions at the bottom of the page).
Buffer Key MyISAM set to 0, no MyISAM table detected
Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: https://bit.ly/2TcGgtU
Variables to adjust:
*** MySQL's maximum memory usage is dangerously high ***
*** Add RAM before increasing MySQL buffer variables ***
max_connections (> 151)
wait_timeout (< 28800)
interactive_timeout (< 28800)
skip-name-resolve=1
join_buffer_size (> 256.0K, or always use indexes with JOINs)
key_buffer_size=0
innodb_buffer_pool_size (>= 1.5G) if possible.
innodb_log_file_size should be (=16M) if possible, so InnoDB total log file size equals 25% of buffer pool size.
欢迎大家提出建议。
编辑：
mysql> SHOW TABLE STATUS WHERE name LIKE "audio";
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
| audio | InnoDB | 10 | Dynamic | 969276 | 164 | 159039488 | 0 | 90898432 | 7340032 | 1644653 | 2023-04-13 14:41:27 | 2023-04-17 14:56:21 | NULL | utf8mb4_0900_ai_ci | NULL | | |
PHP
$sql = "SELECT * FROM audio
WHERE associated_incident IS null
AND archive IS NULL
AND temp_skip IS null
AND length >= 3
AND (locked <> 1 or locked IS NULL)
ORDER BY `audio`.`id` DESC
LIMIT 1;
";
$result = mysqli_query($conn, $sql) or die(mysqli_error());
$audio = mysqli_fetch_assoc($result);
if ($audio) {
$audio_id = $audio['id'];
$audio_timestamp = $audio['timestamp'];
$audio_hash = $audio['hash'];
$audio_length = $audio['length'];
$ulockexpr = date("Y-m-d H:i:s", strtotime('+ 5min'));
$ulockuser = $_SESSION['login_user'];
$sql_lock = "UPDATE audio SET locked=1, lockexpr = '$ulockexpr', lockuser = '$ulockuser' WHERE id = $audio_id";
答案1
更改为innodb_buffer_pool_size = 500M——旧的默认值 128M 太低了。
如果这样SELECT做是为了找到要处理的行，然后您这样做UPDATE，那么使用事务：
START TRANSACTION;
SELECT ... FROM audio ... FOR UPDATE;
... process ...
UPDATE audio SET ... WHERE id = ...;
COMMIT;
这START - FOR UPDATE - COMMIT将阻止两个线程抓取同一项目进行工作。
考虑使用ORDER BY timestamp DESC而不是ORDER BY id DESC。
可能需要此配置才能看到有问题的语句：
log_slow_admin_statements = ON
表上有哪些索引？请提供SHOW CREATE TABLE。
降低 [是的，降低]max_connections至仅50。 和降低 Apache 设置，使其不会产生超过 40 个并发子进程。我看到了 hit Max_used_connections。max_connections发生这种情况时，MySQL 会崩溃，直到问题解决。降低限制将使其更快地开始解决。