MySQL CPU 使用率过高导致服务器崩溃

MySQL CPU 使用率过高导致服务器崩溃

前提:

我不是 DBA,也不熟悉服务器,但我是公司里唯一能够使用这些东西的人。

我们拥有一台 Windows VPS,其规格如下:

CPU: Intel Xeon E5-2630 v4 2.20GHz
RAM: 60GB
SO: Windows Server 2016 Datacenter
HDD: 2TB SSD

在这里我托管了我的 Web 应用程序,它们访问托管在同一服务器上的数据库,这些 Web 应用程序被 +- 1000 个用户使用,他们需要通过 Web 应用程序 API 从数据库获取数据,MySQL 版本是:8.0.20 (MySQL Comminity Server - GPL)

这是my.ini

# Other default tuning values
# MySQL Server Instance Configuration File
# ----------------------------------------------------------------------
# Generated by the MySQL Server Instance Configuration Wizard
# 
# Installation Instructions
# ----------------------------------------------------------------------
# 
# On Linux you can copy this file to /etc/my.cnf to set global options,
# mysql-data-dir/my.cnf to set server-specific options
# (@localstatedir@ for this installation) or to
# ~/.my.cnf to set user-specific options.
# 
# On Windows you should keep this file in the installation directory 
# of your server (e.g. C:\Program Files\MySQL\MySQL Server X.Y). To
# make sure the server reads the config file use the startup option 
# "--defaults-file". 
# 
# To run the server from the command line, execute this in a 
# command line shell, e.g.
# mysqld --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
# 
# To install the server as a Windows service manually, execute this in a 
# command line shell, e.g.
# mysqld --install MySQLXY --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
# 
# And then execute this in a command line shell to start the server, e.g.
# net start MySQLXY
# 
# Guidelines for editing this file
# ----------------------------------------------------------------------
# 
# In this file, you can use all long options that the program supports.
# If you want to know the options a program supports, start the program
# with the "--help" option.
# 
# More detailed information about the individual options can also be
# found in the manual.
# 
# For advice on how to change settings please see
# https://dev.mysql.com/doc/refman/8.0/en/server-configuration-defaults.html
# 
# CLIENT SECTION
# ----------------------------------------------------------------------
# 
# The following options will be read by MySQL client applications.
# Note that only client applications shipped by MySQL are guaranteed
# to read this section. If you want your own MySQL client program to
# honor these values, you need to specify it as an option during the
# MySQL client library initialization.
# 
[client]

# pipe=

# socket=MYSQL

port=3306

[mysql]
no-beep=

# default-character-set=

# SERVER SECTION
# ----------------------------------------------------------------------
# 
# The following options will be read by the MySQL Server. Make sure that
# you have installed the server correctly (see above) so it reads this 
# file.=
# 
# server_type=2
[mysqld]

# The next three options are mutually exclusive to SERVER_PORT below.
# skip-networking=
# enable-named-pipe=
# shared-memory=

# shared-memory-base-name=MYSQL

# The Pipe the MySQL Server will use
# socket=MYSQL

# The TCP/IP Port the MySQL Server will listen on
port=3306

# Path to installation directory. All paths are usually resolved relative to this.
# basedir="C:/Program Files/MySQL/MySQL Server 8.0/"

# Path to the database root
datadir=C:/ProgramData/MySQL/MySQL Server 8.0/Data

# The default character set that will be used when a new schema or table is
# created and no character set is defined
# character-set-server=

# The default authentication plugin to be used when connecting to the server
default_authentication_plugin=mysql_native_password

# The default storage engine that will be used when create new tables when
default-storage-engine=INNODB

# Set the SQL mode to strict
sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION"

# General and Slow logging.
log-output=FILE

general-log=0

general_log_file="VMI384596.log"

slow-query-log=1

slow_query_log_file="VMI384596-slow.log"

long_query_time=10

# Error Logging.
log-error="VMI384596.err"

# ***** Group Replication Related *****
# Specifies the base name to use for binary log files. With binary logging
# enabled, the server logs all statements that change data to the binary
# log, which is used for backup and replication.
log-bin="VMI384596-bin"

# ***** Group Replication Related *****
# Sets the binary logging format, and can be any one of STATEMENT, ROW,
# or MIXED. ROW is suggested for Group Replication.
# binlog_format=

# ***** Group Replication Related *****
# Causes the master to write a checksum for each event in the binary log.
# binlog_checksum supports the values NONE (disabled) and CRC32.
# The default is CRC32. When disabled (value NONE), the server verifies
# that it is writing only complete events to the binary log by writing
# and checking the event length (rather than a checksum) for each event.
# NONE must be used with Group Replication.
# binlog_checksum=

# ***** Group Replication Related *****
# The base name for the relay log. The server creates relay log files in
# sequence by adding a numeric suffix to the base name. If you specify this
# option, the value specified is also used as the base name for the relay log
# index file. Relay logs increase speed by using load-balancing between disks.
# relay_log=

# ***** Group Replication Related *****
# Specifies the server ID. For servers that are used in a replication topology,
# you must specify a unique server ID for each replication server, in the
# range from 1 to 2^32 - 1. �Unique� means that each ID must be different
# from every other ID in use by any other replication master or slave.
server-id=1

# ***** Group Replication Related *****
# The host name or IP address of the slave to be reported to the master
# during slave registration. This value appears in the output of SHOW SLAVE HOSTS
# on the master server. Leave the value unset if you do not want the slave to
# register itself with the master.
# report_host=0.0

# ***** Group Replication Related *****
# The TCP/IP port number for connecting to the slave, to be reported to the master during
# slave registration. Set this only if the slave is listening on a nondefault port or if
# you have a special tunnel from the master or other clients to the slave.
report_port=3306

# ***** Group Replication Related *****
# This option specifies whether global transaction identifiers (GTIDs) are
# used to identify transactions. ON must be used with Group Replication.
# gtid_mode=

# ***** Group Replication Related *****
# When enabled, the server enforces GTID consistency by allowing execution of
# only statements that can be safely logged using a GTID. You must set this
# option to ON before enabling GTID based replication.
# enforce_gtid_consistency=

# ***** Group Replication Related *****
# Whether updates received by a slave server from a master server should be
# logged to the slave's own binary log. Binary logging must be enabled on
# the slave for this variable to have any effect. ON must be used with
# Group Replication.
# log_slave_updates=

# ***** Group Replication Related *****
# Determines whether the slave server logs master status and connection information
# to an InnoDB table in the mysql database, or to a file in the data directory.
# The TABLE setting is required when multiple replication channels are configured.
# master_info_repository=

# ***** Group Replication Related *****
# Determines whether the slave server logs its position in the relay logs to an InnoDB
# table in the mysql database, or to a file in the data directory. The TABLE setting is
# required when multiple replication channels are configured.
# relay_log_info_repository=

# ***** Group Replication Related *****
# Defines the algorithm used to hash the writes extracted during a transaction. If you
# are using Group Replication, this variable must be set to XXHASH64 because the process
# of extracting the writes from a transaction is required for conflict detection on all
# group members.
# transaction_write_set_extraction=

# NOTE: Modify this value after Server initialization won't take effect.
lower_case_table_names=1

# Secure File Priv.
secure-file-priv="C:/ProgramData/MySQL/MySQL Server 8.0/Uploads"

# The maximum amount of concurrent sessions the MySQL server will
# allow. One of these connections will be reserved for a user with
# SUPER privileges to allow the administrator to login even if the
# connection limit has been reached.
max_connections = 2000

# The number of open tables for all threads. Increasing this value
# increases the number of file descriptors that mysqld requires.
# Therefore you have to make sure to set the amount of open files
# allowed to at least 4096 in the variable "open-files-limit" in
# section [mysqld_safe]
table_open_cache=2000

# Maximum size for internal (in-memory) temporary tables. If a table
# grows larger than this value, it is automatically converted to disk
# based table This limitation is for a single table. There can be many
# of them.
tmp_table_size = 4G

# How many threads we should keep in a cache for reuse. When a client
# disconnects, the client's threads are put in the cache if there aren't
# more than thread_cache_size threads from before.  This greatly reduces
# the amount of thread creations needed if you have a lot of new
# connections. (Normally this doesn't give a notable performance
# improvement if you have a good thread implementation.)
thread_cache_size=10

# *** MyISAM Specific options
# The maximum size of the temporary file MySQL is allowed to use while
# recreating the index (during REPAIR, ALTER TABLE or LOAD DATA INFILE.
# If the file-size would be bigger than this, the index will be created
# through the key cache (which is slower).
myisam_max_sort_file_size=10G

# The size of the buffer that is allocated when sorting MyISAM indexes
# during a REPAIR TABLE or when creating indexes with CREATE INDEX
# or ALTER TABLE.
myisam_sort_buffer_size=256K

# Size of the Key Buffer, used to cache index blocks for MyISAM tables.
# Do not set it larger than 30% of your available memory, as some memory
# is also required by the OS to cache rows. Even if you're not using
# MyISAM tables, you should still set it to 8-64M as it will also be
# used for internal temporary disk tables.
key_buffer_size = 64M

# Size of the buffer used for doing full table scans of MyISAM tables.
# Allocated per thread, if a full scan is needed.
read_buffer_size=64K

read_rnd_buffer_size=256K

# *** INNODB Specific options ***
# innodb_data_home_dir=

# Use this option if you have a MySQL server with InnoDB support enabled
# but you do not plan to use it. This will save memory and disk space
# and speed up some things.
# skip-innodb=

# If set to 1, InnoDB will flush (fsync) the transaction logs to the
# disk at each commit, which offers full ACID behavior. If you are
# willing to compromise this safety, and you are running small
# transactions, you may set this to 0 or 2 to reduce disk I/O to the
# logs. Value 0 means that the log is only written to the log file and
# the log file flushed to disk approximately once per second. Value 2
# means the log is written to the log file at each commit, but the log
# file is only flushed to disk approximately once per second.
innodb_flush_log_at_trx_commit = 1

# The size of the buffer InnoDB uses for buffering log data. As soon as
# it is full, InnoDB will have to flush it to disk. As it is flushed
# once per second anyway, it does not make sense to have it very large
# (even with long transactions).


# InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and
# row data. The bigger you set this the less disk I/O is needed to
# access data in tables. On a dedicated database server you may set this
# parameter up to 80% of the machine physical memory size. Do not set it
# too large, though, because competition of the physical memory may
# cause paging in the operating system.  Note that on 32bit systems you
# might be limited to 2-3.5G of user level memory per process, so do not
# set it too high.
innodb_buffer_pool_size=38G

# Size of each log file in a log group. You should set the combined size
# of log files to about 25%-100% of your buffer pool size to avoid
# unneeded buffer pool flush activity on log file overwrite. However,
# note that a larger logfile size will increase the time needed for the
# recovery process.
innodb_log_file_size=48M

# Number of threads allowed inside the InnoDB kernel. The optimal value
# depends highly on the application, hardware as well as the OS
# scheduler properties. A too high value may lead to thread thrashing.
innodb_thread_concurrency=21

# The increment size (in MB) for extending the size of an auto-extend InnoDB system tablespace file when it becomes full.
innodb_autoextend_increment=64

# The number of regions that the InnoDB buffer pool is divided into.
# For systems with buffer pools in the multi-gigabyte range, dividing the buffer pool into separate instances can improve concurrency,
# by reducing contention as different threads read and write to cached pages.
innodb_buffer_pool_instances=8

# Determines the number of threads that can enter InnoDB concurrently.
innodb_concurrency_tickets=5000

# Specifies how long in milliseconds (ms) a block inserted into the old sublist must stay there after its first access before
# it can be moved to the new sublist.
innodb_old_blocks_time=1000

# It specifies the maximum number of .ibd files that MySQL can keep open at one time. The minimum value is 10.
innodb_open_files=300

# When this variable is enabled, InnoDB updates statistics during metadata statements.
innodb_stats_on_metadata=0

# When innodb_file_per_table is enabled (the default in 5.6.6 and higher), InnoDB stores the data and indexes for each newly created table
# in a separate .ibd file, rather than in the system tablespace.
innodb_file_per_table=1

# Use the following list of values: 0 for crc32, 1 for strict_crc32, 2 for innodb, 3 for strict_innodb, 4 for none, 5 for strict_none.
innodb_checksum_algorithm = none

skip-innodb-doublewrite=

# The number of outstanding connection requests MySQL can have.
# This option is useful when the main MySQL thread gets many connection requests in a very short time.
# It then takes some time (although very little) for the main thread to check the connection and start a new thread.
# The back_log value indicates how many requests can be stacked during this short time before MySQL momentarily
# stops answering new requests.
# You need to increase this only if you expect a large number of connections in a short period of time.
back_log=80

# If this is set to a nonzero value, all tables are closed every flush_time seconds to free up resources and
# synchronize unflushed data to disk.
# This option is best used only on systems with minimal resources.
flush_time=0

# The minimum size of the buffer that is used for plain index scans, range index scans, and joins that do not use
# indexes and thus perform full table scans.
join_buffer_size=256K

# The maximum size of one packet or any generated or intermediate string, or any parameter sent by the
# mysql_stmt_send_long_data() C API function.
max_allowed_packet=4M

# If more than this many successive connection requests from a host are interrupted without a successful connection,
# the server blocks that host from performing further connections.
max_connect_errors=100

# Changes the number of file descriptors available to mysqld.
# You should try increasing the value of this option if mysqld gives you the error "Too many open files".
open_files_limit=4161

# If you see many sort_merge_passes per second in SHOW GLOBAL STATUS output, you can consider increasing the
# sort_buffer_size value to speed up ORDER BY or GROUP BY operations that cannot be improved with query optimization
# or improved indexing.
sort_buffer_size = 256K

# The number of table definitions (from .frm files) that can be stored in the definition cache.
# If you use a large number of tables, you can create a large table definition cache to speed up opening of tables.
# The table definition cache takes less space and does not use file descriptors, unlike the normal table cache.
# The minimum and default values are both 400.
table_definition_cache=1400

# Specify the maximum size of a row-based binary log event, in bytes.
# Rows are grouped into events smaller than this size if possible. The value should be a multiple of 256.
binlog_row_event_max_size=8K

# If the value of this variable is greater than 0, a replication slave synchronizes its master.info file to disk.
# (using fdatasync()) after every sync_master_info events.
sync_master_info=10000

# If the value of this variable is greater than 0, the MySQL server synchronizes its relay log to disk.
# (using fdatasync()) after every sync_relay_log writes to the relay log.
sync_relay_log=10000

# If the value of this variable is greater than 0, a replication slave synchronizes its relay-log.info file to disk.
# (using fdatasync()) after every sync_relay_log_info transactions.
sync_relay_log_info=10000

# Load mysql plugins at start."plugin_x ; plugin_y".
# plugin_load=

# The TCP/IP Port the MySQL Server X Protocol will listen on.
# loose_mysqlx_port=33060

# Size of the Key Buffer, used to cache index blocks for MyISAM tables.
# Do not set it larger than 30% of your available memory, as some memory
# is also required by the OS to cache rows. Even if you're not using
# MyISAM tables, you should still set it to 8-64M as it will also be
# used for internal temporary disk tables.

我应该如何通过限制 CPU 使用率来提高 MySQL 性能并防止服务器崩溃?

答案1

变量和状态中没有明显的解释。

全球状况和变量的分析:

观察结果:

  • 版本:8.0.20
  • 60 GB 内存
  • 正常运行时间 = 04:49:11;一些 GLOBAL STATUS 值可能尚无意义。
  • 您正在 Windows 上运行。
  • 4.89 查询/秒:3.43 问题/秒

更重要的问题:

几乎没有发生任何事情。我很难想象 MySQL 导致了崩溃。

降低max_connections到 500。(自启动以来,并发连接数不超过 23 个。)

tmp_table_size = 500M——对于您拥有的 RAM 数量来说,这个值目前太高了,很危险。

innodb_doublewrite = ON

详细信息和其他观察结果:

( innodb_lru_scan_depth * innodb_page_cleaners ) = 1,024 * 4 = 4,096-- 页面清理器每秒的工作量。-- “InnoDB:page_cleaner:预期循环花费 1000 毫秒...”可通过降低 lru_scan_depth 来解决:考虑 1000/innodb_page_cleaners(现在为 4)。还请检查交换。

( innodb_lru_scan_depth ) = 1,024 -- “InnoDB:page_cleaner:预期循环花费 1000ms...” 可以通过降低 lru_scan_depth 来修复

( Innodb_buffer_pool_pages_free * 16384 / innodb_buffer_pool_size ) = 2,478,311 * 16384 / 38912M = 99.5%-- 缓冲池可用 -- buffer_pool_size 大于工作集;可以减小它

( innodb_io_capacity ) = 200-- 刷新时,使用这么多的 IOP。-- 读取可能会很缓慢或不稳定。

( Innodb_buffer_pool_pages_free / Innodb_buffer_pool_pages_total ) = 2,478,311 / 2490368 = 99.5%-- 缓冲池的百分比当前未使用 -- innodb_buffer_pool_size(现在为 40802189312)是否大于必要的?

( innodb_io_capacity_max / innodb_io_capacity ) = 2,000 / 200 = 10-- 容量:最大/普通 -- 推荐 2。最大容量应大约等于 I/O 子系统可以处理的 IOP。(如果驱动器类型未知,2000/200 可能是合理的一对。)

( Innodb_buffer_pool_bytes_data / innodb_buffer_pool_size ) = 196,214,784 / 38912M = 0.48%-- 数据占用缓冲池的百分比 -- 很小的百分比可能表明 buffer_pool 不必要地大。

( innodb_doublewrite ) = innodb_doublewrite = OFF-- 额外的 I/O,但在崩溃时有额外的安全性。-- 对于 FusionIO、Galera、Replicas 和 ZFS 来说,关闭是可以的。

( Innodb_os_log_written / (Uptime / 3600) / innodb_log_files_in_group / innodb_log_file_size ) = 3,944,448 / (17351 / 3600) / 2 / 48M = 0.00813-- 比率 -- (见会议纪要)

( Uptime / 60 * innodb_log_file_size / Innodb_os_log_written ) = 17,351 / 60 * 48M / 3944448 = 3,690-- InnoDB 日志轮换之间的分钟数从 5.6.8 开始,可以动态更改;请确保也更改 my.cnf。--(轮换之间建议 60 分钟有点武断。)调整 innodb_log_file_size(现在为 50331648)。(无法在 AWS 中更改。)

( innodb_flush_method ) = innodb_flush_method = unbuffered-- InnoDB 应如何要求操作系统写入块。建议使用 O_DIRECT 或 O_ALL_DIRECT (Percona) 来避免双重缓冲。(至少对于 Unix 而言。)有关 O_ALL_DIRECT 的注意事项,请参阅 chrischandler

( innodb_io_capacity ) = 200-- 磁盘每秒的 I/O 操作数。慢速驱动器为 100;旋转驱动器为 200;SSD 为 1000-2000;乘以 RAID 因子。

( innodb_adaptive_hash_index ) = innodb_adaptive_hash_index = ON-- 通常应为 ON。-- 在某些情况下,OFF 更好。另请参阅 innodb_adaptive_hash_index_parts(现在是 8)(5.7.9 之后)和 innodb_adaptive_hash_index_partitions(MariaDB 和 Percona)。ON 已与罕见的崩溃有关(错误 73890)。10.5.0 决定默认为 OFF。

( innodb_print_all_deadlocks ) = innodb_print_all_deadlocks = OFF-- 是否记录所有死锁。-- 如果您受到死锁的困扰,请启用此功能。注意:如果您有大量死锁,这可能会将大量数据写入磁盘。

( max_connections ) = 2,000-- 最大连接数(线程)。影响各种分配。-- 如果 max_connections(现在为 2000)太高,并且各种内存设置都很高,则可能会耗尽 RAM。

( bulk_insert_buffer_size ) = 8 / 61440M = 0.01%-- 多行 INSERT 和 LOAD DATA 的缓冲区 -- 太大可能会影响 RAM 大小。太小可能会妨碍此类操作。

( tmp_table_size ) = 4096M-- 大小限制记忆用于支持 SELECT 的临时表 - 减少 tmp_table_size(现在为 4294967296)以避免 RAM 耗尽。可能不超过 64M。

( Select_full_join / Com_select ) = 15,198 / 31082 = 48.9%-- 无索引连接的选择百分比 -- 为 JOIN 中使用的表添加合适的索引。

( Com_admin_commands / Queries ) = 25,348 / 84808 = 29.9%-- “管理员”命令的查询百分比。-- 发生了什么?

( long_query_time ) = 10-- 定义“慢速”查询的截止时间(秒)。-- 建议 2

( log_slow_slave_statements ) = log_slow_slave_statements = OFF-- (5.6.11, 5.7.1) 默认情况下,复制的语句不会显示在慢速日志中;这会导致它们显示出来。-- 在慢速日志中查看可能干扰副本读取的写入会很有帮助。

( back_log ) = 80--(从 5.6.6 开始自动调整大小;基于 max_connections)-- 在进行大量连接时,提升到 min(150,max_connections(现在是 2000)) 可能会有所帮助。

( Max_used_connections / max_connections ) = 23 / 2000 = 1.1%-- 连接的峰值百分比 -- 由于几个内存因素可以基于 max_connections(现在为 2000)扩大,所以最好不要将该设置得太高。

( Com_change_db / Connections ) = 25,413 / 311 = 81.7-- 每个连接的数据库切换 -- (次要)考虑使用“db.table”语法

( Aborted_connects / Connections ) = 227 / 311 = 73.0%——也许有黑客正在试图闯入?(尝试连接)

异常小:

10 * read_buffer_size = 0.6MB
Com_insert = 4.4 /HR
Handler_read_next = 16 /sec
Innodb_buffer_pool_reads * innodb_page_size / innodb_buffer_pool_size = 0.47%
Innodb_dblwr_pages_written = 0
Innodb_rows_updated = 0.62 /HR
back_log / max_connections = 4.0%
innodb_doublewrite_files = 0
innodb_doublewrite_pages = 0

异常大:

Com_create_db = 0.21 /HR
Com_create_table = 92 /HR
Com_show_charsets = 1.7 /HR
Com_show_plugins = 0.41 /HR
Com_show_storage_engines = 0.41 /HR
Innodb_buffer_pool_pages_free = 2.48e+6
Innodb_system_rows_deleted = 0.1 /sec
Innodb_system_rows_inserted = 0.1 /sec
Innodb_system_rows_updated = 0.32 /sec
Ssl_accepts = 304
Ssl_default_timeout = 7,200
Ssl_finished_accepts = 304
Ssl_session_cache_hits = 290
Ssl_session_cache_timeouts = 5
Ssl_verify_depth = 4.29e+9
Ssl_verify_mode = 5
gtid_executed_compression_period = 0.058 /sec
innodb_thread_concurrency = 21
max_error_count = 1,024
max_length_for_sort_data = 4,096
optimizer_trace_offset = --1
performance_schema_max_cond_classes = 100
performance_schema_max_mutex_classes = 300
performance_schema_max_rwlock_classes = 60
performance_schema_max_stage_classes = 175
performance_schema_max_statement_classes = 218
performance_schema_max_thread_classes = 100

异常字符串:

event_scheduler = ON
ft_boolean_syntax = + -><()~*:\"\"&
have_query_cache = NO
innodb_fast_shutdown = 1
innodb_temp_tablespaces_dir = .\\#innodb_temp\\
lower_case_file_system = ON
lower_case_table_names = 1
mysqlx_compression_algorithms = DEFLATE_STREAM,LZ4_MESSAGE,ZSTD_STREAM
optimizer_trace = enabled=off,one_line=off
optimizer_trace_features = greedy_search=on, range_optimizer=on, dynamic_range=on, repeated_subselect=on
protocol_compression_algorithms = zlib,zstd,uncompressed
slave_rows_search_algorithms = INDEX_SCAN,HASH_SCAN

相关内容