我正在跟踪生产服务器上 MySQL 性能缓慢的问题,并遇到了一个奇怪的问题 - 相同的 MySQL 查询在CentOS 7.4.1708 3.14.32-xxxx-grs-ipv6-64
具有 125GB 内存的生产服务器上执行时间更长,而在同一数据库转储上的本地实例上运行的相同查询执行速度要快得多,具体来说,在本地环境(osX box 16 GB 内存)上完成大约需要 10-20 秒,在生产机器上完成大约需要 3-5 分钟。两台机器都使用默认的 MariaDB 配置,并且查询是测试时唯一正在运行的查询。
Mysql数据目录挂载在/home/mysql_data上这可能是原因吗?
使用了相同的 MySQL 转储,查询是从控制台运行的,MySQL dbname < test.sql
因此瓶颈显然是数据库,mariadb 是从 macos 上的端口和 linux 上的 repo 安装的(我也尝试过自定义编译版本),没有区别。
我开始对硬件产生怀疑,任何提示或建议都将不胜感激,附上pt-summary
输出
# Percona Toolkit System Summary Report ######################
Date | 2019-12-09 09:35:10 UTC (local TZ: EST -0500)
Hostname | ...
Uptime | 727 days, 23:51, 3 users, load average: 1.29, 1.48, 1.75
System | Supermicro; X10DRH; v123456789 (Rack Mount Chassis)
Service Tag | 123456789
Platform | Linux
Release | CentOS Linux release 7.4.1708 (Core)
Kernel | 3.14.32-xxxx-grs-ipv6-64
Architecture | CPU = 64-bit, OS = 64-bit
Threading | NPTL 2.17
Compiler | GNU CC version 4.8.5 20150623 (Red Hat 4.8.5-16).
SELinux | Disabled
Virtualized | VMWare
# Processor ##################################################
Processors | physical = 2, cores = 16, virtual = 32, hyperthreading = yes
Speeds | 1x2596.593, 1x2597.156, 1x2600.437, 1x2601.093, 1x2605.312, 1x2607.093, 1x2610.375, 1x2613.937, 1x2619.562, 1x2655.750, 1x2752.312, 1x2759.343, 1x2769.937, 1x2780.906, 1x2835.000, 1x2841.281, 1x2851.593, 1x2855.437, 1x2880.000, 1x2882.812, 1x2896.968, 1x2913.187, 1x2923.500, 1x2945.625, 1x2951.062, 1x2974.312, 1x3004.500, 1x3029.343, 1x3062.625, 1x3076.218, 1x3166.593, 1x3191.250
Models | 32xIntel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
Caches | 32x20480 KB
# Memory #####################################################
Total | 125.9G
Free | 2.3G
Used | physical = 14.5G, swap allocated = 0.0, swap used = 0.0, virtual = 14.5G
Shared | 4.1G
Buffers | 109.1G
Caches | 106.7G
Dirty | 636 kB
UsedRSS | 15.4G
Swappiness | 1
DirtyPolicy | 30, 10
DirtyStatus | 0, 0
Locator Size Speed Form Factor Type Type Detail
========= ======== ================= ============= ============= ===========
P1_DIMMA1 16384 MB 2133 MHz DIMM DDR4 Synchronous
P1_DIMMB1 16384 MB 2133 MHz DIMM DDR4 Synchronous
P1_DIMMC1 16384 MB 2133 MHz DIMM DDR4 Synchronous
P1_DIMMD1 16384 MB 2133 MHz DIMM DDR4 Synchronous
P2_DIMME1 16384 MB 2133 MHz DIMM DDR4 Synchronous
P2_DIMMF1 16384 MB 2133 MHz DIMM DDR4 Synchronous
P2_DIMMG1 16384 MB 2133 MHz DIMM DDR4 Synchronous
P2_DIMMH1 16384 MB 2133 MHz DIMM DDR4 Synchronous
P1_DIMMA2 {EMPTY} Unknown DIMM DDR4 Synchronous
P1_DIMMB2 {EMPTY} Unknown DIMM DDR4 Synchronous
P1_DIMMC2 {EMPTY} Unknown DIMM DDR4 Synchronous
P1_DIMMD2 {EMPTY} Unknown DIMM DDR4 Synchronous
P2_DIMME2 {EMPTY} Unknown DIMM DDR4 Synchronous
P2_DIMMF2 {EMPTY} Unknown DIMM DDR4 Synchronous
P2_DIMMG2 {EMPTY} Unknown DIMM DDR4 Synchronous
P2_DIMMH2 {EMPTY} Unknown DIMM DDR4 Synchronous
# Mounted Filesystems ########################################
Filesystem Size Used Type Opts Mountpoint
/dev/md2 421G 62% ext4 rw,relatime,quota,usrquota,grpquota,data=ordered /R
/dev/md2 421G 62% ext4 rw,relatime,quota,usrquota,grpquota,data=ordered /R
devtmpfs 63G 0% devtmpfs rw,relatime,size=65953940k,nr_inodes=16488485,mode=755 /dev
tmpfs 13G 0% tmpfs rw /run/user/0
tmpfs 13G 0% tmpfs rw,nosuid,nodev,mode=755 /run/user/0
tmpfs 13G 0% tmpfs rw,nosuid,nodev,relatime,size=13197492k,mode=700 /run/user/0
tmpfs 13G 0% tmpfs ro,nosuid,nodev,noexec,mode=755 /run/user/0
tmpfs 63G 0% tmpfs rw /dev/shm
tmpfs 63G 0% tmpfs rw,nosuid,nodev,mode=755 /dev/shm
tmpfs 63G 0% tmpfs rw,nosuid,nodev,relatime,size=13197492k,mode=700 /dev/shm
tmpfs 63G 0% tmpfs ro,nosuid,nodev,noexec,mode=755 /dev/shm
tmpfs 63G 0% tmpfs rw /sys/fs/cgroup
tmpfs 63G 0% tmpfs rw,nosuid,nodev,mode=755 /sys/fs/cgroup
tmpfs 63G 0% tmpfs rw,nosuid,nodev,relatime,size=13197492k,mode=700 /sys/fs/cgroup
tmpfs 63G 0% tmpfs ro,nosuid,nodev,noexec,mode=755 /sys/fs/cgroup
tmpfs 63G 7% tmpfs rw /run
tmpfs 63G 7% tmpfs rw,nosuid,nodev,mode=755 /run
tmpfs 63G 7% tmpfs rw,nosuid,nodev,relatime,size=13197492k,mode=700 /run
tmpfs 63G 7% tmpfs ro,nosuid,nodev,noexec,mode=755 /run
# Disk Schedulers And Queue Size #############################
dm-0 | 128
md0 | 128
md1 | 128
md2 | 128
nbd0 | [deadline] 128
nbd1 | [deadline] 128
nbd10 | [deadline] 128
nbd11 | [deadline] 128
nbd12 | [deadline] 128
nbd13 | [deadline] 128
nbd14 | [deadline] 128
nbd15 | [deadline] 128
nbd2 | [deadline] 128
nbd3 | [deadline] 128
nbd4 | [deadline] 128
nbd5 | [deadline] 128
nbd6 | [deadline] 128
nbd7 | [deadline] 128
nbd8 | [deadline] 128
nbd9 | [deadline] 128
sda | [deadline] 128
sdb | [deadline] 128
# Disk Partioning ############################################
Device Type Start End Size
============ ==== ========== ========== ==================
/dev/dm-0 Disk 107374182400
/dev/md1 Disk 20970405888
/dev/md2 Disk 458591502336
/dev/sda Disk 480103981056
/dev/sda1 Part 4096 40962047 20970470912
/dev/sda2 Part 40962048 936648703 458591567360
/dev/sda3 Part 936648704 937695231 535821824
/dev/sdb Disk 480103981056
/dev/sdb1 Part 4096 40962047 20970470912
/dev/sdb2 Part 40962048 936648703 458591567360
/dev/sdb3 Part 936648704 937695231 535821824
# Kernel Inode State #########################################
dentry-state | 19102026 19047745 45 0 0 0
file-nr | 3328 0 512000
inode-nr | 1262883 278591
# LVM Volumes ################################################
Unable to collect information
# LVM Volume Groups ##########################################
Unable to collect information
# RAID Controller ############################################
Controller | Fusion-MPT SAS
# Network Config #############################################
Controller | Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
Controller | Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
FIN Timeout | 60
Port Range | 61000
# Interface Statistics #######################################
interface rx_bytes rx_packets rx_errors tx_bytes tx_packets tx_errors
========= ========= ========== ========== ========== ========== ==========
lo 12500000000000 8000000000 0 12500000000000 8000000000 0
bond0 0 0 0 0 0 0
dummy0 0 0 0 0 0 0
ifb0 0 0 0 0 0 0
ifb1 0 0 0 0 0 0
eth0 1000000000000 7000000000 0 5000000000000 4500000000 0
eth1 0 0 0 0 0 0
teql0 0 0 0 0 0 0
tunl0@NONE 0 0 0 0 0 0
sit0@NONE 0 0 0 0 0 0
ip6tnl0@NONE 0 0 0 0 0 0
docker0 175000000 450000 0 250000000 450000 0
# Network Devices ############################################
Device Speed Duplex
========= ========= =========
bond0 Unknown! Unknown!
eth0 1000Mb/s Full
eth1 Unknown! Unknown!
docker0
# Network Connections ########################################
Connections from remote IP addresses
35.158.140.87 2
45.94.25.205 1
46.38.144.17 4
46.38.144.32 2
46.38.144.57 3
46.38.144.146 2
46.38.144.179 3
46.38.144.202 1
46.229.168.150 1
54.36.148.89 1
54.36.148.155 1
54.36.148.199 1
54.36.149.27 1
54.36.150.48 1
54.36.150.58 1
54.36.150.85 1
54.36.150.138 1
63.135.176.20 2
64.250.124.70 2
66.249.64.104 1
66.249.64.106 1
66.249.64.234 1
66.249.64.236 1
74.121.199.234 2
92.118.38.38 4
121.66.252.155 1
127.0.0.1 80
136.145.228.221 2
160.10.5.26 2
162.247.242.5 3
178.62.77.163 6
188.163.83.193 3
203.125.15.242 1
Connections to local IP addresses
54.39.202.16 35
127.0.0.1 80
158.69.185.152 1
167.114.208.91 25
Connections to top 10 local ports
22 4
25 4
443 20
51554 1
52622 1
587 15
59914 1
80 2
8891 2
9000 60
States of connections
CLOSE_WAIT 1
ESTABLISHED 25
LISTEN 35
TIME_WAIT 125
# Top Processes ##############################################
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12492 mysql 20 0 61.652g 0.011t 15876 S 93.8 8.6 312:39.96 mysqld
30259 root 20 0 1023576 28600 4676 S 12.5 0.0 38262:50 newrelic-i+
1 root 20 0 191620 3876 1664 S 0.0 0.0 1429:05 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:32.51 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 107:31.39 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+
8 root 20 0 0 0 0 S 0.0 0.0 1407:32 rcu_sched
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
10 root rt 0 0 0 0 S 0.0 0.0 0:37.04 migration/0
# Notable Processes ##########################################
PID OOM COMMAND
29337 -17 sshd
26454 -17 systemd-udevd
27610 -17 postgres
# Simplified and fuzzy rounded vmstat (wait please) ##########
procs ---swap-- -----io---- ---system---- --------cpu--------
r b si so bi bo ir cs us sy il wa st
2 0 0 0 2 80 0 0 5 0 95 0 0
2 0 0 0 0 900 2500 4000 5 1 94 0 0
3 0 0 0 0 1000 2250 4500 10 0 89 0 0
1 0 0 0 0 600 1500 2000 6 0 94 0 0
7 0 0 0 0 250 2000 2000 9 1 90 0 0
# Memory mamagement ##########################################
Unable to get Transparent huge pages status.
# The End ####################################################
sysctl 内存设置:
vm.admin_reserve_kbytes = 8192
vm.block_dump = 0
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 30
vm.dirty_writeback_centisecs = 500
vm.drop_caches = 0
vm.heap_stack_gap = 65536
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 256 256 32
vm.max_map_count = 65530
vm.memory_failure_early_kill = 0
vm.memory_failure_recovery = 1
vm.min_free_kbytes = 45951
vm.min_slab_ratio = 5
vm.min_unmapped_ratio = 1
vm.mmap_min_addr = 4096
vm.nr_pdflush_threads = 0
vm.numa_zonelist_order = default
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.scan_unevictable_pages = 0
vm.stat_interval = 1
vm.swappiness = 1
vm.user_reserve_kbytes = 131072
vm.vfs_cache_pressure = 100
vm.zone_reclaim_mode = 0