使用 iostat、vmstat、mpstat 识别瓶颈

Question

我正在做一些 Apache 和 MySQL 性能测试，然后升级单个组件并再次运行测试以查看瓶颈所在。我有一个本地二手服务器仓库（位于犹他州林顿的 Tams），明天将借给我大约 20 个处理器进行这些测试，以比较更多核心与更高时钟速度。然后我将使用 NVMe RAID 与 SSD RAID、HDD RAID 与单个 HDD 进行测试。

我编写了一个脚本，它遍历并整合了来自 iostat、vmstat、mpstat 等其他各种程序的统计数据。当我查看结果时，我不明白为什么 512 个线程的速度会如此之慢。使用这些工具查看 CPU、HDD、内存上的负载时，似乎与 256 个线程之间没有太大区别。

对于这个，我使用 Apache Benchmark 的命令：ab -kc 500 -n 1000 -l -s 60 http://localhost/stress.php所有测量都在测试开始 5 秒后进行，以便测试能够全面展开。

Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, Cores: 6, Sockets 2, DDR4, (free-mh): 62G 12G 18G 4.5M 32G 49G, NVMe RAID, TestType: Apache PHP, Threads: 16, Server Load: 0.94, 0.28, 0.15, Threads: 16,   Time per request: 125.523 [ms] (mean), Concurrency Level: 16, Requests per second 127.47 [#/sec] (mean)Time per request:       125.523 [ms] (mean),  Connect: 0    0   0.1      0       1, Processing: 59  124 177.8    121    2924, CPU Core Load: %usr,1.89,1.94,1.98,1.95,1.93,1.92,1.93,1.94,1.93,1.94,1.93,1.95,1.92,1.87,1.88,1.85,1.83,1.85,1.82,1.84,1.82,1.85,1.84,1.85,1.82,, CPU Idle%: %idle,97.66,97.59,97.60,97.60,97.64,97.63,97.64,97.60,97.64,97.59,97.64,97.60,97.65,97.66,97.70,97.69,97.74,97.69,97.75,97.70,97.75,97.70,97.73,97.62,97.75,,Waiting on Disk: %iowait,0.04,0.05,0.04,0.04,0.04,0.04,0.04,0.04,0.03,0.04,0.03,0.04,0.03,0.05,0.04,0.04,0.04,0.04,0.04,0.04,0.03,0.04,0.04,0.11,0.04,, vmstatProcs: 17, vmstatSwapIn: 0, vmstatSwapOut: 0, vmstatIoIn: 7, vmstatIoOut: 42, iostatR-S: 1.72, iostatW-S: 13.37, iostatR-MBS: 0.25, iostatW-MBS: 0.52, iostatHDD-Util: 0.00
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, Cores: 6, Sockets 2, DDR4, (free-mh): 62G 12G 17G 4.5M 32G 49G, NVMe RAID, TestType: Apache PHP, Threads: 32, Server Load: 3.98, 0.97, 0.37, Threads: 32,   Time per request: 194.680 [ms] (mean), Concurrency Level: 32, Requests per second 164.37 [#/sec] (mean)Time per request:       194.680 [ms] (mean),  Connect: 0    0   0.5      0       4, Processing: 65  193 380.9    143    4300, CPU Core Load: %usr,1.89,1.94,1.99,1.95,1.93,1.92,1.94,1.95,1.94,1.94,1.93,1.95,1.93,1.88,1.88,1.85,1.84,1.85,1.82,1.85,1.82,1.85,1.84,1.86,1.82,, CPU Idle%: %idle,97.66,97.58,97.60,97.60,97.64,97.63,97.63,97.60,97.64,97.59,97.64,97.60,97.65,97.66,97.70,97.69,97.74,97.68,97.75,97.70,97.75,97.70,97.72,97.62,97.74,,Waiting on Disk: %iowait,0.04,0.05,0.04,0.04,0.04,0.04,0.04,0.04,0.03,0.04,0.03,0.04,0.03,0.05,0.04,0.04,0.04,0.04,0.04,0.04,0.03,0.04,0.04,0.11,0.04,, vmstatProcs: 32, vmstatSwapIn: 0, vmstatSwapOut: 0, vmstatIoIn: 7, vmstatIoOut: 42, iostatR-S: 1.72, iostatW-S: 13.37, iostatR-MBS: 0.25, iostatW-MBS: 0.52, iostatHDD-Util: 0.00
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, Cores: 6, Sockets 2, DDR4, (free-mh): 62G 12G 17G 4.5M 32G 49G, NVMe RAID, TestType: Apache PHP, Threads: 64, Server Load: 7.13, 1.72, 0.62, Threads: 64,   Time per request: 384.736 [ms] (mean), Concurrency Level: 64, Requests per second 166.35 [#/sec] (mean)Time per request:       384.736 [ms] (mean),  Connect: 0    1   3.0      0      14, Processing: 130  378 547.1    311    5501, CPU Core Load: %usr,1.90,1.95,1.99,1.95,1.93,1.93,1.94,1.95,1.94,1.95,1.94,1.95,1.93,1.88,1.89,1.86,1.84,1.86,1.83,1.85,1.83,1.86,1.84,1.86,1.83,, CPU Idle%: %idle,97.66,97.58,97.59,97.59,97.63,97.62,97.63,97.60,97.63,97.58,97.64,97.59,97.64,97.65,97.69,97.69,97.73,97.68,97.75,97.69,97.75,97.69,97.72,97.61,97.74,,Waiting on Disk: %iowait,0.04,0.05,0.04,0.04,0.04,0.04,0.04,0.04,0.03,0.04,0.03,0.04,0.03,0.05,0.04,0.04,0.04,0.04,0.04,0.04,0.03,0.04,0.04,0.11,0.04,, vmstatProcs: 59, vmstatSwapIn: 0, vmstatSwapOut: 0, vmstatIoIn: 7, vmstatIoOut: 42, iostatR-S: 1.72, iostatW-S: 13.37, iostatR-MBS: 0.25, iostatW-MBS: 0.52, iostatHDD-Util: 0.00
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, Cores: 6, Sockets 2, DDR4, (free-mh): 62G 12G 17G 4.5M 32G 49G, NVMe RAID, TestType: Apache PHP, Threads: 128, Server Load: 11.64, 2.82, 0.99, Threads: 128,   Time per request: 786.459 [ms] (mean), Concurrency Level: 128, Requests per second 162.75 [#/sec] (mean)Time per request:       786.459 [ms] (mean),  Connect: 0    1   4.0      0      14, Processing: 149  742 1175.8    468    6124, CPU Core Load: %usr,1.90,1.95,1.99,1.96,1.94,1.93,1.94,1.95,1.94,1.95,1.94,1.96,1.93,1.88,1.89,1.86,1.84,1.86,1.83,1.85,1.83,1.86,1.85,1.86,1.83,, CPU Idle%: %idle,97.65,97.58,97.59,97.59,97.63,97.62,97.63,97.59,97.63,97.58,97.63,97.59,97.64,97.65,97.69,97.68,97.73,97.68,97.74,97.69,97.74,97.69,97.72,97.61,97.74,,Waiting on Disk: %iowait,0.04,0.05,0.04,0.04,0.04,0.04,0.04,0.04,0.03,0.04,0.03,0.04,0.03,0.05,0.04,0.04,0.04,0.04,0.04,0.04,0.03,0.04,0.04,0.11,0.04,, vmstatProcs: 101, vmstatSwapIn: 0, vmstatSwapOut: 0, vmstatIoIn: 7, vmstatIoOut: 42, iostatR-S: 1.72, iostatW-S: 13.37, iostatR-MBS: 0.25, iostatW-MBS: 0.52, iostatHDD-Util: 0.00
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, Cores: 6, Sockets 2, DDR4, (free-mh): 62G 13G 17G 4.5M 32G 48G, NVMe RAID, TestType: Apache PHP, Threads: 256, Server Load: 30.81, 7.33, 2.49, Threads: 256,   Time per request: 2309.182 [ms] (mean), Concurrency Level: 256, Requests per second 110.86 [#/sec] (mean)Time per request:       2309.182 [ms] (mean),  Connect: 0    3   5.2      0      14, Processing: 148 1546 1985.2    830    9003, CPU Core Load: %usr,1.91,1.95,2.00,1.96,1.94,1.93,1.95,1.96,1.95,1.96,1.94,1.96,1.94,1.89,1.90,1.86,1.85,1.86,1.83,1.86,1.83,1.86,1.85,1.87,1.84,, CPU Idle%: %idle,97.65,97.57,97.59,97.59,97.63,97.62,97.62,97.59,97.62,97.58,97.63,97.58,97.64,97.65,97.69,97.68,97.73,97.67,97.74,97.69,97.74,97.69,97.71,97.61,97.73,,Waiting on Disk: %iowait,0.04,0.05,0.04,0.04,0.04,0.04,0.04,0.04,0.03,0.04,0.03,0.04,0.03,0.05,0.04,0.04,0.04,0.04,0.04,0.04,0.03,0.04,0.04,0.11,0.04,, vmstatProcs: 77, vmstatSwapIn: 0, vmstatSwapOut: 0, vmstatIoIn: 7, vmstatIoOut: 42, iostatR-S: 1.72, iostatW-S: 13.37, iostatR-MBS: 0.25, iostatW-MBS: 0.52, iostatHDD-Util: 0.00
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, Cores: 6, Sockets 2, DDR4, (free-mh): 62G 13G 16G 4.5M 32G 48G, NVMe RAID, TestType: Apache PHP, Threads: 512, Server Load: 46.33, 11.29, 3.82, Threads: 512,   Time per request: 28000.550 [ms] (mean), Concurrency Level: 512, Requests per second 18.29 [#/sec] (mean)Time per request:       28000.550 [ms] (mean),  Connect: 0   11  15.6      0      38, Processing: 179 14189 21674.4   1677   54626, CPU Core Load: %usr,1.91,1.96,2.00,1.96,1.94,1.94,1.95,1.96,1.95,1.96,1.95,1.96,1.94,1.89,1.90,1.87,1.85,1.87,1.84,1.86,1.84,1.87,1.85,1.87,1.84,, CPU Idle%: %idle,97.65,97.57,97.58,97.58,97.62,97.61,97.62,97.59,97.62,97.57,97.63,97.58,97.63,97.64,97.68,97.68,97.72,97.67,97.74,97.68,97.74,97.68,97.71,97.60,97.73,,Waiting on Disk: %iowait,0.04,0.05,0.04,0.04,0.04,0.04,0.04,0.04,0.03,0.04,0.03,0.04,0.03,0.05,0.04,0.04,0.04,0.04,0.04,0.04,0.03,0.04,0.04,0.11,0.04,, vmstatProcs: 0, vmstatSwapIn: 0, vmstatSwapOut: 0, vmstatIoIn: 7, vmstatIoOut: 42, iostatR-S: 1.72, iostatW-S: 13.37, iostatR-MBS: 0.25, iostatW-MBS: 0.52, iostatHDD-Util: 0.00
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, Cores: 6, Sockets 2, DDR4, (free-mh): 62G 14G 16G 4.5M 32G 48G, NVMe RAID, TestType: Apache PHP, Threads: 750, Server Load: 37.39, 13.46, 4.95, Threads: 750,   Time per request: 40777.912 [ms] (mean), Concurrency Level: 750, Requests per second 18.39 [#/sec] (mean)Time per request:       40777.912 [ms] (mean),  Connect: 0   13  17.8      0      43, Processing: 150 17962 23352.7   1807   54307, CPU Core Load: %usr,1.91,1.96,2.00,1.97,1.95,1.94,1.95,1.96,1.95,1.96,1.95,1.97,1.94,1.89,1.90,1.87,1.85,1.87,1.84,1.86,1.84,1.87,1.86,1.87,1.84,, CPU Idle%: %idle,97.64,97.57,97.58,97.58,97.62,97.61,97.62,97.58,97.62,97.57,97.62,97.58,97.63,97.64,97.68,97.67,97.72,97.67,97.73,97.68,97.73,97.68,97.71,97.60,97.73,,Waiting on Disk: %iowait,0.04,0.05,0.04,0.04,0.04,0.04,0.04,0.04,0.03,0.04,0.03,0.04,0.03,0.05,0.04,0.04,0.04,0.04,0.04,0.04,0.03,0.04,0.04,0.11,0.04,, vmstatProcs: 0, vmstatSwapIn: 0, vmstatSwapOut: 0, vmstatIoIn: 7, vmstatIoOut: 42, iostatR-S: 1.72, iostatW-S: 13.36, iostatR-MBS: 0.25, iostatW-MBS: 0.52, iostatHDD-Util: 0.00

PHP 脚本正在执行 json_decode()，多次对数千条记录进行排序，对某些数组项使用 preg_replace()，然后使用以下命令生成随机数：

for($z = 0; $z < 10; $z++) {
    $r = array(0,0,0,0,0,0,0,0,0,0,0);
    for ($i=0;$i<100000;$i++) {
      $n = mt_rand(0,10000);
      if ($n<=10) {
        $r[$n]++;
      }
    }
    print_r($r); 
    print_r("<BR>");
}

我想知道是否还有其他一些命令需要我查看输出来确定这一点。当前命令集：

vmstat -w -S m 
iostat -dxmh /dev/md0
mpstat -P ALL
free -h 
lscpu
top -b -n

#Between each test clear cache
sync; echo 3 > /proc/sys/vm/drop_caches 
swapoff -a && swapon -a

Answer 1

很好，您测量了响应时间，即使是对于这个微基准测试。主机级利用率指标并不能说明全部情况，在这里肯定不能。

在 256 个线程之前，性能就已经下降很多。在 32 个线程之后，每秒请求数实际上下降了，因此第三次运行开始变得更糟。而且 CPU 利用率或 iowait 并没有真正相应增加。因此，可能存在对其他资源的争用。

保持开放的心态并检查一切利用率和饱和度。以下是一个非常不完整的列表：

• 它随机地做某事，这会耗尽并阻塞吗/dev/random？
• 它是否同时使用任何共享资源，如 flocks 或 UNIX IPC？
• 如果打印变量，那么这个输出会去哪里，I/O 有多慢？
• 如何管理众多工作线程，换句话说，您的 Apache MPM 是什么以及如何配置它？
• 这个基准测试是否使用数据库？DBMS 调优本身就是一个主题，请仔细研究一下。

由于这是 Linux，其性能工具很好，因此可以考虑对 CPU 上的所有内容进行采样并查看调用图。参考Gregg 的 Linux 性能笔记关于如何制作 CPU 火焰图。尽可能使用 eBPF，并perf record在较早的内核上使用。阅读图表以了解代码、运行时间和数据库以及操作系统之间的时间消耗情况。例如，如果许多采样堆栈与文件 I/O 相关，您可能会看到与文件系统相关的函数。

另外，关于看所有事情的话题，考虑一下网络数据正在进行。此特定工具以最少的配置收集大量数据点。我认为，仅内置警报集就值得安装它。netdata 也可以指向 Apache httpd mod_status因此您可以将连接和工作人员与主机指标一起绘制成图表。