我有一个 EC2 t2.micro 实例,用作 Web 应用程序的测试实例,
操作系统:Ubuntu 22.04.1 LTS
kswapd0 进程的 CPU 消耗偶尔会跳到 100% 附近,导致实例冻结,需要重新启动。
注意:“/proc/sys/vm/swappiness”的默认值为 60
我读到过 kswapd0 是将内存从物理内存交换到虚拟内存的过程,因此理论上,它应该只在内存使用率高时启动。
神秘的是,似乎没有任何进程正在经历内存狂潮,我已经用 atop 记录了资源使用情况,下面是 3 个 atop 快照,间隔 1 分钟。
以下是快照的摘要,我们看到可用内存正在减少,而 CPU 使用率正在增加:
快照 1 顶部:cpu 5%,可用内存 242.9M
快照2 顶部:cpu 38%,可用内存 242.5M
快照3:cpu 99%,可用内存 65.4M
这是完整的快照,我们可以看到其他变量,在快照 2、3 上,DSK、MEM 和 CPU 线处于读取状态(这里未显示颜色......)
此外,快照中的进程列表按内存使用情况排序,我希望看到一个内存使用率高的罪魁祸首进程,但没有一个进程消耗的内存超过 8%。
ATOP - ip-172-31-3-66 2022/10/18 00:00:01 ----------------- 1d15h10m16s elapsed
PRC | sys 2m27s | user 6m23s | | | #proc 148 | | #trun 12 | #tslpi 138 | | #tslpu 44 | #zombie 0 | | clones 92804 | | | #exit 13 |
CPU | sys 0% | | user 5% | irq 0% | | | idle 94% | | wait 0% | | steal 0% | guest 0% | | | curf 2.40GHz | |
CPL | avg1 0.01 | | avg5 0.10 | | avg15 0.15 | | | | csw 12835910 | | intr 5959928 | | | | numcpu 1 | |
MEM | tot 966.2M | free 242.9M | cache 302.3M | dirty 1.4M | buff 9.4M | slab 59.9M | slrec 26.0M | | | shmem 138.0M | shrss 0.0M | shswp 0.0M | | | | numnode 1 |
SWP | tot 0.0M | | free 0.0M | | swcac 0.0M | | | | | | | | vmcom 1.1G | | vmlim 483.1M | |
PAG | scan 6306350 | steal 4335e3 | | stall 0 | compact 0 | numamig 0 | | migrate 13e5 | | | | | swin 0 | swout 0 | | oomkill 0 |
PSI | cpusome 4% | memsome 0% | | memfull 0% | iosome 0% | | iofull 0% | cs 0/1/3 | | ms 0/0/0 | mf 0/0/0 | | is 0/0/0 | if 0/0/0 | | |
DSK | xvda | busy 0% | | read 219927 | write 497224 | discrd 0 | | KiB/r 33 | KiB/w 7 | KiB/d 0 | | MBr/s 0.1 | MBw/s 0.0 | avq 1.51 | | avio 0.57 ms |
DSK | xvda1 | busy 0% | | read 219453 | write 497222 | discrd 0 | | KiB/r 33 | KiB/w 7 | KiB/d 0 | | MBr/s 0.1 | MBw/s 0.0 | avq 1.51 | | avio 0.57 ms |
DSK | xvda15 | busy 0% | | read 222 | write 2 | discrd 0 | | KiB/r 42 | KiB/w 0 | KiB/d 0 | | MBr/s 0.0 | MBw/s 0.0 | avq 0.68 | | avio 1.43 ms |
DSK | xvda14 | busy 0% | | read 55 | write 0 | discrd 0 | | KiB/r 46 | KiB/w 0 | KiB/d 0 | | MBr/s 0.0 | MBw/s 0.0 | avq 0.64 | | avio 4.22 ms |
NET | transport | tcpi 1240508 | | tcpo 1316373 | udpi 148741 | udpo 154146 | | tcpao 18388 | tcppo 10357 | tcprs 4413 | | tcpie 294 | tcpor 3684 | udpnp 5399 | | udpie 0 |
NET | network | | ipi 1400059 | ipo 1361002 | | ipfrw 0 | deliv 1400e3 | | | | | | | icmpi 5402 | icmpo 54 | |
NET | lo ---- | | pcki 1152470 | pcko 1152470 | | sp 0 Mbps | si 158 Kbps | so 158 Kbps | | | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | eth0 ---- | | pcki 291231 | pcko 213817 | | sp 0 Mbps | si 8 Kbps | so 14 Kbps | | | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
*** System and Process Activity since Boot *** Rawfile view
PID TID MINFLT MAJFLT VSTEXT VSLIBS VDATA VSTACK LOCKSZ VSIZE RSIZE PSIZE VGROW RGROW SWAPSZ RUID EUID MEM CMD 1/9
30705 - 29496 291 2.7M 10.2M 90.1M 132.0K 0.0K 241.5M 78.0M 71.4M 241.5M 78.0M 0B flask flask 8% gunicorn
30704 - 28672 137 2.7M 10.2M 87.9M 132.0K 0.0K 239.3M 75.6M 69.1M 239.3M 75.6M 0B flask flask 8% gunicorn
170 - 12586 441557 92.0K 8.8M 49.2M 132.0K 0.0K 227.3M 44.1M 34.2M 227.3M 44.1M 0B root root 5% systemd-journa
30741 - 58604 984 5.8M 16.5M 7.1M 132.0K 0.0K 215.5M 33.5M 14.7M 215.5M 33.5M 0B postgres postgres 3% postgres
30739 - 49786 259 5.8M 16.5M 7.7M 132.0K 0.0K 216.1M 32.2M 14.1M 216.1M 32.2M 0B postgres postgres 3% postgres
212 - 5069 13 80.0K 6.2M 18.0M 132.0K 282.5M 282.5M 26.7M 22.7M 282.5M 26.7M 0B root root 3% multipathd
496 - 13376 1966 18.7M 1.8M 166.2M 132.0K 0.0K 710.3M 22.8M 25.3M 710.3M 22.8M 0B root root 2% snapd
30699 - 11046 22 2.7M 5.6M 21.0M 132.0K 0.0K 39.1M 22.0M 16.7M 39.1M 22.0M 0B flask flask 2% gunicorn
658 - 197263 2185 5.8M 16.3M 2.0M 132.0K 0.0K 210.1M 15.6M 4.1M 210.1M 15.6M 0B postgres postgres 2% postgres
92773 - 430 0 63.6M 3.4M 8.7M 132.0K 0.0K 78.0M 14.5M 9.4M 78.0M 14.5M 0B flask flask 2% node
720 - 872 52 5.8M 16.3M 2.2M 132.0K 0.0K 210.3M 14.0M 4.0M 210.3M 14.0M 0B postgres postgres 1% postgres
476 - 3103 56 2.7M 8.4M 10.4M 132.0K 0.0K 32.4M 13.2M 9.6M 32.4M 13.2M 0B root root 1% networkd-dispa
638 - 2551 27 2.7M 11.0M 17.1M 132.0K 0.0K 107.5M 11.0M 8.2M 107.5M 11.0M 0B root root 1% unattended-upg
1 - 156940 747 896.0K 8.8M 19.4M
ATOP - ip-172-31-3-66 2022/10/18 00:00:01 ----------------- 1s elapsed
PRC | sys 0.03s | user 0.02s | | | #proc 149 | | #trun 12 | #tslpi 148 | | #tslpu 44 | #zombie 0 | | clones 12 | | | #exit 0 |
CPU | sys 38% | | user 62% | irq 0% | | | idle 0% | | wait 0% | | steal 0% | guest 0% | | | curf 2.40GHz | |
CPL | avg1 0.01 | | avg5 0.10 | | avg15 0.15 | | | | csw 297 | | intr 84 | | | | numcpu 1 | |
MEM | tot 966.2M | free 242.5M | cache 304.1M | dirty 1.4M | buff 9.4M | slab 60.0M | slrec 26.1M | | | shmem 138.0M | shrss 0.0M | shswp 0.0M | | | | numnode 1 |
SWP | tot 0.0M | | free 0.0M | | swcac 0.0M | | | | | | | | vmcom 1.1G | | vmlim 483.1M | |
PSI | cpusome 11% | memsome 0% | | memfull 0% | iosome 1% | | iofull 0% | cs 0/1/3 | | ms 0/0/0 | mf 0/0/0 | | is 0/0/0 | if 0/0/0 | | |
DSK | xvda | busy 31% | | read 44 | write 3 | discrd 0 | | KiB/r 41 | KiB/w 29 | KiB/d 0 | | MBr/s 1.8 | MBw/s 0.1 | avq 0.80 | | avio 0.85 ms |
DSK | xvda1 | busy 31% | | read 44 | write 3 | discrd 0 | | KiB/r 41 | KiB/w 29 | KiB/d 0 | | MBr/s 1.8 | MBw/s 0.1 | avq 0.80 | | avio 0.85 ms |
PID TID MINFLT MAJFLT VSTEXT VSLIBS VDATA VSTACK LOCKSZ VSIZE RSIZE PSIZE VGROW RGROW SWAPSZ RUID EUID MEM CMD 1/6
30705 - 0 0 2.7M 10.2M 90.1M 132.0K 0.0K 241.5M 78.0M 71.3M 0B 0B 0B flask flask 8% gunicorn
30704 - 0 0 2.7M 10.2M 87.9M 132.0K 0.0K 239.3M 75.6M 69.0M 0B 0B 0B flask flask 8% gunicorn
170 - 2 6 92.0K 8.8M 49.2M 132.0K 0.0K 227.3M 44.1M 34.2M 0B 0B 0B root root 5% systemd-journa
30741 - 0 0 5.8M 16.5M 7.1M 132.0K 0.0K 215.5M 33.5M 14.7M 0B 0B 0B postgres postgres 3% postgres
30739 - 0 0 5.8M 16.5M 7.7M 132.0K 0.0K 216.1M 32.2M 14.1M 0B 0B 0B postgres postgres 3% postgres
212 - 0 0 80.0K 6.2M 18.0M 132.0K 282.5M 282.5M 26.7M 22.6M 0B 0B 0B root root 3% multipathd
496 - 0 0 18.7M 1.8M 166.2M 132.0K 0.0K 710.3M 22.8M 25.3M 0B 0B 0B root root 2% snapd
30699 - 0 0 2.7M 5.6M 21.0M 132.0K 0.0K 39.1M 22.0M 16.7M 0B 0B 0B flask flask 2% gunicorn
92773 - 416 0 63.6M 3.5M 43.0M 132.0K 0.0K 304.3M 20.5M 10.7M 226.3M 5.9M 0B flask flask 2% node
92779 - 454 0 63.6M 3.4M 41.9M 132.0K 0.0K 303.2M 19.6M 9.5M 233.3M 9.7M 0B flask flask 2% node
658 - 0 0 5.8M 16.3M 2.0M 132.0K 0.0K 210.1M 15.6M 4.1M 0B 0B 0B postgres postgres 2% postgres
720 - 0 0 5.8M 16.3M 2.2M 132.0K 0.0K 210.3M 14.0M 4.0M 0B 0B 0B postgres postgres 1% postgres
476 - 0 0 2.7M 8.4M 10.4M 132.0K 0.0K 32.4M 13.2M 9.5M 0B 0B 0B root root 1% networkd-dispa
638 - 0 0 2.7M 11.0M 17.1M 132.0K 0.0K 107.5M 11.0M 8.2M 0B 0B 0B root root 1% unattended-upg
1 - 43 0 896.0K 8.8M 19.4M 1.0M 0.0K 99.8M 10.1M 6.7M 0B 28.0K 0B root root 1% systemd
92751 - 543 0 172.0K 2.5M 4.7M 948.0K 9.7M 9.7M 9.6M 6.6M 1.2M 1.2M 0B root root 1% atop
1178 - 0 0 8.9M 8.0K 103.1M 132.0K 0.0K 1.2G 8.4M 12.8M 0B 0B 0B root root 1% ssm-agent-work
424 - 0 0 308.0K 12.0M 3.5M 132.0K 0.0K 24.8M 8.2M 4.9M 0B 0B 0B systemd- systemd- 1% systemd-resolv
721 - 0 0 5.8M 16.3M 2.0M
ATOP - ip-172-31-3-66 2022/10/18 00:02:01 ----------------- 1m0s elapsed
PRC | sys 59.57s | user 0.37s | | | #proc 145 | | #trun 2 | #tslpi 115 | | #tslpu 73 | #zombie 0 | | clones 4 | | | #exit 0 |
CPU | sys 99% | | user 0% | irq 1% | | | idle 0% | | wait 0% | | steal 0% | guest 0% | | | curf 2.40GHz | |
CPL | avg1 23.95 | | avg5 8.69 | | avg15 3.26 | | | | csw 121213 | | intr 73529 | | | | numcpu 1 | |
MEM | tot 966.2M | free 65.4M | cache 154.7M | dirty 0.0M | buff 0.1M | slab 58.9M | slrec 24.8M | | | shmem 138.0M | shrss 0.0M | shswp 0.0M | | | | numnode 1 |
SWP | tot 0.0M | | free 0.0M | | swcac 0.0M | | | | | | | | vmcom 1.4G | | vmlim 483.1M | |
PAG | scan 70955e3 | steal 1948e3 | | stall 0 | compact 0 | numamig 0 | | migrate 0 | | | | | swin 0 | swout 0 | | oomkill 0 |
PSI | cpusome 25% | memsome 99% | | memfull 83% | iosome 100% | | iofull 0% | cs 27/23/12 | | ms 99/83/31 | mf 82/70/26 | | is 100/84/31 | if 0/0/0 | | |
DSK | xvda | busy 100% | | read 57692 | write 69 | discrd 0 | | KiB/r 65 | KiB/w 5 | KiB/d 0 | | MBr/s 61.0 | MBw/s 0.0 | avq 22.51 | | avio 1.04 ms |
DSK | xvda1 | busy 100% | | read 57692 | write 69 | discrd 0 | | KiB/r 65 | KiB/w 5 | KiB/d 0 | | MBr/s 61.0 | MBw/s 0.0 | avq 22.51 | | avio 1.04 ms |
NET | transport | tcpi 0 | | tcpo 0 | udpi 8 | udpo 8 | | tcpao 0 | tcppo 0 | tcprs 0 | | tcpie 0 | tcpor 0 | udpnp 0 | | udpie 0 |
NET | network | | ipi 8 | ipo 8 | | ipfrw 0 | deliv 8 | | | | | | | icmpi 0 | icmpo 0 | |
NET | lo ---- | | pcki 8 | pcko 8 | | sp 0 Mbps | si 0 Kbps | so 0 Kbps | | | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | eth0 ---- | | pcki 1 | pcko 1 | | sp 0 Mbps | si 0 Kbps | so 0 Kbps | | | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
PID TID MINFLT MAJFLT VSTEXT VSLIBS VDATA VSTACK LOCKSZ VSIZE RSIZE PSIZE VGROW RGROW SWAPSZ RUID EUID MEM CMD 1/7
30705 - 729 3985 2.7M 10.2M 90.1M 132.0K 0.0K 241.5M 74.4M 69.6M 0B 132.0K 0B flask flask 8% gunicorn
30704 - 697 4022 2.7M 10.2M 87.9M 132.0K 0.0K 239.3M 72.0M 67.2M 0B -0.4M 0B flask flask 7% gunicorn
92777 - 396 2827 2.7M 10.1M 42.1M 132.0K 0.0K 69.6M 43.8M 41.5M 0B 700.0K 0B flask flask 5% python
92780 - 642 2963 2.7M 10.1M 42.1M 132.0K 0.0K 69.6M 43.6M 41.5M 0B 772.0K 0B flask flask 5% python
92776 - 23 2432 2.7M 10.1M 42.4M 132.0K 0.0K 69.6M 43.3M 40.9M 4.3M 772.0K 0B flask flask 4% python
92786 - 622 2998 2.7M 6.1M 42.0M 132.0K 0.0K 60.8M 43.0M 41.0M 0B -0.4M 0B flask flask 4% python
92782 - 715 2977 2.7M 6.1M 42.0M 132.0K 0.0K 60.8M 42.7M 41.0M 0B -0.1M 0B flask flask 4% python
92783 - 583 2985 2.7M 6.1M 42.0M 132.0K 0.0K 60.8M 42.6M 41.0M 0B -0.3M 0B flask flask 4% python
92775 - 737 2983 2.7M 6.1M 42.0M 132.0K 0.0K 60.8M 42.6M 41.0M 0B 600.0K 0B flask flask 4% python
92767 - 675 2969 2.7M 6.1M 42.0M 132.0K 0.0K 60.8M 42.2M 40.6M 0B -0.4M 0B flask flask 4% python
92778 - 238 2779 2.7M 6.1M 42.0M 132.0K 0.0K 60.8M 42.1M 40.8M 0B -0.2M 0B flask flask 4% python
30741 - 0 0 5.8M 16.5M 7.1M 132.0K 0.0K 215.5M 27.5M 12.7M 0B 0B 0B postgres postgres 3% postgres
212 - 0 0 80.0K 6.2M 18.0M 132.0K 282.5M 282.5M 26.7M 22.6M 0B 0B 0B root root 3% multipathd
30739 - 0 0 5.8M 16.5M 7.7M 132.0K 0.0K 216.1M 26.6M 12.2M 0B 0B 0B postgres postgres 3% postgres
30699 - 563 2647 2.7M 5.6M 21.0M 132.0K 0.0K 39.1M 22.4M 16.5M 0B 512.0K 0B flask flask 2% gunicorn
496 - 2 70 18.7M 1.8M 166.2M 132.0K 0.0K 710.3M 16.5M 16.7M 0B 0B 0B root root 2% snapd
658 - 33 441 5.8M 16.3M 2.0M 132.0K 0.0K 210.1M 13.0M 3.8M 0B 300.0K 0B postgres postgres 1% postgres
720 - 47 1995 5.8M 16.3M 2.2M 132.0K 0.0K 210.3M 12.9M 3.7M 0B -0.6M 0B postgres postgres 1% postgres