如何识别丢失的使用内存

Question

我正在运行 22.04，最近出现了 systemd-oomd 问题12我正在尝试弄清楚我的 RAM 都用在哪儿了。我有 16GB 的 RAM，应该足够了，但我的 Chrome、Gnome Shell、Gnome Terminal 总是被 oomd 杀死。

我的进程表 ( ps -ef) 与 中的内存使用情况不符/proc/meminfo，因此我尝试查找我不知道的其他内存分配，以查明我的 RAM 被用在何处。我知道 ipcs、slabtop，但内存分配不合逻辑。

根据我对ps输出的阅读（如下），我只使用了 8.4gb 的 RAM，但查看时meminfo我使用了 15+gb。

ps uax | awk '{ print $6 }' | sort -n | awk '{s+=$1} END {print s}'
4551528

slabtop -s c -o | tail -n +7 | awk '{s =s+$7} END {print s}'
1593640

ipcs
------ 消息队列 -------- key msqid 所有者权限已用字节数消息 ------ 共享内存段 -------- key shmid 所有者权限字节数 nattch 状态 0x511c029a 196610 ruckc 600 16 1 0xca270022 196611 ruckc 600 65536 1 0x00000000 196632 ruckc 600 524288 2 dest ------ 信号量数组 -------- key semid 所有者权限 nsems 0x00105b6b 2 root 600 1 0x00105b6c 3 root 666 2 0xcc270022 8 ruckc 600 1 0xcb270022 9 ruckc 600 1

cat /proc/meminfo

MemTotal：16088108 kB MemFree：4183228 kB MemAvailable：5130228 kB 缓冲区：1084 kB 缓存：2270828 kB SwapCached：22000 kB 活动：620544 kB 非活动：2647548 kB 活动（匿名）：138556 kB 非活动（匿名）：1932644 kB 活动（文件）：481988 kB 非活动（文件）：714904 kB 不可驱逐：900756 kB Mlocked：72936 kB SwapTotal：2097148 kB SwapFree：1967088 kB 脏：2856 kB 写回：0 kB AnonPages：1875816 kB Mapped：562296 kB Shmem：1072444 kB KReclaimable：135272 kB Slab：1624092 kB SReclaimable：135272 kB SUnreclaim：1488820 kB KernelStack：18176 kB PageTables：35684 kB NFS_Unstable：0 kB Bounce：0 kB WritebackTmp：0 kB CommitLimit：10141200 kB Committed_AS：9789584 kB VmallocTotal：34359738367 kB VmallocUsed：591532 kB VmallocChunk：0 kB Percpu：10848 kB HardwareCorrupted：0 kB AnonHugePages：0 kB ShmemHugePages：0 kB ShmemPmdMapped：0 kB FileHugePages：0 kB FilePmdMapped：0 kB HugePages_Total：0 HugePages_Free：0 HugePages_Rsvd：0 HugePages_Surp：0 Hugepagesize：2048 kB Hugetlb：0 kB DirectMap4k：3318960 kB DirectMap2M：13158400 kB DirectMap1G：1048576 kB

free -h
总共使用 空闲 共享 buff/缓存 可用 内存：15Gi 9.1Gi 4.0Gi 1.0Gi 2.3Gi 4.9Gi 交换：2.0Gi 127Mi 1.9Gi

oomd 日志：journalctl -xeu systemd-oomd.service | grep Killed

Apr 13 10:51:39 laptop systemd-oomd[3562]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-google\x2dchrome-11396.scope due to memory used (16109408256) / total (16474247168) and swap used (1982627840) / total (2147479552) being more than 90.00%
Apr 13 13:09:57 laptop systemd-oomd[3363]: Killed /user.slice/user-1000.slice/[email protected]/session.slice/[email protected] due to memory used (16195358720) / total (16474247168) and swap used (2053111808) / total (2147479552) being more than 90.00%
Apr 14 13:21:21 laptop systemd-oomd[3248]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-google\x2dchrome-41482.scope due to memory used (16056369152) / total (16474247168) and swap used (1945407488) / total (2147479552) being more than 90.00%
Apr 18 09:58:14 laptop systemd-oomd[3457]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-org.gnome.Nautilus-0.scope due to memory used (16223395840) / total (16474243072) and swap used (2027683840) / total (2147479552) being more than 90.00%
Apr 18 09:58:15 laptop systemd-oomd[3457]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-org.gnome.Nautilus-0.scope due to memory used (16091160576) / total (16474243072) and swap used (2145886208) / total (2147479552) being more than 90.00%
Apr 18 09:58:15 laptop systemd-oomd[3457]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-org.gnome.Nautilus-0.scope due to memory used (16159051776) / total (16474243072) and swap used (2073485312) / total (2147479552) being more than 90.00%
Apr 18 09:58:15 laptop systemd-oomd[3457]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-org.gnome.Nautilus-0.scope due to memory used (16212561920) / total (16474243072) and swap used (2008236032) / total (2147479552) being more than 90.00%
Apr 18 09:58:15 laptop systemd-oomd[3457]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-org.gnome.Nautilus-0.scope due to memory used (15865094144) / total (16474243072) and swap used (1974681600) / total (2147479552) being more than 90.00%
Apr 20 16:00:06 laptop systemd-oomd[3457]: Killed /user.slice/user-1000.slice/[email protected]/session.slice/[email protected] due to memory used (16120066048) / total (16474243072) and swap used (1999306752) / total (2147479552) being more than 90.00%
Apr 20 16:04:18 laptop systemd-oomd[3457]: Killed /user.slice/user-1000.slice/[email protected]/session.slice/[email protected] due to memory used (16117116928) / total (16474243072) and swap used (2039713792) / total (2147479552) being more than 90.00%
Apr 26 08:07:19 laptop systemd-oomd[3457]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-code-2704032.scope due to memory used (16189804544) / total (16474243072) and swap used (1975377920) / total (2147479552) being more than 90.00%
May 03 12:17:16 laptop systemd-oomd[3253]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-jetbrains\x2dtoolbox-10574.scope due to memory used (15012593664) / total (16474185728) and swap used (1944969216) / total (2147479552) being more than 90.00%
May 04 22:31:52 laptop systemd-oomd[3253]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-google\x2dchrome-1560985.scope due to memory used (15096467456) / total (16474185728) and swap used (2009534464) / total (2147479552) being more than 90.00%
May 05 12:34:52 laptop systemd-oomd[3354]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-google\x2dchrome-47763.scope due to memory used (16104947712) / total (16474189824) and swap used (1948254208) / total (2147479552) being more than 90.00%
May 09 10:26:05 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/session.slice/[email protected] due to memory used (16139558912) / total (16474193920) and swap used (2147348480) / total (2147479552) being more than 90.00%
May 10 19:40:25 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-google\x2dchrome-950608.scope due to memory used (16102866944) / total (16474193920) and swap used (2071912448) / total (2147479552) being more than 90.00%
May 10 19:49:30 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-org.gnome.Terminal.slice/vte-spawn-16c14424-f9a9-4c04-8ff2-0755377a7539.scope due to memory used (16022802432) / total (16474193920) and swap used (1948741632) / total (2147479552) being more than 90.00%
May 11 09:00:04 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-org.gnome.Terminal.slice/vte-spawn-71b574b8-17e5-408b-b91f-431a458d55bb.scope due to memory used (15930691584) / total (16474193920) and swap used (2041720832) / total (2147479552) being more than 90.00%
May 11 10:20:57 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-google\x2dchrome-2201754.scope due to memory used (15575441408) / total (16474193920) and swap used (1939279872) / total (2147479552) being more than 90.00%
May 11 12:05:21 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-org.gnome.Terminal.slice/vte-spawn-24f78600-4612-4805-9b29-a96ca4f432cb.scope due to memory used (15433670656) / total (16474193920) and swap used (1950101504) / total (2147479552) being more than 90.00%
May 11 12:05:51 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-google\x2dchrome-2814868.scope due to memory used (15329484800) / total (16474193920) and swap used (1946755072) / total (2147479552) being more than 90.00%
May 11 12:06:29 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-netbeans-2803190.scope due to memory used (15826632704) / total (16474193920) and swap used (2145792000) / total (2147479552) being more than 90.00%
May 11 14:33:29 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-google\x2dchrome-2909372.scope due to memory used (15613919232) / total (16474193920) and swap used (1933402112) / total (2147479552) being more than 90.00%
May 11 14:56:16 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/session.slice/[email protected] due to memory used (16104726528) / total (16474193920) and swap used (1968328704) / total (2147479552) being more than 90.00%
May 12 08:25:14 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-jetbrains\x2dpycharm-3483077.scope due to memory used (15616532480) / total (16474193920) and swap used (1937936384) / total (2147479552) being more than 90.00%
May 12 11:15:23 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-google\x2dchrome-3361444.scope due to memory used (15812100096) / total (16474193920) and swap used (1955221504) / total (2147479552) being more than 90.00%
May 12 17:58:07 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-google\x2dchrome-722172.scope due to memory used (16039403520) / total (16474193920) and swap used (1988845568) / total (2147479552) being more than 90.00%
May 13 11:30:57 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-google\x2dchrome-1218538.scope due to memory used (16068583424) / total (16474193920) and swap used (2087342080) / total (2147479552) being more than 90.00%
May 13 11:30:59 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/session.slice/[email protected] due to memory used (16112533504) / total (16474193920) and swap used (1946112000) / total (2147479552) being more than 90.00%
May 18 15:47:46 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-org.gnome.Terminal.slice/vte-spawn-921c3683-819b-4f55-ba40-5bcd85681825.scope due to memory used (16058290176) / total (16474193920) and swap used (2002198528) / total (2147479552) being more than 90.00%
May 20 09:03:27 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-google\x2dchrome-1534842.scope due to memory used (15276462080) / total (16474193920) and swap used (1937162240) / total (2147479552) being more than 90.00%
Jun 03 12:15:36 laptop systemd-oomd[3298]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-jetbrains\x2dwebstorm-541749.scope due to memory used (15719739392) / total (16474193920) and swap used (1967783936) / total (2147479552) being more than 90.00%
Jun 08 00:21:43 laptop systemd-oomd[3768]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-google\x2dchrome-16072.scope due to memory used (15445958656) / total (16474218496) and swap used (1942192128) / total (2147479552) being more than 90.00%
Jun 08 07:14:28 laptop systemd-oomd[3768]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-teams-2369963.scope due to memory used (15633932288) / total (16474218496) and swap used (1964466176) / total (2147479552) being more than 90.00%
Jun 08 07:16:38 laptop systemd-oomd[3768]: Killed /user.slice/user-1000.slice/[email protected]/app.slice/app-gnome-jetbrains\x2dgoland-2978322.scope due to memory used (15238217728) / total (16474218496) and swap used (1939480576) / total (2147479552) being more than 90.00%
Jun 13 10:14:46 laptop systemd-oomd[7467]: Killed /user.slice/user-1000.slice/[email protected]/session.slice/[email protected] due to memory used (15284772864) / total (16474218496) and swap used (1937956864) / total (2147479552) being more than 90.00%

Answer 1

检查日志后发现您正在使用 netbeans、pycharm 等……它们也因 OOM 而被终止。看起来您使用的内存比您拥有的内存要多得多。netbeans 或 pycharm 的 Java 进程使用的内存可能比您想象的要多。您可以扩大或限制为这些应用程序保留给 Java 的内存。

• 你能运行free -h吗？我们可以轻松看到你有多少可用内存，缓冲区中有多少……

• 可以在以下位置找到有关 chrome 或任何给定其 PID 的进程的信息/proc/PID/stat（如虚拟内存大小或驻留集）。每个字段的解释如下：http://brokestream.com/procstat.html

• htop 可能会派上用场，因为你可以按内存使用情况、组进行排序

• 您可能有兴趣检查内存从/到交换区传输的频率。如果进程正在争用内存，则系统可能会非常忙于在交换区之间/到交换区传输，无法足够快地分配资源，最终导致 OOM。您可以从 /proc/vmstat 检查 pswpin 和 pswpout。我这里有一个小程序可以做到这一点：https://blog.carlesmateo.com/2021/05/27/swap-swappiness-servers-not-responding/

• 你能检查一下你在 chrome 上使用了哪些插件吗？16GB 的 RAM 听起来很多，但其实并不是那么多。浏览器需要大量内存。而且很多时候，广告会占用大量资源。

我的建议是使用较少的 Chrome 标签，特别是如果你编程的话，因为它很容易耗尽内存。

更新 2022-06-23：

• 我从free -h输出中看到有一些交换正在使用。请检查您的 swappiness 值cat /proc/sys/vm/swappiness

• Swappiness 定义了用于改善 IO 的内存与出于此原因从交换中获取内存之间的平衡。基本上，如果您的 swappiness 为 60，您可以预期，当 60% 的 RAM 内存正在使用时，交换将用于为缓冲区提供更多内存。在我看来，这是一个巨大的错误，因为使用交换的性能损失高于使用缓存/缓冲区的好处。在您的例子中，您有近 5GB 的可用内存，但您的系统使用了（缓慢的）交换。我会将 swappiness 设置为 10，我希望您的体验会有所改善，并且可能会解决您的问题。sudo sysctl vm.swappiness=10如果您对这个新值感到满意，请在/etc/sysctl.conf

• 你可以用iotop --only它来实时查看在读写交换时浪费了多少带宽