识别 Linux 中导致 oom(内存不足)内核终止进程的高内存分配器/泄漏进程

识别 Linux 中导致 oom(内存不足)内核终止进程的高内存分配器/泄漏进程

我已经找到了大量有关 oom 的信息,但没有太多信息可以确定问题的根本原因。 OOm Killer 根据其评分来杀死进程,但它杀死的进程不一定是占用内存的进程。在我的嵌入式系统中,只有一个日志日志,我可以依赖这个难以重现的问题。我如何从中推断出内存占用过程?我如何理解 OOM 杀手的日志转储?

Jan 16 14:30:41 Esystem kernel: steaming_device_driver invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jan 16 14:30:41 Esystem kernel: CPU: 0 PID: 386 Comm: steaming_device_driver Tainted: G           O      5.4.47-1.3.0-511_02111356_a81cba7f+DEBUG+g5ec03d06f54e #1
Jan 16 14:30:41 Esystem kernel: Hardware name: i.MX8MNano DDR4 board (DT)
Jan 16 14:30:41 Esystem kernel: Call trace:
Jan 16 14:30:41 Esystem kernel:  dump_backtrace+0x0/0x140
Jan 16 14:30:41 Esystem kernel:  show_stack+0x14/0x20
Jan 16 14:30:41 Esystem kernel:  dump_stack+0xb4/0xf8
Jan 16 14:30:41 Esystem kernel:  dump_header+0x44/0x1ec
Jan 16 14:30:41 Esystem kernel:  oom_kill_process+0x1d4/0x1d8
Jan 16 14:30:41 Esystem kernel:  out_of_memory+0x170/0x4e0
Jan 16 14:30:41 Esystem kernel:  __alloc_pages_slowpath+0x954/0x9f8
Jan 16 14:30:41 Esystem kernel:  __alloc_pages_nodemask+0x21c/0x280
Jan 16 14:30:41 Esystem kernel:  alloc_pages_current+0x7c/0xe8
Jan 16 14:30:41 Esystem kernel:  __page_cache_alloc+0x80/0xa8
Jan 16 14:30:41 Esystem kernel:  pagecache_get_page+0x150/0x300
Jan 16 14:30:41 Esystem kernel:  filemap_fault+0x544/0x950
Jan 16 14:30:41 Esystem kernel:  ext4_filemap_fault+0x30/0x8b8
Jan 16 14:30:41 Esystem kernel:  __do_fault+0x4c/0x188
Jan 16 14:30:41 Esystem kernel:  __handle_mm_fault+0xb5c/0x10a0
Jan 16 14:30:41 Esystem kernel:  handle_mm_fault+0xdc/0x1a8
Jan 16 14:30:41 Esystem kernel:  do_page_fault+0x130/0x460
Jan 16 14:30:41 Esystem kernel:  do_translation_fault+0x5c/0x78
Jan 16 14:30:41 Esystem kernel:  do_mem_abort+0x3c/0x98
Jan 16 14:30:41 Esystem kernel:  do_el0_ia_bp_hardening+0x38/0xb8
Jan 16 14:30:41 Esystem kernel:  el0_ia+0x18/0x1c
Jan 16 14:30:41 Esystem kernel: Mem-Info:
Jan 16 14:30:41 Esystem kernel: active_anon:95298 inactive_anon:96 isolated_anon:0
                                     active_file:141 inactive_file:467 isolated_file:0
                                     unevictable:0 dirty:0 writeback:0 unstable:0
                                     slab_reclaimable:2677 slab_unreclaimable:8348
                                     mapped:112 shmem:205 pagetables:846 bounce:0
                                     free:1314 free_pcp:0 free_cma:0
Jan 16 14:30:41 Esystem kernel: Node 0 active_anon:381192kB inactive_anon:384kB active_file:564kB inactive_file:1868kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:448kB dirty:0kB writeback:0kB shmem:820kB shmem_thp: 0kB shmem_pmdmapp
Jan 16 14:30:41 Esystem kernel: Node 0 DMA32 free:5256kB min:7092kB low:7636kB high:8180kB active_anon:381192kB inactive_anon:384kB active_file:564kB inactive_file:1868kB unevictable:0kB writepending:0kB present:491520kB managed:454112kB mlocked:0kB 
Jan 16 14:30:41 Esystem kernel: lowmem_reserve[]: 0 0 0
Jan 16 14:30:41 Esystem kernel: Node 0 DMA32: 112*4kB (UMEC) 184*8kB (UE) 117*16kB (UE) 46*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB = 5264kB
Jan 16 14:30:41 Esystem kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jan 16 14:30:41 Esystem kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=32768kB
Jan 16 14:30:41 Esystem kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jan 16 14:30:41 Esystem kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=64kB
Jan 16 14:30:41 Esystem kernel: 812 total pagecache pages
Jan 16 14:30:41 Esystem kernel: 0 pages in swap cache
Jan 16 14:30:41 Esystem kernel: Swap cache stats: add 0, delete 0, find 0/0
Jan 16 14:30:41 Esystem kernel: Free swap  = 0kB
Jan 16 14:30:41 Esystem kernel: Total swap = 0kB
Jan 16 14:30:41 Esystem kernel: 122880 pages RAM
Jan 16 14:30:41 Esystem kernel: 0 pages HighMem/MovableOnly
Jan 16 14:30:41 Esystem kernel: 9352 pages reserved
Jan 16 14:30:41 Esystem kernel: 32768 pages cma reserved
Jan 16 14:30:41 Esystem kernel: 0 pages hwpoisoned
Jan 16 14:30:41 Esystem kernel: Tasks state (memory values in pages):
Jan 16 14:30:41 Esystem kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jan 16 14:30:41 Esystem kernel: [    292]     0   292     9543      157    98304        0          -250 systemd-journal
Jan 16 14:30:41 Esystem kernel: [    306]     0   306     3159      191    57344        0         -1000 systemd-udevd
Jan 16 14:30:41 Esystem kernel: [    317]   993   317     4079      123    57344        0             0 systemd-network
Jan 16 14:30:41 Esystem kernel: [    331]     0   331    19289       37    45056        0             0 rngd
Jan 16 14:30:41 Esystem kernel: [    332]   992   332     1926       99    53248        0             0 systemd-resolve
Jan 16 14:30:41 Esystem kernel: [    333]   991   333    20408      110    65536        0             0 systemd-timesyn
Jan 16 14:30:41 Esystem kernel: [    339]     0   339      880       65    49152        0             0 auto-update.sh
Jan 16 14:30:41 Esystem kernel: [    340]   995   340     1210       86    45056        0             0 avahi-daemon
Jan 16 14:30:41 Esystem kernel: [    345]     0   345      799       31    40960        0             0 klogd
Jan 16 14:30:41 Esystem kernel: [    347]   995   347     1179       63    45056        0             0 avahi-daemon
Jan 16 14:30:41 Esystem kernel: [    348]     0   348      799       26    45056        0             0 syslogd
Jan 16 14:30:41 Esystem kernel: [    350]   996   350     1128      149    49152        0          -900 dbus-daemon
Jan 16 14:30:41 Esystem kernel: [    352]     0   352     2151      174    61440        0             0 ofonod
Jan 16 14:30:41 Esystem kernel: [    365]   998   365      926       65    45056        0             0 rpcbind
Jan 16 14:30:41 Esystem kernel: [    372]     0   372    37478       22    53248        0             0 tee-supplicant
Jan 16 14:30:41 Esystem kernel: [    381]     0   381     1924      122    57344        0             0 systemd-logind
Jan 16 14:30:41 Esystem kernel: [    385]     0   385     1037       59    49152        0             0 xsystrack
Jan 16 14:30:41 Esystem kernel: [    386]     0   386    98589      172   131072        0             0 steaming_device_driver
Jan 16 14:30:41 Esystem kernel: [    387]     0   387   131366      145   151552        0             0 user_psu_daemon
Jan 16 14:30:41 Esystem kernel: [    389]     0   389   203209     3492   274432        0             0 xproc_manager.py
Jan 16 14:30:41 Esystem kernel: [    431]     0   431      511       37    45056        0             0 hciattach
Jan 16 14:30:41 Esystem kernel: [    434]     0   434     1696      150    57344        0             0 bluetoothd
Jan 16 14:30:41 Esystem kernel: [    456]     0   456     2524      190    53248        0             0 wpa_supplicant
Jan 16 14:30:41 Esystem kernel: [    457]   997   457      791      145    45056        0             0 rpc.statd
Jan 16 14:30:41 Esystem kernel: [    460]     0   460     9884      162    69632        0             0 tcf-agent
Jan 16 14:30:41 Esystem kernel: [    461]     0   461      648       24    40960        0             0 xinetd
Jan 16 14:30:41 Esystem kernel: [    466]     0   466     1252       32    45056        0             0 agetty
Jan 16 14:30:41 Esystem kernel: [    467]     0   467      509       26    45056        0             0 agetty
Jan 16 14:30:41 Esystem kernel: [    477]     0   477      509       26    40960        0             0 agetty
Jan 16 14:30:41 Esystem kernel: [    489]     0   489     4640       18    45056        0             0 umtprd
Jan 16 14:30:41 Esystem kernel: [    541]     0   541   206089    20731   385024        0             0 application1
Jan 16 14:30:41 Esystem kernel: [    574]     0   574   309918    60424  1114112        0             0 python3
Jan 16 14:30:41 Esystem kernel: [    658]     0   658    71768     2840   143360        0             0 updater
Jan 16 14:30:41 Esystem kernel: [ 102299]     0 102299     1935      155    57344        0             0 sshd
Jan 16 14:30:41 Esystem kernel: [ 102302]     0 102302     2320      242    65536        0             0 systemd
Jan 16 14:30:41 Esystem kernel: [ 102303]     0 102303    23402      523    69632        0             0 (sd-pam)
Jan 16 14:30:41 Esystem kernel: [ 102312]     0 102312      940      128    45056        0             0 sh
Jan 16 14:30:41 Esystem kernel: [ 102342]     0 102342    26140       94   241664        0             0 journalctl
Jan 16 14:30:41 Esystem kernel: [ 120859]     0 120859   108920     3034   155648        0             0 sch_manage
Jan 16 14:30:41 Esystem kernel: [ 120885]     0 120885      855       51    49152        0             0 sh
Jan 16 14:30:41 Esystem kernel: [ 120886]     0 120886      880       65    49152        0             0 auto-update.sh
Jan 16 14:30:41 Esystem kernel: [ 120888]     0 120888      499       23    36864        0             0 ls
Jan 16 14:30:41 Esystem kernel: [ 120889]     0 120889      480       19    45056        0             0 head
Jan 16 14:30:41 Esystem kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/nw.service,task=python3,pid=574,uid=0
Jan 16 14:30:41 Esystem kernel: Out of memory: Killed process 574 (python3) total-vm:1239672kB, anon-rss:241684kB, file-rss:0kB, shmem-rss:12kB, UID:0 pgtables:1088kB oom_score_adj:0
Jan 16 14:30:41 Esystem kernel: sched: RT throttling activated
Jan 16 14:30:41 Esystem kernel: oom_reaper: reaped process 574 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB
Jan 16 14:30:40 Esystem kernel[345]: [53284.724775] steaming_device_driver invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0

我该如何解释这一点?有这方面的文档吗?

答案1

RSS 是驻留集大小,换句话说,进程的 RAM 消耗。进程 ID 541(应用1)和574(蟒蛇3)就 RSS 而言是迄今为止最大的。其中574人被判处死刑。

现在,“python3”并不是非常具体,但是如果您知道系统上正在运行的东西,您可能可以弄清楚那是什么进程。

关于文档,我会搜索oom 网站:kernel.org首先,或者查看 linuxtoday 或 Linux Journal 等出版物。

看完之后OOM 文档,对我来说很明显,虚拟内存而不是物理内存用于确定应终止哪个进程。警告:我还没有检查该文档页面是否仍然是最新的。

相关内容