理解openwrt系统上的OOM killer原因

理解openwrt系统上的OOM killer原因

这是一个busybox/openwrt系统。

OOM-killer 会间歇性地终止进程。有时是 collectd,有时是 snmpd,以及如下例中的 uhttpd。

通过不运行 collectd(图表显示为 *rtg 默认值),我可以释放大约 37% 的空间,但是我不确定这是否会有什么作用,只会让问题随着时间的推移而进一步加剧。

我的 sysctl“vm.min_free_kbytes”显示 16384,并且根据下面的输出,据我解释,我有 32760kB 可用空间,那么为什么它会重新启动?

任何能够解释重启原因或提出建议的帮助都将不胜感激。

我可以(也许我应该)将 vm.min_free_kbytes 设置得更低一些?但是,下面的输出中并没有明显地显示我的可用内存不足。它显示“DMA free:32760kB min:16384kB”。内核版本是 4.9.17

 kern.warn kernel: [72064.465528] observer invoked oom-killer: gfp_mask=0x24080c0(GFP_KERNEL|__GFP_ZERO), nodemask=0, order=1, oom_score_adj=0
 kern.debug kernel: [72064.465537] CPU: 0 PID: 25658 Comm: observer Not tainted 4.9.17 #0
 kern.debug kernel: [72064.465539] Call Trace:
 kern.debug kernel: [72064.465547] [c54dfd40] [c03d1798] dump_stack+0x84/0xb0 (unreliable)
 kern.debug kernel: [72064.465563] [c54dfd50] [c03cfae0] dump_header.isra.4+0x54/0x180
 kern.debug kernel: [72064.465570] [c54dfd90] [c0097aec] oom_kill_process+0x88/0x3f0
 kern.debug kernel: [72064.465575] [c54dfdd0] [c0098350] out_of_memory+0x37c/0x3b0
 kern.debug kernel: [72064.465583] [c54dfe00] [c009bbf0] __alloc_pages_nodemask+0x904/0x9cc
 kern.debug kernel: [72064.465588] [c54dfeb0] [c009bcd4] __get_free_pages+0x1c/0x44
 kern.debug kernel: [72064.465598] [c54dfec0] [c001c660] mm_init+0xcc/0x144
 kern.debug kernel: [72064.465604] [c54dfee0] [c00de2b4] do_execveat_common+0x254/0x574
 kern.debug kernel: [72064.465609] [c54dff30] [c00de600] do_execve+0x2c/0x3c
 kern.debug kernel: [72064.465618] [c54dff40] [c000cff8] ret_from_syscall+0x0/0x3c
 kern.debug kernel: [72064.465624] --- interrupt: c01 at 0xfd41cd4
 kern.debug kernel: [72064.465624]     LR = 0x10024200
 kern.debug kernel: [72064.465626] Mem-Info:
 kern.debug kernel: [72064.465638] active_anon:1298 inactive_anon:26 isolated_anon:0
 kern.debug kernel: [72064.465638]  active_file:11 inactive_file:18 isolated_file:0
 kern.debug kernel: [72064.465638]  unevictable:15509 dirty:0 writeback:0 unstable:0
 kern.debug kernel: [72064.465638]  slab_reclaimable:691 slab_unreclaimable:2447
 kern.debug kernel: [72064.465638]  mapped:1622 shmem:163 pagetables:166 bounce:0
 kern.debug kernel: [72064.465638]  free:8190 free_pcp:72 free_cma:0
 kern.debug kernel: [72064.465648] Node 0 active_anon:5192kB inactive_anon:104kB active_file:44kB inactive_file:72kB unevictable:62036kB isolated(anon):0kB isolated(file):0kB mapped:6488kB dirty:0kB writeback:0kB shmem:652kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
 kern.debug kernel: [72064.465661] DMA free:32760kB min:16384kB low:20480kB high:24576kB active_anon:5192kB inactive_anon:104kB active_file:44kB inactive_file:72kB unevictable:62036kB writepending:0kB present:262144kB managed:200856kB mlocked:0kB slab_reclaimable:2764kB slab_unreclaimable:9788kB kernel_stack:920kB pagetables:664kB bounce:0kB free_pcp:288kB local_pcp:136kB free_cma:0kB
 kern.emerg kernel: lowmem_reserve[]: 0 0 0 0
 kern.debug kernel: [72064.465669] DMA: 306*4kB (UMEH) 258*8kB (UMH) 177*16kB (UMEH) 35*32kB (UMEH) 65*64kB (MEH) 47*128kB (MH) 2*256kB (H) 1*512kB (H) 2*1024kB (H) 2*2048kB (H) 2*4096kB (H) 0*8192kB 0*16384kB = 32776kB
 kern.emerg kernel: 15702 total pagecache pages
 kern.debug kernel: [72064.465707] 0 pages in swap cache
 kern.debug kernel: [72064.465710] Swap cache stats: add 0, delete 0, find 0/0
 kern.debug kernel: [72064.465711] Free swap  = 0kB
 kern.debug kernel: [72064.465713] Total swap = 0kB
 kern.debug kernel: [72064.465715] 65536 pages RAM
 kern.debug kernel: [72064.465716] 0 pages HighMem/MovableOnly
 kern.debug kernel: [72064.465718] 15322 pages reserved
 kern.info kernel: [72064.465720] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
 kern.info kernel: [72064.465730] [  834]     0   834      700      462       7       0        0             0 ubusd
 kern.info kernel: [72064.465735] [  835]     0   835      641      105       7       0        0             0 askfirst
 kern.info kernel: [72064.465740] [ 1548]     0  1548      594      378       6       0        0             0 allocpts
 kern.info kernel: [72064.465746] [ 2858]     0  2858      774      438       7       0        0             0 logd
 kern.info kernel: [72064.465751] [ 2873]     0  2873     1042      492       7       0        0             0 observer
 kern.info kernel: [72064.465756] [ 2883]     0  2883      921      497       7       0        0             0 rpcd
 kern.info kernel: [72064.465761] [ 2936]     0  2936      819      522       7       0        0             0 netifd
 kern.info kernel: [72064.465766] [ 2998]     0  2998      753      394       7       0        0             0 odhcpd
 kern.info kernel: [72064.465771] [ 3106]     0  3106      699      408       7       0        0             0 dropbear
 kern.info kernel: [72064.465776] [ 3112]     0  3112      917       31       6       0        0             0 netserver
 kern.info kernel: [72064.465782] [ 4134]   453  4134      639      430       7       0        0             0 dnsmasq
 kern.info kernel: [72064.465787] [ 4226]     0  4226     1269      382       7       0        0             0 hostapd
 kern.info kernel: [72064.465792] [ 4232]     0  4232     1269      367       8       0        0             0 hostapd
 kern.info kernel: [72064.465797] [ 4456]     0  4456     1328      785       8       0        0             0 uhttpd
 kern.info kernel: [72064.465802] [ 4628]     0  4628      789      465       7       0        0             0 lldpd
 kern.info kernel: [72064.465807] [ 4632]   121  4632      705      357       7       0        0             0 lldpd
 kern.info kernel: [72064.465812] [ 4700]     0  4700     1165      536       7       0        0             0 ntpd
 kern.info kernel: [72064.465817] [ 4786]     0  4786    18795      703      19       0        0             0 collectd
 kern.info kernel: [72064.465823] [25656]     0 25656     1042      215       7       0        0             0 observer
 kern.info kernel: [72064.465828] [25658]     0 25658     1042       42       6       0        0             0 observer
 kern.info kernel: [72064.465833] [25659]     0 25659     1043      103       7       0        0             0 wc
 kern.err kernel: [72064.465836] Out of memory: Kill process 4456 (uhttpd) score 15 or sacrifice child
 kern.err kernel: [72064.473474] Killed process 4456 (uhttpd) total-vm:5312kB, anon-rss:284kB, file-rss:2856kB, shmem-rss:0kB

相关内容