我正在运行 Debian 8 Jessie 开发服务器。16GB 内存。
我看过关于 oom-killer 的其他帖子，但没有一个与我在这里问的问题真正相关
我正在尝试追踪导致oom-killer事件发生的进程。有一个内存转储，但没有列标题 - 所以我不知道这些数字代表什么意思。我没有通过 Google 搜索找到此信息。
有几行看起来像是潜在的问题，但我不知道是否能够解释这些信息。
我特别好奇这是什么-999 docker-containe意思......
倾倒
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.996382] sshd invoked oom-killer: gfp_mask=0x2000d0, order=2, oom_score_adj=-1000
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.996458] sshd cpuset=/ mems_allowed=0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.996589] CPU: 5 PID: 943 Comm: sshd Tainted: G C 3.16.0-6-amd64 #1 Debian 3.16.56-1+deb8u1
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.996661] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997200] 0000000000000000 ffffffff81533b49 00000000002000d0 0000000000000000
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997417] ffffffff815326ab ffffffff81537f8e 0000000000000200 ffffffff8106b413
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997634] ffffffff810c91a4 ffffffff8115c7f3 ffff8800083ea210 ffffffff8172faca
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997849] Call Trace:
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997905] [<ffffffff81533b49>] ? dump_stack+0x5d/0x78
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997962] [<ffffffff815326ab>] ? dump_header+0x95/0x1fd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998019] [<ffffffff81537f8e>] ? mutex_lock+0xe/0x30
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998078] [<ffffffff8106b413>] ? put_online_cpus+0x23/0x90
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998135] [<ffffffff810c91a4>] ? rcu_oom_notify+0xc4/0xe0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998195] [<ffffffff8115c7f3>] ? do_try_to_free_pages+0x483/0x520
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998254] [<ffffffff8114a5fd>] ? oom_kill_process+0x21d/0x370
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998312] [<ffffffff8114a19d>] ? find_lock_task_mm+0x3d/0xa0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998369] [<ffffffff8114adbe>] ? out_of_memory+0x4be/0x4f0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998427] [<ffffffff81151776>] ? __alloc_pages_nodemask+0xca6/0xcf0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998486] [<ffffffff8106875a>] ? copy_process.part.25+0x11a/0x1ae0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998544] [<ffffffff8106a2d3>] ? do_fork+0xc3/0x400
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998601] [<ffffffff8105a4cb>] ? __do_page_fault+0x1ab/0x470
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998659] [<ffffffff81539ef9>] ? stub_clone+0x69/0x90
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998715] [<ffffffff81539b5c>] ? system_call_fast_compare_end+0x1c/0x21
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998783] Mem-Info:
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079179] [20093] 1006 20093 306152 5057 107 1873 0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079247] [20120] 0 20120 13507 38 32 73 0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079315] [20123] 1006 20123 1085 7 8 15 0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079383] [20124] 1006 20124 306153 4858 110 2030 0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079452] [20557] 0 20557 13507 91 32 20 0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079520] [20560] 1006 20560 1085 22 8 0 0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079588] [20562] 1006 20562 306109 4393 110 2423 0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079656] [21447] 0 21447 13507 93 32 18 0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079725] [21450] 1006 21450 1085 23 8 0 0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079793] [21452] 1006 21452 305668 6278 108 261 0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079861] [21489] 0 21489 13507 93 32 18 0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079929] [21492] 1006 21492 1085 22 8 0 0 sh
(snip ... a bazillion node, sh, cron)
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176046] [23607] 0 23607 23318 237 49 0 0 sshd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176114] [ 2963] 1010 2963 8982 203 23 0 0 systemd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176183] [ 3198] 1010 3198 49655 573 31 154 0 (sd-pam)
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176251] [ 3843] 1010 3843 23318 240 46 0 0 sshd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176319] [ 4188] 1010 4188 6510 867 18 0 0 bash
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176387] [ 6721] 0 6721 13507 97 32 14 0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176455] [28824] 0 28824 23319 237 50 0 0 sshd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176523] [ 9005] 110 9005 9560 142 21 0 0 pickup
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176591] [ 9898] 1006 9898 1085 22 8 0 0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176659] [ 9900] 1006 9900 300400 6495 94 0 0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176727] [21244] 1010 21244 23319 242 47 0 0 sshd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176795] [22039] 1010 22039 3177 42 12 0 0 sftp-server
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176864] [31202] 0 31202 23318 238 48 0 0 sshd
(snip ... a bazillion node, sh, cron)
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.182092] [13192] 0 13192 1849 209 8 0 -999 docker-containe
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.182163] [13208] 0 13208 352255 84126 319 0 0 zcashd
(snip ... a bazillion node, sh, cron)
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184048] [13334] 0 13334 1085 30 8 0 0 sessionclean
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184118] [13335] 1006 13335 11718 53 19 0 0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184185] [13336] 0 13336 9884 49 18 0 0 php
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184255] [13342] 0 13342 1085 30 8 0 0 sessionclean
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184324] [13344] 0 13344 6447 29 11 0 0 sort
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184392] [13345] 0 13345 6447 29 11 0 0 sort
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184461] [13346] 0 13346 1085 28 8 0 0 sessionclean
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184530] [13347] 0 13347 12216 139 28 0 0 sshd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184599] [13362] 0 13362 11068 74 19 0 0 docker
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184668] [13363] 0 13363 1085 28 8 0 0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184736] [13364] 1006 13364 1085 28 8 0 0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184806] [13365] 0 13365 8808 55 23 25 0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184875] [13366] 0 13366 8808 55 23 25 0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184945] [13372] 111 13372 5874 67 16 76 0 nrpe
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.185015] [13373] 0 13373 9884 49 18 0 0 php7.2
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.185083] [13374] 1006 13374 11728 197 15 0 0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.185153] [13375] 0 13375 9884 48 20 0 0 php
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.185221] [13378] 0 13378 7185 33 18 39 0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.185289] [13380] 0 13380 7185 33 18 39 0 cron
答案1
使用源代码！在 Debian 源代码中搜索短语“invoked oom-killer”会找到 mm/oom_kill.c它有需要打印的 9 列标题。
* Dumps the current memory state of all eligible tasks. Tasks not in the same
* memcg, not in the same cpuset, or bound to a disjoint set of mempolicy nodes
* are not shown.
* State information includes task's pid, uid, tgid, vm size, rss, nr_ptes,
* swapents, oom_score_adj value, and name.
*/
static void dump_tasks(const struct mem_cgroup *memcg, const nodemask_t *nodemask)
{
struct task_struct *p;
struct task_struct *task;
pr_info("[ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name\n");
这已在 UNIX SE 上讨论过，请参阅问题使用 /var/log/messages 调试内存不足问题
-999 之所以引人注目，是因为该进程（名称为 docker）是这组行中唯一一个 oom 分数被调低的进程。（在我看来，这似乎是一个贪婪的 init 脚本。）但它的常驻内存（rss）很小，所以它可能不是一个很大的内存消耗者。
还要查看Killed process行数以及它的大小。OOM 会终止释放最多内存资源的任务。
监控系统范围内的内存消耗，并确定主机是否有足够的内存。基本操作是/proc/meminfo随时间捕获并计算任务数量。监控工具将有助于收集数据。如果您没有这样的工具，请考虑从网络数据的可用内存、每个用户的内存以及进程计数图表。
您有相当多的东西在运行，cron、php、node 和 docker...验证任何应该运行的计划任务和应用程序是否运行正常。