LXC libvirt 内存 cgroup 内存不足

LXC libvirt 内存 cgroup 内存不足

我正在使用 LXC 和 virt-manager (libvirt) 来托管 Bind DNS 服务器,分配了 512MB RAM。虚拟机位于 ZFS 文件系统上(Linux 上的 ZFS)

最近我遇到了内存不足的情况,但我找不到哪个进程消耗了内存。在日志中,大多数内存消耗inactive_anon超过 300MB,并且active_anon

有人能帮我找出导致这种情况的原因吗?如何释放该内存

我不认为重新启动 LXC VM 是一个解决方案。

看一下第二个代码示例: inactive_anon:216272KB active_anon:266004KB

环境:

Linux 0.6.5.7-8-jessie 上的 ZFS compression:lz4 dedup:off

Debian 8.6 Jessie x86_64,英特尔奔腾 G4400,16GB RAM

Linux 版本 4.3.0-0.bpo.1-amd64 ([电子邮件保护]) (gcc 版本 4.9.2 (Debian 4.9.2-10) ) #1 SMP Debian 4.3.3-7~bpo8+1 (2016-01-19)

Memory cgroup out of memory: Kill process 3273 (named) score 113 or sacrifice child
Killed process 3273 (named) total-vm:282176kB, anon-rss:59172kB, file-rss:0kB
mandb invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
mandb cpuset=machine-lxc\x2d03_dns.scope mems_allowed=0
CPU: 1 PID: 27944 Comm: mandb Tainted: P        W  O    4.3.0-0.bpo.1-amd64 #1 Debian 4.3.3-7~bpo8+1
Hardware name: System manufacturer System Product Name/H110M-A, BIOS 0501 11/16/2015
 0000000000000000 000000009e067aef ffffffff812e1889 ffff8801917b7e40
 ffffffff811ccbba ffff880409835700 ffff880436d13280 ffff880100000000
 0000000000015d80 ffffffff81163b12 ffff88003697f1b0 000000009e067aef
Call Trace:
 [<ffffffff812e1889>] ? dump_stack+0x40/0x57
 [<ffffffff811ccbba>] ? dump_header+0xa7/0x23b
 [<ffffffff81163b12>] ? find_lock_task_mm+0x52/0xc0
 [<ffffffff811640ab>] ? oom_kill_process+0x1fb/0x400
 [<ffffffff810fd37f>] ? css_next_descendant_pre+0x1f/0x50
 [<ffffffff811c5320>] ? mem_cgroup_iter+0x1b0/0x3b0
 [<ffffffff811c740f>] ? mem_cgroup_out_of_memory+0x28f/0x2d0
 [<ffffffff811c7fcd>] ? mem_cgroup_oom_synchronize+0x32d/0x340
 [<ffffffff811c38d0>] ? mem_cgroup_begin_page_stat+0x90/0x90
 [<ffffffff81164804>] ? pagefault_out_of_memory+0x44/0xc0
 [<ffffffff8158c5b8>] ? page_fault+0x28/0x30
Task in /machine.slice/machine-lxc\x2d03_dns.scope killed as a result of limit of /machine.slice/machine-lxc\x2d03_dns.scope
memory: usage 524288kB, limit 524288kB, failcnt 48845
memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
Memory cgroup stats for /machine.slice/machine-lxc\x2d03_dns.scope: cache:413924KB rss:110364KB rss_huge:0KB mapped_file:7860KB dirty:0KB writeback:0KB inactive_anon:187496KB active_anon:336768KB inactive_file:0KB active_file:0KB unevictable:0KB
[ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[ 6048]     0  6048    41741      999      47       3        0             0 libvirt_lxc
[ 6049]     0  6049     6967      290      19       3        0             0 systemd
[ 6071]     0  6071     8242     2032      22       3        0             0 systemd-journal
[ 6147]     0  6147    13795      168      31       3        0         -1000 sshd
[ 6148]     0  6148     6476       60      17       3        0             0 cron
[ 6150]     0  6150    65721      354      30       3        0             0 rsyslogd
[ 6154]     0  6154     3166       39      11       3        0             0 agetty
[27938]     0 27938    10191       86      24       3        0             0 cron
[27939]     0 27939     1084       21       8       3        0             0 sh
[27940]     0 27940     1056       19       7       3        0             0 run-parts
[27941]     0 27941     1084       21       7       3        0             0 man-db
[27944]     6 27944    37727    26396      79       3        0             0 mandb



Memory cgroup out of memory: Kill process 15882 (mandb) score 80 or sacrifice child
Killed process 15882 (mandb) total-vm:99416kB, anon-rss:42008kB, file-rss:8kB
apt-get invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
apt-get cpuset=machine-lxc\x2d03_dns.scope mems_allowed=0
CPU: 0 PID: 15849 Comm: apt-get Tainted: P        W  O    4.3.0-0.bpo.1-amd64 #1 Debian 4.3.3-7~bpo8+1
Hardware name: System manufacturer System Product Name/H110M-A, BIOS 0501 11/16/2015
 0000000000000000 0000000026367b10 ffffffff812e1889 ffff880108f63e40
 ffffffff811ccbba ffffffff81111d60 0000000000000000 0000000026367b10
 0000000000000000 ffffffff81163b12 ffff88011e372ab0 0000000026367b10
Call Trace:
 [<ffffffff812e1889>] ? dump_stack+0x40/0x57
 [<ffffffff811ccbba>] ? dump_header+0xa7/0x23b
 [<ffffffff81111d60>] ? kprobe_flush_task+0x90/0x120
 [<ffffffff81163b12>] ? find_lock_task_mm+0x52/0xc0
 [<ffffffff811640ab>] ? oom_kill_process+0x1fb/0x400
 [<ffffffff810fd37f>] ? css_next_descendant_pre+0x1f/0x50
 [<ffffffff811c5320>] ? mem_cgroup_iter+0x1b0/0x3b0
 [<ffffffff811c740f>] ? mem_cgroup_out_of_memory+0x28f/0x2d0
 [<ffffffff811c7fcd>] ? mem_cgroup_oom_synchronize+0x32d/0x340
 [<ffffffff811c38d0>] ? mem_cgroup_begin_page_stat+0x90/0x90
 [<ffffffff81164804>] ? pagefault_out_of_memory+0x44/0xc0
 [<ffffffff8158c5b8>] ? page_fault+0x28/0x30
Task in /machine.slice/machine-lxc\x2d03_dns.scope killed as a result of limit of /machine.slice/machine-lxc\x2d03_dns.scope
memory: usage 482284kB, limit 524288kB, failcnt 82266
memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
Memory cgroup stats for /machine.slice/machine-lxc\x2d03_dns.scope: cache:463692KB rss:18592KB rss_huge:0KB mapped_file:15208KB dirty:0KB writeback:0KB inactive_anon:216272KB active_anon:266004KB inactive_file:0KB active_file:4KB unevictable:0KB
[ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[ 6048]     0  6048    41741     1000      47       3        0             0 libvirt_lxc
[ 6049]     0  6049     6967      291      19       3        0             0 systemd
[ 6147]     0  6147    13795      168      31       3        0         -1000 sshd
[ 6148]     0  6148     6476       59      17       3        0             0 cron
[ 6150]     0  6150    65721      354      30       3        0             0 rsyslogd
[ 6154]     0  6154     3166       39      11       3        0             0 agetty
[25585]     0 25585    12338     5113      29       3        0             0 systemd-journal
[15728]     0 15728    20680      262      43       3        0             0 sshd
[15730]     0 15730     5064      124      15       3        0             0 bash
[15756]     0 15756    20647      277      44       3        0             0 sshd
[15760]     0 15760     5064       78      15       3        0             0 bash
[15849]     0 15849    16558     1418      38       3        0             0 apt-get
[15869]     0 15869     6901     1485      19       3        0             0 dpkg
[15881]     0 15881     1084       21       8       3        0             0 man-db.postinst
Memory cgroup out of memory: Kill process 25585 (systemd-journal) score 38 or sacrifice child
Killed process 25585 (systemd-journal) total-vm:49352kB, anon-rss:252kB, file-rss:20200kB

在虚拟机中完成一些工作之后

Memory cgroup out of memory: Kill process 15882 (mandb) score 80 or sacrifice child
Killed process 15882 (mandb) total-vm:99416kB, anon-rss:42008kB, file-rss:8kB
apt-get invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
apt-get cpuset=machine-lxc\x2d03_dns.scope mems_allowed=0
CPU: 0 PID: 15849 Comm: apt-get Tainted: P        W  O    4.3.0-0.bpo.1-amd64 #1 Debian 4.3.3-7~bpo8+1
Hardware name: System manufacturer System Product Name/H110M-A, BIOS 0501 11/16/2015
 0000000000000000 0000000026367b10 ffffffff812e1889 ffff880108f63e40
 ffffffff811ccbba ffffffff81111d60 0000000000000000 0000000026367b10
 0000000000000000 ffffffff81163b12 ffff88011e372ab0 0000000026367b10
Call Trace:
 [<ffffffff812e1889>] ? dump_stack+0x40/0x57
 [<ffffffff811ccbba>] ? dump_header+0xa7/0x23b
 [<ffffffff81111d60>] ? kprobe_flush_task+0x90/0x120
 [<ffffffff81163b12>] ? find_lock_task_mm+0x52/0xc0
 [<ffffffff811640ab>] ? oom_kill_process+0x1fb/0x400
 [<ffffffff810fd37f>] ? css_next_descendant_pre+0x1f/0x50
 [<ffffffff811c5320>] ? mem_cgroup_iter+0x1b0/0x3b0
 [<ffffffff811c740f>] ? mem_cgroup_out_of_memory+0x28f/0x2d0
 [<ffffffff811c7fcd>] ? mem_cgroup_oom_synchronize+0x32d/0x340
 [<ffffffff811c38d0>] ? mem_cgroup_begin_page_stat+0x90/0x90
 [<ffffffff81164804>] ? pagefault_out_of_memory+0x44/0xc0
 [<ffffffff8158c5b8>] ? page_fault+0x28/0x30
Task in /machine.slice/machine-lxc\x2d03_dns.scope killed as a result of limit of /machine.slice/machine-lxc\x2d03_dns.scope
memory: usage 482284kB, limit 524288kB, failcnt 82266
memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
Memory cgroup stats for /machine.slice/machine-lxc\x2d03_dns.scope: cache:463692KB rss:18592KB rss_huge:0KB mapped_file:15208KB dirty:0KB writeback:0KB inactive_anon:216272KB active_anon:266004KB inactive_file:0KB active_file:4KB unevictable:0KB
[ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[ 6048]     0  6048    41741     1000      47       3        0             0 libvirt_lxc
[ 6049]     0  6049     6967      291      19       3        0             0 systemd
[ 6147]     0  6147    13795      168      31       3        0         -1000 sshd
[ 6148]     0  6148     6476       59      17       3        0             0 cron
[ 6150]     0  6150    65721      354      30       3        0             0 rsyslogd
[ 6154]     0  6154     3166       39      11       3        0             0 agetty
[25585]     0 25585    12338     5113      29       3        0             0 systemd-journal
[15728]     0 15728    20680      262      43       3        0             0 sshd
[15730]     0 15730     5064      124      15       3        0             0 bash
[15756]     0 15756    20647      277      44       3        0             0 sshd
[15760]     0 15760     5064       78      15       3        0             0 bash
[15849]     0 15849    16558     1418      38       3        0             0 apt-get
[15869]     0 15869     6901     1485      19       3        0             0 dpkg
[15881]     0 15881     1084       21       8       3        0             0 man-db.postinst
Memory cgroup out of memory: Kill process 25585 (systemd-journal) score 38 or sacrifice child
Killed process 25585 (systemd-journal) total-vm:49352kB, anon-rss:252kB, file-rss:20200kB

相关内容