我在我的服务器上发现了一个奇怪的东西。使用的交换空间明显高于总交换空间。
free -m
total used free shared buffers cached
Mem: 1869 1788 80 71 35 487
-/+ buffers/cache: 1265 603
Swap: 2046 1759218603 9146
显示swapon -s交换大小为 2.0 GB：
swapon -s
Filename Type Size Used Priority
/dev/sdb1 partition 2096124 46584 -1
以下是主要交换消费者的信息：
for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r |head -n 10
scanner 11988 kB
scanner 11944 kB
scanner 11936 kB
scanner 11932 kB
php-fpm 4428 kB
log 2920 kB
snmpd 2700 kB
cma 1996 kB
cma 1996 kB
cma 1996 kB
记忆信息：
cat /proc/meminfo
MemTotal: 1914596 kB
MemFree: 672796 kB
Buffers: 19100 kB
Cached: 387924 kB
SwapCached: 5348 kB
Active: 470276 kB
Inactive: 446516 kB
Active(anon): 310168 kB
Inactive(anon): 278548 kB
Active(file): 160108 kB
Inactive(file): 167968 kB
Unevictable: 3896 kB
Mlocked: 3896 kB
SwapTotal: 2096124 kB
SwapFree: 9384400 kB
Dirty: 48 kB
Writeback: 0 kB
AnonPages: 510888 kB
Mapped: 44264 kB
Shmem: 76232 kB
Slab: 261036 kB
SReclaimable: 217824 kB
SUnreclaim: 43212 kB
KernelStack: 4272 kB
PageTables: 14600 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3053420 kB
Committed_AS: 1391372 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 281060 kB
VmallocChunk: 34359452940 kB
HardwareCorrupted: 0 kB
AnonHugePages: 172032 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 8192 kB
DirectMap2M: 2088960 kB
[编辑] 正如这里所问的，free没有 -m 参数和uname -a
free
total used free shared buffers cached
Mem: 1914596 1741152 173444 75800 8740 377320
-/+ buffers/cache: 1355092 559504
Swap: 2096124 1801439850 10696044
uname -a
Linux myserver 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu Aug 13 22:55:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
有关信息我还在我的 dmesg 中发现了这一点：
Out of memory: Kill process 28579 (php-fpm) score 137 or sacrifice child
Killed process 28579, UID 516, (php-fpm) total-vm:1439832kB, anon-rss:68204kB, file-rss:4072kB
hpet1: lost 1 rtc interrupts
vmtoolsd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
vmtoolsd cpuset=/ mems_allowed=0
Pid: 2024, comm: vmtoolsd Tainted: G W -- ------------ 2.6.32-573.3.1.el6.x86_64 #1
Call Trace:
[<ffffffff810d6dd1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
[<ffffffff8112a5d0>] ? dump_header+0x90/0x1b0
[<ffffffff81232cbc>] ? security_real_capable_noaudit+0x3c/0x70
[<ffffffff8112aa52>] ? oom_kill_process+0x82/0x2a0
[<ffffffff8112a991>] ? select_bad_process+0xe1/0x120
[<ffffffff8112ae90>] ? out_of_memory+0x220/0x3c0
[<ffffffff8113786c>] ? __alloc_pages_nodemask+0x93c/0x950
[<ffffffff810f5709>] ? delayacct_end+0x89/0xa0
[<ffffffff811275f0>] ? sync_page_killable+0x0/0x40
[<ffffffff8117035a>] ? alloc_pages_vma+0x9a/0x150
[<ffffffff81163882>] ? read_swap_cache_async+0xf2/0x160
[<ffffffff811643d9>] ? valid_swaphandles+0x69/0x160
[<ffffffff81163977>] ? swapin_readahead+0x87/0xc0
[<ffffffff81152abd>] ? handle_pte_fault+0x6dd/0xb20
[<ffffffff811b45c0>] ? mntput_no_expire+0x30/0x110
[<ffffffff81153199>] ? handle_mm_fault+0x299/0x3d0
[<ffffffff8119fd7b>] ? putname+0x2b/0x40
[<ffffffff81155bb6>] ? find_vma+0x46/0x80
[<ffffffff8104f156>] ? __do_page_fault+0x146/0x500
[<ffffffff81290271>] ? cpumask_any_but+0x31/0x50
[<ffffffff81197750>] ? vfs_fstatat+0x50/0xa0
[<ffffffff81014a19>] ? read_tsc+0x9/0x10
[<ffffffff8106343d>] ? thread_group_times+0x3d/0x120
[<ffffffff811978cb>] ? vfs_stat+0x1b/0x20
[<ffffffff8153e95e>] ? do_page_fault+0x3e/0xa0
[<ffffffff8153bd05>] ? page_fault+0x25/0x30
Mem-Info:
Node 0 DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
CPU 2: hi: 0, btch: 1 usd: 0
CPU 3: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 110
CPU 1: hi: 186, btch: 31 usd: 62
CPU 2: hi: 186, btch: 31 usd: 48
CPU 3: hi: 186, btch: 31 usd: 78
active_anon:322906 inactive_anon:108241 isolated_anon:480
active_file:1427 inactive_file:1298 isolated_file:0
unevictable:994 dirty:0 writeback:1 unstable:0
free:13249 slab_reclaimable:2892 slab_unreclaimable:9700
mapped:3333 shmem:10113 pagetables:6259 bounce:0
Node 0 DMA free:8352kB min:332kB low:412kB high:496kB active_anon:3544kB inactive_anon:3652kB active_file:0kB inactive_file:8kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15300kB mlocked:0kB dirty:0kB writeback:0kB mapped:8kB shmem:56kB slab_reclaimable:16kB slab_unreclaimable:36kB kernel_stack:0kB pagetables:52kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11333 all_unreclaimable? yes
lowmem_reserve[]: 0 2004 2004 2004
Node 0 DMA32 free:44644kB min:44720kB low:55900kB high:67080kB active_anon:1288080kB inactive_anon:429184kB active_file:5708kB inactive_file:5184kB unevictable:3976kB isolated(anon):2048kB isolated(file):0kB present:2052192kB mlocked:3976kB dirty:0kB writeback:4kB mapped:13324kB shmem:40396kB slab_reclaimable:11552kB slab_unreclaimable:38764kB kernel_stack:4480kB pagetables:24984kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2629632 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 12*4kB 14*8kB 6*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8352kB
Node 0 DMA32: 435*4kB 377*8kB 269*16kB 344*32kB 142*64kB 39*128kB 23*256kB 3*512kB 1*1024kB 1*2048kB 0*4096kB = 44644kB
13751 total pagecache pages
196 pages in swap cache
Swap cache stats: add 35252293, delete 35252097, find 25799626/28229091
Free swap = 8734852kB
Total swap = 2096124kB
524272 pages RAM
45687 pages reserved
34271 pages shared
456140 pages non-shared
怎么会这样？是不是服务器出了什么问题？
答案1
解决方案是升级到 kernel-2.6.32-573.7.1.el6 或更高版本。
正如所提到的内核错误修复更新和解决方案文章做yum update了工作
下面是 kernel-2.6.32-573.1.1.el6.x86_64 的错误报告摘录，该错误导致 Swap free 大于 swap total：
get_swap_page() 锁定中的先前更改删除了 swap_lock 自旋锁的使用。这可能会导致 nr_swap_pages 损坏和 /proc/meminfo 文件中的 SwapFree 信息无效，其中 SwapFree 的大小可能超过 SwapTotal 的大小。此更新对 nr_swap_pages 使用原子变量，并且 /proc/meminfo 中的 SwapFree 大小现在正确。（BZ#1259362）
答案2
看起来这个问题已经在 ServerFault 上被提及了，网址为“免费报告虚假的交换使用量“Red Hat 在其（非公开）勘误表跟踪器中也对此进行了描述https://access.redhat.com/solutions/1571043。我无法访问 Red Hat 的完整解决方案文本，因此无法向您提供更多详细信息。