我遇到了 oom_killer 活动，无法理解dmesg输出
[Fri Jan 11 02:11:48 2019] lowmem_reserve[]: 0 2811 64249 64249
[Fri Jan 11 02:11:48 2019] Node 0 DMA32 free:248704kB min:2956kB low:3692kB high:4432kB active_anon:2031516kB inactive_anon:524376kB active_file:160kB inactive_file:100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2881076kB mlocked:0kB dirty:0kB writeback:0kB mapped:2564kB shmem:56560kB slab_reclaimable:44392kB slab_unreclaimable:9560kB kernel_stack:3904kB pagetables:9288kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:427 all_unreclaimable? yes
[Fri Jan 11 02:11:48 2019] lowmem_reserve[]: 0 0 61437 61437
[Fri Jan 11 02:11:48 2019] Node 0 Normal free:64472kB min:64608kB low:80760kB high:96912kB active_anon:59250028kB inactive_anon:2533808kB active_file:5060kB inactive_file:8272kB unevictable:0kB isolated(anon):0kB isolated(file):352kB present:63963136kB managed:62912504kB mlocked:0kB dirty:0kB writeback:4kB mapped:72292kB shmem:1480340kB slab_reclaimable:245132kB slab_unreclaimable:99196kB kernel_stack:29440kB pagetables:217044kB unstable:0kB bounce:0kB free_pcp:188kB local_pcp:188kB free_cma:0kB writeback_tmp:0kB pages_scanned:37993 all_unreclaimable? yes
[Fri Jan 11 02:11:48 2019] lowmem_reserve[]: 0 0 0 0
[Fri Jan 11 02:11:48 2019] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15860kB
[Fri Jan 11 02:11:48 2019] Node 0 DMA32: 723*4kB (UEM) 544*8kB (UEM) 728*16kB (UEM) 820*32kB (UEM) 714*64kB (UEM) 441*128kB (UEM) 233*256kB (UEM) 64*512kB (UEM) 9*1024kB (UEM) 0*2048kB 0*4096kB = 248908kB
[Fri Jan 11 02:11:48 2019] Node 0 Normal: 12711*4kB (UEM) 1791*8kB (UEM) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 65188kB
[Fri Jan 11 02:11:48 2019] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Fri Jan 11 02:11:48 2019] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Fri Jan 11 02:11:48 2019] 420932 total pagecache pages
[Fri Jan 11 02:11:48 2019] 33002 pages in swap cache
[Fri Jan 11 02:11:48 2019] Swap cache stats: add 1061688, delete 1028686, find 21081912/21097467
[Fri Jan 11 02:11:48 2019] Free swap = 0kB
[Fri Jan 11 02:11:48 2019] Total swap = 2097148kB
[Fri Jan 11 02:11:48 2019] 16777086 pages RAM
[Fri Jan 11 02:11:48 2019] 0 pages HighMem/MovableOnly
[Fri Jan 11 02:11:48 2019] 324714 pages reserved
[Fri Jan 11 02:11:48 2019] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[Fri Jan 11 02:11:48 2019] [ 633] 0 633 46199 30241 97 47 0 systemd-journal
[Fri Jan 11 02:11:48 2019] [ 675] 0 675 12641 143 26 40 -1000 systemd-udevd
[Fri Jan 11 02:11:48 2019] [ 821] 0 821 119828 16 36 147 0 lvmetad
[Fri Jan 11 02:11:48 2019] [ 1315] 0 1315 14423 35 28 83 -1000 auditd
[Fri Jan 11 02:11:48 2019] [ 1318] 0 1318 5927 0 16 55 0 rpc.idmapd
[Fri Jan 11 02:11:48 2019] [ 1339] 169 1339 3768 8 13 36 0 lsmd
[Fri Jan 11 02:11:48 2019] [ 1340] 81 1340 7766 64 21 56 -900 dbus-daemon
[Fri Jan 11 02:11:48 2019] [ 1341] 0 1341 25997 84 45 323 0 VGAuthService
[Fri Jan 11 02:11:48 2019] [ 1342] 0 1342 58916 119 60 238 0 vmtoolsd
[Fri Jan 11 02:11:48 2019] [ 1343] 0 1343 1618 23 9 19 0 rngd
[Fri Jan 11 02:11:48 2019] [ 1344] 0 1344 5520 28 17 40 0 systemd-logind
[Fri Jan 11 02:11:48 2019] [ 1345] 997 1345 135183 304 62 1078 0 polkitd
[Fri Jan 11 02:11:48 2019] [ 1347] 0 1347 63153 417 30 86 0 rsyslogd
[Fri Jan 11 02:11:48 2019] [ 1350] 0 1350 5409 43 15 41 0 irqbalance
[Fri Jan 11 02:11:48 2019] [ 1391] 0 1391 67265 16 43 639 0 gssproxy
[Fri Jan 11 02:11:48 2019] [ 1408] 38 1408 8570 51 22 110 0 ntpd
[Fri Jan 11 02:11:48 2019] [ 1471] 0 1471 10296 2 23 100 0 rpc.gssd
[Fri Jan 11 02:11:48 2019] [ 1483] 0 1483 8099 5 21 56 0 atd
[Fri Jan 11 02:11:48 2019] [ 1486] 0 1486 33193 35 22 132 0 crond
[Fri Jan 11 02:11:48 2019] [ 1489] 0 1489 29146 2 14 38 0 agetty
[Fri Jan 11 02:11:48 2019] [ 1622] 0 1622 141697 171 96 2507 0 tuned
[Fri Jan 11 02:11:48 2019] [ 1624] 0 1624 72433 412 96 3258 0 salt-minion
[Fri Jan 11 02:11:48 2019] [ 1629] 0 1629 49494 0 52 288 0 cupsd
[Fri Jan 11 02:11:48 2019] [ 1632] 29 1632 10594 2 25 206 0 rpc.statd
[Fri Jan 11 02:11:48 2019] [ 1638] 0 1638 28910 22 14 19 0 rhsmcertd
[Fri Jan 11 02:11:48 2019] [ 1639] 0 1639 27059 27 55 220 -1000 sshd
[Fri Jan 11 02:11:48 2019] [ 1649] 0 1649 6248 0 16 58 0 xinetd
[Fri Jan 11 02:11:48 2019] [ 1671] 32 1671 16265 55 36 96 0 rpcbind
[Fri Jan 11 02:11:48 2019] [ 1701] 0 1701 10655 110 23 107 0 rpc.mountd
[Fri Jan 11 02:11:48 2019] [ 1710] 0 1710 42838 110 30 141 0 pbx_exchange
[Fri Jan 11 02:11:48 2019] [ 2118] 0 2118 46898 84 37 95 0 vnetd
[Fri Jan 11 02:11:48 2019] [ 2398] 0 2398 37270 1007 49 92 0 adclient
[Fri Jan 11 02:11:48 2019] [ 2399] 0 2399 24721 38 19 81 0 cdcwatch
[Fri Jan 11 02:11:48 2019] [ 2547] 0 2547 54892 113 50 191 0 bpcd
[Fri Jan 11 02:11:48 2019] [ 2688] 0 2688 22946 81 44 182 0 master
[Fri Jan 11 02:11:48 2019] [ 2714] 89 2714 23017 38 48 230 0 qmgr
[Fri Jan 11 02:11:48 2019] [ 3250] 0 3250 189579 431 122 906 0 nbdisco
[Fri Jan 11 02:11:48 2019] [ 3723] 0 3723 195675 8308 156 1030 0 salt-minion
[Fri Jan 11 02:11:48 2019] [ 4057] 0 4057 22743 13 14 38 0 nqmagt
[Fri Jan 11 02:11:48 2019] [ 4062] 0 4062 135901 1057 46 750 0 agent
[Fri Jan 11 02:11:48 2019] [ 4069] 0 4069 469383 1598 119 3347 0 VigilEntAgent
[Fri Jan 11 02:11:48 2019] [ 4337] 0 4337 102252 564 112 4317 0 salt-minion
[Fri Jan 11 02:11:48 2019] [23762] 0 23762 4095 229 7 74 0 symcfgd
[Fri Jan 11 02:11:48 2019] [23795] 0 23795 90053 82831 186 682 0 rtvscand
[Fri Jan 11 02:11:48 2019] [23866] 0 23866 37105 2054 43 3179 0 smcd
[Fri Jan 11 02:11:48 2019] [29541] 57317 29541 37128 4 31 75 0 ps
[Fri Jan 11 02:11:48 2019] [ 2842] 57317 2842 37128 5 29 74 0 ps
[Fri Jan 11 02:11:48 2019] [30378] 57317 30378 37128 11 31 67 0 ps
[Fri Jan 11 02:11:48 2019] [27560] 57317 27560 28312 3 13 68 0 bash
[Fri Jan 11 02:11:48 2019] [30652] 57317 30652 28284 5 14 59 0 bash
[Fri Jan 11 02:11:48 2019] [30665] 57317 30665 2599863 271425 886 63297 0 java
[Fri Jan 11 02:11:48 2019] [ 4753] 57317 4753 28313 10 13 61 0 bash
[Fri Jan 11 02:11:48 2019] [ 7977] 57317 7977 28285 8 14 57 0 bash
[Fri Jan 11 02:11:48 2019] [ 7990] 57317 7990 2599863 327013 930 32865 0 java
[Fri Jan 11 02:11:48 2019] [ 8440] 57317 8440 28313 11 12 62 0 bash
[Fri Jan 11 02:11:48 2019] [ 8829] 57317 8829 28285 12 12 54 0 bash
[Fri Jan 11 02:11:48 2019] [ 8842] 57317 8842 2599863 299812 952 72150 0 java
[Fri Jan 11 02:11:48 2019] [ 8451] 57317 8451 37128 79 31 0 0 ps
[Fri Jan 11 02:11:48 2019] [ 2876] 57317 2876 37128 79 29 0 0 ps
[Fri Jan 11 02:11:48 2019] [19806] 57317 19806 37129 79 29 0 0 ps
[Fri Jan 11 02:11:48 2019] [24141] 57317 24141 37129 78 30 0 0 ps
[Fri Jan 11 02:11:48 2019] [22801] 57317 22801 37128 79 30 0 0 ps
[Fri Jan 11 02:11:48 2019] [20694] 57317 20694 37128 79 30 0 0 ps
[Fri Jan 11 02:11:48 2019] [13828] 57317 13828 37128 78 29 0 0 ps
[Fri Jan 11 02:11:48 2019] [ 3884] 57317 3884 37129 79 29 0 0 ps
[Fri Jan 11 02:11:48 2019] [16585] 57317 16585 37128 79 31 0 0 ps
[Fri Jan 11 02:11:48 2019] [10773] 57317 10773 37128 79 31 0 0 ps
[Fri Jan 11 02:11:48 2019] [ 984] 57317 984 37129 78 29 0 0 ps
[Fri Jan 11 02:11:48 2019] [26228] 57317 26228 37128 79 30 0 0 ps
[Fri Jan 11 02:11:48 2019] [28863] 57316 28863 16283613 4609499 17059 0 0 java
[Fri Jan 11 02:11:48 2019] [14614] 57317 14614 37128 78 28 0 0 ps
[Fri Jan 11 02:11:48 2019] [22044] 57317 22044 37128 79 31 0 0 ps
[Fri Jan 11 02:11:48 2019] [28610] 57317 28610 37129 79 30 0 0 ps
[Fri Jan 11 02:11:48 2019] [ 2186] 57317 2186 37129 79 29 0 0 ps
[Fri Jan 11 02:11:48 2019] [14078] 57317 14078 4485 77 10 0 0 wrapper
[Fri Jan 11 02:11:48 2019] [14080] 57317 14080 1598728 119038 475 0 0 java
[Fri Jan 11 02:11:48 2019] [14158] 57317 14158 4492 79 10 0 0 wrapper
[Fri Jan 11 02:11:48 2019] [14160] 57317 14160 1594484 143688 488 0 0 java
[Fri Jan 11 02:11:48 2019] [14238] 57317 14238 4492 77 10 0 0 wrapper
[Fri Jan 11 02:11:48 2019] [14240] 57317 14240 1599042 163554 517 0 0 java
[Fri Jan 11 02:11:48 2019] [14322] 57317 14322 4492 80 10 0 0 wrapper
[Fri Jan 11 02:11:48 2019] [14324] 57317 14324 1598785 158818 529 0 0 java
[Fri Jan 11 02:11:48 2019] [14402] 57317 14402 4492 77 10 0 0 wrapper
[Fri Jan 11 02:11:48 2019] [14405] 57317 14405 1598785 156120 511 0 0 java
[Fri Jan 11 02:11:48 2019] [23366] 0 23366 50567 756 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [23367] 0 23367 14342 101 33 0 0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23368] 0 23368 14342 103 31 0 0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23369] 0 23369 14342 96 31 0 0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23370] 0 23370 14342 101 30 0 0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23371] 0 23371 14342 102 32 0 0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23372] 0 23372 14342 102 33 0 0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23373] 0 23373 14342 102 32 0 0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23374] 0 23374 14342 104 30 0 0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23375] 0 23375 14342 103 31 0 0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23376] 0 23376 14342 103 32 0 0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23377] 0 23377 14342 102 31 0 0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23378] 0 23378 14342 102 33 0 0 rotatelogs
[Fri Jan 11 02:11:48 2019] [ 5477] 57316 5477 7630798 2313242 9166 0 0 java
[Fri Jan 11 02:11:48 2019] [ 2224] 9618 2224 52716 795 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 8329] 9618 8329 52716 795 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [10808] 9618 10808 69100 797 99 0 0 httpd
[Fri Jan 11 02:11:48 2019] [15676] 57316 15676 8922598 1626467 4977 0 0 java
[Fri Jan 11 02:11:48 2019] [30309] 89 30309 22972 254 45 0 0 pickup
[Fri Jan 11 02:11:48 2019] [30581] 57317 30581 5204496 687312 1524 0 0 java
[Fri Jan 11 02:11:48 2019] [30690] 57317 30690 5027787 649120 1415 0 0 java
[Fri Jan 11 02:11:48 2019] [31537] 57317 31537 5204498 775719 1715 0 0 java
[Fri Jan 11 02:11:48 2019] [32658] 57317 32658 5171025 650715 1415 0 0 java
[Fri Jan 11 02:11:48 2019] [ 2292] 57316 2292 7069828 1488690 3871 0 0 java
[Fri Jan 11 02:11:48 2019] [ 2322] 57317 2322 5203973 644654 1405 0 0 java
[Fri Jan 11 02:11:48 2019] [ 3794] 57317 3794 28284 63 12 0 0 sh
[Fri Jan 11 02:11:48 2019] [ 3799] 57317 3799 1072293 165518 445 0 0 java
[Fri Jan 11 02:11:48 2019] [ 4291] 9618 4291 52716 790 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 4310] 9618 4310 52716 787 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 4620] 9618 4620 52716 787 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 4721] 9618 4721 52716 788 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 4847] 9618 4847 52716 789 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 4867] 9618 4867 52716 788 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 5021] 9618 5021 52716 786 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 5072] 9618 5072 52716 789 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 5075] 9618 5075 69100 788 99 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 6618] 9618 6618 52716 788 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 6623] 9618 6623 52716 788 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 6758] 9618 6758 69092 788 100 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 6760] 9618 6760 52716 787 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 6917] 9618 6917 52716 787 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 7003] 9618 7003 52716 788 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 7787] 9618 7787 52716 787 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 7822] 9618 7822 52716 787 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 7834] 9618 7834 52716 786 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 7837] 9618 7837 52716 787 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 7940] 9618 7940 52716 786 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 7972] 9618 7972 52716 786 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 7974] 9618 7974 52716 786 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 7975] 9618 7975 52716 786 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 8215] 9618 8215 52716 778 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 8219] 9618 8219 52716 778 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 8314] 9618 8314 52716 778 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 8360] 9618 8360 69092 778 100 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 8361] 9618 8361 69092 780 102 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 8362] 9618 8362 52716 778 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 8363] 9618 8363 69092 779 100 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 8428] 9618 8428 69100 779 99 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 8429] 9618 8429 52716 780 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 8478] 9618 8478 52716 778 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 8482] 9618 8482 52716 778 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 8575] 9618 8575 52716 778 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 8626] 9618 8626 52716 778 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 8627] 9618 8627 52716 780 98 0 0 httpd
[Fri Jan 11 02:11:48 2019] [ 9707] 57317 9707 28284 60 13 0 0 sh
[Fri Jan 11 02:11:48 2019] [ 9710] 57317 9710 5013780 105123 317 0 0 java
[Fri Jan 11 02:11:48 2019] [ 9771] 57317 9771 28284 61 14 0 0 sh
[Fri Jan 11 02:11:48 2019] [ 9775] 57317 9775 5012346 98369 304 0 0 java
[Fri Jan 11 02:11:48 2019] [ 9820] 57317 9820 28284 60 12 0 0 sh
[Fri Jan 11 02:11:48 2019] [ 9822] 57317 9822 5010296 86077 270 0 0 java
[Fri Jan 11 02:11:48 2019] [ 9859] 57317 9859 37228 89 29 0 0 ps
[Fri Jan 11 02:11:48 2019] [ 9860] 57317 9860 11215 75 25 0 0 ps
[Fri Jan 11 02:11:48 2019] [ 9861] 57317 9861 28276 39 13 0 0 sh
[Fri Jan 11 02:11:48 2019] Out of memory: Kill process 28863 (java) score 272 or sacrifice child
[Fri Jan 11 02:11:48 2019] Killed process 28863 (java) total-vm:65134452kB, anon-rss:18437996kB, file-rss:0kB, shmem-rss:0kB
我们对 PID 28863 感兴趣
那么为什么表格中和汇总行中显示的数值不同？
[Fri Jan 11 02:11:48 2019] [28863] 57316 28863 16283613 4609499 17059 0 0 java
[Fri Jan 11 02:11:48 2019] Out of memory: Kill process 28863 (java) score 272 or sacrifice child
[Fri Jan 11 02:11:48 2019] Killed process 28863 (java) total-vm:65134452kB, anon-rss:18437996kB, file-rss:0kB, shmem-rss:0kB
表格显示虚拟内存为 16283613，而摘要显示为 65134452kB。为什么？
答案1
这是因为表格上显示的数字是页数，而摘要行中显示的是实际大小。将页数 (16283613) 乘以页大小 (4k) 即可得出结果。