分析

分析

我对 OOM 终止程序日志有疑问。我们预计会有大量 OOM 终止程序。

生态系统

我的生态系统如下所示:

4 cores我有一台带有和 的服务器8 GB of RAM。我在那里运行着以下最耗资源的程序:

  1. PostgreSQL 数据库
  2. 两个带有进程的应用程序称为vega——由 Go 代码编译的本机二进制文件。

我无法将应用程序与 PostgreSQL 隔离,因为应用程序使用 PostgreSQL 作为内部存储并且必须共享文件系统。

OOM 日志

我有以下日志syslog

Dec 21 00:43:18 n00.testnet.vega.xyz kernel: vega invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: CPU: 0 PID: 742994 Comm: vega Not tainted 5.4.0-122-generic #138-Ubuntu
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Hardware name: DigitalOcean Droplet/Droplet, BIOS 20171212 12/12/2017
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Call Trace:
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  dump_stack+0x6d/0x8b
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  dump_header+0x4f/0x1eb
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  oom_kill_process.cold+0xb/0x10
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  out_of_memory+0x1cf/0x4d0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  __alloc_pages_slowpath+0xd5e/0xe50
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  __alloc_pages_nodemask+0x2d0/0x320
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  alloc_pages_current+0x87/0xe0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  __page_cache_alloc+0x72/0x90
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  pagecache_get_page+0xbf/0x300
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  filemap_fault+0x6b2/0xa50
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  ? unlock_page_memcg+0x12/0x20
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  ? page_add_file_rmap+0xff/0x1a0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  ? xas_load+0xd/0x80
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  ? xas_find+0x17f/0x1c0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  ? filemap_map_pages+0x24c/0x380
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  ext4_filemap_fault+0x32/0x50
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  __do_fault+0x3c/0x170
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  do_fault+0x24b/0x640
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  __handle_mm_fault+0x4c5/0x7a0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  handle_mm_fault+0xca/0x200
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  do_user_addr_fault+0x1f9/0x450
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  __do_page_fault+0x58/0x90
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  ? exit_to_usermode_loop+0xbf/0x160
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  do_page_fault+0x2c/0xe0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  do_async_page_fault+0x39/0x70
Dec 21 00:43:18 n00.testnet.vega.xyz kernel:  async_page_fault+0x34/0x40
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: RIP: 0033:0x458f17
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Code: Bad RIP value.
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: RSP: 002b:000000c03e2c5340 EFLAGS: 00010212
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: RAX: 00000000016e4eb6 RBX: 0000000000000000 RCX: 000000000552ee80
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: RDX: 000000000000f32b RSI: 0000000000019db8 RDI: 0000000004b89160
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: RBP: 000000c03e2c5350 R08: 000000000000f32c R09: 0000000004c02ac0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: R10: 0000000001c0a9e5 R11: 00000000051b6ab8 R12: 0000000000000000
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: R13: 0000000000000000 R14: 000000c03e2b4340 R15: 0000000000000000
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Mem-Info:
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: active_anon:1919323 inactive_anon:1905 isolated_anon:0
                                              active_file:172 inactive_file:73 isolated_file:0
                                              unevictable:4617 dirty:4 writeback:0 unstable:0
                                              slab_reclaimable:35806 slab_unreclaimable:25291
                                              mapped:526606 shmem:524808 pagetables:9410 bounce:0
                                              free:25658 free_pcp:1747 free_cma:0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 active_anon:7677292kB inactive_anon:7620kB active_file:688kB inactive_file:292kB unevictable:18468kB isolated(anon):0kB isolated(file):0kB mapped:2106424kB dirty:16kB writeback:0kB shmem:2099232kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 1214464kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 DMA free:15908kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: lowmem_reserve[]: 0 2965 7907 7907 7907
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 DMA32 free:44580kB min:25292kB low:31612kB high:37932kB active_anon:2954028kB inactive_anon:1292kB active_file:0kB inactive_file:888kB unevictable:0kB writepending:8kB present:3129172kB managed:3063636kB mlocked:0kB kernel_stack:432kB pagetables:11840kB bounce:0kB free_pcp:3164kB local_pcp:728kB free_cma:0kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: lowmem_reserve[]: 0 0 4942 4942 4942
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 Normal free:42144kB min:42156kB low:52692kB high:63228kB active_anon:4723264kB inactive_anon:6328kB active_file:1324kB inactive_file:500kB unevictable:18468kB writepending:8kB present:5242880kB managed:5068608kB mlocked:18468kB kernel_stack:4784kB pagetables:25800kB bounce:0kB free_pcp:3824kB local_pcp:976kB free_cma:0kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: lowmem_reserve[]: 0 0 0 0 0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 DMA32: 554*4kB (UME) 483*8kB (UME) 831*16kB (UME) 318*32kB (UME) 129*64kB (UME) 33*128kB (UE) 3*256kB (UM) 0*512kB 2*1024kB (UM) 0*2048kB 0*4096kB = 44848kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 Normal: 1590*4kB (UMEH) 1296*8kB (UMEH) 955*16kB (UMEH) 296*32kB (UMEH) 7*64kB (MH) 3*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 42312kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: 527191 total pagecache pages
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: 0 pages in swap cache
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Swap cache stats: add 0, delete 0, find 0/0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Free swap  = 0kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Total swap = 0kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: 2097011 pages RAM
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: 0 pages HighMem/MovableOnly
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: 59973 pages reserved
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: 0 pages cma reserved
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: 0 pages hwpoisoned
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Tasks state (memory values in pages):
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    378]     0   378    86394      867   663552        0          -250 systemd-journal
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    536]     0   536    70033     4486    94208        0         -1000 multipathd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    643]   102   643    22718      531    81920        0             0 systemd-timesyn
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    679]   100   679     4796      730    77824        0             0 systemd-network
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    698]   101   698     6207     1775    90112        0             0 systemd-resolve
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    722]     0   722     5040      746    61440        0         -1000 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    827]     0   827    61160     1228   106496        0             0 accounts-daemon
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    835]     0   835     2134      560    53248        0             0 cron
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    836]   103   836     2003      991    61440        0          -900 dbus-daemon
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    842]     0   842   324985      236   172032        0          -900 droplet-agent
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    847]     0   847    20481      761    57344        0             0 irqbalance
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    849]     0   849     7469     2819    94208        0             0 networkd-dispat
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    852]     0   852    59077      794   106496        0             0 polkitd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    859]   104   859    56086      526    86016        0             0 rsyslogd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    865]     0   865   292647     4440   303104        0          -900 snapd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    868]     0   868     4395     1039    77824        0             0 systemd-logind
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    871]     0   871    98871      525   131072        0             0 udisksd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    872]     0   872      948      504    45056        0             0 atd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    873]     0   873      622      143    45056        0             0 none
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    874]     0   874   375917     4672   311296        0          -999 containerd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    888]     0   888     1838      402    53248        0             0 agetty
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    911]     0   911    79705      516   126976        0             0 ModemManager
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    912]     0   912     1457      387    45056        0             0 agetty
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    917]     0   917      652      118    45056        0             0 sshguard
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    919]     0   919      652       24    45056        0             0 sshguard
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    920]     0   920     1034      330    45056        0             0 sshg-parser
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    921]     0   921    19538      806    53248        0             0 sshg-blocker
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    922]     0   922      652       24    45056        0             0 sshguard
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    923]     0   923      652      443    45056        0             0 sshg-fw-iptable
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    924]     0   924   159700      971   741376        0             0 journalctl
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    942]     0   942    27031     2689   102400        0             0 unattended-upgr
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [    948]     0   948     3043      729    61440        0         -1000 sshd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 553186]     0 553186   361527     5005   356352        0          -500 dockerd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 631350]  1002 631350     4762      823    69632        0             0 systemd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 631357]  1002 631357    43209     1272   110592        0             0 (sd-pam)
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 631443]  1002 631443     2219      455    49152        0             0 screen
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 631444]  1002 631444     2606      999    57344        0             0 bash
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 633355]  1002 633355   195674    22178   282624        0             0 metrics-collect
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 664318]     0 664318   750436    17304  2252800        0             0 grafana-agent
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670974]   112 670974   543687    15391   262144        0          -900 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670976]   112 670976   543938   520398  4308992        0             0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670977]   112 670977   543720   516982  4304896        0             0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670978]   112 670978   543687     4689   180224        0             0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670979]   112 670979   543855      865   180224        0             0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670980]   112 670980    18189      615   139264        0             0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670981]   112 670981   543794      781   172032        0             0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670982]   112 670982   543792      776   159744        0             0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670983]   112 670983   544464     2378   204800        0             0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 671211]  1020 671211   183655     2671   159744        0             0 visor
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 671236]  1020 671236   784480   495752  5017600        0             0 vega
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 671237]  1020 671237  1107523   811623  7364608        0             0 vega
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 671310]   995 671310   188400     4246   200704        0             0 caddy
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 671387]   112 671387   544345     7955   290816        0             0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 671406]   112 671406   546181   525444  4345856        0             0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 671434]   112 671434   544577   514269  4337664        0             0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 671506]   112 671506   565894    10351   462848        0             0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743022]     0 743022     5040      684    57344        0             0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743023]     0 743023     5040      684    57344        0             0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743024]     0 743024     5040      684    57344        0             0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743025]     0 743025     5040      661    57344        0             0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743026]     0 743026     5040      661    57344        0             0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743027]     0 743027     5040      477    57344        0             0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743028]     0 743028     5040      661    57344        0             0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743029]     0 743029     5040      661    57344        0             0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743030]     0 743030     5040      658    57344        0             0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743033]     0 743033     5040      668    57344        0             0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743034]     0 743034     5040      668    57344        0             0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743035]     0 743035     5040      668    57344        0             0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743039]     0 743039     5040      758    57344        0             0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743040]     0 743040     5040      758    57344        0             0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743047]     0 743047     3043      821    61440        0             0 sshd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743049]   109 743049     3043      563    57344        0             0 sshd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/vegavisor.service,task=vega,pid=671237,uid=1020
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Out of memory: Killed process 671237 (vega) total-vm:4430092kB, anon-rss:3246492kB, file-rss:0kB, shmem-rss:0kB, UID:1020 pgtables:7192kB oom_score_adj:0

Grafana 图表

grafana 内存使用情况

分析

我对其进行了分析,以下是我的发现:

内存碎片

Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 Normal: 1590*4kB (UMEH) 1296*8kB (UMEH) 955*16kB (UMEH) 296*32kB (UMEH) 7*64kB (MH) 3*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 42312kB

这一行看上去没有问题,因为我们仍然有一些块continuous memory,但是可用内存量大约为42Mb

释放普通空间中的内存

 Node 0 Normal free:42144kB min:42156kB low:52692kB high:63228kB active_anon:4723264kB inactive_anon:6328kB active_file:1324kB inactive_file:500kB unevictable:18468kB writepending:8kB present:5242880kB managed:5068608kB mlocked:18468kB kernel_stack:4784kB pagetables:25800kB bounce:0kB free_pcp:3824kB local_pcp:976kB free_cma:0kB

the free memory(42144kB)据我所知,由于低于,因此调用了 OOM 杀手min threshold(42156kB)

进程表转储

然后我切换到进程使用的内存并编写了简单的脚本:

#!/bin/python
import re

results = []

page_size_bytes = 4096
idx_total_vm = 9
idx_rss = 10
idx_name = -2

gigabyte = 1024**3 # bytes

with open("/home/daniel/tmp/oom-dump-21-12-2022.txt") as file:
    for line in file.readlines():
        line_parts = re.split('\s+', line)

        try:
            results.append((line_parts[idx_name], int(line_parts[idx_rss])*page_size_bytes, int(line_parts[idx_total_vm])*page_size_bytes))
        except:
            continue

results.sort(key= lambda item: item[1], reverse=True)

print("%15s : %9s, %9s" % ("process name", "rss", "vsz"))
for row in results:
    print("%15s : %3f GB , %3f GB" % (row[0], row[1]/gigabyte, row[2]/gigabyte))


total_rss = sum([row[1] for row in results])
total_vsz = sum([row[2] for row in results])
print("%15s : %3f GB , %3f GB" % ("total", total_rss/gigabyte, total_vsz/gigabyte))

它给出了以下结果:

   process name :       rss,       vsz
           vega : 3.096096 GB , 4.224865 GB
       postgres : 2.004410 GB , 2.083515 GB
       postgres : 1.985161 GB , 2.074959 GB
       postgres : 1.972130 GB , 2.074127 GB
       postgres : 1.961781 GB , 2.077396 GB
           vega : 1.891144 GB , 2.992554 GB
metrics-collect : 0.084602 GB , 0.746437 GB
  grafana-agent : 0.066010 GB , 2.862686 GB
       postgres : 0.058712 GB , 2.074001 GB
       postgres : 0.039486 GB , 2.158714 GB
       postgres : 0.030346 GB , 2.076511 GB
        dockerd : 0.019093 GB , 1.379116 GB
       postgres : 0.017887 GB , 2.074001 GB
     containerd : 0.017822 GB , 1.434010 GB
     multipathd : 0.017113 GB , 0.267155 GB
          snapd : 0.016937 GB , 1.116360 GB
          caddy : 0.016197 GB , 0.718689 GB
networkd-dispat : 0.010754 GB , 0.028492 GB
unattended-upgr : 0.010258 GB , 0.103115 GB
          visor : 0.010189 GB , 0.700588 GB
       postgres : 0.009071 GB , 2.076965 GB
systemd-resolve : 0.006771 GB , 0.023678 GB
       (sd-pam) : 0.004852 GB , 0.164829 GB
accounts-daemon : 0.004684 GB , 0.233307 GB
 systemd-logind : 0.003963 GB , 0.016766 GB
           bash : 0.003811 GB , 0.009941 GB
    dbus-daemon : 0.003780 GB , 0.007641 GB
     journalctl : 0.003704 GB , 0.609207 GB
systemd-journal : 0.003307 GB , 0.329567 GB
       postgres : 0.003300 GB , 2.074642 GB
        systemd : 0.003139 GB , 0.018166 GB
           sshd : 0.003132 GB , 0.011608 GB
   sshg-blocker : 0.003075 GB , 0.074532 GB
        polkitd : 0.003029 GB , 0.225361 GB
       postgres : 0.002979 GB , 2.074409 GB
       postgres : 0.002960 GB , 2.074402 GB
     irqbalance : 0.002903 GB , 0.078129 GB
  systemd-udevd : 0.002892 GB , 0.019226 GB
  systemd-udevd : 0.002892 GB , 0.019226 GB
  systemd-udevd : 0.002846 GB , 0.019226 GB
systemd-network : 0.002785 GB , 0.018295 GB
           sshd : 0.002781 GB , 0.011608 GB
  systemd-udevd : 0.002609 GB , 0.019226 GB
  systemd-udevd : 0.002609 GB , 0.019226 GB
  systemd-udevd : 0.002609 GB , 0.019226 GB
  systemd-udevd : 0.002548 GB , 0.019226 GB
  systemd-udevd : 0.002548 GB , 0.019226 GB
  systemd-udevd : 0.002548 GB , 0.019226 GB
  systemd-udevd : 0.002522 GB , 0.019226 GB
  systemd-udevd : 0.002522 GB , 0.019226 GB
  systemd-udevd : 0.002522 GB , 0.019226 GB
  systemd-udevd : 0.002522 GB , 0.019226 GB
  systemd-udevd : 0.002510 GB , 0.019226 GB
       postgres : 0.002346 GB , 0.069386 GB
              0 : 0.002148 GB , 0.011608 GB
           cron : 0.002136 GB , 0.008141 GB
systemd-timesyn : 0.002026 GB , 0.086662 GB
       rsyslogd : 0.002007 GB , 0.213951 GB
        udisksd : 0.002003 GB , 0.377163 GB
   ModemManager : 0.001968 GB , 0.304050 GB
            atd : 0.001923 GB , 0.003616 GB
  systemd-udevd : 0.001820 GB , 0.019226 GB
         screen : 0.001736 GB , 0.008465 GB
sshg-fw-iptable : 0.001690 GB , 0.002487 GB
         agetty : 0.001534 GB , 0.007011 GB
         agetty : 0.001476 GB , 0.005558 GB
    sshg-parser : 0.001259 GB , 0.003944 GB
  droplet-agent : 0.000900 GB , 1.239719 GB
           none : 0.000546 GB , 0.002373 GB
       sshguard : 0.000450 GB , 0.002487 GB
       sshguard : 0.000092 GB , 0.002487 GB
       sshguard : 0.000092 GB , 0.002487 GB
          total : 13.465000 GB , 46.040333 GB

问题

我的第一个问题是:

13.46GB为什么我可以看到系统上的总内存使用情况 (RSS) 8GB?RSS 是否包括共享库/内存?

假设我有 2GB 的 PostgreSQL 共享内存;那么我的使用量将约为 8GB。这样对吗?

我的第二个问题是:

我在 grafana 中有一个间隙,因为我每 60 秒收集一次指标,所以它不会显示峰值,除非峰值是 Prometheus 抓取指标时出现的。

您是否知道一些导出器或工具可以将 Prometheus 内存分配指标导出为计数器?因此,每次内存分配都会增加指标的值,然后我可以计算rate分配的值或increase value every given time period

或者也许您可以推荐一些其他的东西来显示我的监控差距带来的异常分配?

相关内容