有大量可用交换内存的 Linux OOM

有大量可用交换内存的 Linux OOM

我的服务器会因为内存不足而定期终止 MariaDB。问题是,有大量的交换内存可用,并且交换量设置为 100。无论如何,从内核转储来看,没有任何迹象表明存在任何类型的大量内存使用。MariaDB 在 Docker 容器内运行。这里可能发生了什么?我该如何进一步调试?

注意:我从下面的转储中删除了大多数进程。MariaDB 是迄今为止消耗内存最多的进程。

kernel: [9484097.392673] Purging GPU memory, 0 pages freed, 0 pages still pinned, 1 pages left available.
kernel: [9484097.392877] bash invoked oom-killer: gfp_mask=0x40cc0(GFP_KERNEL|__GFP_COMP), order=3, oom_score_adj=0
kernel: [9484097.392881] CPU: 4 PID: 2030589 Comm: bash Tainted: P            E     5.11.0-27-generic #29~20.04.1-Ubuntu
kernel: [9484097.392884] Hardware name: Dell Inc. OptiPlex 5090/0FXD80, BIOS 1.1.51 08/26/2021
kernel: [9484097.392885] Call Trace:
kernel: [9484097.392887]  dump_stack+0x74/0x92
kernel: [9484097.392891]  dump_header+0x4f/0x1f6
kernel: [9484097.392894]  oom_kill_process.cold+0xb/0x10
kernel: [9484097.392897]  out_of_memory.part.0+0x1ee/0x460
kernel: [9484097.392900]  out_of_memory+0x6d/0xd0
kernel: [9484097.392902]  __alloc_pages_slowpath.constprop.0+0xc4d/0xd20
kernel: [9484097.392905]  __alloc_pages_nodemask+0x2a0/0x300
kernel: [9484097.392907]  alloc_pages_current+0x87/0xe0
kernel: [9484097.392910]  kmalloc_order+0x2d/0xa0
kernel: [9484097.392913]  kmalloc_order_trace+0x1d/0x80
kernel: [9484097.392915]  __kmalloc+0x3f3/0x470
kernel: [9484097.392919]  0xffffffffc0e2a362
kernel: [9484097.392925]  cshook_systemcalltable_pre_compat_sys_ioctl+0x2e6c0/0x37310 [falcon_lsm_serviceable]
kernel: [9484097.392930]  cshook_network_ops_inet6_sockraw_release+0x1ce48/0x208a0 [falcon_lsm_serviceable]
kernel: [9484097.392933]  ? cshook_security_inode_free_security+0x73d8/0x9a10 [falcon_lsm_serviceable]
kernel: [9484097.392936]  cshook_network_ops_inet6_sockraw_release+0x1de32/0x208a0 [falcon_lsm_serviceable]
kernel: [9484097.392940]  ? LoadImageEventSource_GetImageHashMem+0x4bf5/0x8ab0 [falcon_lsm_serviceable]
kernel: [9484097.392944]  cshook_network_ops_inet6_sockraw_release+0x1dfe5/0x208a0 [falcon_lsm_serviceable]
kernel: [9484097.392947]  _ZdlPv+0x3bd26/0x4fd50 [falcon_lsm_serviceable]
kernel: [9484097.392949]  ? _ZdlPv+0x3bb40/0x4fd50 [falcon_lsm_serviceable]
kernel: [9484097.392952]  _ZdlPv+0x3c5de/0x4fd50 [falcon_lsm_serviceable]
kernel: [9484097.392955]  _ZdlPv+0x3c669/0x4fd50 [falcon_lsm_serviceable]
kernel: [9484097.392957]  cshook_systemcalltable_pre_compat_sys_ioctl+0x21582/0x37310 [falcon_lsm_serviceable]
kernel: [9484097.392960]  ? 0xffffffffc0e2b56e
kernel: [9484097.392964]  cshook_security_sb_free_security+0x3f93/0x4540 [falcon_lsm_serviceable]
kernel: [9484097.392968]  cshook_security_sb_free_security+0x3135/0x4540 [falcon_lsm_serviceable]
kernel: [9484097.392973]  cshook_security_sb_free_security+0x3950/0x4540 [falcon_lsm_serviceable]
kernel: [9484097.392977]  cshook_security_file_permission+0x8ca/0x1060 [falcon_lsm_serviceable]
kernel: [9484097.392981]  ? 0xffffffffc0e2a362
kernel: [9484097.392985]  ? cshook_systemcalltable_pre_compat_sys_ioctl+0x2e6f4/0x37310 [falcon_lsm_serviceable]
kernel: [9484097.392988]  cshook_security_file_permission+0xde7/0x1060 [falcon_lsm_serviceable]
kernel: [9484097.392992]  cshook_security_bprm_check_security+0x69/0xe0 [falcon_lsm_serviceable]
kernel: [9484097.392996]  pinnedhook_security_bprm_check_security+0x3c/0x70 [falcon_lsm_pinned_16206]
kernel: [9484097.393001]  security_bprm_check+0x27/0x40
kernel: [9484097.393004]  exec_binprm+0xe7/0x420
kernel: [9484097.393006]  bprm_execve+0x16d/0x290
kernel: [9484097.393008]  do_execveat_common.isra.0+0x189/0x1c0
kernel: [9484097.393011]  __x64_sys_execve+0x37/0x50
kernel: [9484097.393013]  do_syscall_64+0x38/0x90
kernel: [9484097.393016]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
kernel: [9484097.393018] RIP: 0033:0x7f511d5da0fb
kernel: [9484097.393020] Code: f8 01 0f 8e bd fe ff ff 5b 48 8d 3d df 59 13 00 5d 41 5c e9 97 62 fa ff 0f 1f 80 00 00 00 00 f3 0f 1e fa b8 3b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 05 dd 12 00 f7 d8 64 89 01 48
kernel: [9484097.393022] RSP: 002b:00007ffc5e6504e8 EFLAGS: 00000246 ORIG_RAX: 000000000000003b
kernel: [9484097.393024] RAX: ffffffffffffffda RBX: 0000558b0590aff0 RCX: 00007f511d5da0fb
kernel: [9484097.393025] RDX: 0000558b060bd670 RSI: 0000558b060be660 RDI: 0000558b060bd540
kernel: [9484097.393026] RBP: 0000558b060bd540 R08: 0000000000000000 R09: 0000558b060be660
kernel: [9484097.393027] R10: 00007f511d708d10 R11: 0000000000000246 R12: 0000558b060bd540
kernel: [9484097.393028] R13: 0000558b060be660 R14: 0000558b060bd670 R15: 0000558b0590af3c
kernel: [9484097.393030] Mem-Info:
kernel: [9484097.393032] active_anon:32657 inactive_anon:388766 isolated_anon:0
kernel: [9484097.393032]  active_file:236257 inactive_file:954342 isolated_file:0
kernel: [9484097.393032]  unevictable:4660 dirty:2311 writeback:0
kernel: [9484097.393032]  slab_reclaimable:183875 slab_unreclaimable:75067
kernel: [9484097.393032]  mapped:86112 shmem:2143 pagetables:12640 bounce:0
kernel: [9484097.393032]  free:42867 free_pcp:376 free_cma:0
kernel: [9484097.393036] Node 0 active_anon:130628kB inactive_anon:1555064kB active_file:945028kB inactive_file:3817368kB unevictable:18640kB isolated(anon):0kB isolated(file):0kB mapped:344448kB dirty:9244kB writeback:0kB shmem:8572kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:40928kB pagetables:50560kB all_unreclaimable? no
kernel: [9484097.393039] Node 0 DMA free:14884kB min:136kB low:168kB high:200kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
kernel: [9484097.393042] lowmem_reserve[]: 0 1402 7587 7587 7587
kernel: [9484097.393045] Node 0 DMA32 free:72236kB min:12468kB low:15584kB high:18700kB reserved_highatomic:0KB active_anon:11312kB inactive_anon:337820kB active_file:155040kB inactive_file:673224kB unevictable:48kB writepending:588kB present:1589708kB managed:1523848kB mlocked:48kB bounce:0kB free_pcp:1052kB local_pcp:40kB free_cma:0kB
kernel: [9484097.393049] lowmem_reserve[]: 0 0 6184 6184 6184
kernel: [9484097.393052] Node 0 Normal free:84348kB min:54976kB low:68720kB high:82464kB reserved_highatomic:2048KB active_anon:119316kB inactive_anon:1216372kB active_file:789988kB inactive_file:3145876kB unevictable:18592kB writepending:8656kB present:6512640kB managed:6340644kB mlocked:18476kB bounce:0kB free_pcp:452kB local_pcp:0kB free_cma:0kB
kernel: [9484097.393055] lowmem_reserve[]: 0 0 0 0 0
kernel: [9484097.393058] Node 0 DMA: 3*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 1*2048kB (M) 3*4096kB (M) = 14884kB
kernel: [9484097.393068] Node 0 DMA32: 6103*4kB (UE) 1997*8kB (UE) 2008*16kB (UE) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 72516kB
kernel: [9484097.393076] Node 0 Normal: 5120*4kB (UEH) 2433*8kB (UEH) 2892*16kB (UEH) 1*32kB (H) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 86248kB
kernel: [9484097.393085] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
kernel: [9484097.393087] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
kernel: [9484097.393088] 1217872 total pagecache pages
kernel: [9484097.393089] 23192 pages in swap cache
kernel: [9484097.393090] Swap cache stats: add 3011895, delete 2988503, find 103821394/104463504
kernel: [9484097.393091] Free swap  = 3874508kB
kernel: [9484097.393092] Total swap = 4194300kB
kernel: [9484097.393092] 2029585 pages RAM
kernel: [9484097.393093] 0 pages HighMem/MovableOnly
kernel: [9484097.393093] 59485 pages reserved
kernel: [9484097.393094] 0 pages hwpoisoned
kernel: [9484097.393094] Tasks state (memory values in pages):
kernel: [9484097.393095] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
kernel: [9484097.393111] [    687]     0   687      698        0    45056       30             0 falcond
kernel: [9484097.393113] [    688]     0   688   505660     5971   327680     1633             0 falcon-sensor
kernel: [9484097.393137] [    855]     0   855   541254     4791   421888     1144          -999 containerd
kernel: [9484097.393146] [   1283]     0  1283   749675    11418   655360     1711          -500 dockerd
kernel: [9484097.393215] [ 360705]   999 360705  1707021    70258  1396736       21             0 mariadbd
kernel: [9484097.393940] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=2a8e9308e3efb49deadde95237efaeb3f2fc3421ad65a3a5b18750d8d8f7deb5,mems_allowed=0,global_oom,task_memcg=/docker/99fa2d54134f8e550f363b25927323e7711008dda5da43cbb16a7ec9df2802c1,task=mariadbd,pid=360705,uid=999
kernel: [9484097.394218] Out of memory: Killed process 360705 (mariadbd) total-vm:6828084kB, anon-rss:259524kB, file-rss:21508kB, shmem-rss:0kB, UID:999 pgtables:1364kB oom_score_adj:0
kernel: [9484097.407371] oom_reaper: reaped process 360705 (mariadbd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

编辑:根据要求添加有关系统的信息

H/W path             Device       Class          Description
============================================================
                                  system         OptiPlex 7050 (07A1)
/0                                bus            0NW6H5
/0/0                              memory         64KiB BIOS
/0/9                              memory         8GiB System Memory
/0/9/0                            memory         4GiB DIMM DDR4 Synchronous Unbuffered (Unregistered) 2400 MHz (0.4 ns)
/0/9/1                            memory         4GiB DIMM DDR4 Synchronous Unbuffered (Unregistered) 2400 MHz (0.4 ns)
/0/9/2                            memory         [empty]
/0/9/3                            memory         [empty]
/0/f                              memory         256KiB L1 cache
/0/10                             memory         1MiB L2 cache
/0/11                             memory         8MiB L3 cache
/0/12                             processor      Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
/0/100                            bridge         Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers
/0/100/2                          display        HD Graphics 630
/0/100/14                         bus            200 Series/Z370 Chipset Family USB 3.0 xHCI Controller
/0/100/14/0          usb1         bus            xHCI Host Controller
/0/100/14/0/3                     bus            Hub
/0/100/14/0/3/1                   multimedia     USB PnP Audio Device
/0/100/14/0/3/2                   multimedia     USB PnP Audio Device
/0/100/14/0/3/3                   multimedia     USB PnP Audio Device
/0/100/14/0/3/4                   multimedia     USB PnP Audio Device
/0/100/14/0/3/5                   bus            Hub
/0/100/14/0/3/5/1                 multimedia     USB PnP Audio Device
/0/100/14/0/3/5/2                 multimedia     USB PnP Audio Device
/0/100/14/0/3/5/3                 multimedia     USB PnP Audio Device
/0/100/14/0/3/5/4                 multimedia     USB PnP Audio Device
/0/100/14/0/3/5/5                 bus            Hub
/0/100/14/0/3/5/5/1               multimedia     USB PnP Audio Device
/0/100/14/0/3/5/5/2               multimedia     USB PnP Audio Device
/0/100/14/0/3/5/5/3               multimedia     USB PnP Audio Device
/0/100/14/0/3/5/5/4               multimedia     USB PnP Audio Device
/0/100/14/0/3/5/5/5               generic        FT232R USB UART
/0/100/14/0/3/5/5/6               multimedia     USB PnP Audio Device
/0/100/14/0/3/5/5/7               multimedia     USB PnP Audio Device
/0/100/14/0/3/5/6                 multimedia     USB PnP Audio Device
/0/100/14/0/3/5/7                 multimedia     USB PnP Audio Device
/0/100/14/1          usb2         bus            xHCI Host Controller
/0/100/14.2                       generic        200 Series PCH Thermal Subsystem
/0/100/15                         generic        200 Series PCH Serial IO I2C Controller #0
/0/100/16                         communication  200 Series PCH CSME HECI #1
/0/100/17                         storage        SATA Controller [RAID mode]
/0/100/1d                         bridge         200 Series PCH PCI Express Root Port #9
/0/100/1f                         bridge         200 Series PCH LPC Controller (Q270)
/0/100/1f.2                       memory         Memory controller
/0/100/1f.4                       bus            200 Series/Z370 Chipset Family SMBus Controller
/0/100/1f.6          eth0         network        Ethernet Connection (5) I219-LM
/0/1                              system         PnP device PNP0c02
/0/2                              communication  PnP device PNP0501
/0/3                              input          PnP device PNP0303
/0/4                              input          PnP device PNP0f03
/0/5                              system         PnP device PNP0c02
/0/6                              system         PnP device PNP0c02
/0/7                              system         PnP device PNP0b00
/0/8                              generic        PnP device INT3f0d
/0/a                              system         PnP device PNP0c02
/0/b                              system         PnP device PNP0c02
/0/c                              system         PnP device PNP0c02
/0/d                              system         PnP device PNP0c02
/0/e                 scsi0        storage        
/0/e/0.0.0           /dev/sda     disk           1TB ST1000DM010-2EP1
/0/e/0.0.0/1         /dev/sda1    volume         511MiB Windows FAT volume
/0/e/0.0.0/2         /dev/sda2    volume         931GiB EXT4 volume
/0/13                scsi1        storage        
/0/13/0.0.0          /dev/cdrom   disk           DVD+-RW DU-8A5LH
/1                   veth2e771c6  network        Ethernet interface
/2                   veth7cb2db4  network        Ethernet interface
/3                   vethf37e821  network        Ethernet interface
/4                   vethd70ba99  network        Ethernet interface
/5                   veth2bcd630  network        Ethernet interface
/6                   vethd242ecd  network        Ethernet interface
/7                   veth7ed6a4b  network        Ethernet interface

Distributor ID: Ubuntu
Description:    Ubuntu-Server 20.04.3 2021.09.15 customized using Cubic on 2021-09-15 14:33
Release:        20.04
Codename:       focal

相关内容