我的服务器会因为内存不足而定期终止 MariaDB。问题是,有大量的交换内存可用,并且交换量设置为 100。无论如何,从内核转储来看,没有任何迹象表明存在任何类型的大量内存使用。MariaDB 在 Docker 容器内运行。这里可能发生了什么?我该如何进一步调试?
注意:我从下面的转储中删除了大多数进程。MariaDB 是迄今为止消耗内存最多的进程。
kernel: [9484097.392673] Purging GPU memory, 0 pages freed, 0 pages still pinned, 1 pages left available.
kernel: [9484097.392877] bash invoked oom-killer: gfp_mask=0x40cc0(GFP_KERNEL|__GFP_COMP), order=3, oom_score_adj=0
kernel: [9484097.392881] CPU: 4 PID: 2030589 Comm: bash Tainted: P E 5.11.0-27-generic #29~20.04.1-Ubuntu
kernel: [9484097.392884] Hardware name: Dell Inc. OptiPlex 5090/0FXD80, BIOS 1.1.51 08/26/2021
kernel: [9484097.392885] Call Trace:
kernel: [9484097.392887] dump_stack+0x74/0x92
kernel: [9484097.392891] dump_header+0x4f/0x1f6
kernel: [9484097.392894] oom_kill_process.cold+0xb/0x10
kernel: [9484097.392897] out_of_memory.part.0+0x1ee/0x460
kernel: [9484097.392900] out_of_memory+0x6d/0xd0
kernel: [9484097.392902] __alloc_pages_slowpath.constprop.0+0xc4d/0xd20
kernel: [9484097.392905] __alloc_pages_nodemask+0x2a0/0x300
kernel: [9484097.392907] alloc_pages_current+0x87/0xe0
kernel: [9484097.392910] kmalloc_order+0x2d/0xa0
kernel: [9484097.392913] kmalloc_order_trace+0x1d/0x80
kernel: [9484097.392915] __kmalloc+0x3f3/0x470
kernel: [9484097.392919] 0xffffffffc0e2a362
kernel: [9484097.392925] cshook_systemcalltable_pre_compat_sys_ioctl+0x2e6c0/0x37310 [falcon_lsm_serviceable]
kernel: [9484097.392930] cshook_network_ops_inet6_sockraw_release+0x1ce48/0x208a0 [falcon_lsm_serviceable]
kernel: [9484097.392933] ? cshook_security_inode_free_security+0x73d8/0x9a10 [falcon_lsm_serviceable]
kernel: [9484097.392936] cshook_network_ops_inet6_sockraw_release+0x1de32/0x208a0 [falcon_lsm_serviceable]
kernel: [9484097.392940] ? LoadImageEventSource_GetImageHashMem+0x4bf5/0x8ab0 [falcon_lsm_serviceable]
kernel: [9484097.392944] cshook_network_ops_inet6_sockraw_release+0x1dfe5/0x208a0 [falcon_lsm_serviceable]
kernel: [9484097.392947] _ZdlPv+0x3bd26/0x4fd50 [falcon_lsm_serviceable]
kernel: [9484097.392949] ? _ZdlPv+0x3bb40/0x4fd50 [falcon_lsm_serviceable]
kernel: [9484097.392952] _ZdlPv+0x3c5de/0x4fd50 [falcon_lsm_serviceable]
kernel: [9484097.392955] _ZdlPv+0x3c669/0x4fd50 [falcon_lsm_serviceable]
kernel: [9484097.392957] cshook_systemcalltable_pre_compat_sys_ioctl+0x21582/0x37310 [falcon_lsm_serviceable]
kernel: [9484097.392960] ? 0xffffffffc0e2b56e
kernel: [9484097.392964] cshook_security_sb_free_security+0x3f93/0x4540 [falcon_lsm_serviceable]
kernel: [9484097.392968] cshook_security_sb_free_security+0x3135/0x4540 [falcon_lsm_serviceable]
kernel: [9484097.392973] cshook_security_sb_free_security+0x3950/0x4540 [falcon_lsm_serviceable]
kernel: [9484097.392977] cshook_security_file_permission+0x8ca/0x1060 [falcon_lsm_serviceable]
kernel: [9484097.392981] ? 0xffffffffc0e2a362
kernel: [9484097.392985] ? cshook_systemcalltable_pre_compat_sys_ioctl+0x2e6f4/0x37310 [falcon_lsm_serviceable]
kernel: [9484097.392988] cshook_security_file_permission+0xde7/0x1060 [falcon_lsm_serviceable]
kernel: [9484097.392992] cshook_security_bprm_check_security+0x69/0xe0 [falcon_lsm_serviceable]
kernel: [9484097.392996] pinnedhook_security_bprm_check_security+0x3c/0x70 [falcon_lsm_pinned_16206]
kernel: [9484097.393001] security_bprm_check+0x27/0x40
kernel: [9484097.393004] exec_binprm+0xe7/0x420
kernel: [9484097.393006] bprm_execve+0x16d/0x290
kernel: [9484097.393008] do_execveat_common.isra.0+0x189/0x1c0
kernel: [9484097.393011] __x64_sys_execve+0x37/0x50
kernel: [9484097.393013] do_syscall_64+0x38/0x90
kernel: [9484097.393016] entry_SYSCALL_64_after_hwframe+0x44/0xa9
kernel: [9484097.393018] RIP: 0033:0x7f511d5da0fb
kernel: [9484097.393020] Code: f8 01 0f 8e bd fe ff ff 5b 48 8d 3d df 59 13 00 5d 41 5c e9 97 62 fa ff 0f 1f 80 00 00 00 00 f3 0f 1e fa b8 3b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 05 dd 12 00 f7 d8 64 89 01 48
kernel: [9484097.393022] RSP: 002b:00007ffc5e6504e8 EFLAGS: 00000246 ORIG_RAX: 000000000000003b
kernel: [9484097.393024] RAX: ffffffffffffffda RBX: 0000558b0590aff0 RCX: 00007f511d5da0fb
kernel: [9484097.393025] RDX: 0000558b060bd670 RSI: 0000558b060be660 RDI: 0000558b060bd540
kernel: [9484097.393026] RBP: 0000558b060bd540 R08: 0000000000000000 R09: 0000558b060be660
kernel: [9484097.393027] R10: 00007f511d708d10 R11: 0000000000000246 R12: 0000558b060bd540
kernel: [9484097.393028] R13: 0000558b060be660 R14: 0000558b060bd670 R15: 0000558b0590af3c
kernel: [9484097.393030] Mem-Info:
kernel: [9484097.393032] active_anon:32657 inactive_anon:388766 isolated_anon:0
kernel: [9484097.393032] active_file:236257 inactive_file:954342 isolated_file:0
kernel: [9484097.393032] unevictable:4660 dirty:2311 writeback:0
kernel: [9484097.393032] slab_reclaimable:183875 slab_unreclaimable:75067
kernel: [9484097.393032] mapped:86112 shmem:2143 pagetables:12640 bounce:0
kernel: [9484097.393032] free:42867 free_pcp:376 free_cma:0
kernel: [9484097.393036] Node 0 active_anon:130628kB inactive_anon:1555064kB active_file:945028kB inactive_file:3817368kB unevictable:18640kB isolated(anon):0kB isolated(file):0kB mapped:344448kB dirty:9244kB writeback:0kB shmem:8572kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:40928kB pagetables:50560kB all_unreclaimable? no
kernel: [9484097.393039] Node 0 DMA free:14884kB min:136kB low:168kB high:200kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
kernel: [9484097.393042] lowmem_reserve[]: 0 1402 7587 7587 7587
kernel: [9484097.393045] Node 0 DMA32 free:72236kB min:12468kB low:15584kB high:18700kB reserved_highatomic:0KB active_anon:11312kB inactive_anon:337820kB active_file:155040kB inactive_file:673224kB unevictable:48kB writepending:588kB present:1589708kB managed:1523848kB mlocked:48kB bounce:0kB free_pcp:1052kB local_pcp:40kB free_cma:0kB
kernel: [9484097.393049] lowmem_reserve[]: 0 0 6184 6184 6184
kernel: [9484097.393052] Node 0 Normal free:84348kB min:54976kB low:68720kB high:82464kB reserved_highatomic:2048KB active_anon:119316kB inactive_anon:1216372kB active_file:789988kB inactive_file:3145876kB unevictable:18592kB writepending:8656kB present:6512640kB managed:6340644kB mlocked:18476kB bounce:0kB free_pcp:452kB local_pcp:0kB free_cma:0kB
kernel: [9484097.393055] lowmem_reserve[]: 0 0 0 0 0
kernel: [9484097.393058] Node 0 DMA: 3*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 1*2048kB (M) 3*4096kB (M) = 14884kB
kernel: [9484097.393068] Node 0 DMA32: 6103*4kB (UE) 1997*8kB (UE) 2008*16kB (UE) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 72516kB
kernel: [9484097.393076] Node 0 Normal: 5120*4kB (UEH) 2433*8kB (UEH) 2892*16kB (UEH) 1*32kB (H) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 86248kB
kernel: [9484097.393085] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
kernel: [9484097.393087] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
kernel: [9484097.393088] 1217872 total pagecache pages
kernel: [9484097.393089] 23192 pages in swap cache
kernel: [9484097.393090] Swap cache stats: add 3011895, delete 2988503, find 103821394/104463504
kernel: [9484097.393091] Free swap = 3874508kB
kernel: [9484097.393092] Total swap = 4194300kB
kernel: [9484097.393092] 2029585 pages RAM
kernel: [9484097.393093] 0 pages HighMem/MovableOnly
kernel: [9484097.393093] 59485 pages reserved
kernel: [9484097.393094] 0 pages hwpoisoned
kernel: [9484097.393094] Tasks state (memory values in pages):
kernel: [9484097.393095] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
kernel: [9484097.393111] [ 687] 0 687 698 0 45056 30 0 falcond
kernel: [9484097.393113] [ 688] 0 688 505660 5971 327680 1633 0 falcon-sensor
kernel: [9484097.393137] [ 855] 0 855 541254 4791 421888 1144 -999 containerd
kernel: [9484097.393146] [ 1283] 0 1283 749675 11418 655360 1711 -500 dockerd
kernel: [9484097.393215] [ 360705] 999 360705 1707021 70258 1396736 21 0 mariadbd
kernel: [9484097.393940] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=2a8e9308e3efb49deadde95237efaeb3f2fc3421ad65a3a5b18750d8d8f7deb5,mems_allowed=0,global_oom,task_memcg=/docker/99fa2d54134f8e550f363b25927323e7711008dda5da43cbb16a7ec9df2802c1,task=mariadbd,pid=360705,uid=999
kernel: [9484097.394218] Out of memory: Killed process 360705 (mariadbd) total-vm:6828084kB, anon-rss:259524kB, file-rss:21508kB, shmem-rss:0kB, UID:999 pgtables:1364kB oom_score_adj:0
kernel: [9484097.407371] oom_reaper: reaped process 360705 (mariadbd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
编辑:根据要求添加有关系统的信息
H/W path Device Class Description
============================================================
system OptiPlex 7050 (07A1)
/0 bus 0NW6H5
/0/0 memory 64KiB BIOS
/0/9 memory 8GiB System Memory
/0/9/0 memory 4GiB DIMM DDR4 Synchronous Unbuffered (Unregistered) 2400 MHz (0.4 ns)
/0/9/1 memory 4GiB DIMM DDR4 Synchronous Unbuffered (Unregistered) 2400 MHz (0.4 ns)
/0/9/2 memory [empty]
/0/9/3 memory [empty]
/0/f memory 256KiB L1 cache
/0/10 memory 1MiB L2 cache
/0/11 memory 8MiB L3 cache
/0/12 processor Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
/0/100 bridge Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers
/0/100/2 display HD Graphics 630
/0/100/14 bus 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller
/0/100/14/0 usb1 bus xHCI Host Controller
/0/100/14/0/3 bus Hub
/0/100/14/0/3/1 multimedia USB PnP Audio Device
/0/100/14/0/3/2 multimedia USB PnP Audio Device
/0/100/14/0/3/3 multimedia USB PnP Audio Device
/0/100/14/0/3/4 multimedia USB PnP Audio Device
/0/100/14/0/3/5 bus Hub
/0/100/14/0/3/5/1 multimedia USB PnP Audio Device
/0/100/14/0/3/5/2 multimedia USB PnP Audio Device
/0/100/14/0/3/5/3 multimedia USB PnP Audio Device
/0/100/14/0/3/5/4 multimedia USB PnP Audio Device
/0/100/14/0/3/5/5 bus Hub
/0/100/14/0/3/5/5/1 multimedia USB PnP Audio Device
/0/100/14/0/3/5/5/2 multimedia USB PnP Audio Device
/0/100/14/0/3/5/5/3 multimedia USB PnP Audio Device
/0/100/14/0/3/5/5/4 multimedia USB PnP Audio Device
/0/100/14/0/3/5/5/5 generic FT232R USB UART
/0/100/14/0/3/5/5/6 multimedia USB PnP Audio Device
/0/100/14/0/3/5/5/7 multimedia USB PnP Audio Device
/0/100/14/0/3/5/6 multimedia USB PnP Audio Device
/0/100/14/0/3/5/7 multimedia USB PnP Audio Device
/0/100/14/1 usb2 bus xHCI Host Controller
/0/100/14.2 generic 200 Series PCH Thermal Subsystem
/0/100/15 generic 200 Series PCH Serial IO I2C Controller #0
/0/100/16 communication 200 Series PCH CSME HECI #1
/0/100/17 storage SATA Controller [RAID mode]
/0/100/1d bridge 200 Series PCH PCI Express Root Port #9
/0/100/1f bridge 200 Series PCH LPC Controller (Q270)
/0/100/1f.2 memory Memory controller
/0/100/1f.4 bus 200 Series/Z370 Chipset Family SMBus Controller
/0/100/1f.6 eth0 network Ethernet Connection (5) I219-LM
/0/1 system PnP device PNP0c02
/0/2 communication PnP device PNP0501
/0/3 input PnP device PNP0303
/0/4 input PnP device PNP0f03
/0/5 system PnP device PNP0c02
/0/6 system PnP device PNP0c02
/0/7 system PnP device PNP0b00
/0/8 generic PnP device INT3f0d
/0/a system PnP device PNP0c02
/0/b system PnP device PNP0c02
/0/c system PnP device PNP0c02
/0/d system PnP device PNP0c02
/0/e scsi0 storage
/0/e/0.0.0 /dev/sda disk 1TB ST1000DM010-2EP1
/0/e/0.0.0/1 /dev/sda1 volume 511MiB Windows FAT volume
/0/e/0.0.0/2 /dev/sda2 volume 931GiB EXT4 volume
/0/13 scsi1 storage
/0/13/0.0.0 /dev/cdrom disk DVD+-RW DU-8A5LH
/1 veth2e771c6 network Ethernet interface
/2 veth7cb2db4 network Ethernet interface
/3 vethf37e821 network Ethernet interface
/4 vethd70ba99 network Ethernet interface
/5 veth2bcd630 network Ethernet interface
/6 vethd242ecd network Ethernet interface
/7 veth7ed6a4b network Ethernet interface
。
Distributor ID: Ubuntu
Description: Ubuntu-Server 20.04.3 2021.09.15 customized using Cubic on 2021-09-15 14:33
Release: 20.04
Codename: focal