我的服务器运行的是 Debian 6 64 位，内核是 2.6.32-5-amd64，每天有几次服务器负载变得非常高，我有下面这个调用跟踪，有人能向我解释一下问题是什么吗？
[521565.551017] Call Trace:
[521565.551019] <IRQ> [<ffffffff810ba7b3>] ? __alloc_pages_nodemask+0x59b/0x5fc
[521565.551029] [<ffffffff81248c08>] ? skb_gro_receive+0x17e/0x3cf
[521565.551034] [<ffffffff810e6b26>] ? new_slab+0x5b/0x1ca
[521565.551037] [<ffffffff810e6e85>] ? __slab_alloc+0x1f0/0x39b
[521565.551039] [<ffffffff8124896e>] ? __alloc_skb+0x3e/0x15a
[521565.551041] [<ffffffff8124896e>] ? __alloc_skb+0x3e/0x15a
[521565.551044] [<ffffffff810e7261>] ? kmem_cache_alloc_node+0x8b/0x10b
[521565.551046] [<ffffffff8124896e>] ? __alloc_skb+0x3e/0x15a
[521565.551051] [<ffffffff8119bf48>] ? swiotlb_map_page+0x0/0xc4
[521565.551053] [<ffffffff8124996c>] ? __netdev_alloc_skb+0x29/0x45
[521565.551063] [<ffffffffa0058b00>] ? tg3_alloc_rx_skb+0x96/0x11e [tg3]
[521565.551067] [<ffffffffa006291e>] ? tg3_poll_work+0x45a/0xb49 [tg3]
[521565.551072] [<ffffffff8127362c>] ? ip_local_deliver_finish+0x146/0x1e9
[521565.551076] [<ffffffff8124fc8a>] ? napi_gro_flush+0x1e/0x39
[521565.551080] [<ffffffffa0063139>] ? tg3_poll+0x8f/0x1a8 [tg3]
[521565.551084] [<ffffffff811f6f71>] ? mix_pool_bytes_extract+0x57/0x14a
[521565.551087] [<ffffffff8124fd8e>] ? net_rx_action+0xae/0x1c9
[521565.551091] [<ffffffff81053d2b>] ? __do_softirq+0xdd/0x1a6
[521565.551095] [<ffffffff81011cac>] ? call_softirq+0x1c/0x30
[521565.551097] [<ffffffff8101322b>] ? do_softirq+0x3f/0x7c
[521565.551100] [<ffffffff81053b9b>] ? irq_exit+0x36/0x76
[521565.551102] [<ffffffff81012922>] ? do_IRQ+0xa0/0xb6
[521565.551104] [<ffffffff810114d3>] ? ret_from_intr+0x0/0x11
[521565.551106] <EOI> [<ffffffff81017318>] ? mwait_idle+0x72/0x7d
[521565.551110] [<ffffffff810172c8>] ? mwait_idle+0x22/0x7d
[521565.551113] [<ffffffff8100fe97>] ? cpu_idle+0xa2/0xda
[521565.551115] Mem-Info:
[521565.551116] Node 0 DMA per-cpu:
[521565.551118] CPU 0: hi: 0, btch: 1 usd: 0
[521565.551120] CPU 1: hi: 0, btch: 1 usd: 0
[521565.551121] CPU 2: hi: 0, btch: 1 usd: 0
[521565.551123] CPU 3: hi: 0, btch: 1 usd: 0
[521565.551124] CPU 4: hi: 0, btch: 1 usd: 0
[521565.551126] CPU 5: hi: 0, btch: 1 usd: 0
[521565.551127] CPU 6: hi: 0, btch: 1 usd: 0
[521565.551129] CPU 7: hi: 0, btch: 1 usd: 0
[521565.551130] Node 0 DMA32 per-cpu:
[521565.551132] CPU 0: hi: 186, btch: 31 usd: 0
[521565.551133] CPU 1: hi: 186, btch: 31 usd: 0
[521565.551135] CPU 2: hi: 186, btch: 31 usd: 0
[521565.551136] CPU 3: hi: 186, btch: 31 usd: 0
[521565.551138] CPU 4: hi: 186, btch: 31 usd: 0
[521565.551139] CPU 5: hi: 186, btch: 31 usd: 0
[521565.551141] CPU 6: hi: 186, btch: 31 usd: 0
[521565.551142] CPU 7: hi: 186, btch: 31 usd: 0
[521565.551143] Node 0 Normal per-cpu:
[521565.551145] CPU 0: hi: 186, btch: 31 usd: 0
[521565.551147] CPU 1: hi: 186, btch: 31 usd: 0
[521565.551148] CPU 2: hi: 186, btch: 31 usd: 30
[521565.551150] CPU 3: hi: 186, btch: 31 usd: 0
[521565.551151] CPU 4: hi: 186, btch: 31 usd: 0
[521565.551153] CPU 5: hi: 186, btch: 31 usd: 1
[521565.551154] CPU 6: hi: 186, btch: 31 usd: 0
[521565.551155] CPU 7: hi: 186, btch: 31 usd: 0
[521565.551159] active_anon:50248 inactive_anon:44179 isolated_anon:0
[521565.551160] active_file:1929286 inactive_file:1929366 isolated_file:0
[521565.551161] unevictable:0 dirty:307090 writeback:56879 unstable:0
[521565.551162] free:18514 slab_reclaimable:68418 slab_unreclaimable:51386
[521565.551163] mapped:260756 shmem:95 pagetables:4659 bounce:0
[521565.551165] Node 0 DMA free:15852kB min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15296kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[521565.551173] lowmem_reserve[]: 0 2991 16121 16121
[521565.551175] Node 0 DMA32 free:53496kB min:3012kB low:3764kB high:4516kB active_anon:13668kB inactive_anon:24464kB active_file:1297884kB inactive_file:1298024kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3063648kB mlocked:0kB dirty:290652kB writeback:41076kB mapped:221156kB shmem:16kB slab_reclaimable:71508kB slab_unreclaimable:55600kB kernel_stack:360kB pagetables:1048kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[521565.551184] lowmem_reserve[]: 0 0 13130 13130
[521565.551187] Node 0 Normal free:4708kB min:13224kB low:16528kB high:19836kB active_anon:187324kB inactive_anon:152252kB active_file:6419260kB inactive_file:6419440kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13445120kB mlocked:0kB dirty:937708kB writeback:186440kB mapped:821868kB shmem:364kB slab_reclaimable:202164kB slab_unreclaimable:149944kB kernel_stack:1480kB pagetables:17588kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[521565.551196] lowmem_reserve[]: 0 0 0 0
[521565.551198] Node 0 DMA: 1*4kB 1*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15852kB
[521565.551204] Node 0 DMA32: 121*4kB 276*8kB 539*16kB 1197*32kB 4*64kB 4*128kB 0*256kB 2*512kB 0*1024kB 1*2048kB 0*4096kB = 53460kB
[521565.551211] Node 0 Normal: 612*4kB 80*8kB 43*16kB 16*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 4800kB
[521565.551217] 3883344 total pagecache pages
[521565.551218] 24509 pages in swap cache
[521565.551220] Swap cache stats: add 96013, delete 71504, find 812823/817958
[521565.551221] Free swap = 4737200kB
[521565.551222] Total swap = 4878304kB
[521565.594382] 4194304 pages RAM
[521565.594383] 77961 pages reserved
[521565.594384] 2084547 pages shared
[521565.594385] 2278383 pages non-shared
[521565.594387] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[521565.594390] cache: kmalloc-256, object size: 256, buffer size: 256, default order: 1, min order: 0
[521565.594393] node 0: slabs: 500, objs: 15104, free: 8
[521598.698442] swapper: page allocation failure. order:4, mode:0x4020
[521598.698451] Pid: 0, comm: swapper Not tainted 2.6.32-5-amd64 #1
答案1
我当然不能确定，但从那条痕迹来看，我会说这是
- 网络相关
- 表示你的内核无法分配内存来处理传入的数据
- 特别是与入站流量相关的
我会先运行流量跟踪，将入站网络流量与缓慢时期关联起来。这可能会让您对正在发生的事情有所了解。
这也可能与网络流量完全无关，只是内存问题。检查 的输出free -m。