我遇到了一个非常奇怪的问题,我的 Debian Squeeze 总是在晚上(柏林时间)冻结。这是我从某个时间得到的结果,在执行几次后,它就会冻结,必须进行硬重置。
来自 /var/log/messages
Dec 11 01:36:11 srv156 kernel: [125983.204251] CPU 1:
Dec 11 01:36:11 srv156 kernel: [125983.204251] Modules linked in: xt_multiport nf_conntrack_ipv4 nf_defrag_ipv4 xt_recent xt_state nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables hwmon_vid snd_hda_codec_atihdmi snd_hda_intel snd_hda_codec snd_hwdep snd_pcm radeon snd_timer ttm drm_kms_helper snd k10temp i2c_piix4 soundcore snd_page_alloc edac_core parport_pc drm i2c_algo_bit i2c_core shpchp pci_hotplug pcspkr edac_mce_amd parport wmi evdev processor button ext3 jbd mbcache raid1 md_mod sd_mod crc_t10dif ata_generic ahci ohci_hcd pata_atiixp e100 mii libata xhci floppy ehci_hcd thermal thermal_sys usbcore scsi_mod nls_base [last unloaded: i2c_dev]
Dec 11 01:36:11 srv156 kernel: [125983.204251] Pid: 758, comm: flush-9:0 Tainted: G B 2.6.32-5-amd64 #1 GA-78LMT-USB3
Dec 11 01:36:11 srv156 kernel: [125983.204251] RIP: 0010:[<ffffffff810b3506>] [<ffffffff810b3506>] find_get_pages_tag+0x66/0xdd
Dec 11 01:36:11 srv156 kernel: [125983.204251] RSP: 0018:ffff8804235e7b30 EFLAGS: 00000286
Dec 11 01:36:11 srv156 kernel: [125983.204251] RAX: ffffffffffffffff RBX: ffff8804235e7c00 RCX: 0000000000000000
Dec 11 01:36:11 srv156 kernel: [125983.204251] RDX: 0000000000040000 RSI: ffffea000496b2a8 RDI: ffffea000496b2a0
Dec 11 01:36:11 srv156 kernel: [125983.204251] RBP: ffffffff8101166e R08: ffff8804235e7af0 R09: 0000000000000000
Dec 11 01:36:11 srv156 kernel: [125983.204251] R10: 0000000000000000 R11: 0000000000040000 R12: ffff8804235e7c08
Dec 11 01:36:11 srv156 kernel: [125983.204251] R13: 0000000d22678a20 R14: ffff8804235e7af0 R15: 00000000091b9060
Dec 11 01:36:11 srv156 kernel: [125983.204251] FS: 0000000000000000(0000) GS:ffff880010440000(0000) knlGS:000000007ebf7b70
Dec 11 01:36:11 srv156 kernel: [125983.204522] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Dec 11 01:36:11 srv156 kernel: [125983.204522] CR2: 00000000dec86000 CR3: 0000000001001000 CR4: 00000000000006e0
Dec 11 01:36:11 srv156 kernel: [125983.204522] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec 11 01:36:11 srv156 kernel: [125983.204522] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec 11 01:36:11 srv156 kernel: [125983.204522] Call Trace:
Dec 11 01:36:11 srv156 kernel: [125983.204522] [<ffffffff810bb792>] ? pagevec_lookup_tag+0x1a/0x21
Dec 11 01:36:11 srv156 kernel: [125983.204522] [<ffffffff810ba330>] ? write_cache_pages+0x162/0x327
Dec 11 01:36:11 srv156 kernel: [125983.204522] [<ffffffff810b9d48>] ? __writepage+0x0/0x25
Dec 11 01:36:11 srv156 kernel: [125983.204522] [<ffffffff8110758a>] ? writeback_single_inode+0xe7/0x2da
Dec 11 01:36:11 srv156 kernel: [125983.204522] [<ffffffff81108290>] ? writeback_inodes_wb+0x424/0x4ff
Dec 11 01:36:11 srv156 kernel: [125983.204522] [<ffffffff81108497>] ? wb_writeback+0x12c/0x1ab
Dec 11 01:36:11 srv156 kernel: [125983.204522] [<ffffffff8110870d>] ? wb_do_writeback+0x14f/0x165
Dec 11 01:36:11 srv156 kernel: [125983.204522] [<ffffffff81108754>] ? bdi_writeback_task+0x31/0xaa
Dec 11 01:36:11 srv156 kernel: [125983.204522] [<ffffffff810c8664>] ? bdi_start_fn+0x0/0xd0
Dec 11 01:36:11 srv156 kernel: [125983.204522] [<ffffffff810c86d4>] ? bdi_start_fn+0x70/0xd0
Dec 11 01:36:11 srv156 kernel: [125983.204522] [<ffffffff810c8664>] ? bdi_start_fn+0x0/0xd0
Dec 11 01:36:11 srv156 kernel: [125983.204522] [<ffffffff81064ac1>] ? kthread+0x79/0x81
Dec 11 01:36:11 srv156 kernel: [125983.204522] [<ffffffff81011baa>] ? child_rip+0xa/0x20
Dec 11 01:36:11 srv156 kernel: [125983.204522] [<ffffffff81064a48>] ? kthread+0x0/0x81
Dec 11 01:36:11 srv156 kernel: [125983.204522] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
来自 /var/log/syslog
Dec 10 21:20:29 srv156 kernel: [110625.162930] BUG: Bad page map in process java pte:14fa4f067 pmd:424b54067
Dec 10 21:20:29 srv156 kernel: [110625.162937] page:ffffea000496c148 flags:0200000000000878 count:2 mapcount:-1 mapping:ffff88014f8d7de8 index:2f4
Dec 10 21:20:29 srv156 kernel: [110625.162946] addr:0000000009096000 vm_flags:00100077 anon_vma:ffff880422410d40 mapping:(null) index:9096
Dec 10 21:20:29 srv156 kernel: [110625.162955] Pid: 21356, comm: java Tainted: G B 2.6.32-5-amd64 #1
Dec 10 21:20:29 srv156 kernel: [110625.162961] Call Trace:
Dec 10 21:20:29 srv156 kernel: [110625.162966] [<ffffffff810ca4bf>] ? print_bad_pte+0x232/0x24a
Dec 10 21:20:29 srv156 kernel: [110625.162973] [<ffffffff810cb56f>] ? unmap_vmas+0x62d/0x931
Dec 10 21:20:29 srv156 kernel: [110625.162980] [<ffffffff810cfc74>] ? exit_mmap+0xc4/0x148
Dec 10 21:20:29 srv156 kernel: [110625.162986] [<ffffffff8104bbc1>] ? mmput+0x3c/0xdf
Dec 10 21:20:29 srv156 kernel: [110625.162992] [<ffffffff8104f81e>] ? exit_mm+0x102/0x10d
Dec 10 21:20:29 srv156 kernel: [110625.162998] [<ffffffff81051243>] ? do_exit+0x1f8/0x6c9
Dec 10 21:20:29 srv156 kernel: [110625.163004] [<ffffffff81071abb>] ? futex_wake+0xd6/0xe7
Dec 10 21:20:29 srv156 kernel: [110625.163010] [<ffffffff8105178a>] ? do_group_exit+0x76/0x9d
Dec 10 21:20:29 srv156 kernel: [110625.163016] [<ffffffff8105df9f>] ? get_signal_to_deliver+0x310/0x339
Dec 10 21:20:29 srv156 kernel: [110625.163023] [<ffffffff81010037>] ? do_notify_resume+0x87/0x73f
Dec 10 21:20:29 srv156 kernel: [110625.163029] [<ffffffff810cc664>] ? handle_mm_fault+0x7aa/0x80f
Dec 10 21:20:29 srv156 kernel: [110625.163036] [<ffffffff81073f14>] ? compat_sys_futex+0x10d/0x12b
Dec 10 21:20:29 srv156 kernel: [110625.163043] [<ffffffff812fb546>] ? do_page_fault+0x2e0/0x2fc
Dec 10 21:20:29 srv156 kernel: [110625.163049] [<ffffffff81010e0e>] ? int_signal+0x12/0x17
Dec 10 21:20:29 srv156 kernel: [110625.163114] BUG: Bad page state in process java pfn:14fa0c
Dec 10 21:20:29 srv156 kernel: [110625.163120] page:ffffea000496b2a0 flags:020000000002001c count:0 mapcount:-1 mapping:ffff88039dc0db30 index:11e3
Dec 10 21:20:29 srv156 kernel: [110625.164563] Pid: 21356, comm: java Tainted: G B 2.6.32-5-amd64 #1
Dec 10 21:20:29 srv156 kernel: [110625.164570] Call Trace:
Dec 10 21:20:29 srv156 kernel: [110625.164578] [<ffffffff810b71a9>] ? bad_page+0x116/0x129
Dec 10 21:20:29 srv156 kernel: [110625.164586] [<ffffffff810b7692>] ? free_pages_check+0x38/0x57
Dec 10 21:20:29 srv156 kernel: [110625.164595] [<ffffffff810b89cf>] ? free_hot_cold_page+0x46/0x190
Dec 10 21:20:29 srv156 kernel: [110625.164603] [<ffffffff810b8b82>] ? __pagevec_free+0x69/0x7f
Dec 10 21:20:29 srv156 kernel: [110625.164611] [<ffffffff810bba3f>] ? release_pages+0x137/0x18d
Dec 10 21:20:29 srv156 kernel: [110625.164620] [<ffffffff810d8559>] ? free_pages_and_swap_cache+0x57/0x73
Dec 10 21:20:29 srv156 kernel: [110625.164629] [<ffffffff810cb5ed>] ? unmap_vmas+0x6ab/0x931
Dec 10 21:20:29 srv156 kernel: [110625.164637] [<ffffffff810cfc74>] ? exit_mmap+0xc4/0x148
Dec 10 21:20:29 srv156 kernel: [110625.164644] [<ffffffff8104bbc1>] ? mmput+0x3c/0xdf
Dec 10 21:20:29 srv156 kernel: [110625.164652] [<ffffffff8104f81e>] ? exit_mm+0x102/0x10d
Dec 10 21:20:29 srv156 kernel: [110625.164660] [<ffffffff81051243>] ? do_exit+0x1f8/0x6c9
Dec 10 21:20:29 srv156 kernel: [110625.164667] [<ffffffff81071abb>] ? futex_wake+0xd6/0xe7
Dec 10 21:20:29 srv156 kernel: [110625.164675] [<ffffffff8105178a>] ? do_group_exit+0x76/0x9d
Dec 10 21:20:29 srv156 kernel: [110625.164683] [<ffffffff8105df9f>] ? get_signal_to_deliver+0x310/0x339
Dec 10 21:20:29 srv156 kernel: [110625.164692] [<ffffffff81010037>] ? do_notify_resume+0x87/0x73f
Dec 10 21:20:29 srv156 kernel: [110625.164700] [<ffffffff810cc664>] ? handle_mm_fault+0x7aa/0x80f
最后一篇日志是最近才发布的,因为我刚刚发现它。看来 Java 进程做了一些事情,并开始慢慢消耗服务器的所有资源。我不知道这是否是根本原因。
我正在使用 Debian Squeeze。uname -a
Linux srv156 2.6.32-5-amd64 #1 SMP Sun Sep 23 11:00:33 UTC 2012 x86_64 GNU/Linux
我真的非常感谢您的帮助,我不知道还能做些什么。
答案1
看起来像是内存问题 - 坏页。您是否配置了 mcelog - 您是否有任何文件? http://mcelog.org/
答案2
你的内核被B
flag 污染了,意思是:
B:发现某个进程处于坏页状态,这表明虚拟内存子系统已损坏,可能是由 RAM 或缓存内存故障引起的。
还回溯/var/log/syslog
说Bad page map
,所以正如@Jim已经提到的,你应该检查你的内存。
答案3
30 天后,服务器停止冻结。
原来是内存出了问题。感谢大家!
答案4
当您收到呼叫跟踪时,肯定存在问题。
可能是硬件不稳定(尝试进行一些长时间的 memtest86 运行、SMART 短时间和长时间自检、使用供应商诊断程序(如果有)等)或文件系统上的一些损坏的二进制文件。
如果您确定硬件没有问题,我会重新格式化驱动程序(我的意思是 mkfs 一个新的文件系统)并重新安装整个系统。