我甚至找不到从哪里开始研究!崩溃之间没有规律可循。
2014-08-27T13:48:01.033Z cpu0:32957)MCE: 1118: cpu0: MCA error detected via CMCI (Gbl status=0x0): Restart IP: invalid, Error IP: invalid, MCE in progress: no.
2014-08-27T13:48:01.033Z cpu0:32957)MCE: 222: cpu0: bank0: status=0x9000004000010005: (VAL=1, OVFLW=0, UC=0, EN=1, PCC=0, S=0, AR=0), ECC=no, Addr:0x0 (invalid), Misc:0x0 (invalid)
2014-08-27T13:48:01.033Z cpu0:32957)MCE: 231: cpu0: bank0: MCA recoverable error (CE): "Internal Parity Error."
2014-08-27T13:48:01.033Z cpu1:33242)World: 8773: PRDA 0x418040400000 ss 0x0 ds 0x10b es 0x10b fs 0x0 gs 0x13b
2014-08-27T13:48:01.033Z cpu1:33242)World: 8775: TR 0x4020 GDT 0x4123876a1000 (0x402f) IDT 0x418014cf3000 (0xfff)
2014-08-27T13:48:01.033Z cpu1:33242)World: 8776: CR0 0x80010031 CR3 0x11e3e4000 CR4 0x42768
2014-08-27T13:48:01.039Z cpu1:33242)Backtrace for current CPU #1, worldID=33242, ebp=0x4119c0013dd0
2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013dd0:[0x418014c8cf99]PanicvPanicInt@vmkernel#nover+0x575 stack: 0x8, 0x4119c0013e40, 0x41
2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013e30:[0x418014c8d1dd]Panic_NoSave@vmkernel#nover+0x49 stack: 0x800000001, 0x5, 0xbe200000
2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013e90:[0x418014c63e75]IDTReturnPrepare@vmkernel#nover+0x2c5 stack: 0x67374e, 0x6738ce, 0xf
2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013f20:[0x418014c6475f]Int18_MachineCheck@vmkernel#nover+0x163 stack: 0xff8f3bb8, 0x1196300
2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013f30:[0x418014cf1064]gate_entry@vmkernel#nover+0x64 stack: 0x0, 0x13b, 0x0, 0x12a7495c, 0
2014-08-27T13:48:01.040Z cpu1:33242)[45m[33;1mVMware ESXi 5.5.0 [Releasebuild-1892794 x86_64][0m
NOT_REACHED bora/vmkernel/main/idt.c:1165
2014-08-27T13:48:01.040Z cpu1:33242)cr0=0x80010031 cr2=0x12a18be0 cr3=0x11e3e4000 cr4=0x42768
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:0 world:32949 name:"memMapKernel-0" (S)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:1 world:33242 name:"vobd" (U)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:2 world:32791 name:"CmdCompl-2" (S)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:3 world:33210 name:"vmsyslogd" (U)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:4 world:32783 name:"coalesceWorld-0" (S)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:5 world:53216 name:"vmm1:Analytics_VM" (V)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:6 world:33212 name:"vmsyslogd" (U)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:7 world:32784 name:"netCoalesce2World" (S)
2014-08-27T13:48:01.040Z cpu1:33242)@BlueScreen: NOT_REACHED bora/vmkernel/main/idt.c:1165
2014-08-27T13:48:01.040Z cpu1:33242)Code start: 0x418014c00000 VMK uptime: 2:02:57:31.656
2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013dd0:[0x418014c8cf99]PanicvPanicInt@vmkernel#nover+0x575 stack: 0x8
2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013e30:[0x418014c8d1dd]Panic_NoSave@vmkernel#nover+0x49 stack: 0x800000001
2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013e90:[0x418014c63e75]IDTReturnPrepare@vmkernel#nover+0x2c5 stack: 0x67374e
2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013f20:[0x418014c6475f]Int18_MachineCheck@vmkernel#nover+0x163 stack: 0xff8f3bb8
2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013f30:[0x418014cf1064]gate_entry@vmkernel#nover+0x64 stack: 0x0
2014-08-27T13:48:01.042Z cpu1:33242)base fs=0x0 gs=0x418040400000 Kgs=0x0
2014-08-27T13:48:01.042Z cpu1:33242)MC:PCPU0 B:0 S:0x9000004000010005 M:0x0 A:0x0 0
2014-08-27T13:48:01.043Z cpu1:33242)MC:PCPU1 B:8 S:0xbe2000000005110a M:0x9080000086 A:0x118be2600 5
MC:PCPU0: 1 hardware errors seen since boot (1 corrected by hardware)
MC:PCPU1: 1 hardware errors seen since boot (0 corrected by hardware)
2014-08-27T13:48:01.043Z cpu1:33242)PCPU fam:6 model:58 step:9 type:2 name:Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
2014-08-27T13:48:01.043Z cpu1:33242)vmkernel 0x0 .data 0x0 .bss 0x0
2014-08-27T13:48:01.043Z cpu1:33242)chardevs 0x418015171000 .data 0x417fc0000000 .bss 0x417fc0000400
2014-08-27T13:48:01.043Z cpu1:33242)user 0x418015178000 .data 0x417fc0400000 .bss 0x417fc040e180
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_mgmt 0x41801522b000 .data 0x417fc0800000 .bss 0x417fc0800140
2014-08-27T13:48:01.043Z cpu1:33242)vprobe 0x418015231000 .data 0x417fc0c00000 .bss 0x417fc0c0b7c0
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_socket 0x41801526f000 .data 0x417fc1000000 .bss 0x417fc10005c0
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_0_0_0_vmkernel_shim 0x418015274000 .data 0x417fc1400000 .bss 0x417fc14080c0
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_1_0_0_vmkernel_shim 0x418015279000 .data 0x417fc1800000 .bss 0x417fc1808840
2014-08-27T13:48:01.043Z cpu1:33242)procfs 0x41801527e000 .data 0x417fc1c00000 .bss 0x417fc1c00240
2014-08-27T13:48:01.043Z cpu1:33242)vfat 0x418015281000 .data 0x417fc2000000 .bss 0x417fc2002600
2014-08-27T13:48:01.043Z cpu1:33242)procMisc 0x41801528b000 .data 0x417fc2400000 .bss 0x417fc2400000
2014-08-27T13:48:01.043Z cpu1:33242)vmci 0x41801528c000 .data 0x417fc2800000 .bss 0x417fc28057c0
2014-08-27T13:48:01.043Z cpu1:33242)iodm 0x4180152af000 .data 0x417fc2c00000 .bss 0x417fc2c00138
2014-08-27T13:48:01.043Z cpu1:33242)vmkplexer 0x4180152b3000 .data 0x417fc3000000 .bss 0x417fc3000260
2014-08-27T13:48:01.043Z cpu1:33242)vmklinux_9 0x4180152b7000 .data 0x417fc3400000 .bss 0x417fc3408e80
2014-08-27T13:48:01.043Z cpu1:33242)vmklinux_9_2_0_0 0x41801533e000 .data 0x417fc3800000 .bss 0x417fc3807e84
2014-08-27T13:48:01.043Z cpu1:33242)vmklinux_9_2_1_0 0x418015341000 .data 0x417fc3c00000 .bss 0x417fc3c07f98
2014-08-27T13:48:01.043Z cpu1:33242)vmklinux_9_2_2_0 0x418015344000 .data 0x417fc4000000 .bss 0x417fc4008838
2014-08-27T13:48:01.043Z cpu1:33242)iscsi_trans 0x418015347000 .data 0x417fc4400000 .bss 0x417fc4401800
2014-08-27T13:48:01.043Z cpu1:33242)iscsi_trans_compat_shim 0x418015352000 .data 0x417fc4800000 .bss 0x417fc480096c
2014-08-27T13:48:01.043Z cpu1:33242)iscsi_trans_incompat_shim 0x418015353000 .data 0x417fc4c00000 .bss 0x417fc4c007e4
2014-08-27T13:48:01.043Z cpu1:33242)etherswitch 0x418015354000 .data 0x417fc5000000 .bss 0x417fc5013a00
2014-08-27T13:48:01.043Z cpu1:33242)netsched 0x418015389000 .data 0x417fc5400000 .bss 0x417fc5404800
2014-08-27T13:48:01.043Z cpu1:33242)cnic_register 0x41801539b000 .data 0x417fc5800000 .bss 0x417fc58001e0
2014-08-27T13:48:01.043Z cpu1:33242)e1000 0x41801539d000 .data 0x417fc5c00000 .bss 0x417fc5c01240
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_1_0_0_iscsi_shim 0x4180153c3000 .data 0x417fc6000000 .bss 0x417fc6000970
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_0_0_0_iscsi_shim 0x4180153c4000 .data 0x417fc6400000 .bss 0x417fc6400970
2014-08-27T13:48:01.043Z cpu1:33242)random 0x4180153c5000 .data 0x417fc6800000 .bss 0x417fc6800600
2014-08-27T13:48:01.043Z cpu1:33242)usb 0x4180153c9000 .data 0x417fc6c00000 .bss 0x417fc6c01660
2014-08-27T13:48:01.043Z cpu1:33242)ehci-hcd 0x4180153eb000 .data 0x417fc7000000 .bss 0x417fc70002a0
2014-08-27T13:48:01.043Z cpu1:33242)hid 0x4180153f6000 .data 0x417fc7400000 .bss 0x417fc74004e0
2014-08-27T13:48:01.043Z cpu1:33242)healthchk 0x4180153fb000 .data 0x417fc7800000 .bss 0x417fc7811e00
2014-08-27T13:48:01.043Z cpu1:33242)teamcheck 0x418015411000 .data 0x417fc7c00000 .bss 0x417fc7c12240
2014-08-27T13:48:01.043Z cpu1:33242)vlanmtucheck 0x418015424000 .data 0x417fc8000000 .bss 0x417fc8012000
2014-08-27T13:48:01.043Z cpu1:33242)heartbeat 0x418015439000 .data 0x417fc8400000 .bss 0x417fc8411f00
2014-08-27T13:48:01.043Z cpu1:33242)shaper 0x41801544a000 .data 0x417fc8800000 .bss 0x417fc8813e80
2014-08-27T13:48:01.043Z cpu1:33242)lldp 0x41801545d000 .data 0x417fc8c00000 .bss 0x417fc8c00040
2014-08-27T13:48:01.043Z cpu1:33242)cdp 0x418015462000 .data 0x417fc9000000 .bss 0x417fc9013400
2014-08-27T13:48:01.043Z cpu1:33242)ipfix 0x41801547e000 .data 0x417fc9400000 .bss 0x417fc9412540
2014-08-27T13:48:01.043Z cpu1:33242)tcpip4 0x418015492000 .data 0x417fc9800000 .bss 0x417fc9818180
2014-08-27T13:48:01.043Z cpu1:33242)dvsdev 0x418015598000 .data 0x417fc9c00000 .bss 0x417fc9c00030
2014-08-27T13:48:01.043Z cpu1:33242)dvfilter 0x41801559b000 .data 0x417fca000000 .bss 0x417fca000b00
2014-08-27T13:48:01.043Z cpu1:33242)lacp 0x4180155bd000 .data 0x417fca400000 .bss 0x417fca400160
2014-08-27T13:48:01.043Z cpu1:33242)hbr_filter 0x4180155c7000 .data 0x417fca800000 .bss 0x417fca800300
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_1_0_0_dvfilter_shim 0x4180155f3000 .data 0x417fcac00000 .bss 0x417fcac009b0
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_0_0_0_dvfilter_shim 0x4180155f4000 .data 0x417fcb000000 .bss 0x417fcb000930
2014-08-27T13:48:01.043Z cpu1:33242)dvfilter-generic-fastpath 0x4180155f5000 .data 0x417fcb400000 .bss 0x417fcb412380
2014-08-27T13:48:01.043Z cpu1:33242)vmkstatelogger 0x41801560f000 .data 0x417fcb800000 .bss 0x417fcb803a00
2014-08-27T13:48:01.043Z cpu1:33242)esxfw 0x418015633000 .data 0x417fcbc00000 .bss 0x417fcbc12d00
2014-08-27T13:48:01.043Z cpu1:33242)dm 0x418015648000 .data 0x417fcc000000 .bss 0x417fcc000000
2014-08-27T13:48:01.043Z cpu1:33242)nmp 0x41801564a000 .data 0x417fcc400000 .bss 0x417fcc403e50
2014-08-27T13:48:01.043Z cpu1:33242)vmw_satp_local 0x41801566d000 .data 0x417fcc800000 .bss 0x417fcc800028
2014-08-27T13:48:01.043Z cpu1:33242)vmw_satp_default_aa 0x41801566f000 .data 0x417fccc00000 .bss 0x417fccc00000
2014-08-27T13:48:01.043Z cpu1:33242)vmw_psp_lib 0x418015670000 .data 0x417fcd000000 .bss 0x417fcd000290
2014-08-27T13:48:01.043Z cpu1:33242)vmw_psp_fixed 0x418015672000 .data 0x417fcd400000 .bss 0x417fcd400000
2014-08-27T13:48:01.043Z cpu1:33242)vmw_psp_rr 0x418015674000 .data 0x417fcd800000 .bss 0x417fcd800068
2014-08-27T13:48:01.043Z cpu1:33242)vmw_psp_mru 0x418015677000 .data 0x417fcdc00000 .bss 0x417fcdc00000
2014-08-27T13:48:01.043Z cpu1:33242)libata_92 0x418015679000 .data 0x417fce000000 .bss 0x417fce002660
2014-08-27T13:48:01.043Z cpu1:33242)libata_9_2_0_0 0x41801569b000 .data 0x417fce400000 .bss 0x417fce401750
2014-08-27T13:48:01.043Z cpu1:33242)libata_9_2_1_0 0x41801569c000 .data 0x417fce800000 .bss 0x417fce801750
2014-08-27T13:48:01.043Z cpu1:33242)usb-storage 0x41801569d000 .data 0x417fcec00000 .bss 0x417fcec04780
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_1_0_0_nmp_shim 0x4180156a9000 .data 0x417fcf000000 .bss 0x417fcf000ca8
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_0_0_0_nmp_shim 0x4180156aa000 .data 0x417fcf400000 .bss 0x417fcf400ca8
2014-08-27T13:48:01.043Z cpu1:33242)svmmirror 0x4180156ab000 .data 0x417fcf800000 .bss 0x417fcf8000c0
2014-08-27T13:48:01.043Z cpu1:33242)cbt 0x4180156b7000 .data 0x417fcfc00000 .bss 0x417fcfc00080
2014-08-27T13:48:01.043Z cpu1:33242)migrate 0x4180156bb000 .data 0x417fd0000000 .bss 0x417fd0004d40
2014-08-27T13:48:01.043Z cpu1:33242)libfc_92 0x41801571a000 .data 0x417fd0400000 .bss 0x417fd0400b80
2014-08-27T13:48:01.043Z cpu1:33242)libfcoe_92 0x418015733000 .data 0x417fd0800000 .bss 0x417fd08001c0
2014-08-27T13:48:01.043Z cpu1:33242)libfc_9_2_0_0 0x418015739000 .data 0x417fd0c00000 .bss 0x417fd0c00868
2014-08-27T13:48:01.043Z cpu1:33242)libfcoe_9_2_0_0 0x41801573a000 .data 0x417fd1000000 .bss 0x417fd10001f4
2014-08-27T13:48:01.043Z cpu1:33242)libfc_9_2_1_0 0x41801573b000 .data 0x417fd1400000 .bss 0x417fd1400868
2014-08-27T13:48:01.043Z cpu1:33242)libfcoe_9_2_1_0 0x41801573c000 .data 0x417fd1800000 .bss 0x417fd18001f4
2014-08-27T13:48:01.043Z cpu1:33242)ahci 0x41801573d000 .data 0x417fd1c00000 .bss 0x417fd1c00420
2014-08-27T13:48:01.043Z cpu1:33242)sunrpc 0x418015744000 .data 0x417fd2000000 .bss 0x417fd2002b80
2014-08-27T13:48:01.043Z cpu1:33242)nfsclient 0x418015753000 .data 0x417fd2400000 .bss 0x417fd2403940
2014-08-27T13:48:01.043Z cpu1:33242)vmkibft 0x41801576c000 .data 0x417fd2800000 .bss 0x417fd28037c0
2014-08-27T13:48:01.043Z cpu1:33242)lvmdriver 0x41801576f000 .data 0x417fd2c00000 .bss 0x417fd2c03380
2014-08-27T13:48:01.043Z cpu1:33242)deltadisk 0x418015783000 .data 0x417fd3000000 .bss 0x417fd3005c00
2014-08-27T13:48:01.043Z cpu1:33242)tracing 0x4180157ae000 .data 0x417fd3400000 .bss 0x417fd3405b40
2014-08-27T13:48:01.043Z cpu1:33242)rdt 0x4180157b5000 .data 0x417fd3800000 .bss 0x417fd3804e00
2014-08-27T13:48:01.043Z cpu1:33242)vsanutil 0x4180157db000 .data 0x417fd3c00000 .bss 0x417fd3c069c0
2014-08-27T13:48:01.043Z cpu1:33242)lsomcommon 0x4180157fa000 .data 0x417fd4000000 .bss 0x417fd4001680
2014-08-27T13:48:01.043Z cpu1:33242)plog 0x41801582e000 .data 0x417fd4400000 .bss 0x417fd44056c0
2014-08-27T13:48:01.043Z cpu1:33242)vmfs3 0x418015871000 .data 0x417fd4800000 .bss 0x417fd4803840
2014-08-27T13:48:01.043Z cpu1:33242)dvfg-igmp 0x4180158d9000 .data 0x417fd5a00000 .bss 0x417fd5a00208
2014-08-27T13:48:01.043Z cpu1:33242)cmmds_net 0x4180158df000 .data 0x417fd5e00000 .bss 0x417fd5e02f40
2014-08-27T13:48:01.043Z cpu1:33242)cmmds 0x4180158ec000 .data 0x417fd6200000 .bss 0x417fd6204d80
2014-08-27T13:48:01.043Z cpu1:33242)cmmds_resolver 0x418015921000 .data 0x417fd6600000 .bss 0x417fd6600140
2014-08-27T13:48:01.043Z cpu1:33242)vsan 0x41801592d000 .data 0x417fd6a00000 .bss 0x417fd6a1c200
2014-08-27T13:48:01.043Z cpu1:33242)vmklink_mpi 0x418015a48000 .data 0x417fd6e00000 .bss 0x417fd6e02400
2014-08-27T13:48:01.043Z cpu1:33242)swapobj 0x418015a4d000 .data 0x417fd7200000 .bss 0x417fd7203010
2014-08-27T13:48:01.043Z cpu1:33242)osfs 0x418015a55000 .data 0x417fd7600000 .bss 0x417fd7603380
2014-08-27T13:48:01.043Z cpu1:33242)vflash 0x418015a63000 .data 0x417fd7a00000 .bss 0x417fd7a03540
2014-08-27T13:48:01.043Z cpu1:33242)vfc 0x418015a6e000 .data 0x417fd7e00000 .bss 0x417fd7e02ac0
Coredump to disk.
2014-08-27T13:48:01.093Z cpu1:33242)Slot 1 of 1.
2014-08-27T13:48:01.093Z cpu1:33242)Dump: 2212: Using dump slot size 2684354560.
答案1
错误消息看起来像是 CPU 或 RAM 出现了硬件问题。您是否尝试过运行memtest
内置硬件诊断工具(如果可用,可在服务器启动期间选择)?
答案2
这肯定看起来像是硬件错误。您可以尝试使用 mcelog 之类的工具,它只是充当任何机器检查异常 (mce) 的转换器。您还没有提到您使用的操作系统版本,但在 RedHat/Centos 中,您可以通过 yum 安装它
yum install mcelog
并且消息记录在 /var/log/mcelog 中
欲了解更多信息,请参阅主网站http://www.mcelog.org/