Я даже не нашел, с чего начать исследования! Между сбоями нет закономерностей.
2014-08-27T13:48:01.033Z cpu0:32957)MCE: 1118: cpu0: MCA error detected via CMCI (Gbl status=0x0): Restart IP: invalid, Error IP: invalid, MCE in progress: no.
2014-08-27T13:48:01.033Z cpu0:32957)MCE: 222: cpu0: bank0: status=0x9000004000010005: (VAL=1, OVFLW=0, UC=0, EN=1, PCC=0, S=0, AR=0), ECC=no, Addr:0x0 (invalid), Misc:0x0 (invalid)
2014-08-27T13:48:01.033Z cpu0:32957)MCE: 231: cpu0: bank0: MCA recoverable error (CE): "Internal Parity Error."
2014-08-27T13:48:01.033Z cpu1:33242)World: 8773: PRDA 0x418040400000 ss 0x0 ds 0x10b es 0x10b fs 0x0 gs 0x13b
2014-08-27T13:48:01.033Z cpu1:33242)World: 8775: TR 0x4020 GDT 0x4123876a1000 (0x402f) IDT 0x418014cf3000 (0xfff)
2014-08-27T13:48:01.033Z cpu1:33242)World: 8776: CR0 0x80010031 CR3 0x11e3e4000 CR4 0x42768
2014-08-27T13:48:01.039Z cpu1:33242)Backtrace for current CPU #1, worldID=33242, ebp=0x4119c0013dd0
2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013dd0:[0x418014c8cf99]PanicvPanicInt@vmkernel#nover+0x575 stack: 0x8, 0x4119c0013e40, 0x41
2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013e30:[0x418014c8d1dd]Panic_NoSave@vmkernel#nover+0x49 stack: 0x800000001, 0x5, 0xbe200000
2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013e90:[0x418014c63e75]IDTReturnPrepare@vmkernel#nover+0x2c5 stack: 0x67374e, 0x6738ce, 0xf
2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013f20:[0x418014c6475f]Int18_MachineCheck@vmkernel#nover+0x163 stack: 0xff8f3bb8, 0x1196300
2014-08-27T13:48:01.039Z cpu1:33242)0x4119c0013f30:[0x418014cf1064]gate_entry@vmkernel#nover+0x64 stack: 0x0, 0x13b, 0x0, 0x12a7495c, 0
2014-08-27T13:48:01.040Z cpu1:33242)[45m[33;1mVMware ESXi 5.5.0 [Releasebuild-1892794 x86_64][0m
NOT_REACHED bora/vmkernel/main/idt.c:1165
2014-08-27T13:48:01.040Z cpu1:33242)cr0=0x80010031 cr2=0x12a18be0 cr3=0x11e3e4000 cr4=0x42768
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:0 world:32949 name:"memMapKernel-0" (S)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:1 world:33242 name:"vobd" (U)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:2 world:32791 name:"CmdCompl-2" (S)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:3 world:33210 name:"vmsyslogd" (U)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:4 world:32783 name:"coalesceWorld-0" (S)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:5 world:53216 name:"vmm1:Analytics_VM" (V)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:6 world:33212 name:"vmsyslogd" (U)
2014-08-27T13:48:01.040Z cpu1:33242)pcpu:7 world:32784 name:"netCoalesce2World" (S)
2014-08-27T13:48:01.040Z cpu1:33242)@BlueScreen: NOT_REACHED bora/vmkernel/main/idt.c:1165
2014-08-27T13:48:01.040Z cpu1:33242)Code start: 0x418014c00000 VMK uptime: 2:02:57:31.656
2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013dd0:[0x418014c8cf99]PanicvPanicInt@vmkernel#nover+0x575 stack: 0x8
2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013e30:[0x418014c8d1dd]Panic_NoSave@vmkernel#nover+0x49 stack: 0x800000001
2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013e90:[0x418014c63e75]IDTReturnPrepare@vmkernel#nover+0x2c5 stack: 0x67374e
2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013f20:[0x418014c6475f]Int18_MachineCheck@vmkernel#nover+0x163 stack: 0xff8f3bb8
2014-08-27T13:48:01.040Z cpu1:33242)0x4119c0013f30:[0x418014cf1064]gate_entry@vmkernel#nover+0x64 stack: 0x0
2014-08-27T13:48:01.042Z cpu1:33242)base fs=0x0 gs=0x418040400000 Kgs=0x0
2014-08-27T13:48:01.042Z cpu1:33242)MC:PCPU0 B:0 S:0x9000004000010005 M:0x0 A:0x0 0
2014-08-27T13:48:01.043Z cpu1:33242)MC:PCPU1 B:8 S:0xbe2000000005110a M:0x9080000086 A:0x118be2600 5
MC:PCPU0: 1 hardware errors seen since boot (1 corrected by hardware)
MC:PCPU1: 1 hardware errors seen since boot (0 corrected by hardware)
2014-08-27T13:48:01.043Z cpu1:33242)PCPU fam:6 model:58 step:9 type:2 name:Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
2014-08-27T13:48:01.043Z cpu1:33242)vmkernel 0x0 .data 0x0 .bss 0x0
2014-08-27T13:48:01.043Z cpu1:33242)chardevs 0x418015171000 .data 0x417fc0000000 .bss 0x417fc0000400
2014-08-27T13:48:01.043Z cpu1:33242)user 0x418015178000 .data 0x417fc0400000 .bss 0x417fc040e180
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_mgmt 0x41801522b000 .data 0x417fc0800000 .bss 0x417fc0800140
2014-08-27T13:48:01.043Z cpu1:33242)vprobe 0x418015231000 .data 0x417fc0c00000 .bss 0x417fc0c0b7c0
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_socket 0x41801526f000 .data 0x417fc1000000 .bss 0x417fc10005c0
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_0_0_0_vmkernel_shim 0x418015274000 .data 0x417fc1400000 .bss 0x417fc14080c0
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_1_0_0_vmkernel_shim 0x418015279000 .data 0x417fc1800000 .bss 0x417fc1808840
2014-08-27T13:48:01.043Z cpu1:33242)procfs 0x41801527e000 .data 0x417fc1c00000 .bss 0x417fc1c00240
2014-08-27T13:48:01.043Z cpu1:33242)vfat 0x418015281000 .data 0x417fc2000000 .bss 0x417fc2002600
2014-08-27T13:48:01.043Z cpu1:33242)procMisc 0x41801528b000 .data 0x417fc2400000 .bss 0x417fc2400000
2014-08-27T13:48:01.043Z cpu1:33242)vmci 0x41801528c000 .data 0x417fc2800000 .bss 0x417fc28057c0
2014-08-27T13:48:01.043Z cpu1:33242)iodm 0x4180152af000 .data 0x417fc2c00000 .bss 0x417fc2c00138
2014-08-27T13:48:01.043Z cpu1:33242)vmkplexer 0x4180152b3000 .data 0x417fc3000000 .bss 0x417fc3000260
2014-08-27T13:48:01.043Z cpu1:33242)vmklinux_9 0x4180152b7000 .data 0x417fc3400000 .bss 0x417fc3408e80
2014-08-27T13:48:01.043Z cpu1:33242)vmklinux_9_2_0_0 0x41801533e000 .data 0x417fc3800000 .bss 0x417fc3807e84
2014-08-27T13:48:01.043Z cpu1:33242)vmklinux_9_2_1_0 0x418015341000 .data 0x417fc3c00000 .bss 0x417fc3c07f98
2014-08-27T13:48:01.043Z cpu1:33242)vmklinux_9_2_2_0 0x418015344000 .data 0x417fc4000000 .bss 0x417fc4008838
2014-08-27T13:48:01.043Z cpu1:33242)iscsi_trans 0x418015347000 .data 0x417fc4400000 .bss 0x417fc4401800
2014-08-27T13:48:01.043Z cpu1:33242)iscsi_trans_compat_shim 0x418015352000 .data 0x417fc4800000 .bss 0x417fc480096c
2014-08-27T13:48:01.043Z cpu1:33242)iscsi_trans_incompat_shim 0x418015353000 .data 0x417fc4c00000 .bss 0x417fc4c007e4
2014-08-27T13:48:01.043Z cpu1:33242)etherswitch 0x418015354000 .data 0x417fc5000000 .bss 0x417fc5013a00
2014-08-27T13:48:01.043Z cpu1:33242)netsched 0x418015389000 .data 0x417fc5400000 .bss 0x417fc5404800
2014-08-27T13:48:01.043Z cpu1:33242)cnic_register 0x41801539b000 .data 0x417fc5800000 .bss 0x417fc58001e0
2014-08-27T13:48:01.043Z cpu1:33242)e1000 0x41801539d000 .data 0x417fc5c00000 .bss 0x417fc5c01240
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_1_0_0_iscsi_shim 0x4180153c3000 .data 0x417fc6000000 .bss 0x417fc6000970
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_0_0_0_iscsi_shim 0x4180153c4000 .data 0x417fc6400000 .bss 0x417fc6400970
2014-08-27T13:48:01.043Z cpu1:33242)random 0x4180153c5000 .data 0x417fc6800000 .bss 0x417fc6800600
2014-08-27T13:48:01.043Z cpu1:33242)usb 0x4180153c9000 .data 0x417fc6c00000 .bss 0x417fc6c01660
2014-08-27T13:48:01.043Z cpu1:33242)ehci-hcd 0x4180153eb000 .data 0x417fc7000000 .bss 0x417fc70002a0
2014-08-27T13:48:01.043Z cpu1:33242)hid 0x4180153f6000 .data 0x417fc7400000 .bss 0x417fc74004e0
2014-08-27T13:48:01.043Z cpu1:33242)healthchk 0x4180153fb000 .data 0x417fc7800000 .bss 0x417fc7811e00
2014-08-27T13:48:01.043Z cpu1:33242)teamcheck 0x418015411000 .data 0x417fc7c00000 .bss 0x417fc7c12240
2014-08-27T13:48:01.043Z cpu1:33242)vlanmtucheck 0x418015424000 .data 0x417fc8000000 .bss 0x417fc8012000
2014-08-27T13:48:01.043Z cpu1:33242)heartbeat 0x418015439000 .data 0x417fc8400000 .bss 0x417fc8411f00
2014-08-27T13:48:01.043Z cpu1:33242)shaper 0x41801544a000 .data 0x417fc8800000 .bss 0x417fc8813e80
2014-08-27T13:48:01.043Z cpu1:33242)lldp 0x41801545d000 .data 0x417fc8c00000 .bss 0x417fc8c00040
2014-08-27T13:48:01.043Z cpu1:33242)cdp 0x418015462000 .data 0x417fc9000000 .bss 0x417fc9013400
2014-08-27T13:48:01.043Z cpu1:33242)ipfix 0x41801547e000 .data 0x417fc9400000 .bss 0x417fc9412540
2014-08-27T13:48:01.043Z cpu1:33242)tcpip4 0x418015492000 .data 0x417fc9800000 .bss 0x417fc9818180
2014-08-27T13:48:01.043Z cpu1:33242)dvsdev 0x418015598000 .data 0x417fc9c00000 .bss 0x417fc9c00030
2014-08-27T13:48:01.043Z cpu1:33242)dvfilter 0x41801559b000 .data 0x417fca000000 .bss 0x417fca000b00
2014-08-27T13:48:01.043Z cpu1:33242)lacp 0x4180155bd000 .data 0x417fca400000 .bss 0x417fca400160
2014-08-27T13:48:01.043Z cpu1:33242)hbr_filter 0x4180155c7000 .data 0x417fca800000 .bss 0x417fca800300
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_1_0_0_dvfilter_shim 0x4180155f3000 .data 0x417fcac00000 .bss 0x417fcac009b0
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_0_0_0_dvfilter_shim 0x4180155f4000 .data 0x417fcb000000 .bss 0x417fcb000930
2014-08-27T13:48:01.043Z cpu1:33242)dvfilter-generic-fastpath 0x4180155f5000 .data 0x417fcb400000 .bss 0x417fcb412380
2014-08-27T13:48:01.043Z cpu1:33242)vmkstatelogger 0x41801560f000 .data 0x417fcb800000 .bss 0x417fcb803a00
2014-08-27T13:48:01.043Z cpu1:33242)esxfw 0x418015633000 .data 0x417fcbc00000 .bss 0x417fcbc12d00
2014-08-27T13:48:01.043Z cpu1:33242)dm 0x418015648000 .data 0x417fcc000000 .bss 0x417fcc000000
2014-08-27T13:48:01.043Z cpu1:33242)nmp 0x41801564a000 .data 0x417fcc400000 .bss 0x417fcc403e50
2014-08-27T13:48:01.043Z cpu1:33242)vmw_satp_local 0x41801566d000 .data 0x417fcc800000 .bss 0x417fcc800028
2014-08-27T13:48:01.043Z cpu1:33242)vmw_satp_default_aa 0x41801566f000 .data 0x417fccc00000 .bss 0x417fccc00000
2014-08-27T13:48:01.043Z cpu1:33242)vmw_psp_lib 0x418015670000 .data 0x417fcd000000 .bss 0x417fcd000290
2014-08-27T13:48:01.043Z cpu1:33242)vmw_psp_fixed 0x418015672000 .data 0x417fcd400000 .bss 0x417fcd400000
2014-08-27T13:48:01.043Z cpu1:33242)vmw_psp_rr 0x418015674000 .data 0x417fcd800000 .bss 0x417fcd800068
2014-08-27T13:48:01.043Z cpu1:33242)vmw_psp_mru 0x418015677000 .data 0x417fcdc00000 .bss 0x417fcdc00000
2014-08-27T13:48:01.043Z cpu1:33242)libata_92 0x418015679000 .data 0x417fce000000 .bss 0x417fce002660
2014-08-27T13:48:01.043Z cpu1:33242)libata_9_2_0_0 0x41801569b000 .data 0x417fce400000 .bss 0x417fce401750
2014-08-27T13:48:01.043Z cpu1:33242)libata_9_2_1_0 0x41801569c000 .data 0x417fce800000 .bss 0x417fce801750
2014-08-27T13:48:01.043Z cpu1:33242)usb-storage 0x41801569d000 .data 0x417fcec00000 .bss 0x417fcec04780
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_1_0_0_nmp_shim 0x4180156a9000 .data 0x417fcf000000 .bss 0x417fcf000ca8
2014-08-27T13:48:01.043Z cpu1:33242)vmkapi_v2_0_0_0_nmp_shim 0x4180156aa000 .data 0x417fcf400000 .bss 0x417fcf400ca8
2014-08-27T13:48:01.043Z cpu1:33242)svmmirror 0x4180156ab000 .data 0x417fcf800000 .bss 0x417fcf8000c0
2014-08-27T13:48:01.043Z cpu1:33242)cbt 0x4180156b7000 .data 0x417fcfc00000 .bss 0x417fcfc00080
2014-08-27T13:48:01.043Z cpu1:33242)migrate 0x4180156bb000 .data 0x417fd0000000 .bss 0x417fd0004d40
2014-08-27T13:48:01.043Z cpu1:33242)libfc_92 0x41801571a000 .data 0x417fd0400000 .bss 0x417fd0400b80
2014-08-27T13:48:01.043Z cpu1:33242)libfcoe_92 0x418015733000 .data 0x417fd0800000 .bss 0x417fd08001c0
2014-08-27T13:48:01.043Z cpu1:33242)libfc_9_2_0_0 0x418015739000 .data 0x417fd0c00000 .bss 0x417fd0c00868
2014-08-27T13:48:01.043Z cpu1:33242)libfcoe_9_2_0_0 0x41801573a000 .data 0x417fd1000000 .bss 0x417fd10001f4
2014-08-27T13:48:01.043Z cpu1:33242)libfc_9_2_1_0 0x41801573b000 .data 0x417fd1400000 .bss 0x417fd1400868
2014-08-27T13:48:01.043Z cpu1:33242)libfcoe_9_2_1_0 0x41801573c000 .data 0x417fd1800000 .bss 0x417fd18001f4
2014-08-27T13:48:01.043Z cpu1:33242)ahci 0x41801573d000 .data 0x417fd1c00000 .bss 0x417fd1c00420
2014-08-27T13:48:01.043Z cpu1:33242)sunrpc 0x418015744000 .data 0x417fd2000000 .bss 0x417fd2002b80
2014-08-27T13:48:01.043Z cpu1:33242)nfsclient 0x418015753000 .data 0x417fd2400000 .bss 0x417fd2403940
2014-08-27T13:48:01.043Z cpu1:33242)vmkibft 0x41801576c000 .data 0x417fd2800000 .bss 0x417fd28037c0
2014-08-27T13:48:01.043Z cpu1:33242)lvmdriver 0x41801576f000 .data 0x417fd2c00000 .bss 0x417fd2c03380
2014-08-27T13:48:01.043Z cpu1:33242)deltadisk 0x418015783000 .data 0x417fd3000000 .bss 0x417fd3005c00
2014-08-27T13:48:01.043Z cpu1:33242)tracing 0x4180157ae000 .data 0x417fd3400000 .bss 0x417fd3405b40
2014-08-27T13:48:01.043Z cpu1:33242)rdt 0x4180157b5000 .data 0x417fd3800000 .bss 0x417fd3804e00
2014-08-27T13:48:01.043Z cpu1:33242)vsanutil 0x4180157db000 .data 0x417fd3c00000 .bss 0x417fd3c069c0
2014-08-27T13:48:01.043Z cpu1:33242)lsomcommon 0x4180157fa000 .data 0x417fd4000000 .bss 0x417fd4001680
2014-08-27T13:48:01.043Z cpu1:33242)plog 0x41801582e000 .data 0x417fd4400000 .bss 0x417fd44056c0
2014-08-27T13:48:01.043Z cpu1:33242)vmfs3 0x418015871000 .data 0x417fd4800000 .bss 0x417fd4803840
2014-08-27T13:48:01.043Z cpu1:33242)dvfg-igmp 0x4180158d9000 .data 0x417fd5a00000 .bss 0x417fd5a00208
2014-08-27T13:48:01.043Z cpu1:33242)cmmds_net 0x4180158df000 .data 0x417fd5e00000 .bss 0x417fd5e02f40
2014-08-27T13:48:01.043Z cpu1:33242)cmmds 0x4180158ec000 .data 0x417fd6200000 .bss 0x417fd6204d80
2014-08-27T13:48:01.043Z cpu1:33242)cmmds_resolver 0x418015921000 .data 0x417fd6600000 .bss 0x417fd6600140
2014-08-27T13:48:01.043Z cpu1:33242)vsan 0x41801592d000 .data 0x417fd6a00000 .bss 0x417fd6a1c200
2014-08-27T13:48:01.043Z cpu1:33242)vmklink_mpi 0x418015a48000 .data 0x417fd6e00000 .bss 0x417fd6e02400
2014-08-27T13:48:01.043Z cpu1:33242)swapobj 0x418015a4d000 .data 0x417fd7200000 .bss 0x417fd7203010
2014-08-27T13:48:01.043Z cpu1:33242)osfs 0x418015a55000 .data 0x417fd7600000 .bss 0x417fd7603380
2014-08-27T13:48:01.043Z cpu1:33242)vflash 0x418015a63000 .data 0x417fd7a00000 .bss 0x417fd7a03540
2014-08-27T13:48:01.043Z cpu1:33242)vfc 0x418015a6e000 .data 0x417fd7e00000 .bss 0x417fd7e02ac0
Coredump to disk.
2014-08-27T13:48:01.093Z cpu1:33242)Slot 1 of 1.
2014-08-27T13:48:01.093Z cpu1:33242)Dump: 2212: Using dump slot size 2684354560.
Сообщение об ошибке выглядит так, будто у вас аппаратная проблема с процессором или оперативной памятью. Вы пробовали бегать memtest или, если есть, встроенный инструмент диагностики оборудования, который можно выбрать во время загрузки сервера?
Это определенно похоже на аппаратную ошибку. Вы можете попробовать такой инструмент, как mcelog, который просто действует как переводчик для любого исключения проверки машины (mce). Вы не упомянули, какую версию ОС вы используете, но в RedHat / Centos вы можете установить это через ням
yum install mcelog
и сообщение регистрируется в / var / log / mcelog
Для получения дополнительной информации вы можете обратиться к главному веб-сайту. http://www.mcelog.org/