Я запускаю MongoDB с движком WiredTiger в файловой системе ext4. Последнее, что я сделал, - это начал создавать индекс, но потом что-то случилось, и мне нужно было отменить построение индекса и перезапустить машину. Я сделал это, нажав Ctrl + C в консоли MongoDB, на которой выполнялась сборка индекса. Он вежливо спросил меня, хочу ли я остановить построение индекса, и я сказал «да», затем я дождался его закрытия, что заняло несколько секунд, так что казалось, что все в порядке. Затем (думаю, через 13 секунд) я выключил сервер, используя systemctl.
Соответствующий журнал:
2015-06-04T22:41:56.176Z I - [conn1] Index Build: 5950100/253027947 2%
2015-06-04T22:41:59.022Z I - [conn1] Index Build: 5968100/253027947 2%
2015-06-04T22:42:02.002Z I - [conn1] Index Build: 5999700/253027947 2%
2015-06-04T22:42:05.086Z I - [conn1] Index Build: 6043300/253027947 2%
2015-06-04T22:42:08.114Z I - [conn1] Index Build: 6067900/253027947 2%
2015-06-04T22:42:11.006Z I - [conn1] Index Build: 6096800/253027947 2%
2015-06-04T22:42:14.007Z I - [conn1] Index Build: 6153800/253027947 2%
2015-06-04T22:42:16.588Z I COMMAND [conn3] going to kill op: op: 19
2015-06-04T22:42:29.282Z I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2015-06-04T22:42:31.766Z I COMMAND [conn1] command contributions.$cmd command: createIndexes { createIndexes: "contributions", indexes: [ { key: { created_at: 1.0 }, name: "created_at_1" } ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:151 locks:{ Global: { acquireCount: { w: 1 } }, Database: { acquireCount: { W: 1 } }, Collection: { acquireCount: { w: 1 } } } 244229ms
2015-06-04T22:42:31.930Z I CONTROL [signalProcessingThread] now exiting
2015-06-04T22:42:31.931Z I NETWORK [signalProcessingThread] shutdown: going to close listening sockets...
2015-06-04T22:42:31.931Z I NETWORK [signalProcessingThread] closing listening socket: 6
2015-06-04T22:42:31.931Z I NETWORK [signalProcessingThread] closing listening socket: 7
2015-06-04T22:42:31.932Z I NETWORK [signalProcessingThread] closing listening socket: 8
2015-06-04T22:42:31.967Z I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2015-06-04T22:42:31.967Z I NETWORK [signalProcessingThread] shutdown: going to flush diaglog...
2015-06-04T22:42:31.967Z I NETWORK [signalProcessingThread] shutdown: going to close sockets...
2015-06-04T22:42:31.967Z I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down
2015-06-04T22:42:31.995Z I COMMAND [conn2] command contributions.$cmd command: dbStats { dbstats: 1.0, scale: undefined } keyUpdates:0 writeConflicts:0 numYields:0 reslen:196 locks:{ Global: { acquireCount: { r: 1 } }, Database: { acquireCount: { R: 1 }, acquireWaitCount: { R: 1 }, timeAcquiringMicros: { R: 26993306862 } } } 164304ms
2015-06-04T22:42:32.047Z I NETWORK [conn2] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:57879]
2015-06-04T22:42:38.394Z I STORAGE [signalProcessingThread] shutdown: removing fs lock...
2015-06-04T22:42:38.395Z I CONTROL [signalProcessingThread] dbexit: rc: 0
Выглядит пока хорошо? Затем я снова начал запускать сервер:
2015-06-04T22:48:34.118Z I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-06-04T22:48:34.179Z E STORAGE [initandlisten] WiredTiger (0) [1433458114:179527][216:0x7f672d3c9b80], file:WiredTiger.wt, connection: read checksum error [4096B @ 12288, 270058226 != 116927414]
2015-06-04T22:48:34.179Z E STORAGE [initandlisten] WiredTiger (0) [1433458114:179630][216:0x7f672d3c9b80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2015-06-04T22:48:34.179Z E STORAGE [initandlisten] WiredTiger (-31804) [1433458114:179646][216:0x7f672d3c9b80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-06-04T22:48:34.179Z I - [initandlisten] Fatal Assertion 28558
2015-06-04T22:48:34.232Z I CONTROL [initandlisten]
0x1008a19 0xf98405 0xf8408a 0xdf811f 0x13df42e 0x13df571 0x13dfac8 0x134ffc6 0x134ff2e 0x134db1c 0x13514b6 0x135ea83 0x1382ee6 0x13de89d 0x13b5916 0x137ed3b 0xddf117 0xdde7ce 0xa9f8ff 0x7cbf96 0x7cb5cf 0x7d1b54 0x7f672a51a790 0x7cb4e9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"C08A19"},{"b":"400000","o":"B98405"},{"b":"400000","o":"B8408A"},{"b":"400000","o":"9F811F"},{"b":"400000","o":"FDF42E"},{"b":"400000","o":"FDF571"},{"b":"400000","o":"FDFAC8"},{"b":"400000","o":"F4FFC6"},{"b":"400000","o":"F4FF2E"},{"b":"400000","o":"F4DB1C"},{"b":"400000","o":"F514B6"},{"b":"400000","o":"F5EA83"},{"b":"400000","o":"F82EE6"},{"b":"400000","o":"FDE89D"},{"b":"400000","o":"FB5916"},{"b":"400000","o":"F7ED3B"},{"b":"400000","o":"9DF117"},{"b":"400000","o":"9DE7CE"},{"b":"400000","o":"69F8FF"},{"b":"400000","o":"3CBF96"},{"b":"400000","o":"3CB5CF"},{"b":"400000","o":"3D1B54"},{"b":"7F672A4FA000","o":"20790"},{"b":"400000","o":"3CB4E9"}],"processInfo":{ "mongodbVersion" : "3.0.3", "gitVersion" : "nogitversion", "uname" : { "sysname" : "Linux", "release" : "4.0.4-2-ARCH", "version" : "#1 SMP PREEMPT Fri May 22 03:05:23 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF8256A000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "7824FD6A99160FBCC32A18B30D22DA47D857F8B2" }, { "b" : "7F672CFB7000", "path" : "/usr/lib/libsnappy.so.1", "elfType" : 3, "buildId" : "9D4BD954C9638C5AF8F33C9E757D5E37C9FA8ED2" }, { "b" : "7F672CD39000", "path" : "/usr/lib/libboost_program_options.so.1.58.0", "elfType" : 3, "buildId" : "7ED7CC3A2967448967E729DD716DE1E8DC1082A5" }, { "b" : "7F672CB21000", "path" : "/usr/lib/libboost_filesystem.so.1.58.0", "elfType" : 3, "buildId" : "69307A2E931F589C000ABB2535CB443E2E6A7FC3" }, { "b" : "7F672C8FA000", "path" : "/usr/lib/libboost_thread.so.1.58.0", "elfType" : 3, "buildId" : "56D33D6B2101D6AFB786566C7AA105F443DDDEA0" }, { "b" : "7F672C6F6000", "path" : "/usr/lib/libboost_system.so.1.58.0", "elfType" : 3, "buildId" : "0AC74F672BD854B0757E24D168DAB10747E1EEB3" }, { "b" : "7F672C486000", "path" : "/usr/lib/libpcre.so.1", "elfType" : 3, "buildId" : "57C4AA1F4B17299142D8F86475D3A517CA49FA5D" }, { "b" : "7F672C27C000", "path" : "/usr/lib/libpcrecpp.so.0", "elfType" : 3, "buildId" : "261C39169D4A9A6DAED28AD8FEA628073620A943" }, { "b" : "7F672C05F000", "path" : "/usr/lib/libsasl2.so.3", "elfType" : 3, "buildId" : "77D3CE2669A200A63E578D8CF5381BB0162EC0B6" }, { "b" : "7F672BE58000", "path" : "/usr/lib/libatomic.so.1", "elfType" : 3, "buildId" : "BC43111A4C7077951E47D2DE930B3823B53616F9" }, { "b" : "7F672BB54000", "path" : "/usr/lib/libm.so.6", "elfType" : 3, "buildId" : "D428918FE286A2A6D01CA9E0BF2E05A97AF21AD2" }, { "b" : "7F672B937000", "path" : "/usr/lib/libpthread.so.0", "elfType" : 3, "buildId" : "23D7FC97EEAE8C49BF5CEF84119FE79695C661D7" }, { "b" : "7F672B6BC000", "path" : "/usr/lib/libssl.so.1.0.0", "elfType" : 3, "buildId" : "3E18725D6F353BB32DE7A328B11698E801245728" }, { "b" : "7F672B240000", "path" : "/usr/lib/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "A85224F6CF75D52869455B19FE0CA8C9E14DEF8A" }, { "b" : "7F672B038000", "path" : "/usr/lib/librt.so.1", "elfType" : 3, "buildId" : "182D2FA7DB1838557442258CC3B534D0B2840C94" }, { "b" : "7F672AE34000", "path" : "/usr/lib/libdl.so.2", "elfType" : 3, "buildId" : "A707ACB29AB2BE8CDC017FD508B684E16B6FA810" }, { "b" : "7F672AAB2000", "path" : "/usr/lib/libstdc++.so.6", "elfType" : 3, "buildId" : "EB9EE0FBECCB790310F3572088014B20FA2BA931" }, { "b" : "7F672A89C000", "path" : "/usr/lib/libgcc_s.so.1", "elfType" : 3, "buildId" : "C5984028332F1B178982ED90432612CECFD136AD" }, { "b" : "7F672A4FA000", "path" : "/usr/lib/libc.so.6", "elfType" : 3, "buildId" : "B440183D6AEFAC5A9259F6E2DE824E43E85ED341" }, { "b" : "7F672D1BD000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "F68A065085ADDE71B86C49009A1B30068202DED4" }, { "b" : "7F672A2E3000", "path" : "/usr/lib/libresolv.so.2", "elfType" : 3, "buildId" : "6B73CDD765109B316C29E219E337F017E00CF1D4" }, { "b" : "7F672A0CD000", "path" : "/usr/lib/libz.so.1", "elfType" : 3, "buildId" : "7212A4C7A99747A95CEE8EA4675FB62217EF4626" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0x1008a19]
mongod(_ZN5mongo10logContextEPKc+0x105) [0xf98405]
mongod(_ZN5mongo13fassertFailedEi+0xDA) [0xf8408a]
mongod(+0x9F811F) [0xdf811f]
mongod(__wt_eventv+0x44E) [0x13df42e]
mongod(__wt_err+0xA1) [0x13df571]
mongod(__wt_illegal_value+0x58) [0x13dfac8]
mongod(__wt_block_extlist_read+0x66) [0x134ffc6]
mongod(__wt_block_extlist_read_avail+0x1E) [0x134ff2e]
mongod(__wt_block_checkpoint_load+0x29C) [0x134db1c]
mongod(+0xF514B6) [0x13514b6]
mongod(__wt_btree_open+0x573) [0x135ea83]
mongod(__wt_conn_btree_get+0x226) [0x1382ee6]
mongod(__wt_session_get_btree+0x36D) [0x13de89d]
mongod(__wt_metadata_open+0x26) [0x13b5916]
mongod(wiredtiger_open+0xD2B) [0x137ed3b]
mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x407) [0xddf117]
mongod(+0x9DE7CE) [0xdde7ce]
mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x29F) [0xa9f8ff]
mongod(+0x3CBF96) [0x7cbf96]
mongod(_ZN5mongo13initAndListenEi+0xF) [0x7cb5cf]
mongod(main+0x9C4) [0x7d1b54]
libc.so.6(__libc_start_main+0xF0) [0x7f672a51a790]
mongod(_start+0x29) [0x7cb4e9]
----- END BACKTRACE -----
2015-06-04T22:48:34.232Z I - [initandlisten]
***aborting after fassert() failure
Я запустил это, чтобы попытаться исправить (что я в основном получил от эта почта):
sudo -u mongodb mongod --repair --dbpath /data/db --storageEngine wiredTiger
Но это дало мне:
2015-06-05T14:18:16.733-0400 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-06-05T14:18:16.740-0400 E STORAGE [initandlisten] WiredTiger (0) [1433528296:740719][9508:0x7f69443b6b80], file:WiredTiger.wt, connection: read checksum error [4096B @ 12288, 270058226 != 116927414]
2015-06-05T14:18:16.740-0400 E STORAGE [initandlisten] WiredTiger (0) [1433528296:740854][9508:0x7f69443b6b80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2015-06-05T14:18:16.740-0400 E STORAGE [initandlisten] WiredTiger (-31804) [1433528296:740878][9508:0x7f69443b6b80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-06-05T14:18:16.740-0400 I - [initandlisten] Fatal Assertion 28558
2015-06-05T14:18:16.747-0400 I CONTROL [initandlisten]
0x1008a19 0xf98405 0xf8408a 0xdf811f 0x13df42e 0x13df571 0x13dfac8 0x134ffc6 0x134ff2e 0x134db1c 0x13514b6 0x135ea83 0x1382ee6 0x13de89d 0x13b5916 0x137ed3b 0xddf117 0xdde7ce 0xa9f8ff 0x7cbf96 0x7cb5cf 0x7d1b54 0x7f6941507790 0x7cb4e9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"C08A19"},{"b":"400000","o":"B98405"},{"b":"400000","o":"B8408A"},{"b":"400000","o":"9F811F"},{"b":"400000","o":"FDF42E"},{"b":"400000","o":"FDF571"},{"b":"400000","o":"FDFAC8"},{"b":"400000","o":"F4FFC6"},{"b":"400000","o":"F4FF2E"},{"b":"400000","o":"F4DB1C"},{"b":"400000","o":"F514B6"},{"b":"400000","o":"F5EA83"},{"b":"400000","o":"F82EE6"},{"b":"400000","o":"FDE89D"},{"b":"400000","o":"FB5916"},{"b":"400000","o":"F7ED3B"},{"b":"400000","o":"9DF117"},{"b":"400000","o":"9DE7CE"},{"b":"400000","o":"69F8FF"},{"b":"400000","o":"3CBF96"},{"b":"400000","o":"3CB5CF"},{"b":"400000","o":"3D1B54"},{"b":"7F69414E7000","o":"20790"},{"b":"400000","o":"3CB4E9"}],"processInfo":{ "mongodbVersion" : "3.0.3", "gitVersion" : "nogitversion", "uname" : { "sysname" : "Linux", "release" : "4.0.4-2-ARCH", "version" : "#1 SMP PREEMPT Fri May 22 03:05:23 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF90F78000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "7824FD6A99160FBCC32A18B30D22DA47D857F8B2" }, { "b" : "7F6943FA4000", "path" : "/usr/lib/libsnappy.so.1", "elfType" : 3, "buildId" : "9D4BD954C9638C5AF8F33C9E757D5E37C9FA8ED2" }, { "b" : "7F6943D26000", "path" : "/usr/lib/libboost_program_options.so.1.58.0", "elfType" : 3, "buildId" : "7ED7CC3A2967448967E729DD716DE1E8DC1082A5" }, { "b" : "7F6943B0E000", "path" : "/usr/lib/libboost_filesystem.so.1.58.0", "elfType" : 3, "buildId" : "69307A2E931F589C000ABB2535CB443E2E6A7FC3" }, { "b" : "7F69438E7000", "path" : "/usr/lib/libboost_thread.so.1.58.0", "elfType" : 3, "buildId" : "56D33D6B2101D6AFB786566C7AA105F443DDDEA0" }, { "b" : "7F69436E3000", "path" : "/usr/lib/libboost_system.so.1.58.0", "elfType" : 3, "buildId" : "0AC74F672BD854B0757E24D168DAB10747E1EEB3" }, { "b" : "7F6943473000", "path" : "/usr/lib/libpcre.so.1", "elfType" : 3, "buildId" : "A6006C4B8CB56618E21731F3437C6E233497A281" }, { "b" : "7F6943269000", "path" : "/usr/lib/libpcrecpp.so.0", "elfType" : 3, "buildId" : "D4D45AA15DD8E12F9B82072576B74DE99F5681BA" }, { "b" : "7F694304C000", "path" : "/usr/lib/libsasl2.so.3", "elfType" : 3, "buildId" : "77D3CE2669A200A63E578D8CF5381BB0162EC0B6" }, { "b" : "7F6942E45000", "path" : "/usr/lib/libatomic.so.1", "elfType" : 3, "buildId" : "BC43111A4C7077951E47D2DE930B3823B53616F9" }, { "b" : "7F6942B41000", "path" : "/usr/lib/libm.so.6", "elfType" : 3, "buildId" : "D428918FE286A2A6D01CA9E0BF2E05A97AF21AD2" }, { "b" : "7F6942924000", "path" : "/usr/lib/libpthread.so.0", "elfType" : 3, "buildId" : "23D7FC97EEAE8C49BF5CEF84119FE79695C661D7" }, { "b" : "7F69426A9000", "path" : "/usr/lib/libssl.so.1.0.0", "elfType" : 3, "buildId" : "3E18725D6F353BB32DE7A328B11698E801245728" }, { "b" : "7F694222D000", "path" : "/usr/lib/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "A85224F6CF75D52869455B19FE0CA8C9E14DEF8A" }, { "b" : "7F6942025000", "path" : "/usr/lib/librt.so.1", "elfType" : 3, "buildId" : "182D2FA7DB1838557442258CC3B534D0B2840C94" }, { "b" : "7F6941E21000", "path" : "/usr/lib/libdl.so.2", "elfType" : 3, "buildId" : "A707ACB29AB2BE8CDC017FD508B684E16B6FA810" }, { "b" : "7F6941A9F000", "path" : "/usr/lib/libstdc++.so.6", "elfType" : 3, "buildId" : "EB9EE0FBECCB790310F3572088014B20FA2BA931" }, { "b" : "7F6941889000", "path" : "/usr/lib/libgcc_s.so.1", "elfType" : 3, "buildId" : "C5984028332F1B178982ED90432612CECFD136AD" }, { "b" : "7F69414E7000", "path" : "/usr/lib/libc.so.6", "elfType" : 3, "buildId" : "B440183D6AEFAC5A9259F6E2DE824E43E85ED341" }, { "b" : "7F69441AA000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "F68A065085ADDE71B86C49009A1B30068202DED4" }, { "b" : "7F69412D0000", "path" : "/usr/lib/libresolv.so.2", "elfType" : 3, "buildId" : "6B73CDD765109B316C29E219E337F017E00CF1D4" }, { "b" : "7F69410BA000", "path" : "/usr/lib/libz.so.1", "elfType" : 3, "buildId" : "82A1A5B6D7F272E064155BB64576230AEFAFA7B8" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0x1008a19]
mongod(_ZN5mongo10logContextEPKc+0x105) [0xf98405]
mongod(_ZN5mongo13fassertFailedEi+0xDA) [0xf8408a]
mongod(+0x9F811F) [0xdf811f]
mongod(__wt_eventv+0x44E) [0x13df42e]
mongod(__wt_err+0xA1) [0x13df571]
mongod(__wt_illegal_value+0x58) [0x13dfac8]
mongod(__wt_block_extlist_read+0x66) [0x134ffc6]
mongod(__wt_block_extlist_read_avail+0x1E) [0x134ff2e]
mongod(__wt_block_checkpoint_load+0x29C) [0x134db1c]
mongod(+0xF514B6) [0x13514b6]
mongod(__wt_btree_open+0x573) [0x135ea83]
mongod(__wt_conn_btree_get+0x226) [0x1382ee6]
mongod(__wt_session_get_btree+0x36D) [0x13de89d]
mongod(__wt_metadata_open+0x26) [0x13b5916]
mongod(wiredtiger_open+0xD2B) [0x137ed3b]
mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x407) [0xddf117]
mongod(+0x9DE7CE) [0xdde7ce]
mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x29F) [0xa9f8ff]
mongod(+0x3CBF96) [0x7cbf96]
mongod(_ZN5mongo13initAndListenEi+0xF) [0x7cb5cf]
mongod(main+0x9C4) [0x7d1b54]
libc.so.6(__libc_start_main+0xF0) [0x7f6941507790]
mongod(_start+0x29) [0x7cb4e9]
----- END BACKTRACE -----
2015-06-05T14:18:16.747-0400 I - [initandlisten]
***aborting after fassert() failure
Так что, видимо, он настолько сломан, что --repair нужно открыть не открывается?