У нас был двухузловой кластер с GlusterFS.
Это 4-томный кластер, реплика 2, по одному кирпичику на сервер.
Мертвый сервер: nnfs02
Живой сервер со всеми данными: nnfs03
Новый сервер: nnfs01
После того, как nnfs02 умер, мы добавили nnfs01 и смогли удалить блоки для 2 томов из 4, создать новые блоки на nnfs01 с репликой 2 и восстановить том, чтобы на обоих серверах было 100% данных. Названия томов: источник, аватар.
У нас есть два тома, которые мы не можем исправить, и эти два мешают запуску glsuterd. электронные книги / обложки.
Перед перезапуском службы я отсоединил мертвый узел (nnfs02) и попытался удалить его кирпичи. Не повезло.
Журнал отладки: https://gist.github.com/anonymous/d8c412bda8754b4b67a1
[2015-10-21 08:07:35.029300] I [glusterfsd.c:1959:main] 0-glusterd: Started running glusterd version 3.5.5 (glusterd --debug) [2015-10-21 08:07:35.029548] D [glusterfsd.c:596:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol [2015-10-21 08:07:35.032568] I [glusterd.c:1122:init] 0-management: Using /var/lib/glusterd as working directory [2015-10-21 08:07:35.032614] D [glusterd.c:345:glusterd_rpcsvc_options_build] 0-: listen-backlog value: 128 [2015-10-21 08:07:35.032784] D [rpcsvc.c:2183:rpcsvc_init] 0-rpc-service: RPC service inited. [2015-10-21 08:07:35.032802] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2015-10-21 08:07:35.032831] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.5.5/rpc-transport/socket.so [2015-10-21 08:07:35.035483] I [socket.c:3645:socket_init] 0-socket.management: SSL support is NOT enabled [2015-10-21 08:07:35.035505] I [socket.c:3660:socket_init] 0-socket.management: using system polling thread [2015-10-21 08:07:35.035520] D [name.c:557:server_fill_address_family] 0-socket.management: option address-family not specified, defaulting to inet [2015-10-21 08:07:35.035571] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.5.5/rpc-transport/rdma.so [2015-10-21 08:07:35.035607] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.5.5/rpc-transport/rdma.so: cannot open shared object file: No such file or directory [2015-10-21 08:07:35.035619] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine [2015-10-21 08:07:35.035630] W [rpcsvc.c:1535:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2015-10-21 08:07:35.035641] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0 [2015-10-21 08:07:35.035650] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0 [2015-10-21 08:07:35.035660] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0 [2015-10-21 08:07:35.035669] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0 [2015-10-21 08:07:35.035678] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0 [2015-10-21 08:07:35.035687] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0 [2015-10-21 08:07:35.035721] D [rpcsvc.c:2183:rpcsvc_init] 0-rpc-service: RPC service inited. [2015-10-21 08:07:35.035731] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2015-10-21 08:07:35.035754] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.5.5/rpc-transport/socket.so [2015-10-21 08:07:35.035785] D [socket.c:3533:socket_init] 0-socket.management: disabling nodelay [2015-10-21 08:07:35.035797] I [socket.c:3645:socket_init] 0-socket.management: SSL support is NOT enabled [2015-10-21 08:07:35.035806] I [socket.c:3660:socket_init] 0-socket.management: using system polling thread [2015-10-21 08:07:35.035860] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0 [2015-10-21 08:07:35.035873] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0 [2015-10-21 08:07:35.035899] D [glusterd-utils.c:6105:glusterd_sm_tr_log_init] 0-: returning 0 [2015-10-21 08:07:35.035913] D [glusterd.c:1298:init] 0-management: cannot get run-with-valgrind value [2015-10-21 08:07:35.036639] I [glusterd.c:367:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [2015-10-21 08:07:35.037001] D [glusterd.c:400:glusterd_check_gsync_present] 0-glusterd: Returning -1 [2015-10-21 08:07:35.044391] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.044426] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0 [2015-10-21 08:07:35.044473] D [store.c:280:gf_store_retrieve_value] 0-: key operating-version found [2015-10-21 08:07:35.044519] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30504 [2015-10-21 08:07:35.044572] D [glusterd-utils.c:518:glusterd_volinfo_new] 0-management: Returning 0 [2015-10-21 08:07:35.050190] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.050222] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0 [2015-10-21 08:07:35.050245] D [store.c:475:gf_store_iter_new] 0-: Returning with 0 [2015-10-21 08:07:35.050271] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 ... [2015-10-21 08:07:35.050459] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.050482] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=diagnostics.brick-log-level,value:WARNING [2015-10-21 08:07:35.050509] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=diagnostics.client-log-level,value:WARNING [2015-10-21 08:07:35.050553] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.enable-ino32,value:on [2015-10-21 08:07:35.050603] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.addr-namelookup,value:off [2015-10-21 08:07:35.050631] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.disable,value:on [2015-10-21 08:07:35.050655] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-max-file-size,value:2MB [2015-10-21 08:07:35.050678] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-refresh-timeout,value:4 [2015-10-21 08:07:35.050701] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-size,value:256MB [2015-10-21 08:07:35.052059] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.io-thread-count,value:32 [2015-10-21 08:07:35.052089] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2015-10-21 08:07:35.052111] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2015-10-21 08:07:35.052126] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1 [2015-10-21 08:07:35.052164] D [store.c:475:gf_store_iter_new] 0-: Returning with 0 [2015-10-21 08:07:35.052186] D [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0 [2015-10-21 08:07:35.052208] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 .... [2015-10-21 08:07:35.052514] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.057629] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.057658] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0 [2015-10-21 08:07:35.057683] D [store.c:475:gf_store_iter_new] 0-: Returning with 0 [2015-10-21 08:07:35.057710] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 .... [2015-10-21 08:07:35.277883] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.277900] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1 [2015-10-21 08:07:35.277934] D [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0 [2015-10-21 08:07:35.277952] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.284152] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.284173] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0 [2015-10-21 08:07:35.284213] D [store.c:475:gf_store_iter_new] 0-: Returning with 0 [2015-10-21 08:07:35.284248] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 ... [2015-10-21 08:07:35.284297] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.284309] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1 [2015-10-21 08:07:35.284332] D [glusterd-store.c:1636:glusterd_store_retrieve_bricks] 0-: Returning with 0 [2015-10-21 08:07:35.284611] D [glusterd-utils.c:1792:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-10-21 08:07:35.296988] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.297034] D [store.c:280:gf_store_retrieve_value] 0-: key version found [2015-10-21 08:07:35.297055] D [store.c:434:gf_store_handle_destroy] 0-: Returning 0 [2015-10-21 08:07:35.311483] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.311556] D [glusterd-utils.c:1792:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-10-21 08:07:35.311607] D [store.c:348:gf_store_save_value] 0-: returning: 0 [2015-10-21 08:07:35.311633] D [store.c:348:gf_store_save_value] 0-: returning: 0 [2015-10-21 08:07:35.311668] D [glusterd-store.c:2089:glusterd_store_retrieve_volume] 0-: Returning with 0 [2015-10-21 08:07:35.311684] D [glusterd-utils.c:1156:glusterd_volinfo_find] 0-management: Volume source found [2015-10-21 08:07:35.311693] D [glusterd-utils.c:1163:glusterd_volinfo_find] 0-management: Returning 0 [2015-10-21 08:07:35.320306] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.320328] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0 [2015-10-21 08:07:35.320349] D [store.c:475:gf_store_iter_new] 0-: Returning with 0 [2015-10-21 08:07:35.320384] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.320400] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1 [2015-10-21 08:07:35.320418] D [glusterd-store.c:1739:glusterd_store_retrieve_rbstate] 0-: Returning with 0 [2015-10-21 08:07:35.320430] D [glusterd-utils.c:1156:glusterd_volinfo_find] 0-management: Volume source found [2015-10-21 08:07:35.320439] D [glusterd-utils.c:1163:glusterd_volinfo_find] 0-management: Returning 0 [2015-10-21 08:07:35.324899] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.324919] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0 [2015-10-21 08:07:35.324935] D [store.c:475:gf_store_iter_new] 0-: Returning with 0 [2015-10-21 08:07:35.324957] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 ... [2015-10-21 08:07:35.324994] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1 [2015-10-21 08:07:35.325011] D [glusterd-store.c:1850:glusterd_store_retrieve_node_state] 0-: Returning with 0 [2015-10-21 08:07:35.325031] D [glusterd-utils.c:518:glusterd_volinfo_new] 0-management: Returning 0 [2015-10-21 08:07:35.333696] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.333723] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0 [2015-10-21 08:07:35.333743] D [store.c:475:gf_store_iter_new] 0-: Returning with 0 ... [2015-10-21 08:07:35.333787] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.333940] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.333955] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.io-thread-count,value:32 [2015-10-21 08:07:35.333980] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-size,value:256MB [2015-10-21 08:07:35.334003] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-refresh-timeout,value:4 [2015-10-21 08:07:35.334026] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-max-file-size,value:2MB [2015-10-21 08:07:35.334052] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.disable,value:on [2015-10-21 08:07:35.334075] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=diagnostics.client-log-level,value:WARNING [2015-10-21 08:07:35.334097] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=diagnostics.brick-log-level,value:WARNING [2015-10-21 08:07:35.334121] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.addr-namelookup,value:off [2015-10-21 08:07:35.334145] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.enable-ino32,value:on [2015-10-21 08:07:35.334165] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2015-10-21 08:07:35.334186] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2015-10-21 08:07:35.334246] D [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0 [2015-10-21 08:07:35.334267] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 .... [2015-10-21 08:07:35.334290] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.339246] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.339266] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0 [2015-10-21 08:07:35.339286] D [store.c:475:gf_store_iter_new] 0-: Returning with 0 [2015-10-21 08:07:35.339309] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 .... [2015-10-21 08:07:35.339357] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.339369] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1 [2015-10-21 08:07:35.339402] D [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0 [2015-10-21 08:07:35.339417] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.344478] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.344497] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0 [2015-10-21 08:07:35.344517] D [store.c:475:gf_store_iter_new] 0-: Returning with 0 [2015-10-21 08:07:35.344538] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 .... [2015-10-21 08:07:35.344585] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.344597] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1 [2015-10-21 08:07:35.344618] D [glusterd-store.c:1636:glusterd_store_retrieve_bricks] 0-: Returning with 0 [2015-10-21 08:07:35.344832] D [glusterd-utils.c:1792:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-10-21 08:07:35.350431] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.350478] D [store.c:280:gf_store_retrieve_value] 0-: key version found [2015-10-21 08:07:35.350499] D [store.c:434:gf_store_handle_destroy] 0-: Returning 0 [2015-10-21 08:07:35.354961] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.355030] D [glusterd-utils.c:1792:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-10-21 08:07:35.355081] D [store.c:348:gf_store_save_value] 0-: returning: 0 [2015-10-21 08:07:35.355107] D [store.c:348:gf_store_save_value] 0-: returning: 0 [2015-10-21 08:07:35.355142] D [glusterd-store.c:2089:glusterd_store_retrieve_volume] 0-: Returning with 0 [2015-10-21 08:07:35.355162] D [glusterd-utils.c:1156:glusterd_volinfo_find] 0-management: Volume avatar found [2015-10-21 08:07:35.355171] D [glusterd-utils.c:1163:glusterd_volinfo_find] 0-management: Returning 0 [2015-10-21 08:07:35.360423] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.360446] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0 [2015-10-21 08:07:35.360465] D [store.c:475:gf_store_iter_new] 0-: Returning with 0 [2015-10-21 08:07:35.360489] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.360503] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1 [2015-10-21 08:07:35.360521] D [glusterd-store.c:1739:glusterd_store_retrieve_rbstate] 0-: Returning with 0 [2015-10-21 08:07:35.360533] D [glusterd-utils.c:1156:glusterd_volinfo_find] 0-management: Volume avatar found [2015-10-21 08:07:35.360542] D [glusterd-utils.c:1163:glusterd_volinfo_find] 0-management: Returning 0 [2015-10-21 08:07:35.364940] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.364962] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0 [2015-10-21 08:07:35.364980] D [store.c:475:gf_store_iter_new] 0-: Returning with 0 [2015-10-21 08:07:35.365027] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.365040] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1 [2015-10-21 08:07:35.365058] D [glusterd-store.c:1850:glusterd_store_retrieve_node_state] 0-: Returning with 0 [2015-10-21 08:07:35.365073] D [glusterd-utils.c:518:glusterd_volinfo_new] 0-management: Returning 0 [2015-10-21 08:07:35.369827] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.369862] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0 [2015-10-21 08:07:35.369879] D [store.c:475:gf_store_iter_new] 0-: Returning with 0 [2015-10-21 08:07:35.369903] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 .... [2015-10-21 08:07:35.369995] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.370069] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.io-thread-count,value:32 [2015-10-21 08:07:35.370093] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-size,value:256MB [2015-10-21 08:07:35.370117] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-refresh-timeout,value:4 [2015-10-21 08:07:35.370140] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-max-file-size,value:2MB [2015-10-21 08:07:35.370175] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.disable,value:on [2015-10-21 08:07:35.370199] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=diagnostics.client-log-level,value:WARNING [2015-10-21 08:07:35.370222] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=diagnostics.brick-log-level,value:WARNING [2015-10-21 08:07:35.370246] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.addr-namelookup,value:off [2015-10-21 08:07:35.370270] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.enable-ino32,value:on [2015-10-21 08:07:35.370291] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2015-10-21 08:07:35.370311] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2015-10-21 08:07:35.370331] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2 [2015-10-21 08:07:35.370369] D [store.c:475:gf_store_iter_new] 0-: Returning with 0 [2015-10-21 08:07:35.370399] D [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0 [2015-10-21 08:07:35.370421] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 ... [2015-10-21 08:07:35.370445] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.380106] D [store.c:394:gf_store_handle_new] 0-: Returning 0 [2015-10-21 08:07:35.380125] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0 [2015-10-21 08:07:35.380145] D [store.c:475:gf_store_iter_new] 0-: Returning with 0 [2015-10-21 08:07:35.380168] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 ... [2015-10-21 08:07:35.419445] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.419605] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0 [2015-10-21 08:07:35.419619] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.io-thread-count,value:32 [2015-10-21 08:07:35.419643] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-size,value:256MB [2015-10-21 08:07:35.419666] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-refresh-timeout,value:4 [2015-10-21 08:07:35.419688] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-max-file-size,value:2MB [2015-10-21 08:07:35.419713] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.addr-namelookup,value:off [2015-10-21 08:07:35.419747] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.enable-ino32,value:on [2015-10-21 08:07:35.419770] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=diagnostics.client-log-level,value:WARNING [2015-10-21 08:07:35.419792] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=diagnostics.brick-log-level,value:WARNING [2015-10-21 08:07:35.419822] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.disable,value:on [2015-10-21 08:07:35.419844] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2015-10-21 08:07:35.419864] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2015-10-21 08:07:35.419876] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1 [2015-10-21 08:07:35.419902] D [store.c:475:gf_store_iter_new] 0-: Returning with 0 [2015-10-21 08:07:35.419919] D [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0 [2015-10-21 08:07:35.473545] D [glusterd-utils.c:6200:glusterd_peerinfo_new] 0-: returning 0 [2015-10-21 08:07:35.473553] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0 [2015-10-21 08:07:35.473588] D [glusterd-handler.c:2803:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2015-10-21 08:07:35.473614] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-10-21 08:07:35.473632] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.5.5/rpc-transport/socket.so [2015-10-21 08:07:35.473684] I [socket.c:3645:socket_init] 0-management: SSL support is NOT enabled [2015-10-21 08:07:35.473696] I [socket.c:3660:socket_init] 0-management: using system polling thread [2015-10-21 08:07:35.477581] D [common-utils.c:249:gf_resolve_ip6] 0-resolver: returning ip-192.168.97.50 (port-24007) for hostname: nnfs01 and port: 24007 [2015-10-21 08:07:35.477798] D [glusterd-handler.c:2742:glusterd_rpc_create] 0-management: returning 0 [2015-10-21 08:07:35.477829] D [glusterd-handler.c:2803:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2015-10-21 08:07:35.477854] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-10-21 08:07:35.477881] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.5.5/rpc-transport/socket.so [2015-10-21 08:07:35.477939] I [socket.c:3645:socket_init] 0-management: SSL support is NOT enabled [2015-10-21 08:07:35.477952] I [socket.c:3660:socket_init] 0-management: using system polling thread [2015-10-21 08:07:35.481689] D [common-utils.c:249:gf_resolve_ip6] 0-resolver: returning ip-192.168.97.50 (port-24007) for hostname: nnfs01 and port: 24007 [2015-10-21 08:07:35.481773] D [glusterd-handler.c:2742:glusterd_rpc_create] 0-management: returning 0 [2015-10-21 08:07:35.481793] D [glusterd-store.c:2609:glusterd_store_retrieve_peers] 0-: Returning with 0 [2015-10-21 08:07:35.481925] D [glusterd-utils.c:5576:glusterd_friend_find_by_hostname] 0-management: Unable to find friend: nnfs03 [2015-10-21 08:07:35.481991] D [common-utils.c:2931:gf_is_local_addr] 0-management: 127.0.0.1 [2015-10-21 08:07:35.482024] D [store.c:280:gf_store_retrieve_value] 0-: key UUID found [2015-10-21 08:07:35.482046] D [glusterd-store.c:1490:glusterd_retrieve_uuid] 0-: Returning 0 [2015-10-21 08:07:35.482060] I [glusterd.c:138:glusterd_uuid_init] 0-management: retrieved UUID: f42d41b1-61cc-4bdb-9e5e-7079fb9b35b5 [2015-10-21 08:07:35.482079] D [glusterd-utils.c:5611:glusterd_hostname_to_uuid] 0-management: returning 0 [2015-10-21 08:07:35.482090] D [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning 0 [2015-10-21 08:07:35.482111] D [glusterd-utils.c:5527:glusterd_friend_find_by_hostname] 0-management: Friend nnfs01 found.. state: 3 [2015-10-21 08:07:35.482120] D [glusterd-utils.c:5611:glusterd_hostname_to_uuid] 0-management: returning 0 [2015-10-21 08:07:35.482128] D [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning 0 [2015-10-21 08:07:35.482234] D [glusterd-utils.c:5576:glusterd_friend_find_by_hostname] 0-management: Unable to find friend: nnfs03 [2015-10-21 08:07:35.482318] D [common-utils.c:2931:gf_is_local_addr] 0-management: 127.0.0.1 [2015-10-21 08:07:35.482332] D [glusterd-utils.c:5611:glusterd_hostname_to_uuid] 0-management: returning 0 [2015-10-21 08:07:35.482341] D [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning 0 [2015-10-21 08:07:35.482361] D [glusterd-utils.c:5527:glusterd_friend_find_by_hostname] 0-management: Friend nnfs01 found.. state: 3 [2015-10-21 08:07:35.482371] D [glusterd-utils.c:5611:glusterd_hostname_to_uuid] 0-management: returning 0 [2015-10-21 08:07:35.482393] D [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning 0 [2015-10-21 08:07:35.482492] D [glusterd-utils.c:5576:glusterd_friend_find_by_hostname] 0-management: Unable to find friend: nnfs02 [2015-10-21 08:07:35.482560] D [common-utils.c:2931:gf_is_local_addr] 0-management: 192.168.97.51 [2015-10-21 08:07:35.482763] D [common-utils.c:2931:gf_is_local_addr] 0-management: 192.168.97.51 [2015-10-21 08:07:35.482877] D [common-utils.c:2931:gf_is_local_addr] 0-management: 192.168.97.51 [2015-10-21 08:07:35.482983] D [common-utils.c:2947:gf_is_local_addr] 0-management: nnfs02 is not local [2015-10-21 08:07:35.482997] D [glusterd-utils.c:5611:glusterd_hostname_to_uuid] 0-management: returning -1 [2015-10-21 08:07:35.483007] D [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning -1 [2015-10-21 08:07:35.483016] E [glusterd-store.c:2632:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore [2015-10-21 08:07:35.483026] D [glusterd-store.c:2639:glusterd_resolve_all_bricks] 0-: Returning with -1 [2015-10-21 08:07:35.483035] D [glusterd-store.c:2672:glusterd_restore] 0-: Returning -1 [2015-10-21 08:07:35.483048] E [xlator.c:403:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again [2015-10-21 08:07:35.483059] E [graph.c:307:glusterfs_graph_init] 0-management: initializing translator failed [2015-10-21 08:07:35.483069] E [graph.c:502:glusterfs_graph_activate] 0-graph: init failed [2015-10-21 08:07:35.483552] W [glusterfsd.c:1095:cleanup_and_exit] (-->glusterd(main+0x503) [0x7ff9d1c13a93] (-->glusterd(glusterfs_volumes_init+0xd8) [0x7ff9d1c169e8] (-->glusterd(glusterfs_process_volfp+0x106) [0x7ff9d1c168c6]))) 0-: received signum (0), shutting down [2015-10-21 08:07:35.483570] D [glusterfsd-mgmt.c:2025:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given
Мертвые кирпичи все еще записаны в конфигурационном файле тома, например:
[root@nNFS03 ebooks]# cat /var/lib/glusterd/vols/ebooks/info type=2 count=3 status=1 sub_count=3 stripe_count=1 replica_count=3 version=22 transport-type=0 volume-id=4ab9ebcb-2172-4027-8760-XXXXXXXX username=537ddb35-089c-44fe-922e-XXXXXXXX password=6249f135-17ea-453b-a4b3-XXXXXXXX op-version=2 client-op-version=2 performance.io-thread-count=32 performance.cache-size=256MB performance.cache-refresh-timeout=4 performance.cache-max-file-size=2MB nfs.disable=on diagnostics.client-log-level=WARNING diagnostics.brick-log-level=WARNING nfs.addr-namelookup=off nfs.enable-ino32=on brick-0=nnfs02:-data-gluster-ebooks brick-1=nnfs03:-data-gluster-ebooks brick-2=nnfs01:-data-gluster-ebooks
Подводя итог: nnfs02 мертв
nnfs03 - это живой сервер со всеми данными
nnfs01 - это новый действующий сервер, который в настоящее время реплицирует только «аватары» / «исходные» тома. Также следует копировать «электронные книги» и «обложки».
glusterd не запускается. См. Прилагаемый журнал отладки.