Назад | Перейти на главную страницу

Не удалось снова присоединить узел кластера MariaDB Galera к кластеру после обновления программного обеспечения. Параметры wsrep_ * в my.cnf, похоже, игнорируются

Некоторое время я запускал кластер Galera, MariaDB версии 10.0.20, Galera версии 25.3.10, все на Gentoo. Этот кластер работал нормально, узлы несколько раз перезагружались для обслуживания и так далее. Когда я запускал mysql на любом узле, он загружал библиотеку wsrep_provider, которая запускала SST, а затем в конечном итоге присоединялась к кластеру.

Теперь пришло время провести масштабное обновление. Я выбрал "постепенное обновление", описанное в http://galeracluster.com/documentation-webpages/upgrading.html#id1 , снимите выбранный узел и обновите там программное обеспечение.

Для нетерпеливых, краткое изложение того, что следует: mariadb запускается и игнорирует любые параметры конфигурации wsrep_, и если я укажу их вручную в запущенном экземпляре, он, похоже, зависнет.

Два вопроса: - Почему он молча игнорирует мои параметры конфигурации? Как заставить его уважать их? - Что сделать, чтобы запустить репликацию, как включить узел в кластер?

Все поднялось нормально. Сейчас у меня MariaDB 10.1.13, Galera 25.3.15. Конфигурация в /etc/mysql/my.cnf была объединена. Теперь это выглядит так (только раздел mysqld):

[mysqld]
character-set-server            = utf8
user                            = mysql
port                            = 3306
socket                          = /var/run/mysqld/mysqld.sock
pid-file                        = /var/run/mysqld/mysqld.pid
log-error                       = /var/log/mysql/mysqld.err
basedir                         = /usr
datadir                         = /var/lib/mysql
skip-external-locking
key_buffer_size                 = 16M
max_allowed_packet              = 4M
table_open_cache                = 400
sort_buffer_size                = 512K
net_buffer_length               = 16K
read_buffer_size                = 256K
read_rnd_buffer_size            = 512K
myisam_sort_buffer_size         = 8M
lc_messages_dir                 = /usr/share/mysql
lc_messages                     = en_US

log-bin
server-id                       = 1

tmpdir                          = /tmp/

innodb_buffer_pool_size = 128M
innodb_data_file_path = ibdata1:10M:autoextend:max:128M
innodb_log_file_size = 48M
innodb_log_buffer_size = 8M
innodb_log_files_in_group=2
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
innodb_file_per_table

binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
query_cache_size=0
query_cache_type=0
bind-address=0.0.0.0
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="www_cluster"
wsrep_cluster_address="gcomm://192.168.4.28,192.168.5.28,192.168.5.29"
wsrep_sst_method=rsync

Я трижды проверил этот файл, чтобы убедиться, что мои параметры wsrep_ * находятся в разделе [mysqld], как и были. Эти параметры точно такие же, как и до обновления.

Однако служба mysql теперь запускается как обычный автономный сервер, а не кластерный:

2016-04-04 16:30:46 139950286550848 [Warning] No argument was provided to --log-bin and neither --log-basename or --log-bin-index where used;  This may cause repliction to break when this server acts as a master and has its hostname changed! Please use '--log-basename=www1' or '--log-bin=mysqld-bin' to avoid this problem.
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: The InnoDB memory heap is disabled
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Memory barrier is not used
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Compressed tables use zlib 1.2.8
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Using Linux native AIO
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Using generic crc32 instructions
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Completed initialization of buffer pool
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Highest supported file format is Barracuda.
2016-04-04 16:30:47 139950286550848 [Note] InnoDB: 128 rollback segment(s) are active.
2016-04-04 16:30:47 139950286550848 [Note] InnoDB: Waiting for purge to start
2016-04-04 16:30:47 139950286550848 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.28-76.1 started; log sequence number 116616176
2016-04-04 16:30:47 139949849761536 [Note] InnoDB: Dumping buffer pool(s) not yet started
2016-04-04 16:30:47 139950286550848 [Note] Plugin 'FEEDBACK' is disabled.
2016-04-04 16:30:47 139950286550848 [Note] Server socket created on IP: '0.0.0.0'.
2016-04-04 16:30:47 139950286550848 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.1.13-MariaDB'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Source distribution

Если я запрашиваю переменные wsrep_ *, он показывает, что поставщик не загружен:

MariaDB [(none)]> SHOW STATUS LIKE 'wsrep%';
+--------------------------+----------------------+
| Variable_name            | Value                |
+--------------------------+----------------------+
| wsrep_cluster_conf_id    | 18446744073709551615 |
| wsrep_cluster_size       | 0                    |
| wsrep_cluster_state_uuid |                      |
| wsrep_cluster_status     | Disconnected         |
| wsrep_connected          | OFF                  |
| wsrep_local_bf_aborts    | 0                    |
| wsrep_local_index        | 18446744073709551615 |
| wsrep_provider_name      |                      |
| wsrep_provider_vendor    |                      |
| wsrep_provider_version   |                      |
| wsrep_ready              | OFF                  |
| wsrep_thread_count       | 0                    |
+--------------------------+----------------------+
12 rows in set (0.00 sec)

Похоже, он не соблюдал параметры wsrep_provider и wsrep_cluster_address (и я верю, что все остальные wsrep_ *) параметры.

Я попытался установить их глобально при запуске экземпляра mariadb вручную:

MariaDB [(none)]> set global wsrep_provider='/usr/lib/galera/libgalera_smm.so';
Query OK, 0 rows affected (0.02 sec)
MariaDB [(none)]> SHOW STATUS LIKE 'wsrep%';
+------------------------------+-----------------------------------+
| Variable_name                | Value                             |
+------------------------------+-----------------------------------+
| wsrep_apply_oooe             | 0.000000                          |
| wsrep_apply_oool             | 0.000000                          |
| wsrep_apply_window           | 0.000000                          |
| wsrep_causal_reads           | 0                                 |
| wsrep_cert_deps_distance     | 0.000000                          |
| wsrep_cert_index_size        | 0                                 |
| wsrep_cert_interval          | 0.000000                          |
| wsrep_cluster_conf_id        | 18446744073709551615              |
| wsrep_cluster_size           | 0                                 |
| wsrep_cluster_state_uuid     |                                   |
| wsrep_cluster_status         | Disconnected                      |
| wsrep_commit_oooe            | 0.000000                          |
| wsrep_commit_oool            | 0.000000                          |
| wsrep_commit_window          | 0.000000                          |
| wsrep_connected              | OFF                               |
| wsrep_flow_control_paused    | 0.000000                          |
| wsrep_flow_control_paused_ns | 0                                 |
| wsrep_flow_control_recv      | 0                                 |
| wsrep_flow_control_sent      | 0                                 |
| wsrep_incoming_addresses     |                                   |
| wsrep_last_committed         | 18446744073709551615              |
| wsrep_local_bf_aborts        | 0                                 |
| wsrep_local_cached_downto    | 18446744073709551615              |
| wsrep_local_cert_failures    | 0                                 |
| wsrep_local_commits          | 0                                 |
| wsrep_local_index            | 18446744073709551615              |
| wsrep_local_recv_queue       | 0                                 |
| wsrep_local_recv_queue_avg   | 0.000000                          |
| wsrep_local_recv_queue_max   | 0                                 |
| wsrep_local_recv_queue_min   | 0                                 |
| wsrep_local_replays          | 0                                 |
| wsrep_local_send_queue       | 0                                 |
| wsrep_local_send_queue_avg   | 0.000000                          |
| wsrep_local_send_queue_max   | 0                                 |
| wsrep_local_send_queue_min   | 0                                 |
| wsrep_local_state            | 0                                 |
| wsrep_local_state_comment    | Initialized                       |
| wsrep_local_state_uuid       |                                   |
| wsrep_protocol_version       | 18446744073709551615              |
| wsrep_provider_name          | Galera                            |
| wsrep_provider_vendor        | Codership Oy <info@codership.com> |
| wsrep_provider_version       | 3.15(r8459459)                    |
| wsrep_ready                  | OFF                               |
| wsrep_received               | 0                                 |
| wsrep_received_bytes         | 0                                 |
| wsrep_repl_data_bytes        | 0                                 |
| wsrep_repl_keys              | 0                                 |
| wsrep_repl_keys_bytes        | 0                                 |
| wsrep_repl_other_bytes       | 0                                 |
| wsrep_replicated             | 0                                 |
| wsrep_replicated_bytes       | 0                                 |
| wsrep_thread_count           | 0                                 |
+------------------------------+-----------------------------------+
52 rows in set (0.00 sec)

соответствующие записи журнала:

2016-04-04 16:45:19 139950170823424 [Note] WSREP: Stop replication
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Provider was not loaded, in stop replication
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Initial position: 913d310a-9360-11e4-9d31-922a1e5d98fe:63026
2016-04-04 16:45:19 139950170823424 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2016-04-04 16:45:19 139950170823424 [Note] WSREP: wsrep_load(): Galera 3.15(r8459459) by Codership Oy <info@codership.com> loaded successfully.
2016-04-04 16:45:19 139950170823424 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.4.28; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false;
2016-04-04 16:45:19 139949719164672 [Note] WSREP: Service thread queue flushed.
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1

и заполняем адреса кластера:

MariaDB [(none)]> set global wsrep_cluster_address='gcomm://192.168.4.28,192.168.5.28,192.168.5.29';    
Query OK, 0 rows affected (3.55 sec)

MariaDB [(none)]> SHOW STATUS LIKE 'wsrep%';
+------------------------------+-------------------------------------------------------+
| Variable_name                | Value                                                 |
+------------------------------+-------------------------------------------------------+
| wsrep_apply_oooe             | 0.000000                                              |
| wsrep_apply_oool             | 0.000000                                              |
| wsrep_apply_window           | 0.000000                                              |
| wsrep_causal_reads           | 0                                                     |
| wsrep_cert_deps_distance     | 0.000000                                              |
| wsrep_cert_index_size        | 0                                                     |
| wsrep_cert_interval          | 0.000000                                              |
| wsrep_cluster_conf_id        | 471                                                   |
| wsrep_cluster_size           | 3                                                     |
| wsrep_cluster_state_uuid     | 913d310a-9360-11e4-9d31-922a1e5d98fe                  |
| wsrep_cluster_status         | Primary                                               |
| wsrep_commit_oooe            | 0.000000                                              |
| wsrep_commit_oool            | 0.000000                                              |
| wsrep_commit_window          | 0.000000                                              |
| wsrep_connected              | ON                                                    |
| wsrep_evs_delayed            |                                                       |
| wsrep_evs_evict_list         |                                                       |
| wsrep_evs_repl_latency       | 0.0456547/0.049131/0.0560754/0.00491047/3             |
| wsrep_evs_state              | OPERATIONAL                                           |
| wsrep_flow_control_paused    | 0.000000                                              |
| wsrep_flow_control_paused_ns | 0                                                     |
| wsrep_flow_control_recv      | 0                                                     |
| wsrep_flow_control_sent      | 0                                                     |
| wsrep_gcomm_uuid             | fd416235-fa6b-11e5-867c-72011d51ce8d                  |
| wsrep_incoming_addresses     | 192.168.5.28:3306,192.168.5.29:3306,192.168.4.28:3306 |
| wsrep_last_committed         | 18446744073709551615                                  |
| wsrep_local_bf_aborts        | 0                                                     |
| wsrep_local_cached_downto    | 18446744073709551615                                  |
| wsrep_local_cert_failures    | 0                                                     |
| wsrep_local_commits          | 0                                                     |
| wsrep_local_index            | 2                                                     |
| wsrep_local_recv_queue       | 0                                                     |
| wsrep_local_recv_queue_avg   | 0.000000                                              |
| wsrep_local_recv_queue_max   | 1                                                     |
| wsrep_local_recv_queue_min   | 0                                                     |
| wsrep_local_replays          | 0                                                     |
| wsrep_local_send_queue       | 0                                                     |
| wsrep_local_send_queue_avg   | 0.000000                                              |
| wsrep_local_send_queue_max   | 1                                                     |
| wsrep_local_send_queue_min   | 0                                                     |
| wsrep_local_state            | 1                                                     |
| wsrep_local_state_comment    | Joining: receiving State Transfer                     |
| wsrep_local_state_uuid       |                                                       |
| wsrep_protocol_version       | 7                                                     |
| wsrep_provider_name          | Galera                                                |
| wsrep_provider_vendor        | Codership Oy <info@codership.com>                     |
| wsrep_provider_version       | 3.15(r8459459)                                        |
| wsrep_ready                  | OFF                                                   |
| wsrep_received               | 1                                                     |
| wsrep_received_bytes         | 266                                                   |
| wsrep_repl_data_bytes        | 0                                                     |
| wsrep_repl_keys              | 0                                                     |
| wsrep_repl_keys_bytes        | 0                                                     |
| wsrep_repl_other_bytes       | 0                                                     |
| wsrep_replicated             | 0                                                     |
| wsrep_replicated_bytes       | 0                                                     |
| wsrep_thread_count           | 2                                                     |
+------------------------------+-------------------------------------------------------+
57 rows in set (0.00 sec)

соответствующий журнал:

2016-04-04 16:45:19 139949719164672 [Note] WSREP: Service thread queue flushed.
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2016-04-04 16:49:00 139950170520320 [Note] WSREP: Stop replication
2016-04-04 16:49:02 139950170520320 [Note] WSREP: Start replication
2016-04-04 16:49:02 139950170520320 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2016-04-04 16:49:02 139950170520320 [Note] WSREP: protonet asio version 0
2016-04-04 16:49:02 139950170520320 [Note] WSREP: Using CRC-32C for message checksums.
2016-04-04 16:49:02 139950170520320 [Note] WSREP: backend: asio
2016-04-04 16:49:02 139950170520320 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2016-04-04 16:49:02 139950170520320 [Note] WSREP: restore pc from disk failed
2016-04-04 16:49:02 139950170520320 [Note] WSREP: GMCast version 0
2016-04-04 16:49:02 139950170520320 [Note] WSREP: (fd416235, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2016-04-04 16:49:02 139950170520320 [Note] WSREP: (fd416235, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2016-04-04 16:49:02 139950170520320 [Note] WSREP: EVS version 0
2016-04-04 16:49:02 139950170520320 [Note] WSREP: gcomm: connecting to group 'www_cluster', peer '192.168.4.28:,192.168.5.28:,192.168.5.29:'
2016-04-04 16:49:02 139950170520320 [Warning] WSREP: (fd416235, 'tcp://0.0.0.0:4567') address 'tcp://192.168.4.28:4567' points to own listening address, blacklisting
2016-04-04 16:49:02 139950170520320 [Note] WSREP: (fd416235, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2016-04-04 16:49:02 139950170520320 [Note] WSREP: declaring 4c4f4779 at tcp://192.168.5.28:4567 stable
2016-04-04 16:49:02 139950170520320 [Note] WSREP: declaring b75760e5 at tcp://192.168.5.29:4567 stable
2016-04-04 16:49:02 139950170520320 [Note] WSREP: Node 4c4f4779 state prim
2016-04-04 16:49:02 139950170520320 [Note] WSREP: view(view_id(PRIM,4c4f4779,710) memb {
        4c4f4779,0
        b75760e5,0
        fd416235,0
} joined {
} left {
} partitioned {
})
2016-04-04 16:49:02 139950170520320 [Note] WSREP: save pc into disk
2016-04-04 16:49:03 139950170520320 [Note] WSREP: gcomm: connected
2016-04-04 16:49:03 139950170520320 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2016-04-04 16:49:03 139950170520320 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2016-04-04 16:49:03 139950170520320 [Note] WSREP: Opened channel 'www_cluster'
2016-04-04 16:49:03 139948892088064 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3
2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: sent state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8
2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 0 (www2)
2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 1 (galera)
2016-04-04 16:49:03 139949710771968 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.55477S), skipping check
2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 2 ()
2016-04-04 16:49:03 139948892088064 [Note] WSREP: Quorum results:
        version    = 3,
        component  = PRIMARY,
        conf_id    = 470,
        members    = 2/3 (joined/total),
        act_id     = 63180,
        last_appl. = -1,
        protocols  = 0/7/3 (gcs/repl/appl),
        group UUID = 913d310a-9360-11e4-9d31-922a1e5d98fe
2016-04-04 16:49:03 139948892088064 [Note] WSREP: Flow-control interval: [28, 28]
2016-04-04 16:49:03 139948892088064 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 63180)
2016-04-04 16:49:03 139950170217216 [Note] WSREP: State transfer required:
        Group state: 913d310a-9360-11e4-9d31-922a1e5d98fe:63180
        Local state: 00000000-0000-0000-0000-000000000000:-1
2016-04-04 16:49:03 139950170217216 [Note] WSREP: New cluster view: global state: 913d310a-9360-11e4-9d31-922a1e5d98fe:63180, view# 471: Primary, number of nodes: 3, my index: 2, protocol version 3
2016-04-04 16:49:03 139950170217216 [Warning] WSREP: Gap in state sequence. Need state transfer.
2016-04-04 16:49:03 139948883695360 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '192.168.4.28' --datadir '/var/lib/mysql/'  --defaults-file '/etc/mysql/my.cnf'  --parent '6067' --binlog 'mysqld-bin' '
2016-04-04 16:49:03 139950170217216 [Note] WSREP: Prepared SST request: rsync|192.168.4.28:4444/rsync_sst
2016-04-04 16:49:03 139950170217216 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-04-04 16:49:03 139950170217216 [Note] WSREP: REPL Protocols: 7 (3, 2)
2016-04-04 16:49:03 139949719164672 [Note] WSREP: Service thread queue flushed.
2016-04-04 16:49:03 139950170217216 [Note] WSREP: Assign initial position for certification: 63180, protocol version: 3
2016-04-04 16:49:03 139949719164672 [Note] WSREP: Service thread queue flushed.
2016-04-04 16:49:03 139950170217216 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (913d310a-9360-11e4-9d31-922a1e5d98fe): 1 (Operation not permitted)
         at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
2016-04-04 16:49:03 139948892088064 [Note] WSREP: Member 2.0 () requested state transfer from '*any*'. Selected 0.0 (www2)(SYNCED) as donor.
2016-04-04 16:49:03 139948892088064 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 63180)
2016-04-04 16:49:03 139950170217216 [Note] WSREP: Requesting state transfer: success, donor: 0
2016-04-04 16:49:05 139949710771968 [Note] WSREP: (fd416235, 'tcp://0.0.0.0:4567') turning message relay requesting off

И теперь узел продолжает сидеть в «Присоединение: получение передачи состояния».

Однако выбранный узел-донор (www2) сообщает, что он завершил передачу состояния:

160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') address 'tcp://192.168.5.28:4567' pointing to uuid 4c4f4779 is blacklisted, skipping
160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') address 'tcp://192.168.5.28:4567' pointing to uuid 4c4f4779 is blacklisted, skipping
160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') address 'tcp://192.168.5.28:4567' pointing to uuid 4c4f4779 is blacklisted, skipping
160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') address 'tcp://192.168.5.28:4567' pointing to uuid 4c4f4779 is blacklisted, skipping
160404 16:49:04 [Note] WSREP: declaring b75760e5 at tcp://192.168.5.29:4567 stable
160404 16:49:04 [Note] WSREP: declaring fd416235 at tcp://192.168.4.28:4567 stable
160404 16:49:04 [Note] WSREP: Node 4c4f4779 state prim
160404 16:49:04 [Note] WSREP: view(view_id(PRIM,4c4f4779,710) memb {
    4c4f4779,0
    b75760e5,0
    fd416235,0
} joined {
} left {
} partitioned {
})
160404 16:49:04 [Note] WSREP: save pc into disk
160404 16:49:04 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
160404 16:49:04 [Note] WSREP: STATE_EXCHANGE: sent state UUID: feddde16-fa6b-11e5-86f6-fb86e3a56ec8
160404 16:49:04 [Note] WSREP: STATE EXCHANGE: sent state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8
160404 16:49:04 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 0 (www2)
160404 16:49:04 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 1 (galera)
160404 16:49:05 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 2 ()
160404 16:49:05 [Note] WSREP: Quorum results:
    version    = 3,
    component  = PRIMARY,
    conf_id    = 470,
    members    = 2/3 (joined/total),
    act_id     = 63180,
    last_appl. = 63152,
    protocols  = 0/7/3 (gcs/repl/appl),
    group UUID = 913d310a-9360-11e4-9d31-922a1e5d98fe
160404 16:49:05 [Note] WSREP: Flow-control interval: [28, 28]
160404 16:49:05 [Note] WSREP: New cluster view: global state: 913d310a-9360-11e4-9d31-922a1e5d98fe:63180, view# 471: Primary, number of nodes: 3, my index: 0, protocol version 3
160404 16:49:05 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
160404 16:49:05 [Note] WSREP: REPL Protocols: 7 (3, 2)
160404 16:49:05 [Note] WSREP: Service thread queue flushed.
160404 16:49:05 [Note] WSREP: Assign initial position for certification: 63180, protocol version: 3
160404 16:49:05 [Note] WSREP: Service thread queue flushed.
160404 16:49:05 [Note] WSREP: Member 2.0 () requested state transfer from '*any*'. Selected 0.0 (www2)(SYNCED) as donor.
160404 16:49:05 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 63180)
160404 16:49:06 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
160404 16:49:06 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '192.168.4.28:4444/rsync_sst' --auth '(null)' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix ''  --binlog 'mysqld-bin' --gtid '913d310a-9360-11e4-9d31-922a1e5d98fe:63180''
160404 16:49:06 [Note] WSREP: sst_donor_thread signaled with 0
160404 16:49:06 [Note] WSREP: Flushing tables for SST...
160404 16:49:06 [Note] WSREP: Provider paused at 913d310a-9360-11e4-9d31-922a1e5d98fe:63180 (9555)
160404 16:49:06 [Note] WSREP: Tables flushed.
WSREP_SST: [INFO] Preparing binlog files for transfer: (20160404 16:49:07.005)
mysqld-bin.000033
160404 16:49:07 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') turning message relay requesting off
160404 16:51:39 [Note] WSREP: resuming provider at 9555
160404 16:51:39 [Note] WSREP: Provider resumed.
160404 16:51:40 [Note] WSREP: 0.0 (www2): State transfer to 2.0 () complete.
160404 16:51:40 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 63180)
160404 16:51:40 [Note] WSREP: Member 0.0 (www2) synced with group.
160404 16:51:40 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 63180)
160404 16:51:40 [Note] WSREP: Synchronized with group, ready for connections
160404 16:51:40 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Что делать сейчас? Как продолжить работу узла обновления?

Как я обнаружил, в новой версии MariaDB появилась новая опция: wsrep-on. Раньше в этом не было необходимости.

Я добавил эту опцию в файл конфигурации, она перестала игнорировать мои настройки wsrep. Запускал сервер долгое время (около пяти минут), в этот период я ​​видел активность rsync. После этого узел стал пригодным для использования.