В настоящее время у меня есть два сервера Ubuntu, на которых запущены Heartbeat и DRBD. Серверы напрямую подключены перекрестным кабелем со скоростью 1000 Мбит / с на eth1 и имеют доступ к локальной сети IP-камеры на eth0.
Теперь предположим, что один узел не работает, а оставшийся функциональный узел загружается после завершения работы. Узел, который все еще функционирует, не запускает пульс и не предоставляет доступ к ресурсу drbd после холодной загрузки. Мне нужно вручную перезапустить сердцебиение, sudo service heartbeat restart чтобы все заработало.
Как я могу заставить его нормально стартовать с холодного старта, когда присутствует только один сервер?
Здесь ha.cf:
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility none
keepalive 2
deadtime 10
warntime 7
initdead 60
ucast eth1 192.168.2.2
ucast eth0 10.1.10.201
node EMserver1
node EMserver2
respawn hacluster /usr/lib/heartbeat/ipfail
ping 10.1.10.22 10.1.10.21 10.1.10.11
auto_failback off
Немного материала из системного журнала:
harc[4604]: 2012/11/27_13:54:49 info: Running /etc/ha.d//rc.d/status status
mach_down[4632]: 2012/11/27_13:54:49 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down[4632]: 2012/11/27_13:54:49 info: mach_down takeover complete for node emserver2.
Nov 27 13:54:49 EMserver1 heartbeat: [4586]: info: Initial resource acquisition complete (T_RESOURCES(us))
Nov 27 13:54:49 EMserver1 heartbeat: [4586]: info: mach_down takeover complete.
IPaddr[4679]: 2012/11/27_13:54:49 INFO: Resource is stopped
Nov 27 13:54:49 EMserver1 heartbeat: [4605]: info: Local Resource acquisition completed.
harc[4713]: 2012/11/27_13:54:49 info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
ip-request-resp[4713]: 2012/11/27_13:54:49 received ip-request-resp IPaddr::10.1.10.254 OK yes
ResourceManager[4732]: 2012/11/27_13:54:50 info: Acquiring resource group: emserver1 IPaddr::10.1.10.254 drbddisk::r0 Filesystem::/dev/drbd1::/shr::ext4 nfs-kernel-server
IPaddr[4759]: 2012/11/27_13:54:50 INFO: Resource is stopped
ResourceManager[4732]: 2012/11/27_13:54:50 info: Running /etc/ha.d/resource.d/IPaddr 10.1.10.254 start
IPaddr[4816]: 2012/11/27_13:54:50 INFO: Using calculated nic for 10.1.10.254: eth0
IPaddr[4816]: 2012/11/27_13:54:50 INFO: Using calculated netmask for 10.1.10.254: 255.255.255.0
IPaddr[4816]: 2012/11/27_13:54:50 INFO: eval ifconfig eth0:0 10.1.10.254 netmask 255.255.255.0 broadcast 10.1.10.255
IPaddr[4804]: 2012/11/27_13:54:50 INFO: Success
ResourceManager[4732]: 2012/11/27_13:54:50 info: Running /etc/ha.d/resource.d/drbddisk r0 start
Filesystem[4965]: 2012/11/27_13:54:50 INFO: Resource is stopped
ResourceManager[4732]: 2012/11/27_13:54:50 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /shr ext4 start
Filesystem[5039]: 2012/11/27_13:54:50 INFO: Running start for /dev/drbd1 on /shr
Filesystem[5033]: 2012/11/27_13:54:51 INFO: Success
ResourceManager[4732]: 2012/11/27_13:54:51 info: Running /etc/init.d/nfs-kernel-server start
Nov 27 13:55:00 EMserver1 heartbeat: [4586]: info: Local Resource acquisition completed. (none)
Nov 27 13:55:00 EMserver1 heartbeat: [4586]: info: local resource transition completed.
Nov 27 13:57:46 EMserver1 heartbeat: [4586]: info: Heartbeat shutdown in progress. (4586)
Nov 27 13:57:46 EMserver1 heartbeat: [5286]: info: Giving up all HA resources.
ResourceManager[5301]: 2012/11/27_13:57:46 info: Releasing resource group: emserver1 IPaddr::10.1.10.254 drbddisk::r0 Filesystem::/dev/drbd1::/shr::ext4 nfs-kernel-server
ResourceManager[5301]: 2012/11/27_13:57:46 info: Running /etc/init.d/nfs-kernel-server stop
ResourceManager[5301]: 2012/11/27_13:57:46 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /shr ext4 stop
Filesystem[5372]: 2012/11/27_13:57:46 INFO: Running stop for /dev/drbd1 on /shr
Filesystem[5372]: 2012/11/27_13:57:47 INFO: Trying to unmount /shr
Filesystem[5372]: 2012/11/27_13:57:47 INFO: unmounted /shr successfully
Filesystem[5366]: 2012/11/27_13:57:47 INFO: Success
ResourceManager[5301]: 2012/11/27_13:57:47 info: Running /etc/ha.d/resource.d/drbddisk r0 stop
ResourceManager[5301]: 2012/11/27_13:57:47 info: Running /etc/ha.d/resource.d/IPaddr 10.1.10.254 stop
IPaddr[5509]: 2012/11/27_13:57:47 INFO: ifconfig eth0:0 down
IPaddr[5497]: 2012/11/27_13:57:47 INFO: Success
Nov 27 13:57:47 EMserver1 heartbeat: [5286]: info: All HA resources relinquished.
Nov 27 13:57:48 EMserver1 heartbeat: [4586]: info: killing /usr/lib/heartbeat/ipfail process group 4603 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBFIFO process 4589 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBWRITE process 4590 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBREAD process 4591 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBWRITE process 4592 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBREAD process 4593 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBWRITE process 4594 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBREAD process 4595 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBWRITE process 4596 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBREAD process 4597 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBWRITE process 4598 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBREAD process 4599 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4589 exited. 11 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4596 exited. 10 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4598 exited. 9 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4590 exited. 8 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4595 exited. 7 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4591 exited. 6 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4592 exited. 5 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4593 exited. 4 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4597 exited. 3 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4594 exited. 2 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4599 exited. 1 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: emserver1 Heartbeat shutdown complete.
Вот еще немного из журнала
ResourceManager[2576]: 2012/11/28_16:32:42 info: Acquiring resource group: emserver1 IPaddr::10.1.10.254 drbddisk::r0 Filesystem::/dev/drbd1::/shr::ext4 nfs-kernel-server
IPaddr[2602]: 2012/11/28_16:32:42 INFO: Running OK
Filesystem[2653]: 2012/11/28_16:32:43 INFO: Running OK
Nov 28 16:32:52 EMserver1 heartbeat: [1695]: WARN: node emserver2: is dead
Nov 28 16:32:52 EMserver1 heartbeat: [1695]: info: Dead node emserver2 gave up resources.
Nov 28 16:32:52 EMserver1 ipfail: [1807]: info: Status update: Node emserver2 now has status dead
Nov 28 16:32:52 EMserver1 heartbeat: [1695]: info: Link emserver2:eth1 dead.
Nov 28 16:32:53 EMserver1 ipfail: [1807]: info: NS: We are still alive!
Nov 28 16:32:53 EMserver1 ipfail: [1807]: info: Link Status update: Link emserver2/eth1 now has status dead
Nov 28 16:32:55 EMserver1 ipfail: [1807]: info: Asking other side for ping node count.
Nov 28 16:32:55 EMserver1 ipfail: [1807]: info: Checking remote count of ping nodes.
Nov 28 16:32:57 EMserver1 heartbeat: [1695]: info: Heartbeat shutdown in progress. (1695)
Nov 28 16:32:57 EMserver1 heartbeat: [2734]: info: Giving up all HA resources.
ResourceManager[2751]: 2012/11/28_16:32:57 info: Releasing resource group: emserver1 IPaddr::10.1.10.254 drbddisk::r0 Filesystem::/dev/drbd1::/shr::ext4 nfs-kernel-server
ResourceManager[2751]: 2012/11/28_16:32:57 info: Running /etc/init.d/nfs-kernel-server stop
ResourceManager[2751]: 2012/11/28_16:32:57 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /shr ext4 stop
Filesystem[2829]: 2012/11/28_16:32:57 INFO: Running stop for /dev/drbd1 on /shr
Filesystem[2829]: 2012/11/28_16:32:57 INFO: Trying to unmount /shr
Filesystem[2829]: 2012/11/28_16:32:58 INFO: unmounted /shr successfully
Filesystem[2823]: 2012/11/28_16:32:58 INFO: Success
ResourceManager[2751]: 2012/11/28_16:32:58 info: Running /etc/ha.d/resource.d/drbddisk r0 stop
ResourceManager[2751]: 2012/11/28_16:32:58 info: Running /etc/ha.d/resource.d/IPaddr 10.1.10.254 stop
IPaddr[2971]: 2012/11/28_16:32:58 INFO: ifconfig eth0:0 down
IPaddr[2958]: 2012/11/28_16:32:58 INFO: Success
Nov 28 16:32:58 EMserver1 heartbeat: [2734]: info: All HA resources relinquished.
Nov 28 16:32:59 EMserver1 heartbeat: [1695]: info: killing /usr/lib/heartbeat/ipfail process group 1807 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBFIFO process 1777 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBWRITE process 1778 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBREAD process 1779 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBWRITE process 1780 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBREAD process 1781 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBWRITE process 1782 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBREAD process 1783 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBWRITE process 1784 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBREAD process 1785 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBWRITE process 1786 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBREAD process 1787 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1778 exited. 11 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1779 exited. 10 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1780 exited. 9 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1781 exited. 8 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1782 exited. 7 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1783 exited. 6 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1784 exited. 5 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1785 exited. 4 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1786 exited. 3 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1787 exited. 2 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1777 exited. 1 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: emserver1 Heartbeat shutdown complete.
Если бы я перезапустил сердцебиение в этот момент ... элементы управления пульсом ресурсов заработали бы нормально .... пожалуйста, помогите!
Как указано в комментариях, вот мое первоначальное предложение в форме ответа.
Если вы используете достаточно свежую версию Ubuntu (10.04 или новее), вам следует полностью отказаться от Heartbeat и вместо этого использовать Corosync и Pacemaker. Хотя это не ответ или решение вашей конкретной проблемы, но, как вы видели, ваша проблема просто исчезнет.
А отличная вводная статья о кардиостимуляторе был написан в Linux Journal и Кластеры с нуля - отличная отправная точка для запуска и запуска Corosync и Pacemaker. Для Ubuntu 10.04 используйте PPA для официального стабильного обратного стека кластеров: https://launchpad.net/~ubuntu-ha-maintainers/+archive/ppa
Старый стек Heartbeat имеет много концептуальных недостатков и так давно устарел, что не стоит пытаться отлаживать проблемы, которые обязательно возникнут, если вы используете его сегодня. Corosync и Pacemaker были созданы именно по этой причине, и они работают намного лучше, чем Heartbeat, который даже не может справиться с менеджером ресурсов.