当存在故障节点时,Heartbeat 无法从冷启动成功启动资源

当存在故障节点时,Heartbeat 无法从冷启动成功启动资源

我目前有两台运行 Heartbeat 和 DRBD 的 ubuntu 服务器。这些服务器通过 eth1 上的 1000Mbps 交叉电缆直接连接,并可访问 eth0 上的 IP 摄像机 LAN。

现在,假设一个节点已关闭,而剩余的功能节点在关闭后正在启动。仍在运行的节点将不会启动 heartbeat 并从冷启动提供对 drbd 资源的访问。我必须手动重新启动 heartbeat 才能sudo service heartbeat restart使一切正常运行。

当只有一台服务器时,如何让它从冷启动正常启动?

这里是ha.cf

debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility none

keepalive 2
deadtime 10
warntime 7
initdead 60
ucast eth1 192.168.2.2
ucast eth0 10.1.10.201
node EMserver1
node EMserver2
respawn hacluster /usr/lib/heartbeat/ipfail
ping 10.1.10.22 10.1.10.21 10.1.10.11
auto_failback off

来自系统日志的一些材料:

harc[4604]: 2012/11/27_13:54:49 info: Running /etc/ha.d//rc.d/status status
mach_down[4632]:    2012/11/27_13:54:49 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down[4632]:    2012/11/27_13:54:49 info: mach_down takeover complete for node emserver2.
Nov 27 13:54:49 EMserver1 heartbeat: [4586]: info: Initial resource acquisition complete (T_RESOURCES(us))
Nov 27 13:54:49 EMserver1 heartbeat: [4586]: info: mach_down takeover complete.
IPaddr[4679]:   2012/11/27_13:54:49 INFO:  Resource is stopped
Nov 27 13:54:49 EMserver1 heartbeat: [4605]: info: Local Resource acquisition completed.
harc[4713]: 2012/11/27_13:54:49 info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
ip-request-resp[4713]:  2012/11/27_13:54:49 received ip-request-resp IPaddr::10.1.10.254 OK yes
ResourceManager[4732]:  2012/11/27_13:54:50 info: Acquiring resource group: emserver1 IPaddr::10.1.10.254 drbddisk::r0 Filesystem::/dev/drbd1::/shr::ext4 nfs-kernel-server
IPaddr[4759]:   2012/11/27_13:54:50 INFO:  Resource is stopped
ResourceManager[4732]:  2012/11/27_13:54:50 info: Running /etc/ha.d/resource.d/IPaddr 10.1.10.254 start
IPaddr[4816]:   2012/11/27_13:54:50 INFO: Using calculated nic for 10.1.10.254: eth0
IPaddr[4816]:   2012/11/27_13:54:50 INFO: Using calculated netmask for 10.1.10.254: 255.255.255.0
IPaddr[4816]:   2012/11/27_13:54:50 INFO: eval ifconfig eth0:0 10.1.10.254 netmask 255.255.255.0 broadcast 10.1.10.255
IPaddr[4804]:   2012/11/27_13:54:50 INFO:  Success
ResourceManager[4732]:  2012/11/27_13:54:50 info: Running /etc/ha.d/resource.d/drbddisk r0 start
Filesystem[4965]:   2012/11/27_13:54:50 INFO:  Resource is stopped
ResourceManager[4732]:  2012/11/27_13:54:50 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /shr ext4 start
Filesystem[5039]:   2012/11/27_13:54:50 INFO: Running start for /dev/drbd1 on /shr
Filesystem[5033]:   2012/11/27_13:54:51 INFO:  Success
ResourceManager[4732]:  2012/11/27_13:54:51 info: Running /etc/init.d/nfs-kernel-server  start
Nov 27 13:55:00 EMserver1 heartbeat: [4586]: info: Local Resource acquisition completed. (none)
Nov 27 13:55:00 EMserver1 heartbeat: [4586]: info: local resource transition completed.
Nov 27 13:57:46 EMserver1 heartbeat: [4586]: info: Heartbeat shutdown in progress. (4586)
Nov 27 13:57:46 EMserver1 heartbeat: [5286]: info: Giving up all HA resources.
ResourceManager[5301]:  2012/11/27_13:57:46 info: Releasing resource group: emserver1 IPaddr::10.1.10.254 drbddisk::r0 Filesystem::/dev/drbd1::/shr::ext4 nfs-kernel-server
ResourceManager[5301]:  2012/11/27_13:57:46 info: Running /etc/init.d/nfs-kernel-server  stop
ResourceManager[5301]:  2012/11/27_13:57:46 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /shr ext4 stop
Filesystem[5372]:   2012/11/27_13:57:46 INFO: Running stop for /dev/drbd1 on /shr
Filesystem[5372]:   2012/11/27_13:57:47 INFO: Trying to unmount /shr
Filesystem[5372]:   2012/11/27_13:57:47 INFO: unmounted /shr successfully
Filesystem[5366]:   2012/11/27_13:57:47 INFO:  Success
ResourceManager[5301]:  2012/11/27_13:57:47 info: Running /etc/ha.d/resource.d/drbddisk r0 stop
ResourceManager[5301]:  2012/11/27_13:57:47 info: Running /etc/ha.d/resource.d/IPaddr 10.1.10.254 stop
IPaddr[5509]:   2012/11/27_13:57:47 INFO: ifconfig eth0:0 down
IPaddr[5497]:   2012/11/27_13:57:47 INFO:  Success
Nov 27 13:57:47 EMserver1 heartbeat: [5286]: info: All HA resources relinquished.
Nov 27 13:57:48 EMserver1 heartbeat: [4586]: info: killing /usr/lib/heartbeat/ipfail process group 4603 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBFIFO process 4589 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBWRITE process 4590 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBREAD process 4591 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBWRITE process 4592 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBREAD process 4593 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBWRITE process 4594 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBREAD process 4595 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBWRITE process 4596 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBREAD process 4597 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBWRITE process 4598 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: killing HBREAD process 4599 with signal 15
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4589 exited. 11 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4596 exited. 10 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4598 exited. 9 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4590 exited. 8 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4595 exited. 7 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4591 exited. 6 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4592 exited. 5 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4593 exited. 4 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4597 exited. 3 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4594 exited. 2 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: Core process 4599 exited. 1 remaining
Nov 27 13:57:49 EMserver1 heartbeat: [4586]: info: emserver1 Heartbeat shutdown complete.

以下是日志中的其他内容

ResourceManager[2576]:  2012/11/28_16:32:42 info: Acquiring resource group: emserver1 IPaddr::10.1.10.254 drbddisk::r0 Filesystem::/dev/drbd1::/shr::ext4 nfs-kernel-server
IPaddr[2602]:   2012/11/28_16:32:42 INFO:  Running OK
Filesystem[2653]:   2012/11/28_16:32:43 INFO:  Running OK
Nov 28 16:32:52 EMserver1 heartbeat: [1695]: WARN: node emserver2: is dead
Nov 28 16:32:52 EMserver1 heartbeat: [1695]: info: Dead node emserver2 gave up resources.
Nov 28 16:32:52 EMserver1 ipfail: [1807]: info: Status update: Node emserver2 now has status dead
Nov 28 16:32:52 EMserver1 heartbeat: [1695]: info: Link emserver2:eth1 dead.
Nov 28 16:32:53 EMserver1 ipfail: [1807]: info: NS: We are still alive!
Nov 28 16:32:53 EMserver1 ipfail: [1807]: info: Link Status update: Link emserver2/eth1 now has status dead
Nov 28 16:32:55 EMserver1 ipfail: [1807]: info: Asking other side for ping node count.
Nov 28 16:32:55 EMserver1 ipfail: [1807]: info: Checking remote count of ping nodes.
Nov 28 16:32:57 EMserver1 heartbeat: [1695]: info: Heartbeat shutdown in progress. (1695)
Nov 28 16:32:57 EMserver1 heartbeat: [2734]: info: Giving up all HA resources.
ResourceManager[2751]:  2012/11/28_16:32:57 info: Releasing resource group: emserver1 IPaddr::10.1.10.254 drbddisk::r0 Filesystem::/dev/drbd1::/shr::ext4 nfs-kernel-server
ResourceManager[2751]:  2012/11/28_16:32:57 info: Running /etc/init.d/nfs-kernel-server  stop
ResourceManager[2751]:  2012/11/28_16:32:57 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /shr ext4 stop
Filesystem[2829]:   2012/11/28_16:32:57 INFO: Running stop for /dev/drbd1 on /shr
Filesystem[2829]:   2012/11/28_16:32:57 INFO: Trying to unmount /shr
Filesystem[2829]:   2012/11/28_16:32:58 INFO: unmounted /shr successfully
Filesystem[2823]:   2012/11/28_16:32:58 INFO:  Success
ResourceManager[2751]:  2012/11/28_16:32:58 info: Running /etc/ha.d/resource.d/drbddisk r0 stop
ResourceManager[2751]:  2012/11/28_16:32:58 info: Running /etc/ha.d/resource.d/IPaddr 10.1.10.254 stop
IPaddr[2971]:   2012/11/28_16:32:58 INFO: ifconfig eth0:0 down
IPaddr[2958]:   2012/11/28_16:32:58 INFO:  Success
Nov 28 16:32:58 EMserver1 heartbeat: [2734]: info: All HA resources relinquished.
Nov 28 16:32:59 EMserver1 heartbeat: [1695]: info: killing /usr/lib/heartbeat/ipfail process group 1807 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBFIFO process 1777 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBWRITE process 1778 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBREAD process 1779 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBWRITE process 1780 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBREAD process 1781 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBWRITE process 1782 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBREAD process 1783 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBWRITE process 1784 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBREAD process 1785 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBWRITE process 1786 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: killing HBREAD process 1787 with signal 15
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1778 exited. 11 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1779 exited. 10 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1780 exited. 9 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1781 exited. 8 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1782 exited. 7 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1783 exited. 6 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1784 exited. 5 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1785 exited. 4 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1786 exited. 3 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1787 exited. 2 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: Core process 1777 exited. 1 remaining
Nov 28 16:33:01 EMserver1 heartbeat: [1695]: info: emserver1 Heartbeat shutdown complete.

如果我在此时重新启动心跳...资源心跳控制将正常启动....请帮忙!

答案1

正如评论中所要求的,这是我以答案形式提出的初步建议。

如果您运行的是较新的 Ubuntu 版本(10.04 或更新版本),则应完全放弃 Heartbeat,改用 Corosync 和 Pacemaker。虽然这不能解决您的特定问题,但正如您所见,它可以让您的问题消失。

A关于起搏器的精彩介绍文章发表在《Linux Journal》上,从头开始构建集群是启动和运行 Corosync 和 Pacemaker 的绝佳起点。对于 Ubuntu 10.04,请使用 PPA 获取官方稳定的反向移植集群堆栈:https://launchpad.net/~ubuntu-ha-maintainers/+archive/ppa

旧的 Heartbeat 堆栈存在许多概念上的缺陷,而且已经过时很久了,现在不值得尝试调试今天使用它必然会出现的问题。Corosync 和 Pacemaker 正是出于这个原因而创建的,它们的工作效果比 Heartbeat 好得多,后者甚至不是一个资源管理器。

相关内容