我们正在运行一个 Mongo 2.6 副本集，它有 3 个成员：主服务器、辅助服务器、仲裁服务器。几乎每天我们的 MongoDB 都会切换哪个服务器为主服务器，这会导致与该数据库的所有连接中断。如果它这样做是因为其中一个服务器确实宕机了，那就完全没问题了，但问题是，在每种情况下，看起来“宕机”的服务器实际上并没有宕机。它一直处于运行状态。
以下是我们所知的信息：
- 所有 3 台服务器上的进程
mongod均未重新启动或停止。 - 服务器仍始终向 New Relic 报告。
- 从 mongo 日志中我们看到频繁出现心跳失败。
- 服务器实际上在任何时候都不会处于非常高的负载下。我看到每小时大约 10 分钟后 CPU 出现峰值，但这与故障并不完全一致。
以下是show log rs将 while shell 放入当前主目录中的结果。
2015-05-17T15:05:49.339+0000 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: server1:27017
2015-05-17T15:05:49.358+0000 [rsBackgroundSync] replSet syncing to: server1:27017
2015-05-17T15:05:56.444+0000 [rsBackgroundSync] replset setting syncSourceFeedback to server1:27017
2015-05-17T22:11:36.638+0000 [rsHealthPoll] replSet info server1:27017 is down (or slow to respond):
2015-05-17T22:11:36.644+0000 [rsHealthPoll] replSet member server1:27017 is now in state DOWN
2015-05-17T22:11:37.495+0000 [rsMgr] not electing self, we are not freshest
2015-05-17T22:11:38.656+0000 [rsHealthPoll] replSet member server1:27017 is up
2015-05-17T22:11:38.656+0000 [rsHealthPoll] replSet member server1:27017 is now in state PRIMARY
2015-05-17T22:11:39.140+0000 [rsBackgroundSync] replSet syncing to: server1:27017
2015-05-17T22:11:39.147+0000 [rsBackgroundSync] replset setting syncSourceFeedback to server1:27017
2015-05-17T23:05:47.431+0000 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: server1:27017
2015-05-17T23:05:47.431+0000 [rsBackgroundSync] replSet syncing to: server1:27017
2015-05-17T23:05:47.876+0000 [rsBackgroundSync] replset setting syncSourceFeedback to server1:27017
2015-05-18T10:05:46.821+0000 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: server1:27017
2015-05-18T10:05:46.822+0000 [rsBackgroundSync] replSet syncing to: server1:27017
2015-05-18T10:05:51.014+0000 [rsBackgroundSync] replset setting syncSourceFeedback to server1:27017
2015-05-18T22:12:11.433+0000 [rsHealthPoll] replSet info server1:27017 is down (or slow to respond):
2015-05-18T22:12:11.434+0000 [rsHealthPoll] replSet member server1:27017 is now in state DOWN
2015-05-18T22:12:11.507+0000 [rsMgr] replSet info electSelf 3
2015-05-18T22:12:14.708+0000 [rsMgr] replSet PRIMARY
2015-05-18T22:12:14.709+0000 [rsHealthPoll] replSet member server1:27017 is up
2015-05-18T22:12:14.709+0000 [rsHealthPoll] replSet member server1:27017 is now in state PRIMARY
2015-05-18T22:12:21.610+0000 [rsHealthPoll] replSet member server1:27017 is now in state ROLLBACK
2015-05-18T22:12:23.612+0000 [rsHealthPoll] replSet member server1:27017 is now in state SECONDARY
2015-05-19T22:13:13.004+0000 [rsHealthPoll] couldn't connect to server1:27017: couldn't connect to server server1:27017 (x.x.x.x), connection attempt failed
2015-05-19T22:13:24.127+0000 [rsHealthPoll] couldn't connect to server1:27017: couldn't connect to server server1:27017 (x.x.x.x) failed, connection attempt failed
2015-05-19T22:13:29.267+0000 [rsHealthPoll] replset info server1:27017 just heartbeated us, but our heartbeat failed: , not changing state
2015-05-20T22:14:35.832+0000 [rsHealthPoll] replset info server1:27017 just heartbeated us, but our heartbeat failed: , not changing state
您可以看到我们经常收到心跳故障和停机通知，但每次服务器都会在几秒钟内从停机状态恢复到正常运行状态。我真的不知道下一步该从哪里开始查找，以找出导致问题的原因。
答案1
我经常看到这种情况，而且总是在流程之外mongod。DNS 解析器问题、TCP/IP 堆栈问题、网络链接、物理硬件等。从流程中找出解决办法mongod。检查主机操作系统上的网络错误，检查物理链接（如果物理链接在方程中），如果跨区域，检查两台服务器之间的云提供商。这很可能是主机操作系统的问题，与 MongoDB 本身无关。
答案2
这个问题已经解决了。核心问题是我们的托管服务提供商正在运行 VMWare 快照作为备份机制。这些快照导致虚拟机暂时进入停滞状态，我认为技术术语是虚拟机静止。
一旦禁用这些快照，我们就不再遇到任何问题。