尝试 Apache Ignite 集群快照过程(快照创建、恢复)时,恢复操作不成功。
在恢复过程之前,我们已经销毁了要恢复的缓存(根据官方文档),但日志中仍然存在相同的错误消息/Java异常。
Apache Ignite - 快照创建/恢复过程:
./bin/control.sh --snapshot create 20230412_snapshot_test_v1
./bin/control.sh --cache destroy --caches CACHE_XYZ --yes
./bin/control.sh --snapshot restore 20230412_snapshot_test_v1 --start --groups CACHE_XYZ
Apache Ignite - 日志:
The snapshot operation will be aborted due to a handler error
class org.apache.ignite.IgniteCheckedException: The check procedure failed on 3 nodes.
The check procedure failed on nodes:
Node ID: 260115a7-95b4-48a3-bc98-af9d8af0df51 [192.168.XX.YY, 127.0.0.1]
Consistent ID: 547359f6-4501-4727-8563-a8b170a905f3
Exception: java.lang.NegativeArraySizeException
-935925367
Node ID: 33d84ca7-e4e2-4b6e-9c3c-63625c853dc1 [192.168.XX.YY, 127.0.0.1]
Consistent ID: aa7da35d-37b3-401d-8df3-f3c6e2761c85
Exception: java.lang.NegativeArraySizeException
-935925367
Node ID: d512f146-abb2-4e88-a22c-3a640e84be18 [192.168.XX.YY, 127.0.0.1]
Consistent ID: 0b6fafa5-a31f-4fa7-80c5-44c792af03ff
Exception: java.lang.NegativeArraySizeException
-935925367
...
Failed to restore snapshot cache groups
class org.apache.ignite.compute.ComputeUserUndeclaredException: Failed to reduce job results due to undeclared user exception [task=org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotHandlerRestoreTask@4c606da9, err=class org.apache.ignite.IgniteException: The check procedure failed on 3 nodes.
另一个奇怪的点是,由于某种原因无法--snapshot check
连接到集群,但所有其他命令都正常工作。
Apache Ignite - 集群快照验证:
./bin/control.sh --snapshot check 20230412_snapshot_test_v1 --verbose
...
INFO: Client TCP connection established: /127.0.0.1:11211
...
SEVERE: Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.DirectByteBuffer[pos=0 lim=8192 cap=8192], super=AbstractNioClientWorker [idx=0, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-client-0, igniteInstanceName=gridClient, finished=false, heartbeatTs=1682407593249, hashCode=245934772, interrupted=false, runner=grid-nio-worker-tcp-client-0-#5%gridClient%]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, closeSocket=true, outboundMessagesQueueSizeMetric=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:42122, rmtAddr=/127.0.0.1:11211, createTime=1682407593249, closeTime=0, bytesSent=702, bytesRcvd=78929, bytesSent0=702, bytesRcvd0=78929, sndSchedTime=1682407593249, lastSndTime=1682407593249, lastRcvTime=1682407593249, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=GridTcpRestParser [marsh=JdkMarshaller [clsFilter=null], routerClient=false], directMode=false]], accepted=false, markedForClose=false]]]
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1183)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2508)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2273)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1910)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
at java.lang.Thread.run(Thread.java:750)
...
class org.apache.ignite.internal.client.impl.connection.GridClientConnectionResetException: Failed to perform request (connection failed): /127.0.0.1:11211
Apache Ignite - 日志:
[SnapshotPartitionsVerifyTask$VisorVerifySnapshotPartitionsJob] Verify snapshot partitions procedure has been initiated [snpName=20230412_snapshot_test_v1, consId=33303d92-f3db-4b33-be38-798167c2f726]
[IgniteSnapshotManager] Resolved directory for serialized binary metadata: /usr/share/apache-ignite/work/snapshots/20230412_snapshot_test_v1/db/binary_meta/node00-33303d92-f3db-4b33-be38-798167c2f726
... nothing else is printed, the service goes down ...
快照验证失败(上面解释的连接错误)后,每台 Linux 服务器上的 Apache Ignite 服务都会立即停止,集群将关闭,所有内容都需要手动重新启动。
这是一个配置了 Apache ZooKeeper 的三节点 Apache Ignite 集群(每个节点位于不同的服务器上),我们在使用 OpenJDK 1.8.0 的 Linux 上运行的 Apache Ignite 2.13.0 和 2.14.0 上也有同样的问题。
可能是什么原因,恢复过程失败?
答案1
似乎 Apache Ignite 自动集群快照过程的问题已在最新的 Apache Ignite 版本 (v2.15.0) 中得到修复。请检查官方发行说明了解更多信息。