☰

Elasticsearch-Kubernetes

tag-icon • tag-icon cluster kubernetes elasticsearch
Elasticsearch-Kubernetes

在 Kubernetes 上设置具有 2x 数据节点、2x 主节点和 2x 客户端节点（每个容器位于单独的物理节点上）的 Elasticsearch 集群时，我收到以下错误client-node：

[2019-01-28T12:25:08,574][WARN ][o.e.d.z.UnicastZenPing   ] [elasticsearch-client-5f7759d57c-95wwm] [81] failed send ping to {#zen_unicast_elasticsearch-discovery_0#}{LuD6SiE5RCG7Ol9xb0LBFw}{elasticsearch-discovery}{10.233.12.222:9300}
java.lang.IllegalStateException: handshake failed, mismatched cluster name [Cluster [elasticsearch-default]] - {#zen_unicast_elasticsearch-discovery_0#}{LuD6SiE5RCG7Ol9xb0LBFw}{elasticsearch-discovery}{10.233.12.222:9300}
        at org.elasticsearch.transport.TransportService.handshake(TransportService.java:406) ~[elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.transport.TransportService.handshake(TransportService.java:369) ~[elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:400) ~[elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:503) [elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.3.jar:6.2.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

这个错误没有任何意义，因为我没有集群名称 [elasticsearch-default]。

此外，客户端节点在检查 elasticsearch api 端点时会做出响应，并显示配置中设置的正确集群名称：

{
  "name" : "elasticsearch-client-5f7759d57c-q2vd5",
  "cluster_name" : "elasticsearch-test2",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "6.2.3",
    "build_hash" : "c59ff00",
    "build_date" : "2018-03-13T10:06:29.741383Z",
    "build_snapshot" : false,
    "lucene_version" : "7.2.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"

但是当尝试健康检查端点时，它响应：

curl -X 获取 "10.233.43.73:9200/_cluster/health?pretty"

{
  "error" : {
    "root_cause" : [
      {
        "type" : "master_not_discovered_exception",
        "reason" : null
      }
    ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

此外，master-node日志中显示此错误：

[WARN ][o.e.t.n.Netty4Transport  ] [49d98428-ddef-479e-85b2-e00fb6f0acef] exception caught on transport layer [NettyTcpChannel{localAddress=/10.233.67.119:9300, remoteAddress=/10.233.69.29:35712}], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (ff,f4,ff,fd)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:392) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:359) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.logging.LoggingHandler.channelInactive(LoggingHandler.java:167) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1354) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:917) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (ff,f4,ff,fd)
        at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1283) ~[elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:36) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[?:?]
        ... 20 more

ES 图像：txn2/k8s-es:v6.2.3

ES配置文件：

cluster:
  name: ${CLUSTER_NAME}

node:
  master: ${NODE_MASTER}
  data: ${NODE_DATA}
  name: ${NODE_NAME}
  ingest: ${NODE_INGEST}
  max_local_storage_nodes: ${MAX_LOCAL_STORAGE_NODES}

processors: ${PROCESSORS:1}

network.host: ${NETWORK_HOST}

path:
  data: /data/data
  logs: /data/log
  repo: ${REPO_LOCATIONS}

bootstrap:
  memory_lock: ${MEMORY_LOCK}

http:
  enabled: ${HTTP_ENABLE}
  compression: true
  cors:
    enabled: true
    allow-origin: "*"

discovery:
  zen:
    ping.unicast.hosts: ${DISCOVERY_SERVICE}
    minimum_master_nodes: ${NUMBER_OF_MASTERS}

所有环境变量均被正确读取

bash-4.4# printenv | grep CLUSTER
CLUSTER_NAME=elasticsearch-test2

错误master-node表明存在一些网络问题，例如容器无法相互“通信”，但我甚至测试了是否可以从client-node容器到 elasticsearch-discovery 服务 ClusterIP 建立连接：

sudo nsenter -t 4552 -n telnet 10.233.12.222 9300
Trying 10.233.12.222...
Connected to 10.233.12.222.
Escape character is '^]'.

Kubernetes 配置：7 节点集群（3 个主节点 + 4 个从节点）

clientVersion:
  buildDate: 2018-04-27T09:10:24Z
  compiler: gc
  gitCommit: 81753b10df112992bf51bbc2c2f85208aad78335
  gitTreeState: clean
  gitVersion: v1.10.2
  goVersion: go1.9.3
  major: "1"
  minor: "10"
  platform: linux/amd64
serverVersion:
  buildDate: 2018-04-27T09:10:24Z
  compiler: gc
  gitCommit: 81753b10df112992bf51bbc2c2f85208aad78335
  gitTreeState: clean
  gitVersion: v1.10.2
  goVersion: go1.9.3
  major: "1"
  minor: "10"
  platform: linux/amd64
Docker version: 17.3.2

我还可以补充一点，我设法让它在另一个集群上使用完全相同的图像（2 个主服务器 + 2 个从服务器。完全相同的 kubernetes 和 docker 版本）工作。

答案1

我猜你正在使用这掌舵图？

我遇到了同样的错误，似乎是因为只有 2 个主服务器，而新客户端尝试加入时出现仲裁问题。我的解决方案是添加另一个主服务器第一的一旦客户端无法连接到集群，就太晚了，并且扩展主状态集也会失败。

不幸的是，我的解决方案是删除集群（但不删除持久卷）并重新创建它。PV 已重新连接到 ES 节点，停机时间很短。

相关内容