Spark 连接到独立集群上的从属设备

Spark 连接到独立集群上的从属设备

因此,我的 Mac 上有主节点,我可以在 Web 服务器上检查我的主 url,即 spark://private_ip_address:7077。然后,我尝试连接远程服务器上的从属节点。

因此,我从主服务器运行 start-slaves.sh,它会在远程服务器上创建一个日志。因此,我假设 ssh 连接正常。但是,如下面的日志内容所示,无法与主服务器建立连接。

21/01/12 13:06:36 INFO ResourceUtils: ==============================================================
21/01/12 13:06:36 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
21/01/12 13:06:36 INFO WorkerWebUI: Bound WorkerWebUI to 127.0.0.0, and started at http://127.0.0.0:8081
21/01/12 13:06:36 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:06:48 INFO Worker: Retrying connection to master (attempt # 1)
21/01/12 13:06:48 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:00 INFO Worker: Retrying connection to master (attempt # 2)
21/01/12 13:07:00 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:12 INFO Worker: Retrying connection to master (attempt # 3)
21/01/12 13:07:12 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:24 INFO Worker: Retrying connection to master (attempt # 4)
21/01/12 13:07:24 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:36 INFO Worker: Retrying connection to master (attempt # 5)
21/01/12 13:07:36 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:48 INFO Worker: Retrying connection to master (attempt # 6)
21/01/12 13:07:48 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:08:36 ERROR RpcOutboxMessage: Ask terminated before connecting successfully
21/01/12 13:08:36 WARN NettyRpcEnv: Ignored failure: java.io.IOException: Connecting to /master_node_private_ip_address:7077 timed out (120000 ms)
21/01/12 13:08:36 WARN Worker: Failed to connect to master master_node_private_ip_address:7077
org.apache.spark.SparkException: Exception thrown in awaitResult: 
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:302)
    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
    at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
    at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
    at org.apache.spark.deploy.worker.Worker$$anon$1.run(Worker.scala:277)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Connecting to /master_node_private_ip_address:7077 timed out (120000 ms)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:251)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:195)
    at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
    ... 4 more

我尝试从远程 ssh 到本地,但不起作用。我放入本地 Mac 的公共 ssh 密钥是来自远程服务器的 root 用户密钥。但是,我使用特定用户名 ssh 远程,该用户名不是 root,并且没有生成 ssh 密钥。因此,我认为该 ssh 密钥可能不适用于整个系统,并在执行 后尝试 ssh 返回本地 Mac sudo su root。但仍然不起作用。

我如何允许 ssh 从远程到本地?我将非常感激一些帮助,以了解为什么无法正确连接并在 Web 服务器上显示从属节点。

相关内容