将节点添加到 Hadoop 集群 virtualbox

将节点添加到 Hadoop 集群 virtualbox

我正在尝试使用 virtualbox 设置一个迷你 Hadoop 集群:

hosts文件(正确配置的hosts文件):

hadoop@rui-VirtualBox:/home/rui$ cat /etc/hosts | grep 10.
10.0.13.96 hnode1
10.0.13.97 hnode2
10.0.13.98 hnode3
10.0.13.99 hnode4

正确配置了 IP 地址(我手动编辑了 /etc/network/interfaces 文件,之后您可以看到我可以 ssh 到其他机器)

hadoop@rui-VirtualBox:/home/rui$ ifconfig | grep "inet addr"
          inet addr:10.0.13.96  Bcast:10.0.13.255  Mask:255.255.255.0
          inet addr:127.0.0.1  Mask:255.0.0.0

所有节点均无需密码进行 ssh 身份验证:

hadoop@rui-VirtualBox:/home/rui$ ssh hnode2
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

365 packages can be updated.
105 updates are security updates.

Last login: Tue Sep 23 10:52:52 2014 from hnode1
hadoop@rui-VirtualBox:~$ exit
logout
Connection to hnode2 closed.
hadoop@rui-VirtualBox:/home/rui$ ssh hnode3
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

365 packages can be updated.
105 updates are security updates.

Last login: Tue Sep 23 10:52:59 2014 from hnode1
hadoop@rui-VirtualBox:~$ exit
logout
Connection to hnode3 closed.
hadoop@rui-VirtualBox:/home/rui$ ssh hnode4
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

365 packages can be updated.
105 updates are security updates.

Last login: Tue Sep 23 10:53:04 2014 from hnode1
hadoop@rui-VirtualBox:~$ exit
logout
Connection to hnode4 closed.

从属文件定义如下:

hadoop@rui-VirtualBox:/usr/local/hadoop/hadoop-2.4.1/etc/hadoop$ cat slaves
localhost
hnode2
hnode3
hnode4

JPS 表示没有任何东西被破坏(无论是在主节点还是其他节点上)

hadoop@rui-VirtualBox:/home/rui$ cd $HADOOP_PREFIX
hadoop@rui-VirtualBox:/usr/local/hadoop/hadoop-2.4.1$ jps
3283 Jps
hadoop@rui-VirtualBox:/usr/local/hadoop/hadoop-2.4.1$ ssh hnode2
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

365 packages can be updated.
105 updates are security updates.

Last login: Tue Sep 23 10:53:46 2014 from hnode1
hadoop@rui-VirtualBox:~$ jps
1834 Jps
hadoop@rui-VirtualBox:~$ exit
logout
Connection to hnode2 closed.

下一步我启动集群:

hadoop@rui-VirtualBox:/home/rui$ $HADOOP_PREFIX/sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/hadoop/hadoop-2.4.1/logs/hadoop-hadoop-namenode-rui-VirtualBox.out
localhost: starting datanode, logging to /usr/local/hadoop/hadoop-2.4.1/logs/hadoop-hadoop-datanode-rui-VirtualBox.out
hnode2: starting datanode, logging to /usr/local/hadoop/hadoop-2.4.1/logs/hadoop-hadoop-datanode-rui-VirtualBox.out
hnode3: starting datanode, logging to /usr/local/hadoop/hadoop-2.4.1/logs/hadoop-hadoop-datanode-rui-VirtualBox.out
hnode4: starting datanode, logging to /usr/local/hadoop/hadoop-2.4.1/logs/hadoop-hadoop-datanode-rui-VirtualBox.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/hadoop-2.4.1/logs/hadoop-hadoop-secondarynamenode-rui-VirtualBox.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/hadoop-2.4.1/logs/yarn-hadoop-resourcemanager-rui-VirtualBox.out
hnode3: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.4.1/logs/yarn-hadoop-nodemanager-rui-VirtualBox.out
hnode2: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.4.1/logs/yarn-hadoop-nodemanager-rui-VirtualBox.out
hnode4: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.4.1/logs/yarn-hadoop-nodemanager-rui-VirtualBox.out
localhost: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.4.1/logs/yarn-hadoop-nodemanager-rui-VirtualBox.out

我确实做了 JPS,Hadoop 似乎在主节点和从节点上都运行

hadoop@rui-VirtualBox:/home/rui$ jps
2958 DataNode
3282 ResourceManager
3140 SecondaryNameNode
3411 NodeManager
2806 NameNode
3722 Jps
hadoop@rui-VirtualBox:/home/rui$ ssh hnode2
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

365 packages can be updated.
105 updates are security updates.

Last login: Tue Sep 23 11:17:27 2014 from hnode1
hadoop@rui-VirtualBox:~$ jps
2327 Jps
1889 NodeManager
1769 DataNode
hadoop@rui-VirtualBox:~$ exit
logout
Connection to hnode2 closed.

现在当我告诉我的浏览器转到 hhttp://hnode1:50070 时,我看到只有 1 个节点处于活动状态

当我告诉浏览器转到 hhttp://hnode1:8088/cluster 时,我也有一个可用节点。

在此处输入图片描述

顺便说一下,hhtp://hnode2:8088 或 hhttp://hnode2:50070 返回无法通过浏览器连接页面。

嗯,我应该有 4 个节点,但我只有一个。我做错了什么?

我创建了一个可运行的虚拟机,然后克隆了硬盘并创建了新的虚拟机,并为每个虚拟机分配了不同的 IP 地址。但是我没有重新格式化 hdfs(这是问题吗?)

然后对于主节点(hnode1),我编辑了从属文件(这样足够了吗?)

我究竟做错了什么?

相关内容