![将节点添加到 Hadoop 集群 virtualbox](https://linux22.com/image/1424063/%E5%B0%86%E8%8A%82%E7%82%B9%E6%B7%BB%E5%8A%A0%E5%88%B0%20Hadoop%20%E9%9B%86%E7%BE%A4%20virtualbox.png)
我正在尝试使用 virtualbox 设置一个迷你 Hadoop 集群:
hosts文件(正确配置的hosts文件):
hadoop@rui-VirtualBox:/home/rui$ cat /etc/hosts | grep 10.
10.0.13.96 hnode1
10.0.13.97 hnode2
10.0.13.98 hnode3
10.0.13.99 hnode4
正确配置了 IP 地址(我手动编辑了 /etc/network/interfaces 文件,之后您可以看到我可以 ssh 到其他机器)
hadoop@rui-VirtualBox:/home/rui$ ifconfig | grep "inet addr"
inet addr:10.0.13.96 Bcast:10.0.13.255 Mask:255.255.255.0
inet addr:127.0.0.1 Mask:255.0.0.0
所有节点均无需密码进行 ssh 身份验证:
hadoop@rui-VirtualBox:/home/rui$ ssh hnode2
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)
* Documentation: https://help.ubuntu.com/
365 packages can be updated.
105 updates are security updates.
Last login: Tue Sep 23 10:52:52 2014 from hnode1
hadoop@rui-VirtualBox:~$ exit
logout
Connection to hnode2 closed.
hadoop@rui-VirtualBox:/home/rui$ ssh hnode3
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)
* Documentation: https://help.ubuntu.com/
365 packages can be updated.
105 updates are security updates.
Last login: Tue Sep 23 10:52:59 2014 from hnode1
hadoop@rui-VirtualBox:~$ exit
logout
Connection to hnode3 closed.
hadoop@rui-VirtualBox:/home/rui$ ssh hnode4
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)
* Documentation: https://help.ubuntu.com/
365 packages can be updated.
105 updates are security updates.
Last login: Tue Sep 23 10:53:04 2014 from hnode1
hadoop@rui-VirtualBox:~$ exit
logout
Connection to hnode4 closed.
从属文件定义如下:
hadoop@rui-VirtualBox:/usr/local/hadoop/hadoop-2.4.1/etc/hadoop$ cat slaves
localhost
hnode2
hnode3
hnode4
JPS 表示没有任何东西被破坏(无论是在主节点还是其他节点上)
hadoop@rui-VirtualBox:/home/rui$ cd $HADOOP_PREFIX
hadoop@rui-VirtualBox:/usr/local/hadoop/hadoop-2.4.1$ jps
3283 Jps
hadoop@rui-VirtualBox:/usr/local/hadoop/hadoop-2.4.1$ ssh hnode2
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)
* Documentation: https://help.ubuntu.com/
365 packages can be updated.
105 updates are security updates.
Last login: Tue Sep 23 10:53:46 2014 from hnode1
hadoop@rui-VirtualBox:~$ jps
1834 Jps
hadoop@rui-VirtualBox:~$ exit
logout
Connection to hnode2 closed.
下一步我启动集群:
hadoop@rui-VirtualBox:/home/rui$ $HADOOP_PREFIX/sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/hadoop/hadoop-2.4.1/logs/hadoop-hadoop-namenode-rui-VirtualBox.out
localhost: starting datanode, logging to /usr/local/hadoop/hadoop-2.4.1/logs/hadoop-hadoop-datanode-rui-VirtualBox.out
hnode2: starting datanode, logging to /usr/local/hadoop/hadoop-2.4.1/logs/hadoop-hadoop-datanode-rui-VirtualBox.out
hnode3: starting datanode, logging to /usr/local/hadoop/hadoop-2.4.1/logs/hadoop-hadoop-datanode-rui-VirtualBox.out
hnode4: starting datanode, logging to /usr/local/hadoop/hadoop-2.4.1/logs/hadoop-hadoop-datanode-rui-VirtualBox.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/hadoop-2.4.1/logs/hadoop-hadoop-secondarynamenode-rui-VirtualBox.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/hadoop-2.4.1/logs/yarn-hadoop-resourcemanager-rui-VirtualBox.out
hnode3: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.4.1/logs/yarn-hadoop-nodemanager-rui-VirtualBox.out
hnode2: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.4.1/logs/yarn-hadoop-nodemanager-rui-VirtualBox.out
hnode4: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.4.1/logs/yarn-hadoop-nodemanager-rui-VirtualBox.out
localhost: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.4.1/logs/yarn-hadoop-nodemanager-rui-VirtualBox.out
我确实做了 JPS,Hadoop 似乎在主节点和从节点上都运行
hadoop@rui-VirtualBox:/home/rui$ jps
2958 DataNode
3282 ResourceManager
3140 SecondaryNameNode
3411 NodeManager
2806 NameNode
3722 Jps
hadoop@rui-VirtualBox:/home/rui$ ssh hnode2
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)
* Documentation: https://help.ubuntu.com/
365 packages can be updated.
105 updates are security updates.
Last login: Tue Sep 23 11:17:27 2014 from hnode1
hadoop@rui-VirtualBox:~$ jps
2327 Jps
1889 NodeManager
1769 DataNode
hadoop@rui-VirtualBox:~$ exit
logout
Connection to hnode2 closed.
现在当我告诉我的浏览器转到 hhttp://hnode1:50070 时,我看到只有 1 个节点处于活动状态
当我告诉浏览器转到 hhttp://hnode1:8088/cluster 时,我也有一个可用节点。
顺便说一下,hhtp://hnode2:8088 或 hhttp://hnode2:50070 返回无法通过浏览器连接页面。
嗯,我应该有 4 个节点,但我只有一个。我做错了什么?
我创建了一个可运行的虚拟机,然后克隆了硬盘并创建了新的虚拟机,并为每个虚拟机分配了不同的 IP 地址。但是我没有重新格式化 hdfs(这是问题吗?)
然后对于主节点(hnode1),我编辑了从属文件(这样足够了吗?)
我究竟做错了什么?