☰

HDFS 平衡，如何平衡 hdfs 数据？

tag-icon • tag-icon linux hadoop hdfs big-data
HDFS 平衡，如何平衡 hdfs 数据？

我们有 Hadoop 版本 - 2.6.4

在datanode机器上我们可以看到hdfs数据不平衡

在某些磁盘上，我们使用的大小与 sdb 不同11G和 sdd17G

/dev/sdd 20G 3.0G 17G 15%   /grid/sdd 
/dev/sdb 20G 11G 9.3G 53%   /grid/sdb <-- WHY DISK DISK NOT BALANCED AS SDD DISK , WHY DISKS ARE DIFF USED SIZE!!!

在谷歌搜索后，我找到了以下 CLI（来自https://community.hortonworks.com/questions/19694/help-with-exception-from-hdfs-balancer.html）

hdfs balancer -Ddfs.balancer.movedWinWidth=5400000 -Ddfs.balancer.moverThreads=1000 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.datanode.balance.bandwidthPerSec=100000000 -Ddfs.balancer.max-size-to-move=10737418240 -threshold 20 1>/tmp/balancer-out.log 2>/tmp/balancer-debug.log

运行后，我们得到了相同的 hdfs 大小

/dev/sdd 20G 3.0G 17G 15% /grid/sdd 
/dev/sdb 20G 11G 9.3G 53% /grid/sdb


more /tmp/balancer-out.log Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved The cluster is balanced. Exiting... Mar 7, 2019 5:02:34 PM 0 0 B 0 B 0 B Mar 7, 2019 5:02:34 PM Balancing took 1.453 seconds

所以实际上我们没有在 hdfs 中获得平衡

请指教，如何平衡 hdfs 数据，以便所有磁盘都具有相同的使用大小

答案1

NameNode 在选择接收这些块的 DataNode 之前会考虑各种参数。其中一些考虑因素包括：

1. Policy to keep one of the replicas of a block on the same node as the node that is writing the block.
2. Need to spread different replicas of a block across the racks so that cluster can survive loss of whole rack.
3. One of the replicas is usually placed on the same rack as the node writing to the file so that cross-rack network I/O is reduced.
4. Spread HDFS data uniformly across the DataNodes in the cluster.

因此，您遇到的情况可能与上述情况相符。

Apache Balancer 命令。

hdfs balancer [-threshold <threshold>] [-policy <policy>]
 -- threshold *threshold* Percentage of disk capacity. This overwrites the default threshold.
 -- policy *policy* *datanode* (default): Cluster is balanced if each datanode is balanced.  
                    *blockpool*: Cluster is balanced if each block pool in each datanode is balanced.

相关内容