Ubuntu 上的 Hadoop 安装

Question 1

我刚刚在装有 Hadoop 版本 2.7.3 的 VM（Kubuntu 16.04）上运行了它。

先决条件：

Ubuntu 操作系统
具有 sudo 权限的非 root 用户
已安装 Java

脚步：

从以下位置下载 Hadoop 版本这里经过
- 点击压缩包所需版本列
- 点击下面的链接我们建议您从以下镜像站点进行下载：或右键单击并保存链接
  - 如果链接已保存，则使用以下命令下载：
    
    wget http://www-us.apache.org/dist/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz
检查下载的完整性：
- .md5从下载处下载文件页
- 运行shasum -a 256 hadoop-2.7.3.tar.gz并将结果与cat hadoop-2.7.3.tar.gz.mds此行进行比较
```
...
 hadoop-2.7.3.tar.gz: SHA256 = D489DF38 08244B90 6EB38F4D 081BA49E 50C4603D B03EFD5E 594A1E98 B09259C2
...
```

安装Hadoop：

解压并移动：

tar -xzvf hadoop-2.7.3.tar.gz && sudo mv hadoop-2.7.3 /usr/local/hadoop

使用以下命令在您的系统上找到 java：

readlink -f /usr/bin/java | sed "s:bin/java::"
# result
/usr/lib/jvm/java-8-openjdk-amd64/jre/

编辑hadoop-env.sh文件sudo nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh：

选项 1（静态值）：

#export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/

选项 2（动态值随着系统上的 Java 的变化而变化）：

#export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")

运行Hadoop：

/usr/local/hadoop/bin/hadoop

输出：

Usage: hadoop [--config confdir] [COMMAND | CLASSNAME]
CLASSNAME            run the class named CLASSNAME
or
where COMMAND is one of:
fs                   run a generic filesystem user client
version              print the version
jar <jar>            run a jar file
                     note: please use "yarn jar" to launch
                          YARN applications, not this command.
checknative [-a|-h]  check native hadoop and compression libraries availability
distcp <srcurl> <desturl> copy file or directories recursively
archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
classpath            prints the class path needed to get the
credential           interact with credential providers
                     Hadoop jar and the required libraries
daemonlog            get/set the log level for each daemon

测试：

mkdir ~/input
cp /usr/local/hadoop/etc/hadoop/*.xml ~/input

/usr/local/hadoop/bin/hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar grep ~/input ~/grep_example 'principal[.]*';

检查结果中的错误：

 File System Counters
        FILE: Number of bytes read=1247674
        FILE: Number of bytes written=2324248
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
 Map-Reduce Framework
        Map input records=2
        Map output records=2
        Map output bytes=37
        Map output materialized bytes=47
        Input split bytes=114
        Combine input records=0
        Combine output records=0
        Reduce input groups=2
        Reduce shuffle bytes=47
        Reduce input records=2
        Reduce output records=2
        Spilled Records=4
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=61
        Total committed heap usage (bytes)=263520256
  Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
 File Input Format Counters
        Bytes Read=151
 File Output Format Counters
        Bytes Written=37

来源：

https://www.digitalocean.com/community/tutorials/how-to-install-hadoop-in-stand-alone-mode-on-ubuntu-16-04

添加路径到.bashrc：

# hadoop executable
export PATH="/usr/local/hadoop/bin:$PATH"

Answer