尝试运行 spark shell 时出错

尝试运行 spark shell 时出错

我已经关注本手册在我的 Ubuntu 16.04 上运行 spark。我已经下载了 Spark 2.3、Java 9 和 Scala 2.12.5。

在我的/.bashrc中:

export JAVA_HOME=/usr/local/java/jdk-9.0.1  
export SPARK_HOME=/usr/bin/spark  
export PATH=$PATH:SPARK_HOME/bin  
export PATH=$PATH:SPARK_HOME/sbin  
export SBT_HOME=/usr/bin/sbt  
export PATH=$PATH:$SBT_HOME/bin 

在我的spark-env.sh中:

JAVA_HOME=/usr/local/java/jdk-9.0.1  
SPARK_MASTER_IP=10.1.20.241  
SPARK_WORKER_MEMORY=4g  

在文件末尾。

但是当我运行时./spark-shell出现此错误:

2018-03-28 17:17:26 WARN  Utils:66 - Your hostname, ali resolves to a loopback address: 127.0.1.1; using 192.168.1.52 instead (on interface enp0s3)
2018-03-28 17:17:26 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address  
WARNING: An illegal reflective access operation has occurred  
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/usr/bin/spark/jars/hadoop-auth-2.6.5.jar) to method sun.security.krb5.Config.getInstance()  
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

2018-03-28 17:17:28 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Failed to initialize compiler: object java.lang.Object in compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programmatically, settings.usejavacp.value = true.  
Failed to initialize compiler: object java.lang.Object in compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programmatically, settings.usejavacp.value = true.
Exception in thread "main" java.lang.NullPointerException

相关内容