Hive 安装 - 执行 SQL 查询“从“DBS”中选择“DB_ID””时出错

Hive 安装 - 执行 SQL 查询“从“DBS”中选择“DB_ID””时出错

我正在尝试在具有 1 个名称节点和 3 个数据节点的 Hadoop(3.2.0)多节点集群上安装 Apache Hive(3.1.1)。

我已经关注入门教程在 apache 网站上一步一步地操作,但是当运行“hive”命令时,我在 hive.log 文件中得到了一个非常长的堆栈跟踪,同时在实际控制台中还出现了一些其他问题。

控制台错误与多个 SLF4J 绑定有关:

> SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found
> binding in
> [jar:file:/home/hadoop/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation. SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory] Hive Session ID =
> ebd3ca4d-0910-47d7-96a1-e7d1d369f919
> 
> Logging initialized using configuration in
> jar:file:/home/hadoop/hive/lib/hive-common-3.1.1.jar!/hive-log4j2.properties
> Async: true

Hive.log 错误似乎与设置 metastore 数据库有关。但是,我是这个领域的新手,并不完全熟悉所有使用的术语,因此我发现这个堆栈跟踪非常难以理解,并且找不到修复它的方法。下面是堆栈跟踪,但格式经过缩减,以适合这个问题的主体。

2019-07-05T10:24:28,966  INFO [main] conf.HiveConf: Found configuration file file:/home/hadoop/hive/conf/hive-site.xml
2019-07-05T10:24:29,990  INFO [main] SessionState: Hive Session ID = ebd3ca4d-0910-47d7-96a1-e7d1d369f919
2019-07-05T10:24:30,133  INFO [main] SessionState: 
Logging initialized using configuration in jar:file:/home/hadoop/hive/lib/hive-common-3.1.1.jar!/hive-log4j2.properties Async: true
2019-07-05T10:24:31,583  INFO [main] session.SessionState: Created HDFS directory: /tmp/hive/hadoop/ebd3ca4d-0910-47d7-96a1-e7d1d369f919
2019-07-05T10:24:31,618  INFO [main] session.SessionState: Created local directory: /tmp/hadoop/ebd3ca4d-0910-47d7-96a1-e7d1d369f919
2019-07-05T10:24:31,628  INFO [main] session.SessionState: Created HDFS directory: /tmp/hive/hadoop/ebd3ca4d-0910-47d7-96a1-e7d1d369f919/_tmp_space.db
2019-07-05T10:24:31,644  INFO [main] conf.HiveConf: Using the default value passed in for log id: ebd3ca4d-0910-47d7-96a1-e7d1d369f919
2019-07-05T10:24:31,644  INFO [main] session.SessionState: Updating thread name to ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main
2019-07-05T10:24:32,691  INFO [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
2019-07-05T10:24:32,717  WARN [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored
2019-07-05T10:24:32,723  INFO [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] metastore.ObjectStore: ObjectStore, initialize called
2019-07-05T10:24:32,724  INFO [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] conf.MetastoreConf: Found configuration file file:/home/hadoop/hive/conf/hive-site.xml
2019-07-05T10:24:32,725  INFO [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] conf.MetastoreConf: Unable to find config file hivemetastore-site.xml
2019-07-05T10:24:32,725  INFO [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] conf.MetastoreConf: Found configuration file null
2019-07-05T10:24:32,726  INFO [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] conf.MetastoreConf: Unable to find config file metastore-site.xml
2019-07-05T10:24:32,726  INFO [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] conf.MetastoreConf: Found configuration file null
2019-07-05T10:24:33,634  INFO [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] hikari.HikariDataSource: HikariPool-1 - Starting...
2019-07-05T10:24:33,641  WARN [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] util.DriverDataSource: Registered driver with driverClassName=org.apache.derby.jdbc.EmbeddedDriver was not found, trying direct instantiation.
2019-07-05T10:24:33,926  INFO [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] pool.PoolBase: HikariPool-1 - Driver does not support get/set network timeout for connections. (Feature not implemented: No details.)
2019-07-05T10:24:33,938  INFO [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] hikari.HikariDataSource: HikariPool-1 - Start completed.
2019-07-05T10:24:34,001  INFO [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] hikari.HikariDataSource: HikariPool-2 - Starting...
2019-07-05T10:24:34,001  WARN [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] util.DriverDataSource: Registered driver with driverClassName=org.apache.derby.jdbc.EmbeddedDriver was not found, trying direct instantiation.
2019-07-05T10:24:34,007  INFO [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] pool.PoolBase: HikariPool-2 - Driver does not support get/set network timeout for connections. (Feature not implemented: No details.)
2019-07-05T10:24:34,008  INFO [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] hikari.HikariDataSource: HikariPool-2 - Start completed.
2019-07-05T10:24:34,697  INFO [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2019-07-05T10:24:34,811  WARN [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] metastore.MetaStoreDirectSql: Self-test query [select "DB_ID" from "DBS"] failed; direct SQL is disabled
javax.jdo.JDODataStoreException: Error executing SQL query "select "DB_ID" from "DBS"".
...
Caused by: java.sql.SQLSyntaxErrorException: Table/View 'DBS' does not exist.
...
Caused by: org.apache.derby.iapi.error.StandardException: Table/View 'DBS' does not exist.
...
2019-07-05T10:24:34,817  INFO [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] metastore.ObjectStore: Initialized ObjectStore
2019-07-05T10:24:37,667 ERROR [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] metastore.RetryingHMSHandler: MetaException(message:Version information not found in metastore.)
...

2019-07-05T10:24:37,667 ERROR [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] metastore.RetryingHMSHandler: HMSHandler Fatal error: MetaException(message:Version information not found in metastore.)
...

2019-07-05T10:24:37,667  WARN [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] metadata.Hive: Failed to register all functions.
java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
...
Caused by: java.lang.reflect.InvocationTargetException
...
Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Version information not found in metastore.
...
2019-07-05T10:24:37,711 ERROR [ebd3ca4d-0910-47d7-96a1-e7d1d369f919 main] metadata.HiveMaterializedViewsRegistry: Problem connecting to the metastore when initializing the view registry
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
...
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
...
Caused by: java.lang.reflect.InvocationTargetException
...
Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Version information not found in metastore.

我之前没有在这个设备上使用过 Derby 数据库、Java 数据库连接 (JDBC) 或 MySQL - 这可能是依赖关系问题吗?虽然 Hive 教程中没有列出任何内容。

提前致谢。

答案1

我之前遇到过同样的问题,这是我的解决方案。

在我的系统中$HIVE_HOME/scripts/metastore/upgrade/derby/hive-schema-3.1.0.derby.sql,正在使用的数据库是“APP”(因为“ CREATE TABLE "APP"."DBS"”)

但在我看来$HIVE_HOME/conf/hive-site.xml,它是这样的:

  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:derby:;databaseName=metastore_db;create=true</value>
    <description>
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
    </description>
  </property>

因此我用“APP”替换了“metastore_db”,并重新初始化了 derby 元存储。

  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:derby:;databaseName=APP;create=true</value>
    <description>
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
    </description>
  </property>

相关内容