Sqoop import failed because of error “Make sure HIVE_CONF_DIR is set correctly”

Sqoop Import failed after CDH upgrading
参考リンク

Sqoop Import failed after CDH upgrading

Hadoop環境をCDH5.3.xから5.7.xへアップグレードするとsqoopは1.4.5から1.4.6になったが、sqoop importを実行しようとしたところ以下のようなエラーでhive-importに失敗してしまう。MapreduceジョブによるHDFS上の一時ディレクトリへの書き込みまでは問題なく進んでおり、最後のHiveCLIを起動してのLOAD DATAするところでエラーになっているようだった。

16/08/05 23:58:11 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
16/08/05 23:58:11 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50)
at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392)
at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:524)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44)
... 12 more

16/08/05 23:58:11 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.

16/08/05 23:58:11 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf

at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50)

at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392)

at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379)

at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)

at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)

at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:524)

at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)

at org.apache.sqoop.Sqoop.run(Sqoop.java:143)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)

at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)

at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)

at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:190)

at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44)

... 12 more

どうも参照しているHive関連のクラスが見つからないようだ。ソースを追っていくとHiveのLOAD DATAを実行しているのは以下の部分にあたる。

...
/**
* Execute Hive via an external 'bin/hive' process.
* @param filename the Script file to run.
* @param env the environment strings to pass to any subprocess.
* @throws IOException if Hive did not exit successfully.
*/
private void executeExternalHiveScript(String filename, List<String> env)
throws IOException {
// run Hive on the script and note the return code.
String hiveExec = getHiveBinPath();
String[] argv = getHiveArgs(hiveExec, "-f", filename);
LoggingAsyncSink logSink = new LoggingAsyncSink(LOG);
int ret = Executor.exec(argv, env.toArray(new String[0]), logSink, logSink);
if (0 != ret) {
throw new IOException("Hive exited with status " + ret);
}
}
...

...

/**

* Execute Hive via an external 'bin/hive' process.

* @param filename the Script file to run.

* @param env the environment strings to pass to any subprocess.

* @throws IOException if Hive did not exit successfully.

private void executeExternalHiveScript(String filename, List<String> env)

throws IOException {

// run Hive on the script and note the return code.

String hiveExec = getHiveBinPath();

String[] argv = getHiveArgs(hiveExec, "-f", filename);

LoggingAsyncSink logSink = new LoggingAsyncSink(LOG);

int ret = Executor.exec(argv, env.toArray(new String[0]), logSink, logSink);

if (0 != ret) {

throw new IOException("Hive exited with status " + ret);

}

...

getHiveArgsをさらに追っていくと、Exceptionで落ちているところは以下。

...
/**
* Dynamically create hive configuration object.
* @param conf
* @return
* @throws IOException if instantiate HiveConf failed.
*/
public static Configuration getHiveConf(Configuration conf) throws IOException {
try {
Class HiveConfClass = Class.forName(HIVE_CONF_CLASS);
return ((Configuration)(HiveConfClass.getConstructor(Configuration.class, Class.class)
.newInstance(conf, Configuration.class)));
} catch (ClassNotFoundException ex) {
LOG.error("Could not load " + HIVE_CONF_CLASS
+ ". Make sure HIVE_CONF_DIR is set correctly.");
throw new IOException(ex);
} catch (Exception ex) {
LOG.error("Could not instantiate HiveConf instance.", ex);
throw new IOException(ex);
}
}
...

...

/**

* Dynamically create hive configuration object.

* @param conf

* @return

* @throws IOException if instantiate HiveConf failed.

public static Configuration getHiveConf(Configuration conf) throws IOException {

try {

Class HiveConfClass = Class.forName(HIVE_CONF_CLASS);

return ((Configuration)(HiveConfClass.getConstructor(Configuration.class, Class.class)

.newInstance(conf, Configuration.class)));

} catch (ClassNotFoundException ex) {

LOG.error("Could not load " + HIVE_CONF_CLASS

+ ". Make sure HIVE_CONF_DIR is set correctly.");

throw new IOException(ex);

} catch (Exception ex) {

LOG.error("Could not instantiate HiveConf instance.", ex);

throw new IOException(ex);

}

...

HiveConfのクラスが見つからないということで、HIVEに関するクラスパスなどの環境変数を定義しているところを探してみる。

sqoopコマンドを追っていくと以下の箇所で環境変数であるHIVE_HOMEに関する設定を行なっていることがわかる。

if [ -z "${HIVE_HOME}" ]; then
if [ -d "/usr/lib/hive" ]; then
export HIVE_HOME=/usr/lib/hive
elif [ -d ${SQOOP_HOME}/../hive ]; then
export HIVE_HOME=${SQOOP_HOME}/../hive
fi
fi

if [ -z "${HIVE_HOME}" ]; then

if [ -d "/usr/lib/hive" ]; then

export HIVE_HOME=/usr/lib/hive

elif [ -d ${SQOOP_HOME}/../hive ]; then

export HIVE_HOME=${SQOOP_HOME}/../hive

私の環境では、/usr/lib/hive/の下に共通で使うライブラリを置いていたので、上記の処理の中でHIVE_HOMEが/usr/lib/hiveに向いてしまっていたようだった。しかし、私の環境ではCDHをClouderaManagerのParcelでインストールしているため、ここにhive本体は存在していない。

したがって、以下のように改めてHIVE_HOMEの正しいパスを指定してあげると、無事sqoopコマンドを実行することができた。

export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive

1	export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive

もしくは、/usr/lib/hiveをrmまたはmvすれば良いだろう。

参考リンク

http://www.ericlin.me/sqoop-hive-import-failed-after-upgrading-to-cdh5-4-x-or-cdh5-5-x
https://community.cloudera.com/t5/Batch-SQL-Apache-Hive/Error-with-quot-Make-sure-HIVE-CONF-DIR-is-set-correctly-quot/td-p/37423
https://github.com/apache/sqoop/commit/d408252086d4351eac211cfc2203dc15c660ad31