我正在尝试使用

Python编写一个spark作业,它将打开与Impala的jdbc连接,并将Impala直接从Impala加载到Dataframe中.这个问题非常接近,但在scala中：

Calling JDBC to impala/hive from within a spark job and creating a table

我该怎么做呢？其他数据源有很多例子,例如MySQL,PostgreSQL等,但我还没有看到一个用于Impala Python Kerberos的数据源.一个例子会有很大的帮助.谢谢！

尝试使用来自网络的信息,但它没有用.

SPARK笔记本

#!/bin/bash

export PYSPARK_PYTHON=/home/anave/anaconda2/bin/python

export HADOOP_CONF_DIR=/etc/hive/conf

export PYSPARK_DRIVER_PYTHON=/home/anave/anaconda2/bin/ipython

export PYSPARK_DRIVER_PYTHON_OPTS='notebook --ip=* --no-browser'

# use Java8

export JAVA_HOME=/usr/java/latest

export PATH=$JAVA_HOME/bin:$PATH

# JDBC Drivers for Impala

export CLASSPATH=/home/anave/impala_jdbc_2.5.30.1049/Cloudera_ImpalaJDBC41_2.5.30/*.jar:$CLASSPATH

export JDBC_PATH=/home/anave/impala_jdbc_2.5.30.1049/Cloudera_ImpalaJDBC41_2.5.30

# --jars $SRCDIR/spark-csv-assembly-1.4.0-SNAPSHOT.jar \

# --conf spark.sql.parquet.binaryAsString=true \

# --conf spark.sql.hive.convertMetastoreParquet=false

pyspark --master yarn-client \

--driver-memory 4G \

--executor-memory 2G \

# --num-executors 10 \

--jars /home/anave/spark-csv_2.11-1.4.0.jar $JDBC_PATH/*.jar

--driver-class-path $JDBC_PATH/*.jar

Python代码

properties = {

"driver": "com.cloudera.impala.jdbc41.Driver",

"AuthMech": "1",

# "KrbRealm": "EXAMPLE.COM",

# "KrbHostFQDN": "impala.example.com",

"KrbServiceName": "impala"

}

# imp_env is the hostname of the db, works with other impala queries ran inside python

url = "jdbc:impala:imp_env;auth=noSasl"

db_df = sqlContext.read.jdbc(url=url, table='summary', properties=properties)

我收到此错误消息(Full Error Log)：

Py4JJavaError：调用o42.jdbc时发生错误.

：java.lang.ClassNotFoundException：com.cloudera.impala.jdbc41.Driver

最佳答案 您可以使用

--jars $(echo /dir/of/jars/*.jar | tr ' ' ',')

代替

--jars /home/anave/spark-csv_2.11-1.4.0.jar $JDBC_PATH/*.jar

或者对于另一种方法,请参阅我的answer