hive2.3.4

presto0.215

使用hive2.3.4的beeline连接presto报错

$ beeline -d com.facebook.presto.jdbc.PrestoDriver -u "jdbc:presto://localhost:8080/hive"
Error: Unrecognized connection property 'url' (state=,code=0)

增加--verbose查看详细报错

$ beeline -d com.facebook.presto.jdbc.PrestoDriver -u "jdbc:presto://localhost:8080/hive" --verbose
Error: Unrecognized connection property 'url' (state=,code=0)
java.sql.SQLException: Unrecognized connection property 'url'at com.facebook.presto.jdbc.PrestoDriverUri.validateConnectionProperties(PrestoDriverUri.java:316)at com.facebook.presto.jdbc.PrestoDriverUri.<init>(PrestoDriverUri.java:95)at com.facebook.presto.jdbc.PrestoDriverUri.<init>(PrestoDriverUri.java:85)at com.facebook.presto.jdbc.PrestoDriver.connect(PrestoDriver.java:87)at java.sql.DriverManager.getConnection(DriverManager.java:664)at java.sql.DriverManager.getConnection(DriverManager.java:208)at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145)at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209)at org.apache.hive.beeline.Commands.connect(Commands.java:1641)at org.apache.hive.beeline.Commands.connect(Commands.java:1536)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56)at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1273)at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1312)at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:867)at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:776)at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1010)at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.hadoop.util.RunJar.run(RunJar.java:221)at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

这里是connect过程报错,connect参数为

Usage: connect <url> <username> <password> [driver]

跟进hive和presto代码

org.apache.hive.beeline.BeeLine

  boolean dispatch(String line) {
...if (isBeeLine) {if (line.startsWith(COMMAND_PREFIX)) {// handle SQLLine command in beeline which starts with ! and does not end with ;return execCommandWithPrefix(line);} else {return commands.sql(line, getOpts().getEntireLineAsCommand());}} else {return commands.sql(line, getOpts().getEntireLineAsCommand());}

beeline中的命令和sql都会经过dispatch,命令执行的是execCommandWithPrefix,然后会反射调用Commands

org.apache.hive.beeline.Commands

  public boolean connect(String line) throws Exception {String example = "Usage: connect <url> <username> <password> [driver]"+ BeeLine.getSeparator();String[] parts = beeLine.split(line);if (parts == null) {return false;}if (parts.length < 2) {return beeLine.error(example);}String url = parts.length < 2 ? null : parts[1];String user = parts.length < 3 ? null : parts[2];String pass = parts.length < 4 ? null : parts[3];String driver = parts.length < 5 ? null : parts[4];Properties props = new Properties();if (url != null) {String saveUrl = getUrlToUse(url);props.setProperty(JdbcConnectionParams.PROPERTY_URL, saveUrl);}String value = null;if (driver != null) {props.setProperty(JdbcConnectionParams.PROPERTY_DRIVER, driver);} else {value = Utils.parsePropertyFromUrl(url, JdbcConnectionParams.PROPERTY_DRIVER);if (value != null) {props.setProperty(JdbcConnectionParams.PROPERTY_DRIVER, value);}}if (user != null) {props.setProperty(JdbcConnectionParams.AUTH_USER, user);} else {value = Utils.parsePropertyFromUrl(url, JdbcConnectionParams.AUTH_USER);if (value != null) {props.setProperty(JdbcConnectionParams.AUTH_USER, value);}}if (pass != null) {props.setProperty(JdbcConnectionParams.AUTH_PASSWD, pass);} else {value = Utils.parsePropertyFromUrl(url, JdbcConnectionParams.AUTH_PASSWD);if (value != null) {props.setProperty(JdbcConnectionParams.AUTH_PASSWD, value);}}value = Utils.parsePropertyFromUrl(url, JdbcConnectionParams.AUTH_TYPE);if (value != null) {props.setProperty(JdbcConnectionParams.AUTH_TYPE, value);}return connect(props);}public boolean connect(Properties props) throws IOException {String url = getProperty(props, new String[] {JdbcConnectionParams.PROPERTY_URL,"javax.jdo.option.ConnectionURL","ConnectionURL",});String driver = getProperty(props, new String[] {JdbcConnectionParams.PROPERTY_DRIVER,"javax.jdo.option.ConnectionDriverName","ConnectionDriverName",});String username = getProperty(props, new String[] {JdbcConnectionParams.AUTH_USER,"javax.jdo.option.ConnectionUserName","ConnectionUserName",});String password = getProperty(props, new String[] {JdbcConnectionParams.AUTH_PASSWD,"javax.jdo.option.ConnectionPassword","ConnectionPassword",});if (url == null || url.length() == 0) {return beeLine.error("Property \"url\" is required");}if (driver == null || driver.length() == 0) {if (!beeLine.scanForDriver(url)) {return beeLine.error(beeLine.loc("no-driver", url));}}String auth = getProperty(props, new String[] {JdbcConnectionParams.AUTH_TYPE});if (auth == null) {auth = beeLine.getOpts().getAuthType();if (auth != null) {props.setProperty(JdbcConnectionParams.AUTH_TYPE, auth);}}beeLine.info("Connecting to " + url);if (Utils.parsePropertyFromUrl(url, JdbcConnectionParams.AUTH_PRINCIPAL) == null) {String urlForPrompt = url.substring(0, url.contains(";") ? url.indexOf(';') : url.length());if (username == null) {username = beeLine.getConsoleReader().readLine("Enter username for " + urlForPrompt + ": ");}props.setProperty(JdbcConnectionParams.AUTH_USER, username);if (password == null) {password = beeLine.getConsoleReader().readLine("Enter password for " + urlForPrompt + ": ",new Character('*'));}props.setProperty(JdbcConnectionParams.AUTH_PASSWD, password);}try {beeLine.getDatabaseConnections().setConnection(new DatabaseConnection(beeLine, driver, url, props));beeLine.getDatabaseConnection().getConnection();if (!beeLine.isBeeLine()) {beeLine.updateOptsForCli();}beeLine.runInit();beeLine.setCompletions();beeLine.getOpts().setLastConnectedUrl(url);return true;} catch (SQLException sqle) {beeLine.getDatabaseConnections().remove();return beeLine.error(sqle);} catch (IOException ioe) {return beeLine.error(ioe);}}

注意props字段的赋值过程

org.apache.hive.beeline.DatabaseConnection

  public DatabaseConnection(BeeLine beeLine, String driver, String url,Properties info) throws SQLException {this.beeLine = beeLine;this.driver = driver;this.url = url;this.info = info;}boolean connect() throws SQLException {
...Map<String, String> hiveVars = beeLine.getOpts().getHiveVariables();if (hiveVars != null){for (Map.Entry<String, String> var : hiveVars.entrySet()) {info.put(HIVE_VAR_PREFIX + var.getKey(), var.getValue());}}Map<String, String> hiveConfVars = beeLine.getOpts().getHiveConfVariables();if (hiveConfVars != null){for (Map.Entry<String, String> var : hiveConfVars.entrySet()) {info.put(HIVE_CONF_PREFIX + var.getKey(), var.getValue());}}if (isDriverRegistered) {// if the driver registered in the driver manager, get the connection via the driver manager
      setConnection(DriverManager.getConnection(getUrl(), info));} else {

Commands中props赋值给DatabaseConnection的info

java.sql.DriverManager

    public static Connection getConnection(String url,java.util.Properties info) throws SQLException {return (getConnection(url, info, Reflection.getCallerClass()));}private static Connection getConnection(String url, java.util.Properties info, Class<?> caller) throws SQLException {
...Connection con = aDriver.driver.connect(url, info);

DatabaseConnection的info传递给DriverManager然后传递给Driver,下面看具体的PrestoDriver

com.facebook.presto.jdbc.PrestoDriver

    public Connection connect(String url, Properties info)throws SQLException{if (!acceptsURL(url)) {return null;}PrestoDriverUri uri = new PrestoDriverUri(url, info);OkHttpClient.Builder builder = httpClient.newBuilder();uri.setupClient(builder);QueryExecutor executor = new QueryExecutor(builder.build());return new PrestoConnection(uri, executor);}

info传递给PrestoDriverUri进行初始化

com.facebook.presto.jdbc.PrestoDriverUri

    public PrestoDriverUri(String url, Properties driverProperties)throws SQLException{this(parseDriverUrl(url), driverProperties);}private PrestoDriverUri(URI uri, Properties driverProperties)throws SQLException{this.uri = requireNonNull(uri, "uri is null");address = HostAndPort.fromParts(uri.getHost(), uri.getPort());properties = mergeConnectionProperties(uri, driverProperties);validateConnectionProperties(properties);// enable SSL by default for standard portuseSecureConnection = SSL.getValue(properties).orElse(uri.getPort() == 443);initCatalogAndSchema();}private static void validateConnectionProperties(Properties connectionProperties)throws SQLException{for (String propertyName : connectionProperties.stringPropertyNames()) {if (ConnectionProperties.forKey(propertyName) == null) {throw new SQLException(format("Unrecognized connection property '%s'", propertyName));}}for (ConnectionProperty<?> property : ConnectionProperties.allProperties()) {property.validate(connectionProperties);}}

所以报错“Unrecognized connection property 'url' ”意味着ConnectionProperties.forKey("url") == null

com.facebook.presto.jdbc.ConnectionProperties

ConnectionProperties中确实没有url,所以presto与hive的beeline不兼容;

尝试修改PrestoDriverUri.java,去掉校验:

com.facebook.presto.jdbc.PrestoDriverUri

    private static void validateConnectionProperties(Properties connectionProperties)throws SQLException{
//        for (String propertyName : connectionProperties.stringPropertyNames()) {
//            if (ConnectionProperties.forKey(propertyName) == null) {
//                throw new SQLException(format("Unrecognized connection property '%s'", propertyName));
//            }
//        }for (ConnectionProperty<?> property : ConnectionProperties.allProperties()) {property.validate(connectionProperties);}}

将以上几行注释

简单修改jar包中的class的方法,参考:https://www.cnblogs.com/barneywill/p/10552322.html

编译的时候发现一直报错,

presto-jdbc/com/facebook/presto/jdbc/PrestoDriverUri.java:145: error: method setupSocksProxy in class OkHttpUtil cannot be applied to given types;setupSocksProxy(builder, SOCKS_PROXY.getValue(properties));^required: Builder,Optional<com.google.common.net.HostAndPort>found: Builder,Optional<com.facebook.presto.jdbc.internal.guava.net.HostAndPort>reason: argument mismatch; Optional<com.facebook.presto.jdbc.internal.guava.net.HostAndPort> cannot be converted to Optional<com.google.common.net.HostAndPort>
presto-jdbc/com/facebook/presto/jdbc/PrestoDriverUri.java:146: error: method setupHttpProxy in class OkHttpUtil cannot be applied to given types;setupHttpProxy(builder, HTTP_PROXY.getValue(properties));^required: Builder,Optional<com.google.common.net.HostAndPort>found: Builder,Optional<com.facebook.presto.jdbc.internal.guava.net.HostAndPort>reason: argument mismatch; Optional<com.facebook.presto.jdbc.internal.guava.net.HostAndPort> cannot be converted to Optional<com.google.common.net.HostAndPort>
2 errors

然后发现原来是presto-jdbc打包的时候设置了relocation,会将一些包做shade并且改package,详见:

https://github.com/prestodb/presto/blob/master/presto-jdbc/pom.xml

    <properties><air.main.basedir>${project.parent.basedir}</air.main.basedir><shadeBase>com.facebook.presto.jdbc.internal</shadeBase></properties>
...<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-shade-plugin</artifactId><executions><execution><phase>package</phase><goals><goal>shade</goal></goals><configuration><createSourcesJar>true</createSourcesJar><shadeSourcesContent>true</shadeSourcesContent><dependencyReducedPomLocation>${project.build.directory}/pom.xml</dependencyReducedPomLocation><relocations><relocation><pattern>com.facebook.presto.client</pattern><shadedPattern>${shadeBase}.client</shadedPattern></relocation><relocation><pattern>com.facebook.presto.spi</pattern><shadedPattern>${shadeBase}.spi</shadedPattern></relocation><relocation><pattern>com.fasterxml.jackson</pattern><shadedPattern>${shadeBase}.jackson</shadedPattern></relocation><relocation><pattern>com.google.common</pattern><shadedPattern>${shadeBase}.guava</shadedPattern></relocation><relocation><pattern>com.google.thirdparty</pattern><shadedPattern>${shadeBase}.guava</shadedPattern></relocation><relocation><pattern>io.airlift</pattern><shadedPattern>${shadeBase}.airlift</shadedPattern></relocation><relocation><pattern>javax.annotation</pattern><shadedPattern>${shadeBase}.javax.annotation</shadedPattern></relocation><relocation><pattern>javax.inject</pattern><shadedPattern>${shadeBase}.inject</shadedPattern></relocation><relocation><pattern>org.openjdk.jol</pattern><shadedPattern>${shadeBase}.jol</shadedPattern></relocation><relocation><pattern>org.joda.time</pattern><shadedPattern>${shadeBase}.joda.time</shadedPattern></relocation><relocation><pattern>okhttp3</pattern><shadedPattern>${shadeBase}.okhttp3</shadedPattern></relocation><relocation><pattern>okio</pattern><shadedPattern>${shadeBase}.okio</shadedPattern></relocation></relocations>

所以需要手工调整源代码中的package

okhttp3 -> com.facebook.presto.jdbc.internal.okhttp3
com.google.common -> com.facebook.presto.jdbc.internal.guava
com.facebook.presto.client -> com.facebook.presto.jdbc.internal.client

源代码地址:

https://github.com/prestodb/presto/blob/master/presto-jdbc/src/main/java/com/facebook/presto/jdbc/PrestoDriverUri.java

然后再编译就ok了,编译命令:

# javac -cp presto-jdbc-0.215.jar -d presto-jdbc presto-jdbc/com/facebook/presto/jdbc/PrestoDriverUri.java

然后重新打包,再使用beeline连接就ok了

转载于:https://www.cnblogs.com/barneywill/p/10565750.html

【原创】大叔问题定位分享(33)beeline连接presto报错相关推荐

  1. impala java 堆内存配置_【原创】大叔问题定位分享(38)impala报错内存不足

    impala有时查询报错内存不足,并持续一段时间后自动恢复,报错时日志如下: org.apache.hive.service.cli.HiveSQLException: ExecQueryFInsta ...

  2. beeline连接hiveserver2报错:User: root is not allowed to impersonate root

    最近在生产中搭建HA机制的集群,碰到不少坑,会在接下来的时间里好好总结下,先说下遇到的一个大坑.我们的需求是:希望通过hive的thrift服务来实现跨语言访问Hive数据仓库.但是第一步,你得需要在 ...

  3. 登录mysql报错2059_navicat连接mysql报错2059的解决方法

    navicat连接mysql报错2059的解决方法 发布时间:2020-12-21 10:27:37 来源:亿速云 阅读:82 作者:小新 栏目:数据库 这篇文章给大家分享的是有关navicat连接m ...

  4. win7设置ipsec_解决win7连接IPsec报错789和809错误

    解决win-7连接IPsec报错789和809错误 萌新一枚,大神请飘过~~~ 最近遇到一个很沙雕的问题,win7连接IPsec一直报错789,百度方法解决掉后又报错809,809解决掉后继续报789 ...

  5. 分享个老司机网站报错404页面模板源码

    版权声明:转载请注明原创地址 https://blog.csdn.net/u013032788/article/details/80623193 来,来,来,要不要学习下怎么做前端页面,分享个老司机网 ...

  6. pymysql connect 连接mysql 报错keyerror255

    文章目录 摘要 报错信息 解决方案 摘要 pymysql connect 连接mysql 报错keyerror255:最近困了我两个多月的一个难题,搜这个标题进来的都可以看到搜索引擎提供了n^2篇解决 ...

  7. jni连接dll报错

    jni连接dll报错 用了2个月好好的今天突然出现这个问题 Execution protection violation #A fatal error has been detected by the ...

  8. 登录mysql报错2059,navicat连接mysql报错2059如何解决

    navicat连接mysql报错2059,如下图所示: 网上查询过后,发现这个错误出现的原因是在mysql8之前的版本中加密规则为mysql_native_password,而在mysql8以后的加密 ...

  9. natcat for mysql_用Navicat for mysql连接mysql报错1251-解决办法

    TP框架连接mongodb报错及解决办法 mongodb版本3.4.7 1.认证错误:Failed to connect to: localhost:27017: Authentication fai ...

最新文章

  1. LeetCode简单题之二叉树的层平均值
  2. windows查看端口占用以及关闭相应的进程
  3. 浅谈MVC Form认证
  4. python 求和为正整数n的连续整数序列
  5. vs2013 c# 中调用 c 编写的dll出错的可能错误
  6. 最长有序子序列—动态规划算法
  7. 控制HTML Input只能输入数字和小数点
  8. QCon演讲|闲鱼从零到千万DAU的应用架构演进
  9. Oracle19C的dbhome,Windows server 安装Oracle19c (WINDOWS.X64_193000_db_home.zip) 过程碰到的问题总结...
  10. xshell下载及连接Linux
  11. 查看工作组计算机没有服务器,计算机相关:网上邻居问题:当前工作组的服务器列表无法使用...
  12. -函数-MATLAB提供的函数/主子函数/匿名-嵌套函数
  13. idc机房运维巡检_最全的弱电机房工程运维方案,后附所需表格
  14. java微博情感倾向性分析_基于微博的情感倾向性分析方法研究
  15. cx_Oracle报错“DPI-1047: Cannot locate a 64-bit Oracle Client library”
  16. 品味kettle--(一)用eclipse 搭建源码
  17. Android手机投屏后没有声音,乐播投屏电视没有声音怎么办_乐播投屏电视没有声音手机有声音解决办法_3DM手游...
  18. 瑞士酒店管理大学计算机专业,瑞士DCT国际酒店管理学院生活费用详细介绍
  19. 视频:插入U盘后Win7系统蓝屏,报iusb3xhc.sys
  20. Apple MDM Bypass 免越狱绕过MDM配置锁 免费

热门文章

  1. as上的git到码云操作_高校版News||码云联合实训邦发布软件工程在线实训教程
  2. python大数据招聘信息_2017招聘大数据丨Python需求增速达174%,AI人才缺口超百万!...
  3. Python中出现“TabError: inconsistent use of tabs and spaces in indentation”原因和解决方法
  4. linux touch
  5. 廖大python实战教程第七天
  6. Vue 基础的开发环境
  7. 基于FPGA的DDR3多端口读写存储管理系统设计
  8. EnjoyCSS – 在线的,先进的 CSS3 代码生成器
  9. Twips 缇的含义[来自网络]
  10. Object类中hashCode()和equals()方法详解(附图)