LightDB（pg） 通过DBeaver执行SQL发现SQL不能并行
LightDB（pg） 通过DBeaver执行SQL发现SQL不能并行
最近遇到一个问题，在dbeaver上执行explain analyze 发现SQL使用到了并行，且执行时间为30多秒；而去掉explain analyze 实际执行时，发现用了50多秒，然后通过观察后端进程，发现没有并行进程在执行此SQL。通过网上查找资料及自己跟踪代码发现是由于如下的原因导致的。简单来说就是如果设置了fetchsize 或者maxrows 会导致不能并行。
官方文档中也有此说明：
The client sends an Execute message with a non-zero fetch count. See the discussion of the extended query protocol. Since libpq currently provides no way to send such a message, this can only occur when using a client that does not rely on libpq. If this is a frequent occurrence, it may be a good idea to set max_parallel_workers_per_gather to zero in sessions where it is likely, so as to avoid generating query plans that may be suboptimal when run serially.
jdbc 中执行逻辑如下
private void sendOneQuery(SimpleQuery query, SimpleParameterList params, int maxRows,int fetchSize, int flags) throws IOException {boolean asSimple = (flags & QueryExecutor.QUERY_EXECUTE_AS_SIMPLE) != 0;if (asSimple) {assert (flags & QueryExecutor.QUERY_DESCRIBE_ONLY) == 0: "Simple mode does not support describe requests. sql = " + query.getNativeSql()+ ", flags = " + flags;sendSimpleQuery(query, params);return;}assert !query.getNativeQuery().multiStatement: "Queries that might contain ; must be executed with QueryExecutor.QUERY_EXECUTE_AS_SIMPLE mode. "+ "Given query is " + query.getNativeSql();// nb: if we decide to use a portal (usePortal == true) we must also use a named statement// (oneShot == false) as otherwise the portal will be closed under us unexpectedly when// the unnamed statement is next reused.boolean noResults = (flags & QueryExecutor.QUERY_NO_RESULTS) != 0;boolean noMeta = (flags & QueryExecutor.QUERY_NO_METADATA) != 0;boolean describeOnly = (flags & QueryExecutor.QUERY_DESCRIBE_ONLY) != 0;boolean usePortal = (flags & QueryExecutor.QUERY_FORWARD_CURSOR) != 0 && !noResults && !noMeta&& fetchSize > 0 && !describeOnly;boolean oneShot = (flags & QueryExecutor.QUERY_ONESHOT) != 0 && !usePortal;boolean noBinaryTransfer = (flags & QUERY_NO_BINARY_TRANSFER) != 0;boolean forceDescribePortal = (flags & QUERY_FORCE_DESCRIBE_PORTAL) != 0;// Work out how many rows to fetch in this pass.int rows;if (noResults) {rows = 1; // We're discarding any results anyway, so limit data transfer to a minimum} else if (!usePortal) {rows = maxRows; // Not using a portal -- fetchSize is irrelevant} else if (maxRows != 0 && fetchSize > maxRows) {// fetchSize > maxRows, use maxRows (nb: fetchSize cannot be 0 if usePortal == true)rows = maxRows;} else {rows = fetchSize; // maxRows > fetchSize}sendParse(query, params, oneShot);// Must do this after sendParse to pick up any changes to the// query's state.//boolean queryHasUnknown = query.hasUnresolvedTypes();boolean paramsHasUnknown = params.hasUnresolvedTypes();boolean describeStatement = describeOnly|| (!oneShot && paramsHasUnknown && queryHasUnknown && !query.isStatementDescribed());if (!describeStatement && paramsHasUnknown && !queryHasUnknown) {int queryOIDs[] = query.getStatementTypes();int paramOIDs[] = params.getTypeOIDs();for (int i = 0; i < paramOIDs.length; i++) {// Only supply type information when there isn't any// already, don't arbitrarily overwrite user supplied// type information.if (paramOIDs[i] == Oid.UNSPECIFIED) {params.setResolvedType(i + 1, queryOIDs[i]);}}}if (describeStatement) {sendDescribeStatement(query, params, describeOnly);if (describeOnly) {return;}}// Construct a new portal if needed.Portal portal = null;if (usePortal) {String portalName = "C_" + (nextUniqueID++);portal = new Portal(query, portalName);}sendBind(query, params, portal, noBinaryTransfer);// A statement describe will also output a RowDescription,// so don't reissue it here if we've already done so.//if (!noMeta && !describeStatement) {/** don't send describe if we already have cached the row description from previous executions** XXX Clearing the fields / unpreparing the query (in sendParse) is incorrect, see bug #267.* We might clear the cached fields in a later execution of this query if the bind parameter* types change, but we're assuming here that they'll still be valid when we come to process* the results of this query, so we don't send a new describe here. We re-describe after the* fields are cleared, but the result of that gets processed after processing the results from* earlier executions that we didn't describe because we didn't think we had to.** To work around this, force a Describe at each execution in batches where this can be a* problem. It won't cause more round trips so the performance impact is low, and it'll ensure* that the field information available when we decoded the results. This is undeniably a* hack, but there aren't many good alternatives.*/if (!query.isPortalDescribed() || forceDescribePortal) {sendDescribePortal(query, portal);}}sendExecute(query, portal, rows);}
对并发有影响的是上面的rows, 也即是下面发送的limit值
sendExecute:pgStream.sendChar('E'); // ExecutepgStream.sendInteger4(4 + 1 + encodedSize + 4); // message sizeif (encodedPortalName != null) {pgStream.send(encodedPortalName); // portal name}pgStream.sendChar(0); // portal name terminatorpgStream.sendInteger4(limit); // row limit
LightDB 中对上述请求的处理逻辑如下
由于是’E’，然后会执行 exec_execute_message
portal_name = pq_getmsgstring(&input_message);max_rows = pq_getmsgint(&input_message, 4);pq_getmsgend(&input_message);exec_execute_message(portal_name, max_rows);
exec_execute_message /* execute */
exec_execute_message：/** If we re-issue an Execute protocol request against an existing portal,* then we are only fetching more rows rather than completely re-executing* the query from the start. atStart is never reset for a v3 portal, so we* are safe to use this check.*/execute_is_fetch = !portal->atStart;PortalRun(xxx,!execute_is_fetch && max_rows == FETCH_ALL,xxx) //bool run_once = !execute_is_fetch && max_rows == FETCH_ALL最终在执行器部分，会使用到run_once即如下的execute_once，如果execute_once为false，及会多次执行则会导致不能并行。
ExecutePlan:if (!execute_once)use_parallel_mode = false;
参考
聊聊pg jdbc statement的maxRows参数
When Can Parallel Query Be Used?
How the SQL client can influence performance
LightDB（pg） 通过DBeaver执行SQL发现SQL不能并行相关推荐
- DBeaver执行sql脚本报错：CreateProcess error=193, %1 不是有效的 Win32 应用程序。
DBeaver执行sql脚本报错:CreateProcess error=193, %1 不是有效的 Win32 应用程序. 如图: 定位发现DBeaver默认安装的mysql.exe大小为0字节! ...
- DBeaver执行.sql脚本报错：ERROR 1064 (42000) at line 1
DBeaver执行.sql脚本报错,报错内容如下: ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check ...
- Jmeter JDBC执行多条SQL
今天在编写自动化回归脚本的时候,需要在jmeter的jdbc请求中执行多条sql,在百度里搜索了一些文章,按照网上提供的步骤,发现不起作用,后来发现是作者的截图误导了,为了让后面的同学少走弯路,这里我 ...
- 执行多条SQL语句，执行数据库事务（可传入Sql参数）
上篇博客,实例介绍了一个事务执行多条SQL语句函数(int ExecuteSqlTran(List<String> SQLStringList))点击打开链接,方便之余又发现了它的缺陷-- ...
- oracle执行脚本顺序执行吗,【ORACLE】记录通过执行Oracle的执行计划查询SQL脚本中的效率问题 - 不及格的飞鱼...
记录通过执行Oracle的执行计划查询SQL脚本中的效率问题 问题现象: STARiBOSS5.8.1R2版本中,河北对帐JOB执行时,无法生成发票对帐文件. 首先,Quartz表达式培植的启动时间为 ...
- oracle多条sql语句常量,如何在Oracle中一次执行多条sql语句
有时我们需要一次性执行多条sql语句,而用来更新的sql是根据实际情况用代码拼出来的 解决方案是把sql拼成下面这种形式: begin update TB_VG set seq = 1, vessel ...
- mysql sql执行cmd命令行_命令行执行MySQL的sql文件
在项目开发过程中, 可能存在数据库导入导出,如果文件过大,会发现通过数据库(MySQL等)管理工具进行文件导入会报错,超出最大文件的限制. 需要更改工具的文件大小限制,网上有很多教程,比较麻烦,这里介 ...
- c#执行多句oracle,C#一次执行多条SQL语句，Oracle11g数据库
由于经常执行SQL语句,如果一条一条执行效率低下. oarclecmd.CommandText = sqlstr; oraclecmd.ExecuteNonQuery(); sqlstr 可以写成如下 ...
- 使用系统视图发现SQL Server实例信息
介绍 (Introduction) Out of the box, SQL Server comes with a substantial and – release by release – eve ...
最新文章
- Cron 表达式详解
- 读书印记 - 《注意力经济：如何把大众的注意力变成生意》
- 全球新能源汽车行业前景规模及发展趋势预测报告2022-2028年版
- Java设计模式-责任链模式
- Spring MVC概述
- Spring Boot学习总结（19）——使用Redisson实现分布式锁
- LeetCode：递归思想的延伸，从斐波那契数列到爬楼梯模型
- 如何向水晶报表数据源中的存储过程传参数……
- 算法笔记：tarjan算法求强连通分量割点桥
- HTML5的文档声明
- 哪些事是考研前不知道，考研后才知道的？
- panabit连接控制
- ARCore从零到一 (3) 更换AR模型
- 【思前享后】区块链应用
- Centos7文本处理工具
- 苹果笔记本学php,Mac_Mac怎么连接校园网？苹果电脑系统设置可实现连接校园网方法步骤介绍， 在家使用Mac系统的电脑， - phpStudy...
- 如何使用几何体画人体结构？几何体画人体结构技巧！
- STM32CUBEMX创建X-CUBE-BLE1例程
- php有意思的小项目,推荐6个Github上超有意思的前端项目！
- 你的项目刚刚启动？是时候考虑Globalization了！ 1