SQLLDR载数加速,优化参数
转载备忘,地址:http://www.dba-oracle.com/t_optimize_sql_loader_sqlldr_performance.htm
以下为正文和benchmark数据
1.Use Direct Path Loads - The conventional path loader essentially loads the data by using standard insert statements. The direct path loader (direct=true) loads directly into the Oracle data files and creates blocks in Oracle database block format. The fact that SQL is not being issued makes the entire process much less taxing on the database. There are certain cases, however, in which direct path loads cannot be used (clustered tables). To prepare the database for direct path loads, the script $ORACLE_HOME/rdbms/admin/catldr.sql.sql must be executed.
2.Disable Indexes and Constraints. For conventional data loads only, the disabling of indexes and constraints can greatly enhance the performance of SQL*Loader.
3.Use a Larger Bind Array. For conventional data loads only, larger bind arrays limit the number of calls to the database and increase performance. The size of the bind array is specified using the bindsizeparameter. The bind array's size is equivalent to the number of rows it contains (rows=) times the maximum length of each row. Also see the columnarrayrows and streamsize parameters.
4.Use ROWS=n . For conventional data loads only, rows specifies the number of rows per commit and is related to bindsize. Issuing fewer commits will enhance performance, and the larger rows parameter affects performance (see benchmark below).
5.Use Parallel Loads. Available with direct path data loads only, this option allows multiple SQL*Loader jobs to execute concurrently.
$ sqlldr control=first.ctl parallel=true direct=true
$ sqlldr control=second.ctl parallel=true direct=true
6.Use Fixed Width Data. Fixed width data format saves Oracle some processing when parsing the data. The savings can be tremendous, depending on the type of data and number of rows.
7.Disable Archiving During Load. While this may not be feasible in certain environments, disabling database archiving can increase performance considerably.
8.Use unrecoverable. The unrecoverable option (unrecoverable load data) disables the writing of the data to the redo logs. This option is available for direct path loads only.
Benchmark data
From the book "Advanced Oracle Utilities" we see a valid benchmark of SQL*Loader performance.
"Using the table table_with_one_million_rows, the following benchmark tests were performed with the various SQL*Loader options. The table was truncated after each test.
SQL*Loader Option |
Elapsed Time (Seconds) |
Time Reduction |
direct=false rows=64 |
135 |
- |
direct=false bindsize=512000 rows=10000 |
92 |
32% |
direct=false bindsize=512000 rows=10000 DB in noarchivelog mode |
85 |
37% |
direct=true |
47 |
65% |
direct=true unrecoverable |
41 |
70% |
direct=true unrecoverable fixed width data |
41 |
70% |
SQL*Loader test results indicate conventional path loads take longest.
The results above indicate that conventional path loads take the longest. However, the bindsize and rows parameters can aid the performance under these loads. The test involving the conventional load didn’t come close to the performance of the direct path load with the unrecoverable option specified.
It is also worth noting that the fastest import time achieved for this table (earlier) was 67 seconds, compared to 41 for SQL*Loader direct path – a 39% reduction in execution time. This proves that SQL*Loader can load the same data faster than import.
These tests did not compensate for indexes. All database load operations will execute faster when indexes are disabled.
Another SQL*Loader benchmark test
This benchmark byWarren Koch tests a SQL*Loader (sqlldr) import of 2m rows, using direct path with index skip set for the baseline. Here's results for differing values for column array and stream size in SQL*Loader timings.
CONTROL FILE SNIPPET:
OPTIONS (DIRECT=TRUE, SKIP=TRUE, ERRORS=50, rows=500000, COLUMNARRAYROWS=xx, STREAMSIZE=yy)
UNRECOVERABLE LOAD DATATRUNCATE
into table F15_ADPLS_NEXTASSYfields terminated by X'9' optionally enclosed by X'1F'
TRAILING NULLCOLS {…}
This is for a load of 1,964,601 rows (about 2 million) from a delimited file. I know I could achieve much higher speeds going to a fixed width format but my data source precludes that.
Report headings:
columns = COLUMNARRAYROWS Parameter Setting (xx)
Stream = STREAMSIZE parameter setting (yy)
CPU = CPU time in minutes (from sqlldr log)
Elapsed = Elapsed time in minutes (from sqlldr log)
Main = Total stream buffers loaded by SQL*Loader main thread (from sqlldr log)
Load = Total stream buffers loaded by SQL*Loader load thread (from sqlldr log)
Columns Stream Elap time CPU Main Load
100 256,000 04:12.0 02:52.3 19908 0
1,000 256,000 04:14.4 02:57.3 2219 218
5,000 256,000 04:17.3 03:05.9 515 1,350
50,000 256,000 04:19.1 03:09.9 515 1,350
5,000 500,000 04:15.9 03:02.8 515 512
5,000 512,000 04:14.6 03:02.7 515 512
5,000 128,000 04:11.0 03:05.5 515 2,908
5,000 64,000 04:10.0 03:06.7 515 5876
转载于:https://blog.51cto.com/feixiong/1363760
SQLLDR载数加速,优化参数相关推荐
- selenium-java 优化参数设置,无界面化、允许root运行,页面不加载图片
下面是linux下环境selenium的配置 import org.openqa.selenium.WebDriver; import org.openqa.selenium.chrome.Chrom ...
- 卷积神经网络之优化参数(剪子包袱锤)
目录 一.优化参数的三个方法 1.手动修改 2.for循环调参 3.Keras Tunner自动调参工具 介绍 1.安装 2.准备训练数据和加载的库 3.创建HyperParameters对象以及模型 ...
- mysql属性配置提高查询_MYSQL性能优化-安装时优化参数配置提高服务性能
MYSQL性能优化一直是个头痛的问题,目前大多都是直接把页面html静态页面或直接使用了缓存技术,下面我就mysql本身的性能优化来分享一下. 安装时优化参数配置提高服务性能 在Linux下安装Mys ...
- LNMP架构之PHP——MemCache对PHP页面的缓存加速优化
前言 1.什么是MemCache? MemCache是一个自由.源码开放.高性能.分布式的分布式内存对象缓存系统,用于动态Web应用以减轻数据库的负载. 它通过在内存中缓存数据和对象来减少读取数据库的 ...
- MemCache对PHP页面的缓存加速优化
一.MemCache 简介 Memcache 是一个高性能的分布式内存对象缓存系统,用于动态Web应用以减轻数据库负载.它通过在内存中缓存数据和对象来减少读取数据库的次数,从而提高了网站访问的速度. ...
- PHP服务缓存加速优化实战
PHP服务缓存加速优化实战 (1) 操作码介绍及缓存原理: 当客户端请求一个PHP程序的时候,服务器的 PHP 引擎会解析该 PHP 程序,并将其编译为特定的操作码(Operate Code)文件. ...
- 【JVM】优化参数+优化工具
[JVM]优化参数+优化工具 (一)上线前评估的时候JVM设置合适的参数 [1]JVM参数 [2]典型JVM参数配置参考 [3]内存结构分析 (二)什么时候需要JVM调优?具体的指标? [1]如果使用 ...
- 【vite+vue3.2 项目性能优化实战】使用vite-plugin-cdn-import进行CDN加速优化项目体积
CDN(Content Delivery Network)即内容分发网络,是一种通过在全球范围内分布式部署服务器来加速网络内容传输的技术.CDN加速的原理是,当用户请求访问某个资源时,CDN会根据用户 ...
- HiveSQL运行优化参数配置
HiveSQL运行优化参数配置 HiveSQL常用数据处理语句 HiveSQL中复杂数据类型操作 我们平时在使用hive执行一些SQL任务时.经常会遇到执行速度很慢,数据倾斜,资源不够用等问题.那是因 ...
- vsan主机高级优化参数配置
vsan主机高级优化参数配置 https://blog.51cto.com/roberthu/2049330 esxi6.0 vc6.0 vsan6.2高级参数优化 esxcfg-advcfg -s ...
最新文章
- *** WARNING L1: UNRESOLVED EXTERNAL SYMBOL
- Centos5.5完整安装Cacti+Spine
- Android开发ADB的安装
- 《Windows网络与通信程序设计(第3版)》——1.4 网络应用程序设计基础
- mysql卸载权限不够_Linu下启动MySQL结果显示:env: /etc/init.d/mysql:权限不够怎么解决?...
- 不妨问问自己,学习C语言是为了什么?
- Java Short类hashCode()方法及示例
- 击溃音乐服务器第一人!周杰伦新歌首发,QQ音乐服务器一度崩溃
- CentOS7 修改静态IP
- 开源中国iOS客户端学习——(七)MBProgressHUD特效
- 汇编语言通过WMI获取BIOS、主板、硬盘、CPU、网卡的信息
- Oracle 定时任务详解(dbms_job)
- 第1章 引论 - 数据结构与算法分析 c语言描述
- oracle创建视图多表关联,Oracle创建两表关联查询的视图
- 数据分析系列:生存分析(生存曲线分析、Cox回归分析)——附生存分析python代码。
- 内网渗透系列:内网隧道之icmp_tran
- CentOS7 开启路由转发
- 【BZOJ1502】【NOI2005】月下柠檬树
- 【C语言】共用体的定义与使用
- win10系统文件拖拽卡顿_win10电脑中鼠标拖动窗口有延迟如何解决