MySQL千万级数据进行插入,基础数据3千万,插入1千万数据时间约为4.7分钟,10个线程同时插入
测试MySQL千万条数据插入速度
使用多线程,每条线程处理数据100万条,每次插入携带数据7万条进行提交
数据库基数为0,插入1000万条数据,时间为311957毫秒,也就是311.957秒,约为5.2分钟
数据库基础数据为2000万条数据,插入1000万条数据,时间为302545毫秒,也就是502.545秒。约5.1分钟
数据库基础数据为3000万条数据,插入1000万条数据,时间为286377毫秒,也就是286.377秒。约4.7分
1000万条数据日志
===================>>>>>DefaultManagedAwareThreadFactory-1
===================>>>>>DefaultManagedAwareThreadFactory-2
===================>>>>>DefaultManagedAwareThreadFactory-3
===================>>>>>DefaultManagedAwareThreadFactory-4
===================>>>>>DefaultManagedAwareThreadFactory-5
===================>>>>>DefaultManagedAwareThreadFactory-6
===================>>>>>DefaultManagedAwareThreadFactory-7
===================>>>>>DefaultManagedAwareThreadFactory-8
===================>>>>>DefaultManagedAwareThreadFactory-9
===================>>>>>DefaultManagedAwareThreadFactory-10
DefaultManagedAwareThreadFactory-2======结束=====>>>>285025
DefaultManagedAwareThreadFactory-7======结束=====>>>>286669
DefaultManagedAwareThreadFactory-3======结束=====>>>>296607
DefaultManagedAwareThreadFactory-6======结束=====>>>>298840
DefaultManagedAwareThreadFactory-10======结束=====>>>>296657
DefaultManagedAwareThreadFactory-4======结束=====>>>>301761
DefaultManagedAwareThreadFactory-5======结束=====>>>>302579
DefaultManagedAwareThreadFactory-8======结束=====>>>>301438
DefaultManagedAwareThreadFactory-1======结束=====>>>>311957
DefaultManagedAwareThreadFactory-9======结束=====>>>>304187
基础数据2000万,插入1000万条数据日志:
===================>>>>>DefaultManagedAwareThreadFactory-1
===================>>>>>DefaultManagedAwareThreadFactory-2
===================>>>>>DefaultManagedAwareThreadFactory-3
===================>>>>>DefaultManagedAwareThreadFactory-4
===================>>>>>DefaultManagedAwareThreadFactory-5
===================>>>>>DefaultManagedAwareThreadFactory-6
===================>>>>>DefaultManagedAwareThreadFactory-7
===================>>>>>DefaultManagedAwareThreadFactory-8
===================>>>>>DefaultManagedAwareThreadFactory-9
===================>>>>>DefaultManagedAwareThreadFactory-10
DefaultManagedAwareThreadFactory-8======结束=====>>>>276787
DefaultManagedAwareThreadFactory-3======结束=====>>>>284162
DefaultManagedAwareThreadFactory-4======结束=====>>>>284252
DefaultManagedAwareThreadFactory-2======结束=====>>>>291498
DefaultManagedAwareThreadFactory-1======结束=====>>>>297639
DefaultManagedAwareThreadFactory-7======结束=====>>>>292803
DefaultManagedAwareThreadFactory-5======结束=====>>>>297715
DefaultManagedAwareThreadFactory-9======结束=====>>>>297572
DefaultManagedAwareThreadFactory-10======结束=====>>>>296322
DefaultManagedAwareThreadFactory-6======结束=====>>>>302545
基础数据3000万,插入1000万条数据日志:
===================>>>>>DefaultManagedAwareThreadFactory-1
===================>>>>>DefaultManagedAwareThreadFactory-2
===================>>>>>DefaultManagedAwareThreadFactory-3
===================>>>>>DefaultManagedAwareThreadFactory-4
===================>>>>>DefaultManagedAwareThreadFactory-5
===================>>>>>DefaultManagedAwareThreadFactory-6
===================>>>>>DefaultManagedAwareThreadFactory-7
===================>>>>>DefaultManagedAwareThreadFactory-8
===================>>>>>DefaultManagedAwareThreadFactory-9
===================>>>>>DefaultManagedAwareThreadFactory-10
DefaultManagedAwareThreadFactory-6======结束=====>>>>259247
DefaultManagedAwareThreadFactory-2======结束=====>>>>264036
DefaultManagedAwareThreadFactory-3======结束=====>>>>265275
DefaultManagedAwareThreadFactory-7======结束=====>>>>264781
DefaultManagedAwareThreadFactory-10======结束=====>>>>265781
DefaultManagedAwareThreadFactory-9======结束=====>>>>271145
DefaultManagedAwareThreadFactory-5======结束=====>>>>281170
DefaultManagedAwareThreadFactory-1======结束=====>>>>286377
DefaultManagedAwareThreadFactory-4======结束=====>>>>283481
DefaultManagedAwareThreadFactory-8======结束=====>>>>279556
模拟数据测试:
模拟随机产生手机号码:
/*** 返回手机号码*/private static String[] telFirst = "134,135,136,137,138,139,150,151,152,157,158,159,130,131,132,155,156,133,153".split(",");public static int getNum(int start, int end) {return (int) (Math.random() * (end - start + 1) + start);}public static String getTel() {int index = getNum(0, telFirst.length - 1);String first = telFirst[index];String second = String.valueOf(getNum(1, 888) + 10000).substring(1);String third = String.valueOf(getNum(1, 9100) + 10000).substring(1);return first + second + third;}
模拟随机产生出生日期:
/*** 随机出生日期** @return*/public static String randomBirthday() {Calendar birthday = Calendar.getInstance();birthday.set(Calendar.YEAR, (int) (Math.random() * 60) + 1950);birthday.set(Calendar.MONTH, (int) (Math.random() * 12));birthday.set(Calendar.DATE, (int) (Math.random() * 31));StringBuilder builder = new StringBuilder();builder.append(birthday.get(Calendar.YEAR));long month = birthday.get(Calendar.MONTH) + 1;if (month < 10) {builder.append("0");}builder.append(month);long date = birthday.get(Calendar.DATE);if (date < 10) {builder.append("0");}builder.append(date);return builder.toString();}
模拟产生姓名:
private static final String[] Surname= {"赵","钱","孙","李","周","吴","郑","王","冯","陈","褚","卫","蒋","沈","韩","杨","朱","秦","尤","许","何","吕","施","张","孔","曹","严","华","金","魏","陶","姜","戚","谢","邹","喻","柏","水","窦","章","云","苏","潘","葛","奚","范","彭","郎","鲁","韦","昌","马","苗","凤","花","方","俞","任","袁","柳","酆","鲍","史","唐","费","廉","岑","薛","雷","贺","倪","汤","滕","殷","罗","毕","郝","邬","安","常","乐","于","时","傅","皮","卞","齐","康","伍","余","元","卜","顾","孟","平","黄","和","穆","萧","尹","姚","邵","湛","汪","祁","毛","禹","狄","米","贝","明","臧","计","伏","成","戴","谈","宋","茅","庞","熊","纪","舒","屈","项","祝","董","梁","杜","阮","蓝","闵","席","季","麻","强","贾","路","娄","危","江","童","颜","郭","梅","盛","林","刁","钟","徐","邱","骆","高","夏","蔡","田","樊","胡","凌","霍","虞","万","支","柯","昝","管","卢","莫","经","房","裘","缪","干","解","应","宗","丁","宣","贲","邓","郁","单","杭","洪","包","诸","左","石","崔","吉","钮","龚","程","嵇","邢","滑","裴","陆","荣","翁","荀","羊","于","惠","甄","曲","家","封","芮","羿","储","靳","汲","邴","糜","松","井","段","富","巫","乌","焦","巴","弓","牧","隗","山","谷","车","侯","宓","蓬","全","郗","班","仰","秋","仲","伊","宫","宁","仇","栾","暴","甘","钭","厉","戎","祖","武","符","刘","景","詹","束","龙","叶","幸","司","韶","郜","黎","蓟","溥","印","宿","白","怀","蒲","邰","从","鄂","索","咸","籍","赖","卓","蔺","屠","蒙","池","乔","阴","郁","胥","能","苍","双","闻","莘","党","翟","谭","贡","劳","逄","姬","申","扶","堵","冉","宰","郦","雍","却","璩","桑","桂","濮","牛","寿","通","边","扈","燕","冀","浦","尚","农","温","别","庄","晏","柴","瞿","阎","充","慕","连","茹","习","宦","艾","鱼","容","向","古","易","慎","戈","廖","庾","终","暨","居","衡","步","都","耿","满","弘","匡","国","文","寇","广","禄","阙","东","欧","殳","沃","利","蔚","越","夔","隆","师","巩","厍","聂","晁","勾","敖","融","冷","訾","辛","阚","那","简","饶","空","曾","毋","沙","乜","养","鞠","须","丰","巢","关","蒯","相","查","后","荆","红","游","郏","竺","权","逯","盖","益","桓","公","仉","督","岳","帅","缑","亢","况","郈","有","琴","归","海","晋","楚","闫","法","汝","鄢","涂","钦","商","牟","佘","佴","伯","赏","墨","哈","谯","篁","年","爱","阳","佟","言","福","南","火","铁","迟","漆","官","冼","真","展","繁","檀","祭","密","敬","揭","舜","楼","疏","冒","浑","挚","胶","随","高","皋","原","种","练","弥","仓","眭","蹇","覃","阿","门","恽","来","綦","召","仪","风","介","巨","木","京","狐","郇","虎","枚","抗","达","杞","苌","折","麦","庆","过","竹","端","鲜","皇","亓","老","是","秘","畅","邝","还","宾","闾","辜","纵","侴","万俟","司马","上官","欧阳","夏侯","诸葛","闻人","东方","赫连","皇甫","羊舌","尉迟","公羊","澹台","公冶","宗正","濮阳","淳于","单于","太叔","申屠","公孙","仲孙","轩辕","令狐","钟离","宇文","长孙","慕容","鲜于","闾丘","司徒","司空","兀官","司寇","南门","呼延","子车","颛孙","端木","巫马","公西","漆雕","车正","壤驷","公良","拓跋","夹谷","宰父","谷梁","段干","百里","东郭","微生","梁丘","左丘","东门","西门","南宫","第五","公仪","公乘","太史","仲长","叔孙","屈突","尔朱","东乡","相里","胡母","司城","张廖","雍门","毋丘","贺兰","綦毋","屋庐","独孤","南郭","北宫","王孙"};public static String getChineseName() {String str = null;String name = null;int highPos, lowPos;Random random = new Random();//区码,0xA0打头,从第16区开始,即0xB0=11*16=176,16~55一级汉字,56~87二级汉字highPos = (176 + Math.abs(random.nextInt(72)));random=new Random();//位码,0xA0打头,范围第1~94列lowPos = 161 + Math.abs(random.nextInt(94));byte[] bArr = new byte[2];bArr[0] = (new Integer(highPos)).byteValue();bArr[1] = (new Integer(lowPos)).byteValue();try {//区位码组合成汉字str = new String(bArr, "GB2312");int index=random.nextInt(Surname.length-1);//获得一个随机的姓氏name = Surname[index] +str;} catch (UnsupportedEncodingException e) {e.printStackTrace();}return name;}
模拟产生创建时间:
public static Date getTime() {Random rand = new Random();Calendar cal = Calendar.getInstance();cal.set(1900, 0, 1);long start = cal.getTimeInMillis();cal.set(2020, 0, 1);long end = cal.getTimeInMillis();Date d = new Date(start + (long)(rand.nextDouble() * (end - start)));return d;}
优化点:
1:尽量使用MySQL自增ID,InnoDB引擎表是基于B+树的索引组织表,数据记录本身被存于主索引(一颗B+Tree)的叶子节点上。这就要求同一个叶子节点内(大小为一个内存页或磁盘页)的各条数据记录按主键顺序存放,因此每当有一条新的记录插入时,MySQL会根据其主键将其插入适当的节点和位置,如果页面达到装载因子(InnoDB默认为15/16),则开辟一个新的页(节点),如果表使用自增主键,那么每次插入新的记录,记录就会顺序添加到当前索引节点的后续位置,当一页写满,就会自动开辟一个新的页。如果使用非自增主键(如果身份证号或学号等),由于每次插入主键的值近似于随机,因此每次新纪录都要被插到现有索引页得中间某个位置,此时MySQL不得不为了将新记录插到合适位置而移动数据,甚至目标页面可能已经被回写到磁盘上而从缓存中清掉,此时又要从磁盘上读回来,这增加了很多开销,同时频繁的移动、分页操作造成了大量的碎片,得到了不够紧凑的索引结构,后续不得不通过OPTIMIZE TABLE来重建表并优化填充页面。
2:取消所有的索引,尤其是唯一索引。(同上)每当有一条新的记录插入时,MySQL会根据其插入适当的节点和位置,会导致移动数据,造成大量碎片
3:批量插入可以使SQL日志量(MySQL的binlog和innodb的事务让日志)减少了,降低日志刷盘的数据量和频率,从而提高效率。通过批量插入减少SQL语句解析的次数,减少网络传输的IO。
4:使用事务可以提高数据的插入效率,这是因为进行一个INSERT操作时,MySQL内部会建立一个事务,在事务内才进行真正插入处理操作。通过使用事务可以减少创建事务的消耗,所有插入都在执行后才进行提交操作。
5:多线程处理,这个就不要多说了。
MySQL千万级数据进行插入,基础数据3千万,插入1千万数据时间约为4.7分钟,10个线程同时插入相关推荐
- mysql千万级数据查询select、插入insert慢 可能原因总结
mysql千万级数据查询select.插入insert慢 可能原因总结 表连表查询 并 insert ,insert into -select -from- insert into T1(XX,XX) ...
- mysql百万数据join_MySQL百万级、千万级数据多表关联SQL语句调优
本文不涉及复杂的底层数据结构,通过explain解释SQL,并根据可能出现的情况,来做具体的优化,使百万级.千万级数据表关联查询第一页结果能在2秒内完成(真实业务告警系统优化结果). 希望读者能够理解 ...
- 千万级别数据查询优化_从千万级数据查询来聊一聊索引结构和数据库原理
在日常工作中我们不可避免地会遇到慢SQL问题,比如笔者在之前的公司时会定期收到DBA彪哥发来的Oracle AWR报告,并特别提示我某条sql近阶段执行明显很慢,可能要优化一下等.对于这样的问题通常大 ...
- MySQL百万级、千万级数据多表关联SQL语句调优
本文不涉及复杂的底层数据结构,通过explain解释SQL,并根据可能出现的情况,来做具体的优化,使百万级.千万级数据表关联查询第一页结果能在2秒内完成(真实业务告警系统优化结果). 希望读者能够理解 ...
- mysql面试关联查询语句_MySQL百万级、千万级数据多表关联SQL语句调优
作者:成金之路 www.cnblogs.com/uttu/p/6384541.html 本文不涉及复杂的底层数据结构,通过explain解释SQL,并根据可能出现的情况,来做具体的优化,使百万级.千万 ...
- MySQL百万级/千万级数据存储解决方案
MySQL 百万级/千万级数据存储解决方案 百万级.千万级数据处理,个人认为核心关键在于数据存储方案设计,存储方案设计的是否合理,直接影响到数据CRUD操作.总体设计可以考虑一下三个方面进行设计考虑: ...
- 数据库面试要点:关于MySQL数据库千万级数据查询和存储
摘要:百万级.千万级数据处理,核心关键在于数据存储方案设计,存储方案设计的是否合理,直接影响到数据CRUD操作.总体设计可以考虑一下几个方面进行设计考虑: 数据存储结构设计:索引设计:数据主键设计:查 ...
- MySQL千万级数据量优化方案
前言 千万级大表如何优化,这是一个很有技术含量的问题,通常我们的直觉思维都会跳转到拆分或者数据分区.除此之外,还有其他的思路和解决方案.根据本人多年的工作经验,做了如下总结. 方案 "千万级 ...
- MySQL 亿级数据导入导出/数据迁移笔记
最近MySQL的笔记有点多了,主要是公司Oracle比较稳定维护较少,上周被安排做了一个MySQL亿级数据的迁移,趁此记录下学习笔记: 数据迁移,工作原理和技术支持数据导出.BI报表之类的相似,差异较 ...
最新文章
- L2级自动驾驶量产趋势解读
- 自定义Realm实现认证
- helloworld设置成开机自启动的服务
- 多个项目怎么配置到服务器上,多个项目怎么配置到服务器
- java中怎么进行字符串替换?
- First、FirstOrDefault、Single、SingleOrDefault 的区别
- 计算机组成原理201501,计算机组成原理201501.pdf
- Vue系列vue-router的项目打包(三)
- 使用Regsvr32命令修复系统故障
- 中国电信:张志勇辞任公司执行副总裁
- Spring+Dubbo集成Redis的两种解决方案
- GDAL源码剖析(七)之GDAL RasterIO使用说明
- IAPP视频教程全集
- book mac pro怎么重装系统_MacBook Pro怎么重装系统 MacBook Pro重装系统教程
- wordpress博客 qq客服插件
- c语言输入r1 r2垫片的面积,C语言上机实验答案Word版
- Pytest如何并发执行自动化脚本
- 【原】Linux shell script 21是什么意思
- Java开发技巧!java语言程序设计第四版答案
- 服务器错误500 501 502 503 504 505 详解
热门文章
- Excel中单元格中空格的清除方法
- Business Model Generation《商业模式新生代》对软件测试的启发
- 毫米波雷达视觉融合方案综述(数据级/决策级/特征级融合)
- Unity与IOS内购恢复问题
- SSM + MySQL 服装商城
- pyecharts1.0+版本绘制Pie饼图
- delphi安装dpk控件
- html版本绩拼音怎么写,绩的读音_绩的拼音_绩的注音_绩怎么读-98在线字典
- 真三国无双战服务器维护,真三国无双霸怎么进不去
- 高速公路视联网解决方案来了!全面助力行业智能运行监测升级