性能优化之Java多线程批量拆分List导入数据库
| 前言
前两天做了一个导入的功能,导入开始的时候非常慢,导入2w条数据要1分多钟,后来一点一点的优化,从直接把list怼进Mysql中,到分配把list导入Mysql中,到多线程把list导入Mysql中。时间是一点一点的变少了。非常的爽,最后变成了10s以内。下面就展示一下过程。
| 直接把list怼进Mysql
使用mybatis的批量导入操作:
@Transactional(rollbackFor = Exception.class)public int addFreshStudentsNew2(List<FreshStudentAndStudentModel> list, String schoolNo) {if (list == null || list.isEmpty()) {return 0;}List<StudentEntity> studentEntityList = new LinkedList<>();List<EnrollStudentEntity> enrollStudentEntityList = new LinkedList<>();List<AllusersEntity> allusersEntityList = new LinkedList<>();for (FreshStudentAndStudentModel freshStudentAndStudentModel : list) {EnrollStudentEntity enrollStudentEntity = new EnrollStudentEntity();StudentEntity studentEntity = new StudentEntity();BeanUtils.copyProperties(freshStudentAndStudentModel, studentEntity);BeanUtils.copyProperties(freshStudentAndStudentModel, enrollStudentEntity);String operator = TenancyContext.UserID.get();String studentId = BaseUuidUtils.base58Uuid();enrollStudentEntity.setId(BaseUuidUtils.base58Uuid());enrollStudentEntity.setStudentId(studentId);enrollStudentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard());enrollStudentEntity.setOperator(operator);studentEntity.setId(studentId);studentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard());studentEntity.setOperator(operator);studentEntityList.add(studentEntity);enrollStudentEntityList.add(enrollStudentEntity);AllusersEntity allusersEntity = new AllusersEntity();allusersEntity.setId(enrollStudentEntity.getId());allusersEntity.setUserCode(enrollStudentEntity.getNemtCode());allusersEntity.setUserName(enrollStudentEntity.getName());allusersEntity.setSchoolNo(schoolNo);allusersEntity.setTelNum(enrollStudentEntity.getTelNum());allusersEntity.setPassword(enrollStudentEntity.getNemtCode()); //密码设置为考生号allusersEntityList.add(allusersEntity);}enResult = enrollStudentDao.insertAll(enrollStudentEntityList);stuResult = studentDao.insertAll(studentEntityList);allResult = allusersFacade.insertUserList(allusersEntityList);if (enResult > 0 && stuResult > 0 && allResult) {return 10;}return -10;}
Mapper.xml
<insert id="insertAll" parameterType="com.dmsdbj.itoo.basicInfo.entity.EnrollStudentEntity">insert into tb_enroll_student<trim prefix="(" suffix=")" suffixOverrides=",">id, remark, nEMT_aspiration, nEMT_code, nEMT_score, student_id, identity_card_id, level, major, name, nation, secondary_college, operator, sex, is_delete, account_address, native_place, original_place, used_name, pictrue, join_party_date, political_status, tel_num, is_registry, graduate_school, create_time, update_time </trim> values<foreach collection="list" item="item" index="index" separator=",">(#{item.id,jdbcType=VARCHAR},#{item.remark,jdbcType=VARCHAR},#{item.nemtAspiration,jdbcType=VARCHAR},#{item.nemtCode,jdbcType=VARCHAR},#{item.nemtScore,jdbcType=VARCHAR},#{item.studentId,jdbcType=VARCHAR},#{item.identityCardId,jdbcType=VARCHAR},#{item.level,jdbcType=VARCHAR},#{item.major,jdbcType=VARCHAR},#{item.name,jdbcType=VARCHAR},#{item.nation,jdbcType=VARCHAR},#{item.secondaryCollege,jdbcType=VARCHAR},#{item.operator,jdbcType=VARCHAR},#{item.sex,jdbcType=VARCHAR},0,#{item.accountAddress,jdbcType=VARCHAR},#{item.nativePlace,jdbcType=VARCHAR},#{item.originalPlace,jdbcType=VARCHAR},#{item.usedName,jdbcType=VARCHAR},#{item.pictrue,jdbcType=VARCHAR},#{item.joinPartyDate,jdbcType=VARCHAR},#{item.politicalStatus,jdbcType=VARCHAR},#{item.telNum,jdbcType=VARCHAR},#{item.isRegistry,jdbcType=TINYINT},#{item.graduateSchool,jdbcType=VARCHAR},now(),now() ) </foreach> </insert>
代码说明:
底层的mapper是通过逆向工程来生成的,批量插入如下,是拼接成类似:insert into tb_enroll_student()values (),()…….();
这样的缺点是,数据库一般有一个默认的设置,就是每次sql操作的数据不能超过4M。这样插入,数据多的时候,数据库会报错Packet for query is too large (6071393 > 4194304). You can change this value on the server by setting the max_allowed_packet' variable.,
虽然我们可以通过
类似 修改 my.ini 加上 max_allowed_packet =67108864
,67108864=64M
,默认大小4194304 也就是4M
修改完成之后要重启mysql服务,如果通过命令行修改就不用重启mysql服务。
完成本次操作,但是我们不能保证项目单次最大的大小是多少,这样是有弊端的。所以可以考虑进行分组导入。
| 分组把list导入Mysql中
同样适用mybatis批量插入,区别是对每次的导入进行分组计算,然后分多次进行导入:
@Transactional(rollbackFor = Exception.class)public int addFreshStudentsNew2(List<FreshStudentAndStudentModel> list, String schoolNo) {if (list == null || list.isEmpty()) {return 0;}List<StudentEntity> studentEntityList = new LinkedList<>();List<EnrollStudentEntity> enrollStudentEntityList = new LinkedList<>();List<AllusersEntity> allusersEntityList = new LinkedList<>();for (FreshStudentAndStudentModel freshStudentAndStudentModel : list) {EnrollStudentEntity enrollStudentEntity = new EnrollStudentEntity();StudentEntity studentEntity = new StudentEntity();BeanUtils.copyProperties(freshStudentAndStudentModel, studentEntity);BeanUtils.copyProperties(freshStudentAndStudentModel, enrollStudentEntity);String operator = TenancyContext.UserID.get();String studentId = BaseUuidUtils.base58Uuid();enrollStudentEntity.setId(BaseUuidUtils.base58Uuid());enrollStudentEntity.setStudentId(studentId);enrollStudentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard());enrollStudentEntity.setOperator(operator);studentEntity.setId(studentId);studentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard());studentEntity.setOperator(operator);studentEntityList.add(studentEntity);enrollStudentEntityList.add(enrollStudentEntity);AllusersEntity allusersEntity = new AllusersEntity();allusersEntity.setId(enrollStudentEntity.getId());allusersEntity.setUserCode(enrollStudentEntity.getNemtCode());allusersEntity.setUserName(enrollStudentEntity.getName());allusersEntity.setSchoolNo(schoolNo);allusersEntity.setTelNum(enrollStudentEntity.getTelNum());allusersEntity.setPassword(enrollStudentEntity.getNemtCode()); //密码设置为考生号allusersEntityList.add(allusersEntity);}int c = 100;int b = enrollStudentEntityList.size() / c;int d = enrollStudentEntityList.size() % c;int enResult = 0;int stuResult = 0;boolean allResult = false;for (int e = c; e <= c * b; e = e + c) {enResult = enrollStudentDao.insertAll(enrollStudentEntityList.subList(e - c, e));stuResult = studentDao.insertAll(studentEntityList.subList(e - c, e));allResult = allusersFacade.insertUserList(allusersEntityList.subList(e - c, e));}if (d != 0) {enResult = enrollStudentDao.insertAll(enrollStudentEntityList.subList(c * b, enrollStudentEntityList.size()));stuResult = studentDao.insertAll(studentEntityList.subList(c * b, studentEntityList.size()));allResult = allusersFacade.insertUserList(allusersEntityList.subList(c * b, allusersEntityList.size()));}if (enResult > 0 && stuResult > 0 && allResult) {return 10;}return -10;}
代码说明:
这样操作,可以避免上面的错误,但是分多次插入,无形中就增加了操作实践,很容易超时。所以这种方法还是不值得提倡的。
再次改进,使用多线程分批导入。
| 多线程分批导入Mysql
依然使用mybatis的批量导入,不同的是,根据线程数目进行分组,然后再建立多线程池,进行导入。
@Transactional(rollbackFor = Exception.class)public int addFreshStudentsNew(List<FreshStudentAndStudentModel> list, String schoolNo) {if (list == null || list.isEmpty()) {return 0;}List<StudentEntity> studentEntityList = new LinkedList<>();List<EnrollStudentEntity> enrollStudentEntityList = new LinkedList<>();List<AllusersEntity> allusersEntityList = new LinkedList<>();list.forEach(freshStudentAndStudentModel -> {EnrollStudentEntity enrollStudentEntity = new EnrollStudentEntity();StudentEntity studentEntity = new StudentEntity();BeanUtils.copyProperties(freshStudentAndStudentModel, studentEntity);BeanUtils.copyProperties(freshStudentAndStudentModel, enrollStudentEntity);String operator = TenancyContext.UserID.get();String studentId = BaseUuidUtils.base58Uuid();enrollStudentEntity.setId(BaseUuidUtils.base58Uuid());enrollStudentEntity.setStudentId(studentId);enrollStudentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard());enrollStudentEntity.setOperator(operator);studentEntity.setId(studentId);studentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard());studentEntity.setOperator(operator);studentEntityList.add(studentEntity);enrollStudentEntityList.add(enrollStudentEntity);AllusersEntity allusersEntity = new AllusersEntity();allusersEntity.setId(enrollStudentEntity.getId());allusersEntity.setUserCode(enrollStudentEntity.getNemtCode());allusersEntity.setUserName(enrollStudentEntity.getName());allusersEntity.setSchoolNo(schoolNo);allusersEntity.setTelNum(enrollStudentEntity.getTelNum());allusersEntity.setPassword(enrollStudentEntity.getNemtCode()); //密码设置为考生号allusersEntityList.add(allusersEntity);});int nThreads = 50;int size = enrollStudentEntityList.size();ExecutorService executorService = Executors.newFixedThreadPool(nThreads);List<Future<Integer>> futures = new ArrayList<Future<Integer>>(nThreads);for (int i = 0; i < nThreads; i++) {final List<EnrollStudentEntity> EnrollStudentEntityImputList = enrollStudentEntityList.subList(size / nThreads * i, size / nThreads * (i + 1));final List<StudentEntity> studentEntityImportList = studentEntityList.subList(size / nThreads * i, size / nThreads * (i + 1));final List<AllusersEntity> allusersEntityImportList = allusersEntityList.subList(size / nThreads * i, size / nThreads * (i + 1));Callable<Integer> task1 = () -> {studentSave.saveStudent(EnrollStudentEntityImputList,studentEntityImportList,allusersEntityImportList);return 1;};futures.add(executorService.submit(task1));}executorService.shutdown();if (!futures.isEmpty() && futures != null) {return 10;}return -10;}
代码说明:
上面是通过应用ExecutorService 建立了固定的线程数,然后根据线程数目进行分组,批量依次导入。一方面可以缓解数据库的压力,另一个面线程数目多了,一定程度会提高程序运行的时间。缺点就是要看服务器的配置,如果配置好的话就可以开多点线程,配置差的话就开小点。
| 小结
通过使用这个操作真是不断的提高了,项目使用技巧也是不错。加油~~ 多线程哦~~
来源:https://blog.csdn.net/kisscatforever
/article/details/79817039
性能优化之Java多线程批量拆分List导入数据库相关推荐
- 项目案例:Java多线程批量拆分List导入数据库
欢迎关注方志朋的博客,回复"666"获面试宝典 来源:blog.csdn.net/kisscatforever/ article/details/79817039 一.前言 二.直 ...
- 多线程批量拆分List导入数据库
欢迎关注方志朋的博客,回复"666"获面试宝典 来源:https://blog.csdn.net/kisscatforever/article/details/79817039 一 ...
- 性能优化之Java(Android)代码优化
最新最准确内容建议直接访问原文:性能优化之Java(Android)代码优化 本文为Android性能优化的第三篇--Java(Android)代码优化.主要介绍Java代码中性能优化方式及网络优化, ...
- java多线程批量读取文件(七)
新公司入职一个多月了,至今没有事情可以做,十来个新同事都一样抓狂,所以大家都自己学习一些新东西,我最近在看zookeeper,感觉蛮不错的,和微服务的zuul以及eureka功能类似,只是代码复杂了一 ...
- IDEA最全最常用的配置与性能优化(Java必备)
IDEA最全最常用的配置与性能优化(Java必备) 简介 一.性能优化 1.JVM启动参数 2.清空缓存并重建索引 二.优化设置 1.显示方法分隔符 2.忽略大小写提示 3.主题设置 4.设置字体 5 ...
- java多线程批量处理
前言 高并发,几乎是每个程序员都想拥有的经验.原因很简单:随着流量变大,会遇到各种各样的技术问题,比如接口响应超时.CPU load升高.GC频繁.死锁.大数据量存储等等,这些问题能推动我们在技术深度 ...
- 用java将excel表单导入数据库表单----新手入门
构建项目思路 1.利用Excel第三方工具,将Excel文件读取到内存中.使用最简单,方便的工具是apache的poi工具包,自己网上下载 http://poi.apache.org/ ,使用方法网上 ...
- 高时空损耗的Scanner会卡爆程序(记洛谷P1567的Java性能优化,Java语言描述)
写在前面 对性能调优,其实我一个弱鸡,用的也不多,特别是这种OJ连JVM调优都不成. 大佬s勿喷,且看小菜鸡如何在一道OJ题里与Java性能搏斗! 题目要求 P1567题目链接 简单分析 10^9,没 ...
- 虹软AI 人脸识别SDK接入 — 性能优化篇(多线程)
大家都嫌公司以前使用的刷卡门禁太麻烦,正好借这个机会开发一个人脸识别的门禁系统,采用的SDK是虹软公司开发的,接口调用比较简单. 一.虹软SDK接口性能 在配置为i5-7400 .16G内存的PC上测 ...
最新文章
- Spring Cloud应用开发(一:使用Eureka注册服务)
- 怎样熟练使用一项技术
- 算法-- 找到所有数组中消失的数字(Java)
- session、flask session知识的相关收集
- 使用Skywalking实现全链路监
- POJ 2411 Mondriaan's Dream(状态压缩DP)
- IDEA Java Web 推送Tomcat
- 用于角点检测的FAST算法
- jQuery 事件方法(交互)
- java ee 类切换_eclipse的工程类型切换
- JDK8高性能队列“Disruptor“
- 数据结构代码学习笔记(持续更新中)
- Keil 5模块化编程详细步骤
- Ubuntu安装cuda
- 二十三种设计模式之工厂模式(含Java工厂模式的实现)
- 数据质量监控工具-Apache Griffin
- six MySQL 主主
- oracle wallet java_Oracle Wallet初探
- graphpad画生存曲线怎么样去掉删失点_Graphpad Prism 绘制散点图
- Tuxera Disk Manager是什么软件,Tuxera Disk Manager怎么用
热门文章
- 信息系统监理师题库_信息系统监理题库
- 用 rpm-ostree 数据库检查更新信息和更新日志的方法
- mysql链事务_MYSQL 之事务篇
- 回车的ascii码_ASCII码表
- P4113 [HEOI2012]采花 树状数组离线
- 图论 ---- CF1209F. Koala and Notebook(多位数字拆边+BFS)
- c语言断链隐藏dll,通过断链隐藏模块(DLL)
- 模板 - 最小斯坦纳树
- 《PLACEBO》(安慰剂)米津玄師/野田洋次郎 (罗马音、歌词、汉译)
- v3 微信api 请求微信_企业微信API使用基本教程