SpringBoot JPA 批量插入实现，使用原生sql解决SaveAll插入慢的问题

1.项目中遇到了批量插入的操作，但是使用saveAll，数据量5000条而已，速度感人，用户体验贼差，哈哈，现在将解决方法记录一下。

2.创建springboot项目

3.引入pom依赖

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><parent><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-parent</artifactId><version>2.0.6.RELEASE</version><relativePath/> <!-- lookup parent from repository --></parent><groupId>com.frank</groupId><artifactId>jpa</artifactId><version>0.0.1-SNAPSHOT</version><name>spingboot-jpa</name><description>Demo project for Spring Boot</description><properties><java.version>1.8</java.version></properties><dependencies><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-data-jdbc</artifactId></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-data-jpa</artifactId></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-web</artifactId></dependency><dependency><groupId>mysql</groupId><artifactId>mysql-connector-java</artifactId><version>5.1.48</version></dependency><dependency><groupId>org.projectlombok</groupId><artifactId>lombok</artifactId><optional>true</optional></dependency><dependency><groupId>com.alibaba</groupId><artifactId>fastjson</artifactId><version>1.2.47</version></dependency></dependencies><build><plugins><plugin><groupId>org.springframework.boot</groupId><artifactId>spring-boot-maven-plugin</artifactId></plugin></plugins></build></project>

4.application.yml

spring:datasource:driver-class-name: com.mysql.jdbc.Driverurl: jdbc:mysql://127.0.0.1:3306/test?serverTimezone=UTC&useUnicode=true&characterEncoding=utf-8&useSSL=trueusername: rootpassword: 123456jpa:properties:hibernate:hbm2ddl:auto: updatedialect: org.hibernate.dialect.MySQL5InnoDBDialectformat_sql: true# 开启批量插入jdbc:batch_size: 500batch_versioned_data: trueorder_inserts: trueorder_updates: trueshow-sql: true

5.实体类

package com.frank.jpa.entity;import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.Getter;
import lombok.NoArgsConstructor;import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.Id;
import java.io.Serializable;/*** @author 小石潭记* @date 2020/10/4 14:14* @Description: ${todo}*/
@Entity
@Data
public class User implements Serializable {@Id@GeneratedValueprivate Long id;@Column(nullable = false, unique = true)private String userName;@Column(nullable = false)private String passWord;@Column(nullable = false, unique = true)private String email;@Column(nullable = true, unique = true)private String nickName;@Column(nullable = false)private String regTime;public User(String userName, String passWord, String email, String nickName, String regTime) {this.userName = userName;this.passWord = passWord;this.email = email;this.nickName = nickName;this.regTime = regTime;}public User() {}}

6.repository

package com.frank.jpa.repository;import com.frank.jpa.entity.User;
import org.springframework.data.jpa.repository.Modifying;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.PagingAndSortingRepository;
import org.springframework.data.repository.query.Param;
import org.springframework.stereotype.Repository;
import org.springframework.transaction.annotation.Transactional;import java.util.List;/*** @author 小石潭记* @date 2020/10/4 14:15* @Description: ${todo}* 1. 使用jpa的 CrudRepository 基本查询* 2. 使用jpa的 PagingAndSortingRepository 分页查询和排序* 3. 使用jpa的 Repository 自定义声明式查询方法*public interface PersonQueryRepo extends Repository<Person, Long> {// declare query method// 声明式查询方法// 1. count 计数long countByName(String name);// 2. get/find/stream/query/read 查询Person readFirstByAge(int age);// 3. delete/remove 删除@Transactionalint deleteById(long id);}* 4. 使用jpa的 JpaRepository 使用hql、jpql或sql查询，@Query等注解public interface PersonHqlDao extends JpaRepository<Person, Long> {// 使用hql 或者 jpql 查询@Query("from Person where name = ?1 order by id desc")List<Person> listByName(String name);// 前几种方法中均未介绍update操作，要完成update操作，可使用以下方法// 更新时需要加上 @Transactional 和 @Modifying@Transactional@Modifying // QueryExecutionRequestException: Not supported for DML operations@Query("update Person set name=?2 where id=?1")int updateNameById(long id, String name);}*/
@Repository
public interface UserRepository extends PagingAndSortingRepository<User, Long> {List<User> findByUserNameOrEmail(@Param("userName")String userName, @Param("email")String email);User findByUserName(@Param("userName") String userName);@Transactional@Modifying // QueryExecutionRequestException: Not supported for DML operations@Query("update User set user_name=?2 where id=?1")int updateUserNameById(long id, @Param("userName")String userName);
}

7.service

package com.frank.jpa.service;import com.alibaba.fastjson.JSON;
import com.frank.jpa.entity.User;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.util.ObjectUtils;import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;
import java.util.List;/*** @author 小石潭记* @date 2020/10/4 17:03* @Description: ${todo}*/
@Service
@Transactional
@Slf4j
public class UserService {@PersistenceContextprivate EntityManager entityManager;@Autowiredprivate JdbcTemplate jdbcTemplate;/*** 批量插入** @param list 实体类集合* @param <T>  表对应的实体类*/public <T> void batchInsert(List<T> list) {if (!ObjectUtils.isEmpty(list)) {for (int i = 0; i < list.size(); i++) {entityManager.persist(list.get(i));if (i % 50 == 0) {entityManager.flush();entityManager.clear();}}entityManager.flush();entityManager.clear();}}/*** 批量更新** @param list 实体类集合* @param <T>  表对应的实体类*/public <T> void batchUpdate(List<T> list) {if (!ObjectUtils.isEmpty(list)) {for (int i = 0; i < list.size(); i++) {entityManager.merge(list.get(i));if (i % 50 == 0) {entityManager.flush();entityManager.clear();}}entityManager.flush();entityManager.clear();}}public void saveBatch(List<User> list) {/*String sql="insert into user " +"(user_name)" +" values (?)";List<Object[]> batchArgs=new ArrayList<Object[]>();for (int i = 0; i < list.size(); i++) {batchArgs.add(new Object[]{list.get(i)});}jdbcTemplate.batchUpdate(sql, batchArgs);*/StringBuilder insert = new StringBuilder("INSERT INTO `user` (`user_name`, `pass_word`, `nick_name`," +"`email`,`reg_time`) VALUES ");for (int i = 0; i < list.size(); i++) {insert.append("(").append("'").append(list.get(i).getUserName()).append("'").append(",").append("'").append(list.get(i).getPassWord()).append("'").append(",").append("'").append(list.get(i).getNickName()).append("'").append(",").append("'").append(list.get(i).getEmail()).append("'").append(",").append("'").append(list.get(i).getRegTime()).append("'").append(")");if (i < list.size() - 1) {insert.append(",");}}String sql = (String) JSON.toJSON(insert);log.info("SQL语句:{}", JSON.toJSON(insert));try {jdbcTemplate.execute(sql);} catch (Exception e) {log.error("sql解析错误", e.getMessage());}}
}

8.控制器

package com.frank.jpa.web;import com.frank.jpa.entity.User;
import com.frank.jpa.repository.UserRepository;
import com.frank.jpa.service.UserService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;/*** @author 小石潭记* @date 2020/10/4 14:25* @Description: ${todo}*/
@RestController
@Slf4j
public class UserController {@Autowiredprivate UserRepository repository;@Autowiredprivate UserService service;@GetMapping("/save")public void save(){List<User> list = new ArrayList<>();for (int i = 0; i < 500; i++) {User user = new User();user.setUserName("小石潭记" + i);list.add(user);}long start = System.currentTimeMillis();log.info("开始保存", start);// 500 条数据花费9srepository.saveAll(list);long end = System.currentTimeMillis();log.info("耗时{}s", (end-start) / 1000);}@GetMapping("/saveAll")public void saveAll(){List<User> list = new ArrayList<>();for (int i = 0; i < 500; i++) {User user = new User();user.setUserName("frank" + i);list.add(user);}long start = System.currentTimeMillis();log.info("开始保存", start);// 500 条数据花费13s  yml开启批量操作9sservice.batchInsert(list);long end = System.currentTimeMillis();log.info("耗时{}s", (end-start) / 1000);}/***  使用原生的jdbcTemplate批量插入数据  速度比上面的两个都快*/@GetMapping("/saveBatch")public void saveBatch(){List<User> list = new ArrayList<>();// 耗时1792ms 速度很快了for (int i = 0; i < 5000; i++) {User user = new User();user.setUserName("frank" + i);user.setEmail(i + "qq.com");user.setNickName("小石潭记" + i);user.setPassWord("password" + i);list.add(user);}long start = System.currentTimeMillis();log.info("开始保存", start);service.saveBatch(list);long end = System.currentTimeMillis();log.info("耗时{}", end-start);}@GetMapping("/user-info")public List<User> getUserInfo(String userName, String email) {User user = repository.findByUserName(userName);log.info("查询的user,{}", user);List<User> byUserNameOrEmail = repository.findByUserNameOrEmail(userName, email);log.info("查询的byUserNameOrEmail,{}", byUserNameOrEmail);Iterable<User> iterable = repository.findAll();List<User> list = new ArrayList<>();iterable.forEach(single->{list.add(single);});return list;}@GetMapping("/update-user")public int updateUserName(String id, String userName) {long userId = Long.parseLong(id);return repository.updateUserNameById(userId, userName);}}

9.启动类

package com.frank.jpa;import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.domain.EntityScan;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;/*** @EnableJpaRepositories(basePackages = “xxx.xxx.xxx”) //扫描 @Repository 注解；* @ComponentScan(basePackages = “xxx.xxx.xxx”) //扫描 @Controller、@Service 注解；* @EntityScan(basePackages = “xxx.xxx.xxx”) //扫描 @Entity 注解；*/
@SpringBootApplication
public class SpingbootJpaApplication {public static void main(String[] args) {SpringApplication.run(SpingbootJpaApplication.class, args);}}

经过测试，使用saveAll500条数据花费9s时间，yml开启批量插入：500 条数据花费13s yml开启批量操作9s，使用jdbcTemplate插入数据，5000条数据花费1792ms，速度快了很多。

代码地址

SpringBoot JPA 批量插入实现，使用原生sql解决SaveAll插入慢的问题相关推荐

mysql怎样循环插入数据_你向 Mysql 数据库插入 100w 条数据用了多久？
多线程插入(单表) 多线程插入(多表) 预处理SQL 多值插入SQL 事务(N条提交一次) 多线程插入(单表) 问:为何对同一个表的插入多线程会比单线程快?同一时间对一个表的写操作不应该是独占的吗? ...
java 原生sql批量插入,Java对象集合转MySQL批量插入语句
Mybatis的批量插入确实很好用,但是当数据量特别大的时候可能一次批量插入2000条数据就死翘翘了.怎么办呢?当然是原生sql是最快的啦,10000条数据秒入无压力.下面是将对象集合拼接成批量插入M ...
jpa原生query_JPA执行原生SQL语句
前言作业的项目终于告一段落了暂时,这周继续进行日志系统的编写,只可惜这周开始练科三了,一开始是全天练车,导致每天写代码的时间减少了好多,后来时间进行了调整(早上四点半到七点半,晚上五点到七点多),也 ...
springboot jpa sql打印_SpringBoot集成Spring Data JPA以及读写分离
相关代码:github OSCchina JPA是什么 JPA(Java Persistence API)是Sun官方提出的Java持久化规范,它为Java开发人员提供了一种对象/关联映射工具来管理 ...
使用 JPQL 和原生 SQL 查询 JPA 实体（转）
使用 JPQL 和原生 SQL 查询 JPA 实体(转) 博客分类: EJB3.JPA及其相关 SQL JPA 企业应用数据结构 C# 作者:Yuli Vasiliev 了解如何利用 Java 持 ...
继解决Spring data jpa 批量插入重写saveAll()后遇到符号不兼容问题
问题描述问题: 之前为解决Spring data jpa 批量插入/删除(saveAll()/deleteAll())速度慢的问题重写了saveAll()方法,用自定义拼接sql的方法组装sql, ...
java sql sum函数的使用方法,Springboot jpa使用sum()函数返回结果如何接收
1.需求我的需求是统计域名以及域名出现的次数. 之前使用springboot jpa都是把数据库中的表跟实体类绑定,创建继承JpaRepository的接口.如下: @Repository publ ...
Spring JPA 开启原生sql打印
2019独角兽企业重金招聘Python工程师标准>>> 原生spring项目,使用配置文件(非Spring boot项目) <!-- 配置EntityManagerFactor ...
008_Spring Data JPA原生SQL
1. nativeQuery默认的是false, 表示不开启sql查询.表示是否对value中的语句做转义.使用原生sql, 需要开启它. 2. 原生sql中使用的是表名,这要和JPQL语句中使用的是 ...

SpringBoot JPA 批量插入实现，使用原生sql解决SaveAll插入慢的问题

SpringBoot JPA 批量插入实现，使用原生sql解决SaveAll插入慢的问题相关推荐

最新文章

热门文章