Springboot+Mybatis+通用Mapper多数据源实现数据同步
有个需求需要查询远程数据库,然后将远程数据库中的数据抓取到本地,远程数据库中的数据是每天都增加的,所以就需要写个程序自动实现实时抓取。
这里我用到的框架是Springboot2.0+Mybatis+Mapper,涉及的数据库有SqlServer,Oracle,Mysql
添加依赖
<dependencies><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-quartz</artifactId></dependency><dependency><groupId>org.quartz-scheduler</groupId><artifactId>quartz</artifactId><version>2.3.0</version></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-web</artifactId></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-jdbc</artifactId></dependency><dependency><groupId>org.mybatis.spring.boot</groupId><artifactId>mybatis-spring-boot-starter</artifactId><version>1.3.1</version></dependency><!--<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-data-jpa</artifactId></dependency>--><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-devtools</artifactId><scope>runtime</scope></dependency><dependency><groupId>org.projectlombok</groupId><artifactId>lombok</artifactId><optional>true</optional></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-test</artifactId><scope>test</scope></dependency><dependency><groupId>commons-lang</groupId><artifactId>commons-lang</artifactId><version>2.6</version></dependency><!-- sqlserver --><dependency><groupId>com.microsoft.sqlserver</groupId><artifactId>sqljdbc4</artifactId><version>4.4</version></dependency><!-- oracle --><dependency><groupId>com.oracle</groupId><artifactId>ojdbc14</artifactId><version>10.2.0.4.0</version></dependency><!-- 通用Maper --><dependency><groupId>tk.mybatis</groupId><artifactId>mapper-spring-boot-starter</artifactId><version>1.2.4</version></dependency><!--druid --><dependency><groupId>com.alibaba</groupId><artifactId>druid</artifactId><version>1.0.28</version></dependency><!-- log4j --><dependency><groupId>org.slf4j</groupId><artifactId>slf4j-api</artifactId><version>1.7.25</version></dependency>
如果没有log包,可能在配置文件中的DruidDataSource会出现错误提示,但实际上不会影响到程序。
application.yml项目配置
也可以用application.properties格式配置,但是感觉yml格式看起来比较直观.
##端口
server:port:8081
logging:level:cn.javabb.bootdemo.mapper: info
spring:remote-datasource:url: jdbc:sqlserver://66.66.66.66:1433;DatabaseName=dbNameusername: sapassword: 123#使用Druid的数据源type: com.alibaba.druid.pool.DruidDataSourcedriver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriverfilters: statmaxActive: 20initialSize: 1maxWait: 60000minIdle: 1timeBetweenEvictionRunsMillis: 60000minEvictableIdleTimeMillis: 300000validationQuery: select 'x'testWhileIdle: truetestOnBorrow: falsetestOnReturn: falsepoolPreparedStatements: truemaxOpenPreparedStatements: 20local-datasource:url: jdbc:sqlserver://77.77.77.77:1433;DatabaseName=dbNameusername: sapassword: 1234#使用Druid的数据源type: com.alibaba.druid.pool.DruidDataSourcedriver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriverfilters: statmaxActive: 20initialSize: 1maxWait: 60000minIdle: 1timeBetweenEvictionRunsMillis: 60000minEvictableIdleTimeMillis: 300000validationQuery: select 'x'testWhileIdle: truetestOnBorrow: falsetestOnReturn: falsepoolPreparedStatements: truemaxOpenPreparedStatements: 20oracle-datasource:url: jdbc:oracle:thin:@88.88.88.88:1521/sidusername: tigerpassword: tiger#使用Druid的数据源type: com.alibaba.druid.pool.DruidDataSourcedriver-class-name: oracle.jdbc.driver.OracleDriverfilters: statmaxActive: 20initialSize: 1maxWait: 60000minIdle: 1timeBetweenEvictionRunsMillis: 60000minEvictableIdleTimeMillis: 300000validationQuery: select 'x'testWhileIdle: truetestOnBorrow: falsetestOnReturn: falsepoolPreparedStatements: truemaxOpenPreparedStatements: 20
#Mybatis
mybatis:configuration:map-underscore-to-camel-case: truemapper-locations: classpath:mapper/*.xmltypeAliasesPackage: cn.javabb.**.entity
#Mapper
mapper:mappers:- cn.javabb.bootdemo.base.BaseMappernot-empty: falseidentity: sqlserverbefore: true
其中remote-datasource,oracle-datasource和local-datasource为两个不同的数据源,mybatis配置和通用mapper配置一样的,如果不需要用到通用mapper可以去掉配置。
application启动类:
@EnableAsync
@EnableScheduling
@ServletComponentScan
@SpringBootApplication(exclude = {DataSourceAutoConfiguration.class,DataSourceTransactionManagerAutoConfiguration.class, MybatisAutoConfiguration.class})
public class BootDemoApplication {public static void main(String[] args) {SpringApplication.run(BootDemoApplication.class, args);}}
注解说明:
@EnableAsync:开启异步任务支持,如果不需要可以去掉
@EnableScheduling:开启定时任务支持
@ServletComponentScan:用来扫描mapper
@SpringBootApplication:springboot注解支持
注意:如果项目依赖中有,mybatis-spring-boot-starter,会默认自动根据配置文件配置mybatis,如果刚开始项目没有配置DataSource,启动项目就会报错,在这里去掉DataSource的默认自动配置类不加载,使项目在使用的时候选择去DataSource。
数据源配置
有几个数据源配置几个DataSource
配置第一个数据源:local-datasource
import org.apache.ibatis.session.SqlSessionFactory;
import org.mybatis.spring.SqlSessionFactoryBean;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.autoconfigure.jdbc.DataSourceBuilder;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.support.PathMatchingResourcePatternResolver;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import tk.mybatis.spring.annotation.MapperScan;import javax.sql.DataSource;@Configuration
@MapperScan(basePackages = "cn.javabb.bootdemo.mapper.local",sqlSessionFactoryRef = "localSqlSessionFactory")
public class LocalDataSourceConfig {@Bean(name = "localDataSource")@ConfigurationProperties("spring.local-datasource")public DataSource localDataSource(){return DataSourceBuilder.create().build();}@Bean(name = "localSqlSessionFactory")public SqlSessionFactory sqlSessionFactory(@Qualifier("localDataSource") DataSource dataSource) throws Exception {SqlSessionFactoryBean sessionFactoryBean = new SqlSessionFactoryBean();sessionFactoryBean.setDataSource(dataSource);sessionFactoryBean.setMapperLocations(new PathMatchingResourcePatternResolver().getResources("classpath*:mapper/local/*.xml"));return sessionFactoryBean.getObject();}@Bean(name = "localTransactionManager")public DataSourceTransactionManager localTransactionManager(){return new DataSourceTransactionManager(localDataSource());}
}
spring.local-datasource 为application.yml中配置的数据源名称
classpath*:mapper/local/*.xml 指定local数据源的mybatis的xml文件位置
原本一个数据源的时候,可以在启动程序的时候加载DataSource,但是这里是多数据源,需要为每个数据源配置DataSource,并且需要添加MapperScan注解,(因为这里是使用的通用Mapper,所以引入的class应该是mapper的,import tk.mybatis.spring.annotation.MapperScan;,如果没用就引入org的MapperScan)
basePackages这里的包路径是需要用到这个数据源的路径,这个包下所有的接口文件都会使用这个local数据源
配置第二个数据源:remote-datasource
@Configuration
@MapperScan(basePackages = "cn.javabb.bootdemo.mapper.remote",sqlSessionFactoryRef = "remoteSqlSessionFactory")
public class RemoteDataSourceConfig {@Bean(name = "remoteDataSource")@Primary@ConfigurationProperties("spring.remote-datasource")public DataSource remoteDataSource(){return DataSourceBuilder.create().build();}@Bean(name = "remoteSqlSessionFactory")@Primarypublic SqlSessionFactory sqlSessionFactory(@Qualifier("remoteDataSource") DataSource dataSource) throws Exception {SqlSessionFactoryBean sessionFactoryBean = new SqlSessionFactoryBean();sessionFactoryBean.setDataSource(dataSource);sessionFactoryBean.setMapperLocations(new PathMatchingResourcePatternResolver().getResources("classpath*:mapper/remote/*.xml"));return sessionFactoryBean.getObject();}@Bean(name = "remoteTransactionManager")@Primarypublic DataSourceTransactionManager remoteTransactionManager(){return new DataSourceTransactionManager(remoteDataSource());}}
@Primary指定默认数据源。
配置第三个数据源:oracle-datasource
@Configuration
@MapperScan(basePackages = "cn.javabb.bootdemo.mapper.oracle",sqlSessionFactoryRef = "oracleSqlSessionFactory")
public class OracleDataSourceConfig {@Bean(name = "oracleDataSource")@ConfigurationProperties("spring.oracle-datasource")public DataSource oracleDataSource(){return DataSourceBuilder.create().build();}@Bean(name = "oracleSqlSessionFactory")public SqlSessionFactory sqlSessionFactory(@Qualifier("oracleDataSource") DataSource dataSource) throws Exception {SqlSessionFactoryBean sessionFactoryBean = new SqlSessionFactoryBean();sessionFactoryBean.setDataSource(dataSource);sessionFactoryBean.setMapperLocations(new PathMatchingResourcePatternResolver().getResources("classpath*:mapper/oracle/*.xml"));return sessionFactoryBean.getObject();}@Bean(name = "oracleTransactionManager")public DataSourceTransactionManager oracleTransactionManager(){return new DataSourceTransactionManager(oracleDataSource());}
}
接下来所有的接口都应该按照不同的数据源放置,对应的xml也需要按照不同的数据源分开。
按照我项目的实际例子:
Mapper(使用local数据源,下面所有的方法都会在local数据源中查询)
import cn.javabb.bootdemo.entity.*;
import org.apache.ibatis.annotations.Mapper;
import org.springframework.dao.DataAccessException;
import org.springframework.stereotype.Component;/*** @Author QINB imqinbao@163.com* @CreateDate 2019/1/18/018 13:08* @Since V1.0*/
@Component
@Mapper
public interface StLocalMapper {int insertPptnOne(StPptn pptn);int insertRiverOne(StRiver river);int insertRsvrOne(StRsvr rsvr);
}
StLocalMapper
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="cn.javabb.bootdemo.mapper.local.StLocalMapper"><insert id="insertBatch" parameterType="java.util.List">insert into ST_PPTN_R_copy1(stcd,tm,drp,intv,pdr,dyp,wth,tmdown)values<foreach collection="list" item="item" index="index" separator=",">(#{item.stcd},#{item.tm},#{item.drp},#{item.intv},#{item.pdr},#{item.dyp},#{item.wth},#{item.tmdown})</foreach></insert><insert id="insertPptnOne" parameterType="cn.javabb.bootdemo.entity.StPptn">insert into ST_pptn_R(stcd,tm,drp,intv,pdr,dyp,wth,tmdown)values (#{stcd},#{tm},#{drp},#{intv},#{pdr},#{dyp},#{wth},#{tmdown})</insert><insert id="insertRiverOne" parameterType="cn.javabb.bootdemo.entity.StRiver">insert into ST_river_R(STCD,TM,Z,Q,XSA,XSAVV,XSMXV,FLWCHRCD,WPTN,MSQMT,MSAMT,MSVMT,TMDOWN)values (#{STCD},#{TM},#{Z},#{Q},#{XSA},#{XSAVV},#{XSMXV},#{FLWCHRCD},#{WPTN},#{MSQMT},#{MSAMT},#{MSVMT},#{TMDOWN})</insert><insert id="insertRsvrOne" parameterType="cn.javabb.bootdemo.entity.StRsvr">insert into ST_rsvr_R(STCD,TM,RZ,INQ,W,BLRZ,OTQ,RWCHRCD,RWPTN,INQDR,MSQMT,TMDOWN)values (#{STCD},#{TM},#{RZ},#{INQ},#{W},#{BLRZ},#{OTQ},#{RWCHRCD},#{RWPTN},#{INQDR},#{MSQMT},#{TMDOWN})</insert></mapper>
编写一个service调用方法就行了
@Slf4j
@Service
public class StDataSynService {@AutowiredStLocalMapper stLocalMapper;@Asyncpublic void pptnSyn(){String nowDate = DateFormatUtils.format(new Date(),"yyyy-MM-dd HH:mm:ss");String yesterDay = DateFormatUtils.format(DateUtils.addDays(new Date(),-day),"yyyy-MM-dd HH:mm:ss");log.info("[{}]扫描Pptn同步数据....",nowDate);//需要同步的数据,往前推一个小时List<StPptn> remoteList = stRemoteMapper.listPptnRemote(nowDate,yesterDay);int total = 0;if(null!=remoteList && remoteList.size()>0){total = remoteList.size();log.info("获取Pptn数据:{}条。",total);Long startTM = System.currentTimeMillis();//执行同步int n=0;for(StPptn pptn:remoteList){n += stLocalMapper.insertPptnOne(pptn);}if(n>0){if(n == total){log.info("Pptn同步成功.用时:{}秒,需要同步记录:{},成功:{}",(System.currentTimeMillis()-startTM)/1000,total,n);}else{log.info("Pptn同步成功.用时:{}秒,需要同步记录:{},成功:{},失败:{}",(System.currentTimeMillis()-startTM)/1000,total,n,(total-n));}}}else{log.info("Pptn无数据同步");}}
}
@Async:表示这个方法是异步执行,可以去掉,自己去调用service这个方法就行了。
上面的例子中我并没有用到通用Mapper,其实调用通用Mapper的方法会更简单一点
Springboot+Mybatis+通用Mapper多数据源实现数据同步相关推荐
- Springboot集成mybatis通用Mapper与分页插件PageHelper
Springboot集成mybatis通用Mapper与分页插件PageHelper 插件介绍 通用 Mapper 是一个可以实现任意 MyBatis 通用方法的框架,项目提供了常规的增删改查操作以及 ...
- spring boot整合mybatis+通用mapper+pagehelper分页插件
spring boot整合mybatis+通用mapper+pagehelper分页插件 pom依赖 <?xml version="1.0" encoding="U ...
- springboot整合通用mapper操作数据库
1. 建表sql create table user (`id` int PRIMARY KEY auto_increment COMMENT '用户id', `name` VARCHAR(255) ...
- tk mybatis通用mapper
tk mybatis通用mapper 首先要明白, tk mybatis是国人写的对mybatis的增强包, 而不是mybatis官方出的官方包 其github网址如下: https://github ...
- SpringBoot整合通用mapper实现泛型BaseController增删改
SpringBoot整合通用mapper实现泛型BaseController增删改 前言:目前写项目做需求时增删改查一直用得比较频繁,但是在做开发的时候,却并不想一次一次写重复的代码,尝试着整合了一下 ...
- SpringBoot+JdbcTemplate通用Mapper的实现
文章目录 SpringBoot+JdbcTemplate通用Mapper的实现 通用Mapper 针对User的jdbcMapper 针对 User的Service 相应的Controller Spr ...
- SSM项目使用Mybatis通用mapper插件tk.mybatis的用法
SSM项目使用Mybatis通用mapper插件tk.mybatis的用法 https://blog.csdn.net/qq_40060806/article/details/82949722 TKm ...
- 阿里开源数据同步神器DataX异构数据源间数据同步同步MySQL与HDFS相互实战
Datax 实战使用 继上一篇 阿里开源数据同步神器DataX异构数据源间数据同步基础介绍与快速入门之后的实战篇 1.MySQL-To-HDFS 环境 & 准备说明: 描述: 为了快速搭建测试 ...
- SpringBoot+Mybatis配置Druid多数据源
开篇之前,说一句题外话.多数据源和动态数据源的区别. 多数据源,一般用于对接多个业务上独立的数据库(可能异构数据库). 动态数据源,一般用于大型应用对数据切分. 配置参考 如何配置多数据源,网上教程一 ...
最新文章
- Ubuntu 系统安装OpenJDK 7,openjdk8
- tar 解压缩命令详解
- 组建校园网计算机网络设计,小型校园网的设计与组建Word版
- Java对象分配原理
- 说说 Android 的 Material Design 设计(四)——卡片式布局
- android+统计功能,React Native 轻松集成统计功能(Android 篇)
- [Angularjs] 第一步开始一个项目
- TOJ 2353: Billiard
- python实现完整的特征工程,实践论文中的分类模型和方法用于恶意页面的分类与识别
- 高斯正算matlab,MATLAB实现高斯-克吕格投影正算
- 面试官:如何理解TCP/IP协议?
- 【PyQt】PyQt4使用qt-designer进行设计
- 模型预测控制(MPC)解析(五):基于状态估计的预测控制
- 基于STM32串口通信的ESP8266WiFi模块使用
- delphi10.2 将网页页面带格式复制到word中。
- Android编译时冲突报错的完美解决方案
- Latex书籍模板分享
- iPhone 蓝牙通信编程初步(网上收集)
- 热点事件情境下微博舆情反转预测
- 【XShell | Xftp】解决Xshell强制升级