学习目标：

Java批量数据清洗

学习内容：

Java批量数据清洗的方案

学习产出：

1.Service层：

/*** crm2.4商品概况清洗同步老数据商品总销量* @return*/
Map clearDataoOrderCount();

2.ServiceImpl层：

 /***清洗同步老数据** @return*/@Override@Transactional(rollbackFor = Exception.class)public Map clearDataoOrderCount() {long createTimeMillis = System.currentTimeMillis();HashMap<String, Object> map = new HashMap<>();List<Integer> countList = Lists.newArrayList();//1.获取所有数据List<CrmProductSummary> summaryDaoPage = crmProductSummaryDao.listCrmProductSummary();log.info("GoodsOverviewServiceImpl->clearDataoOrderCount->全部数据1->summaryDaoPage:{}", JSON.toJSONString(summaryDaoPage));map.put("countSum->准备清洗总数量", summaryDaoPage.size());log.info("GoodsOverviewServiceImpl->clearDataoOrderCount->countSum->准备清洗总数量->size:{}", JSON.toJSONString(summaryDaoPage.size()));//2.分批清洗if (!CollectionUtil.isEmpty(summaryDaoPage)) {//分批次清洗1000一次int insertLength = summaryDaoPage.size();int i = 0;while (insertLength > 1000) {List<CrmProductSummary> productSummaries = summaryDaoPage.subList(i, i + 1000);List<CrmProductSummary> listSummary = this.clearData(productSummaries);//执行添加if (!CollectionUtil.isEmpty(listSummary)) {this.updateBatchById(listSummary);}i = i + 1000;insertLength = insertLength - 1000;countList.add(listSummary.size());}if (insertLength > 0) {List<CrmProductSummary> productSummaries = summaryDaoPage.subList(i, i + insertLength);List<CrmProductSummary> list = this.clearData(productSummaries);log.info("GoodsOverviewServiceImpl->clearDataoOrderCount->全部数据2->list:{}", JSON.toJSONString(list));//执行添加if (!CollectionUtil.isEmpty(list)) {this.updateBatchById(list);}countList.add(list.size());}}log.info("GoodsOverviewServiceImpl->clearDataoOrderCount->清洗完成->map:{}", JSON.toJSONString(map));map.put("listCountClear->清洗完成总数量", countList);long endTimeMillis = System.currentTimeMillis();map.put("清洗数据耗时:毫秒", endTimeMillis - createTimeMillis);return map;}

 /*** 清洗同步老数据** @return*/@Transactional(rollbackFor = Exception.class)public List<CrmProductSummary> clearData(List<CrmProductSummary> listSummary) {List<CrmProductSummary> list = Lists.newArrayList();//1.判读为空if (!CollectionUtil.isEmpty(listSummary)) {//2.根据集合获取商品idList<String> productIdList = listSummary.stream().map(CrmProductSummary::getProductId).collect(Collectors.toList());log.info("GoodsOverviewServiceImpl->clearData->根据集合获取商品id->productIdList:{}", JSON.toJSONString(productIdList));List<String> dataList = productIdList.stream().distinct().collect(Collectors.toList());log.info("GoodsOverviewServiceImpl->clearData->根据集合获取商品id去重之后->dataList:{}", JSON.toJSONString(dataList));//3.根据商品id远程获取order-pryProductStringListRequest productStringListRequest = new ProductStringListRequest().setIds(dataList);//4.远程获取到数据List<CleanProductTotalSalesVo> totalSalesVoList = this.jsonCleanProductTotalSales(productStringListRequest);log.info("GoodsOverviewServiceImpl->clearData->crm2.4商品概况清洗同步老数据商品总销量->totalSalesVoList:{}", JSON.toJSONString(totalSalesVoList));if (CollectionUtil.isEmpty(totalSalesVoList)) {return list;}//5.list转mapMap<String, CleanProductTotalSalesVo> mapCleanProductTotalSalesVo = totalSalesVoList.stream().collect(Collectors.toMap(CleanProductTotalSalesVo::getProductId, account -> account));if (CollectionUtil.isEmpty(mapCleanProductTotalSalesVo)) {return list;}for (CrmProductSummary summary : listSummary) {//6.根据mysql商品id获取mangodblog.info("GoodsOverviewServiceImpl->clearData->summary->crm2.4商品概况清洗同步老数据商品总销量->productId:{}", JSON.toJSONString(summary.getProductId()));CleanProductTotalSalesVo totalSalesVo = mapCleanProductTotalSalesVo.get(summary.getProductId());log.info("GoodsOverviewServiceImpl->clearData-totalSalesVo>->crm2.4商品概况清洗同步老数据商品总销量->totalSalesVo:{}", JSON.toJSONString(totalSalesVo));//7.不等于空进行数据赋值if (totalSalesVo != null) {summary.setOrderCount(totalSalesVo.getTotalSales());}//8.addlist.add(summary);}}return list;}

 /*** 远程调用** @param productStringListRequest* @return*/@Transactional(rollbackFor = Exception.class)public List<CleanProductTotalSalesVo> jsonCleanProductTotalSales(ProductStringListRequest productStringListRequest) {List<CleanProductTotalSalesVo> cleanProductTotalSalesVoList = Lists.newArrayList();//1.远程获取orser-payString cleanProductTotalSalesJSON = okHttpCli.doPostJSON(urlConfig.getCleanProductTotalSales(), productStringListRequest);log.info("GoodsOverviewServiceImpl->jsonCleanProductTotalSales->远程获取清洗商品概况的商品总销量接口->cleanProductTotalSalesJSON:{}", JSON.toJSONString(cleanProductTotalSalesJSON));//2.进行数据处理，json解析if (!StringUtils.isEmpty(cleanProductTotalSalesJSON)) {//3.业务处理HashMap hashMap = JSON.parseObject(cleanProductTotalSalesJSON, HashMap.class);log.info("GoodsOverviewServiceImpl->jsonCleanProductTotalSales->远程获取orser-pay清洗商品概况的商品总销量接口业务处理->hashMap:{}", JSON.toJSONString(hashMap));if (!CollectionUtil.isEmpty(hashMap) && "200".equals(hashMap.get("code").toString())) {if (hashMap.get("data") != null) {cleanProductTotalSalesVoList = JsonUtil.getList(hashMap.get("data").toString(), CleanProductTotalSalesVo.class);log.info("GoodsOverviewServiceImpl->jsonCleanProductTotalSales->远程获取orser-pay清洗商品概况的商品总销量接口data转map->cleanProductTotalSalesVoList:{}", JSON.toJSONString(cleanProductTotalSalesVoList));if (CollectionUtil.isEmpty(cleanProductTotalSalesVoList)) {throw new BusinessException(EnumResultCode.ERROR.getCode(), "远程获取orser-pay清洗商品概况的商品总销量接口为空");}}}}log.info("GoodsOverviewServiceImpl->jsonCleanProductTotalSales->远程获取orser-pay清洗商品概况的商品总销量接口响应输出->cleanProductTotalSalesVoList:{}", JSON.toJSONString(cleanProductTotalSalesVoList));return cleanProductTotalSalesVoList;}

3.Controller层

    /*** @return {@link Map< String, Object>}* @author * @date 2022/4/8 16:20* @description 清洗数据**/@PostMapping("/clearDataoOrderCount")public ResponseResult<Map> clearDataoOrderCount(){Map map = goodsOverviewService.clearDataoOrderCount();log.info("GoodsOverviewController->clearDataoOrderCount->清洗响应->map:{}", JSON.toJSONString(map));return ResponseResult.buildSuccess(map);}

Java批量数据清洗相关推荐

java批量转换图片格式
废话不多直接上代码,代码其实也不多.... package com.qiao.testImage;import java.awt.image.BufferedImage; import java.io ...
java 批量处理示例_Java异常处理教程（包含示例和最佳实践）
java 批量处理示例异常是可能在程序执行期间发生的错误事件,它会破坏其正常流程. Java提供了一种健壮且面向对象的方式来处理异常情况,称为Java异常处理 . 我们将在本教程中研究以下主题. ...
java 批量处理示例_Java中异常处理的示例
java 批量处理示例 Here, we will analyse some exception handling codes, to better understand the concepts. ...
Java 批量插入数据到数据库(MySQL)中
实现Java批量插入数据库数据: package Proxy;import java.io.BufferedReader; import java.io.File; import java.io.Fi ...
java 批量打印_JAVA批量打印皕杰报表
原标题:JAVA批量打印皕杰报表不使用皕杰报表工具条上的打印按钮,用java怎么实现批量打印皕杰报表呢? 解决方案: 皕杰报表提供了批量打印的工具类ReportToolkits,ReportTool ...
Java 批量导出PPT为图片，并合并生成PDF
Java 批量导出PPT为图片,并合并生成PDF 最近有小伙伴需要把PPT打印,但苦于PPT太多,浪费纸张,所以特别写了这个合并多张PPT到一张A4纸上并生成PDF方便打印的程序: 前期准备把PPT ...
java批量打印标签_java批量打印
下面是实现java批量打印功能的具体工具类,其中的Info类和UserClass为具体的快递面单中要打印的信息,以参数的方式传递过来,循环调用java PrinterJob服务进行打印:这里的快递尺寸 ...
java 批量打印_用java怎样实现批量打印
展开全部下面是实现Java批量打印功能的具体工具类,其中的Info类和UserClass为具体的快递面单中要打印的信息e68a8462616964757a686964616f313333633839 ...
java批量下载demo_Java批量入库Demo
java中往数据库批量插入数据Demo java代码: //入库数据是需要批量入库的List int len =入库数据.size(); //每次循环10条 int incremnet = 10; / ...
java 批量删除_Java中怎样实现批量删除操作？
展开全部进行编写编程代码就能实现批量删除操作. 具体代码如下: [java] public Connection con=null; public PreparedStatement pstmt= ...

Java批量数据清洗

学习目标：

学习内容：

学习产出：

Java批量数据清洗相关推荐

最新文章

热门文章