前端(Javascript) + JAVA 服务端如何处理 HTTP 断点续传

需求场景

为什么需要断点续传？

假如在生产环境客户或操作上传一个很大的文件（可能有1个G），由于现场网络环境很差，上传到0.5个G的时候突然网络断开了，又要重新上传。客户或操作深吸了一口气，重新上传了一次，等了半小时到文件上传90%的时候突然又断开了，这个时候客户或操作不得要奔溃了。哈哈，当然我们做程序的肯定不允许这种事情发生，这个时候肯定要做断点续传。

我下面说几个场景要用到断点续传：
1.大文件，比如说blibili上传一个视频文件，上传一半网络断开，我们可以重新接着上传。
2.带宽不足，上传大文件是很耗费服务贷款的，如果说在很短时间内上传一个很大的文件，很有可能在一段时间内我们的服务器带宽被全不占满。

设计方案

前端将文件拆分为4个文件分别进行上传到后端，将拆分后的数量和当前文件的index同事发送给后端，后端根据当前文件index和总得分片数量进行验证，验证通过之后就进行合并操作。如果中间中断了传输，前端会记录传输的位置，接着当前的传输位置重新上传未上传完的包，这里一定是重新上传不是续传，拆分成了4个100K的包，假如说index=3的包还没上传完就已经断开上传了，那需要重新上传3这个包。

上代码

前端代码（客户端）

前端可以是Java端也可以是javascript也可以是php等等，客户端是不限语言的。你可以同时是服务端，也可以是客户端。

Javascript 客户端实现

split-file.html 前端页面

<!DOCTYPE html>
<html>
<head><meta name="viewport" content="width=device-width, initial-scale=1.0" /><title>uploadFile</title><style></style>
</head>
<body>
<input type="file" id="file" multiple />
<br />
<br />
<button id="btn">上传</button>
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
<script>var uploadFile;document.querySelector("#file").addEventListener("change",(e) => {var files = e.target.files;if (!files.length) return;uploadFile = new CutFileAndUpload({files,apiUpload: (fileData) => {//接口请求 返回当前文件数据/**fileData = {file: file,  //当前文件succeed: 0, //已经上传的片数shardSize: this.size, //以2MB为一个分片shardCount: 0, //总片数start: 0,  //截取开始位置end: 0,  //截取结束位置}*///构造一个表单 表单字段根据后端接口而定let fdata = new FormData();//计算切割文件单个分片let base64 = fileData.file.slice(fileData.start, fileData.end);fdata.append("file", base64, fileData.file.name);fdata.append("name", fileData.file.name);fdata.append("total", fileData.shardCount); //总片数fdata.append("numbers", fileData.succeed + 1); //当前是第几片axios({url: 'http://localhost:8091/upload/',method: "post",data: fdata,headers: {filenameMd5: fileData.filenameMd5,fileTotal: fileData.shardCount}}).then(function (response) {console.log(response);}).catch(function (error) {console.log(error);});console.log("-----------------------------------------------")console.log(base64);console.log("-----------------------------------------------")//接口请求setTimeout(() => {//更新文件数据uploadFile.updateFileData();}, 2000);},progress: (progress, total) => {//progress 当前文件进度百分比//total 总进度百分比console.log(progress, total);},success: () => {//上传成功回调console.log("全部上传完成");e.target.value = "";},});},false);document.querySelector("#btn").addEventListener("click",() => {uploadFile.uploadFile();},false);/**** @param {*} options*/function CutFileAndUpload(options) {this.files = options.files || []; //要上传的文件列表this.progress = options.progress; //上传进度this.success = options.success; //成功回调this.apiUpload = options.apiUpload;this.fileArr = []; //文件列表切割后的文件数据this.fileIndex = 0; //上传到第几个文件this.size = 100 * 1024; //分片单位 以100K为一个分片this.uploading = false; //上传状态this.cutFile();}CutFileAndUpload.prototype = {constructor: CutFileAndUpload,cutFile() {var files = this.files;if (!files.length) {console.log("请选择要上传的文件");return;}for (var i = 0; i < files.length; i++) {var file = files[i];let fileData = {file: file,succeed: 0, //已经上传的片数shardSize: this.size, //分片单位shardCount: 0, //总片数start: 0, //截取开始位置end: 0, //截取结束位置filenameMd5: uuid() // 文件名称};fileData.shardCount = Math.ceil(fileData.file.size / fileData.shardSize); //总片数this.fileArr.push(fileData);}},uploadFile() {if (!this.fileArr.length) {console.log("请选择要上传的文件");return;}var fileData = this.fileArr[this.fileIndex];//计算每一片的起始与结束位置fileData.start = fileData.succeed * fileData.shardSize;fileData.end = Math.min(fileData.file.size,fileData.start + fileData.shardSize);//计算文件单个分片// let base64 = fileData.file.slice(fileData.start, fileData.end);// console.log(fileData);this.uploading = true;//接口请求this.apiUpload && this.apiUpload(fileData);},updateFileData() {//更新文件数据var fileData = this.fileArr[this.fileIndex];fileData.succeed++;var progress = parseInt((fileData.succeed / fileData.shardCount) * 100);var total;if (fileData.succeed === fileData.shardCount) {//单个文件上传完成this.fileIndex++;total = parseInt((this.fileIndex / this.fileArr.length) * 100);this.progress && this.progress(progress, total);if (this.fileIndex == this.fileArr.length) {//列表的全部文件上传完成this.uploading = false;this.fileIndex = 0;this.fileArr = [];this.success && this.success();} else {this.uploadFile();}} else {total = parseInt((this.fileIndex / this.fileArr.length) * 100);this.progress && this.progress(progress, total);this.uploadFile();}},};function uuid() {var s = [];var hexDigits = "0123456789abcdef";for (var i = 0; i < 36; i++) {s[i] = hexDigits.substr(Math.floor(Math.random() * 0x10), 1);}s[14] = "4"; // bits 12-15 of the time_hi_and_version field to 0010s[19] = hexDigits.substr((s[19] & 0x3) | 0x8, 1); // bits 6-7 of the clock_seq_hi_and_reserved to 01s[8] = s[13] = s[18] = s[23] = "-";var uuid = s.join("");return uuid;}
</script>
</body>
</html>

JAVA客户端

FileSplitUtils 处理文件切分合并的工具类，在后面JAVA服务端代码也会用到这个工具类进行处理合并操作。

public class FileSplitUtils {// 100Kprivate static final int BUFFER_SIZE = 100 * 1024;private static final String LINE_SEPARATOR = "-";/*** 切割文件。** @param srcFile* @param partsDir* @throws IOException*/public static void splitFile(File srcFile, File partsDir) throws IOException {//健壮性的判断。if (!(srcFile.exists() && srcFile.isFile())) {throw new RuntimeException("源文件不是正确的文件或者不存在");}if (!partsDir.exists()) {partsDir.mkdirs();}//1，明确目的。目的输出流有多个，只创建引用。FileOutputStream fos = null;//2，使用字节流读取流和源文件关联。try (FileInputStream fis = new FileInputStream(srcFile)) {//3，定义缓冲区。1M.byte[] buf = new byte[BUFFER_SIZE]; // 100K//4，频繁读写操作。int len = 0;int count = 1;//碎片文件的编号。while ((len = fis.read(buf)) != -1) {//创建输出流对象。只要满足了缓冲区大小，碎片数据确定，直接往碎片文件中写数据 。//碎片文件存储到partsDir中，名称为编号+part扩展名。fos = new FileOutputStream(new File(partsDir, (count++) + ".part"));//将缓冲区中的数据写入到碎片文件中。fos.write(buf, 0, len);//直接关闭输出流。fos.close();}/** 将源文件以及切割的一些信息也保存起来随着碎片文件一起发送。* 信息；* 1，源文件的名称（文件类型）* 2，切割的碎片的个数。* 将这些信息单独封装到一个文件中。* 还要一个输出流完成此动作。*/String filename = srcFile.getName();int partCount = count;//创建一个输出流。fos = new FileOutputStream(new File(partsDir, count + ".properties"));//创建一个属性集。Properties prop = new Properties();//将配置信息存储到属性集中。prop.setProperty("filename", srcFile.getName());prop.setProperty("partcount", Integer.toString(partCount));// 将属性集中的信息持久化。prop.store(fos, "part file info");// fos.write(("filename=" + filename + LINE_SEPARATOR).getBytes());// fos.write(("partcount=" + Integer.toString(partCount)).getBytes());} finally {assert fos != null;fos.close();}}public static void mergerFile(File partsDir) throws IOException {/** 虽然合并成功，问题如下：* 1，如何明确碎片的个数，来确定循环的次数，以明确要有多少个输入流对象。* 2，如何知道合并的文件的类型。* 解决方案：应该先读取配置文件。*///1,获取配置文件。File configFile = getConfigFile(partsDir);//2,获取配置文件信息容器。获取配置信息的属性集。Properties prop = getProperties(configFile);//3,将属性集对象传递合并方法中。merge(partsDir, prop);}public static void mergerFile(File partsDir, Properties prop) throws IOException {merge(partsDir, prop);}//根据配置文件获取配置信息属性集。private static Properties getProperties(File configFile) throws IOException {FileInputStream fis = null;Properties prop = new Properties();try {//读取流和配置文件相关联。fis = new FileInputStream(configFile);//将流中的数据加载的集合中。prop.load(fis);} finally {if (fis != null) {try {fis.close();} catch (IOException e) {//写日志，记录异常信息。便于维护。}}}return prop;}//根据碎片目录获取配置文件对象。private static File getConfigFile(File partsDir) {if (!(partsDir.exists() && partsDir.isDirectory())) {throw new RuntimeException(partsDir.toString() + ",不是有效目录");}//1,判断碎片文件目录中是否存在properties文件。使用过滤器完成。File[] files = partsDir.listFiles(pathname -> pathname.getName().endsWith(".properties"));assert files != null;if (files.length != 1) {throw new RuntimeException("properties扩展名的文件不存在，或不唯一");}return files[0];}private static void merge(File partsDir, Properties prop) throws FileNotFoundException,IOException {//获取属性集中的信息。String filename = prop.getProperty("filename");int partCount = Integer.parseInt(prop.getProperty("partcount"));//使用io包中的SequenceInputStream，对碎片文件进行合并，将多个读取流合并成一个读取流。List<FileInputStream> list = new ArrayList<>();for (int i = 1; i < partCount; i++) {list.add(new FileInputStream(new File(partsDir, i + ".part")));}//怎么获取枚举对象呢？List自身是无法获取枚举Enumeration对象的，考虑到Collections中去找。Enumeration<FileInputStream> en = Collections.enumeration(list);try (SequenceInputStream sis = new SequenceInputStream(en); FileOutputStream fos = new FileOutputStream(new File(partsDir, filename))) {//不断的读写。byte[] buf = new byte[4096];int len = 0;while ((len = sis.read(buf)) != -1) {fos.write(buf, 0, len);}} catch (Exception e) {e.printStackTrace();}}}

UploadFileTest

public static void main(String[] args) throws IOException {File file    = new File("/xxx/Documents/picture/1631503405304.jpg");File destDir = new File("/xxx/Project/allens-learn/upload/");// -------------------拆分文件-------------------// FileSplitUtils.splitFile(file, destDir);// -------------------合并文件-------------------// FileSplitUtils.mergerFile(destDir);// 上传文件String result = HttpClientUtil.doHttpPostForFormMutipart("http://localhost:8091/upload/image",null,file);System.out.println("result: {}" + result);
}

这里使用的Http Components 如果对其不太了解的同学可以参考我的另一篇文章:
HTTP Component 5.0 + 教程

HttpClientUtil http 工具类

/*** 上传文件** @param uri* @param getParams* @return*/
public static String doHttpPostForFormMutipart(String uri, Map<String, String> getParams, File file) {CloseableHttpResponse response = null;try {HttpPost httpPost = new HttpPost(uri);// httpPost.setHeader(new BasicHeader("Content-Type", "multipart/form-data"));if (null != getParams && !getParams.isEmpty()) {List<NameValuePair> list = new ArrayList<>();for (Map.Entry<String, String> param : getParams.entrySet()) {list.add(new BasicNameValuePair(param.getKey(), param.getValue()));}}MultipartPart multipartPart = MultipartPartBuilder.create().setBody(new FileBody(file, ContentType.IMAGE_JPEG)).addHeader("content-type", "image/jpeg").build();org.apache.hc.core5.http.HttpEntity httpEntity = MultipartEntityBuilder.create()//.addPart(multipartPart).addPart("file", new FileBody(file, ContentType.IMAGE_JPEG))//.setContentType(ContentType.IMAGE_JPEG).build();// HttpEntity httpEntity = new UrlEncodedFormEntity(list, "utf-8");httpPost.setEntity(httpEntity);response = httpClient.execute(httpPost);int statusCode = response.getCode();if (HttpStatus.SC_OK == statusCode) {HttpEntity entity = response.getEntity();if (null != entity) {String resStr = EntityUtils.toString(entity, "utf-8");return resStr;}} else {HttpEntity entity = response.getEntity();System.out.println(EntityUtils.toString(entity, "utf-8"));}} catch (Exception e) {e.printStackTrace();//log.error("CloseableHttpClient-post-请求异常", e);} finally {try {if (null != response)response.close();} catch (IOException e) {e.printStackTrace();}}return new String("NO CONTENT");
}

后端处理代码 (服务端)

UploadController 处理前端http请求

package com.allens.alibaba.test.controller.upload;import com.allens.alibaba.test.config.UploadProperties;
import com.allens.alibaba.test.service.UploadService;
import io.swagger.annotations.Api;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;import javax.annotation.Resource;
import javax.servlet.http.HttpServletRequest;@RestController
@Api(tags = "/upload")
@RequestMapping("/upload")
@Slf4j
public class UploadController {@Autowiredprivate UploadService uploadService;@Resourceprivate UploadProperties uploadProperties;@PostMapping("/")public ResponseEntity<String> upload(@RequestParam("file") MultipartFile file,@RequestHeader("filenameMd5") String filenameMd5,@RequestHeader("fileTotal") Integer fileTotal,HttpServletRequest servletRequest) throws Exception {log.info("filenameMd5 : {}", filenameMd5);return ResponseEntity.ok(uploadService.uploadImage(file, filenameMd5, fileTotal));}@GetMapping("/merge")public ResponseEntity<Boolean> mergeFile (@RequestHeader("filenameMd5") String filenameMd5) {return ResponseEntity.ok(uploadService.mergeFile(filenameMd5));}}

UploadUtils 上传工具类，负责生成零时文件名称以及获取上传的文件属性。

import java.util.Properties;public class UploadUtils {private static final String PART_SUFFIX = ".part";/*** 文件名称替换工具，将文件名称替换为随机名称** @param oldName* @return*/public static String generateFileName(String oldName) {String suffix = oldName.substring(oldName.lastIndexOf("."));return IDUtils.generateUniqueId() + suffix;}/*** 文件名称替换工具，将文件名称替换为随机名称** @param oldName* @return*/public static String generatePartName(String oldName, String newName) {return newName + PART_SUFFIX;}public static Properties getFileProperties (String originName, String total) {Properties properties = new Properties();properties.setProperty("filename", originName);properties.setProperty("partcount", String.valueOf(total));return properties;}}

UploadService上传服务

import org.springframework.web.multipart.MultipartFile;public interface UploadService {/*** 上传图片* @param file* @return*/public String uploadImage(MultipartFile file,String filename,Integer total)throws Exception;boolean mergeFile(String filenameMd5);
}

UploadServiceImpl 上传服务实现类

@Service
@Slf4j
public class UploadServiceImpl implements UploadService {private static final String UPLOAD_DIR = "/Users/yueyu/Project/allens-learn/upload/";@Autowiredprivate UploadProperties uploadProperties;private static ConcurrentHashMap<String, AtomicInteger> fileCacheMap = new ConcurrentHashMap<>();@Overridepublic String uploadImage(MultipartFile file, String filename, Integer total) throws IOException {log.info("file type is: {}", file.getContentType());//if (!uploadProperties.getAllowTypes().contains(file.getContentType())) {//    throw new IOException("文件上传类型错误！");//}log.info("file size: byte {}", file.getBytes().length);log.info("file size: {}", file.getSize());File fileDir = new File(uploadProperties.getPath() + "/" + filename + "/");if (!fileDir.exists()) {fileDir.mkdir();}AtomicInteger atomicInteger = fileCacheMap.get(filename);if (atomicInteger == null) {atomicInteger = new AtomicInteger(1);fileCacheMap.put(filename, atomicInteger);} else {atomicInteger.incrementAndGet();}try {// String fileNameGenerator = UploadUtils.generateOriginName(file.getOriginalFilename());file.transferTo(new File(String.format("%s/%s/%s",uploadProperties.getPath(),filename,UploadUtils.generatePartName(file.getOriginalFilename(), String.valueOf(atomicInteger.get())))));} catch (Exception e) {e.printStackTrace();atomicInteger.decrementAndGet();}if (atomicInteger.get() == total) {Properties properties = new Properties();properties.setProperty("filename", file.getOriginalFilename());properties.setProperty("partcount", String.valueOf(total));FileSplitUtils.mergerFile(new File(UPLOAD_DIR + filename), properties);}return file.getOriginalFilename();}@Overridepublic boolean mergeFile(String filename) {try {FileSplitUtils.mergerFile(new File(UPLOAD_DIR + filename));} catch (IOException e) {log.error("合并文件失败", e);}return false;}
}

功能验证

选中一个图片上传

① 后端输出日志

② 文件上传目录

可以看到传输了4个部分，然后合并成了一个图片文件。

总结

我偷懒了，有些地方实现的不够完美，如果有需要自己修改修改就行了。像后端文件校验写的过于简单，在集群情况下肯定会出问题