[原创]个人理解，请批判接受，有误请指正。转载请注明出处: https://heyfl.gitee.io/design/zero-copy-file-download.html
背景
现有报表系统异步导出报表，生成的报表会上传到对象存储中，因为安全问题，用户不能直接上对象存储系统中下载文件，需要通过报表服务代劳，因为不需要对其做修改，只需做转发，所以这里考虑使用零拷贝技术进行优化
现有做法
- 把文件数据『下载』下来，然后把对应的文件返回给客户端
- 数据经过两次拷贝，一次是从对象存储下载到报表服务，一次是从报表服务下载到客户端
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| @Controller
public class DownloadController {
@RequestMapping(value = "/download", method = RequestMethod.GET)
public ResponseEntity<byte[]> downloadFile() throws IOException {
HttpClient client = HttpClientBuilder.create().build();
HttpGet request = new HttpGet("http://xxxx/xxxx.zip");
HttpResponse response = client.execute(request);
byte[] zipContent = EntityUtils.toByteArray(response.getEntity());
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_OCTET_STREAM);
headers.setContentDispositionFormData("attachment", "xxx.zip");
headers.setContentLength(zipContent.length);
return new ResponseEntity<byte[]>(zipContent, headers, HttpStatus.OK);
}
}
|
零拷贝优化
- 通过ZeroCopyInputStreamWrapper把文件数据直接『转发』给客户端
- 数据只经过2次拷贝，不需要经过报表服务，直接从对象存储转发给客户端
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| @Controller
public class DownloadController {
@RequestMapping(value = "/download", method = RequestMethod.GET)
public ResponseEntity<InputStreamResource> downloadFile() throws IOException {
CloseableHttpClient client = HttpClients.createDefault();
HttpGet request = new HttpGet("http://xxxx/xxxx.zip");
HttpResponse response = client.execute(request);
HttpEntity entity = response.getEntity();
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_OCTET_STREAM);
headers.setContentDispositionFormData("attachment", "xxx.zip");
headers.setContentLength(entity.getContentLength());
InputStream stream = new ZeroCopyInputStreamWrapper(entity.getContent());
return new ResponseEntity<InputStreamResource>(new InputStreamResource(stream), headers, HttpStatus.OK);
}
}
|
PS
PS: 使用ZeroCopyInputStreamWrapper将HttpEntity的输入流包装成ZeroCopyInputStream，实际上是将HttpEntity的输入流传递给了ZeroCopyInputStream，而不是将响应数据读取到Java的用户内存中
原理
在ZeroCopyInputStreamWrapper中，它通过使用Java NIO的Direct ByteBuffer和底层的通道来实现DMA的零拷贝操作
数据直接从通道读取到Direct ByteBuffer中，跳过了CPU拷贝的过程，实现了高效的数据传输
其他