Amazon Simple Storage Service (Amazon S3)是面向 Internet 的存储服务，具有高扩展性、可靠性、安全性和快速价廉的特点，提供 99.999999999% 的持久性，可存储无限量的数据，每个对象最多包含 5 TB 的数据。S3支持版本控制、对象生命周期管理、加密、静态网站托管、Select SQL查询等。

S3 JAVA SDK

S3 架构设计与编程语言无关，提供 REST 和 SOAP 接口。HTTP 上的 SOAP 支持已弃用，但仍可在 HTTPS 上使用。SOAP 将不支持新 S3 功能，建议使用 REST API。

借助 REST，可以使用标准的 HTTP 请求创建、提取和删除存储桶和对象。直接利用REST API进行代码开发是复杂的，AWS SDK包装了底层REST API，可以简化编程任务。

配置AWS Credentials

为使用AWS SDK，必须提供AWS凭证，在 ~/.aws/credentials (Windows 用户为 C:\Users\USER_NAME.aws\credentials) 中创建：

[default]

aws_access_key_id = your_access_key_id

aws_secret_access_key = your_secret_access_key

POM

UTF-8

com.amazonaws

aws-java-sdk-s3

com.amazonaws

aws-java-sdk-bom

1.11.433

pom

import

如要使用全部的SDK，不需使用BOM，简单声明如下：

com.amazonaws

aws-java-sdk

1.11.433

S3基本操作

演示了createBucket、listBuckets、putObject、getObject、listObjects、deleteObject、deleteBucket等S3基本操作。

package org.itrunner.aws.s3;

import com.amazonaws.HttpMethod;

import com.amazonaws.regions.Regions;

import com.amazonaws.services.s3.AmazonS3;

import com.amazonaws.services.s3.AmazonS3ClientBuilder;

import com.amazonaws.services.s3.model.*;

import java.io.File;

import java.net.URL;

import java.util.Date;

import java.util.List;

public class S3Util {

private static AmazonS3 s3;

static {

s3 = AmazonS3ClientBuilder.standard().withRegion(Regions.CN_NORTH_1).build();

}

private S3Util() {

}

/*

* Create a new S3 bucket - Amazon S3 bucket names are globally unique

*/

public static Bucket createBucket(String bucketName) {

return s3.createBucket(bucketName);

}

/*

* List the buckets in your account

*/

public static List listBuckets() {

return s3.listBuckets();

}

/*

* List objects in your bucket

*/

public static ObjectListing listObjects(String bucketName) {

return s3.listObjects(bucketName);

}

/*

* List objects in your bucket by prefix

*/

public static ObjectListing listObjects(String bucketName, String prefix) {

return s3.listObjects(bucketName, prefix);

}

/*

* Upload an object to your bucket

*/

public static PutObjectResult putObject(String bucketName, String key, File file) {

return s3.putObject(bucketName, key, file);

}

/*

* Download an object - When you download an object, you get all of the object's metadata and a stream from which to read the contents.

* It's important to read the contents of the stream as quickly as possibly since the data is streamed directly from Amazon S3 and your

* network connection will remain open until you read all the data or close the input stream.

*/

public static S3Object get(String bucketName, String key) {

return s3.getObject(bucketName, key);

}

/*

* Delete an object - Unless versioning has been turned on for your bucket, there is no way to undelete an object, so use caution when deleting objects.

*/

public static void deleteObject(String bucketName, String key) {

s3.deleteObject(bucketName, key);

}

/*

* Delete a bucket - A bucket must be completely empty before it can be deleted, so remember to delete any objects from your buckets before

* you try to delete them.

*/

public static void deleteBucket(String bucketName) {

s3.deleteBucket(bucketName);

}

}

生成预签名URL

默认，S3对象为私有，只有所有者具有访问权限。但是，对象所有者可以使用自己的安全凭证来创建预签名的URL，授予有限时间内的对象下载许可，从而与其他用户共享对象，收到预签名URL的任何人都可以访问对象。

当创建预签名URL时，必须提供安全凭证、存储桶名称和对象键、HTTP 方法 (指定为GET来下载对象) 和过期时间。

public String generatePresignedUrl(String bucketName, String key, int minutes) {

// Sets the expiration date

Date expiration = new Date();

long expTimeMillis = expiration.getTime();

expTimeMillis += 1000 * 60 * minutes;

expiration.setTime(expTimeMillis);

// Generate the presigned URL.

GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, key).withMethod(HttpMethod.GET).withExpiration(expiration);

URL url = s3.generatePresignedUrl(generatePresignedUrlRequest);

return url.toString();

}

从对象中选择内容

利用Amazon S3 Select，可以使用SQL语句筛选 S3 对象的内容，检索所需的部分数据。Amazon S3 Select 适用于以CSV或JSON格式存储的对象，这些对象可以通过GZIP或BZIP2压缩和服务器端加密。

S3 Select的要求和限制

要求：

必须拥有所查询的对象的 s3:GetObject 权限。

如果查询的对象已进行加密，则必须使用 https，并必须在请求中提供加密密钥。

限制：

SQL 表达式的最大长度为 256 KB。

结果中记录的最大长度为 1 MB。

SQL语法

Amazon S3 Select 支持部分SQL，语法如下：

SELECT column_name FROM table_name [WHERE condition] [LIMIT number]

其中table_name为S3Object。

SELECT子句支持*。

文件格式为CSV时，引用列可以使用列编号或列名，列编号从1开始：

select s._1 from S3Object s

Select s.name from S3Object s

使用列名时，程序中必须设置FileHeaderInfo为Use。

可以使用双引号指示列名区分大小写：

SELECT s."name" from S3Object s

不使用双引号列名不区分大小写。

比如，CSV文件内容如下：

username,email

Jason,jason@163.com

Coco,coco@163.com

SQL语句可以为：

select s.email from S3Object s where s.username='Jason'

查询CSV文件

以下示例将查询结果保存在outputPath文件中：

public static void selectCsvObjectContent(String bucketName, String csvObjectKey, String sql, String outputPath) throws Exception {

SelectObjectContentRequest request = generateBaseCSVRequest(bucketName, csvObjectKey, sql);

final AtomicBoolean isResultComplete = new AtomicBoolean(false);

try (OutputStream fileOutputStream = new FileOutputStream(new File(outputPath));

SelectObjectContentResult result = s3.selectObjectContent(request)) {

InputStream resultInputStream = result.getPayload().getRecordsInputStream(

new SelectObjectContentEventVisitor() {

/*

* An End Event informs that the request has finished successfully.

*/

@Override

public void visit(SelectObjectContentEvent.EndEvent event) {

isResultComplete.set(true);

}

}

);

copy(resultInputStream, fileOutputStream);

}

/*

* The End Event indicates all matching records have been transmitted. If the End Event is not received, the results may be incomplete.

*/

if (!isResultComplete.get()) {

throw new Exception("S3 Select request was incomplete as End Event was not received.");

}

}

private static SelectObjectContentRequest generateBaseCSVRequest(String bucket, String key, String query) {

SelectObjectContentRequest request = new SelectObjectContentRequest();

request.setBucketName(bucket);

request.setKey(key);

request.setExpression(query);

request.setExpressionType(ExpressionType.SQL);

InputSerialization inputSerialization = new InputSerialization();

CSVInput csvInput = new CSVInput();

csvInput.setFileHeaderInfo(FileHeaderInfo.USE);

inputSerialization.setCsv(csvInput);

inputSerialization.setCompressionType(CompressionType.NONE);

request.setInputSerialization(inputSerialization);

OutputSerialization outputSerialization = new OutputSerialization();

outputSerialization.setCsv(new CSVOutput());

request.setOutputSerialization(outputSerialization);

return request;

}

参考文档