AWS S3（对象存储）接口方法

由于网络上对于S3接口方法的实现很少，所以写了关于S3包含文件分片上传、文件下载、生成预签名url、读取对象，分片读取、分别读取每个版本，保存到本地、取消用户当前授权，并重新授权、得到对象所有版本的ACL、得到对象的ACL、将某个对象授权给某个用户此方法将作用于对象的所有版本、将某个对象授权给某个用户此方法仅作用于对象的当前版本
、为授权用户设置ACL、得到指定对象的版本、读取文件写到本地在版本开启的情况下。如果不指定版本号，读取的是当前版本
、对象复制、删除空目录、删除对象的各版本、按版本号删除、列出对象各版本、删除对象元数据、修改对象元数据、得到对象的用户自定义元数据、检测对象是否存在，通过listobject方法进行检测、列出桶内所有对象、检测桶是否存在等方法
接口方法。AWS S3（对象存储）文件传输接口方法如下，需要提供原生接口方法手册，点个关注，留言或者加微信：lp1075553038：
package com.hitachivantara.example.hcp;import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStreamWriter;
import java.io.UnsupportedEncodingException;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLDecoder;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;import org.apache.commons.io.FileUtils;
import org.apache.http.Header;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;import com.amazonaws.ClientConfiguration;
import com.amazonaws.HttpMethod;
import com.amazonaws.Protocol;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.AccessControlList;
import com.amazonaws.services.s3.model.Bucket;
import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
import com.amazonaws.services.s3.model.CannedAccessControlList;
import com.amazonaws.services.s3.model.CanonicalGrantee;
import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
import com.amazonaws.services.s3.model.CopyObjectRequest;
import com.amazonaws.services.s3.model.CopyObjectResult;
import com.amazonaws.services.s3.model.CreateBucketRequest;
import com.amazonaws.services.s3.model.DeleteObjectRequest;
import com.amazonaws.services.s3.model.DeleteObjectsRequest;
import com.amazonaws.services.s3.model.DeleteObjectsResult;
import com.amazonaws.services.s3.model.DeleteVersionRequest;
import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest;
import com.amazonaws.services.s3.model.GetObjectMetadataRequest;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.Grant;
import com.amazonaws.services.s3.model.GroupGrantee;
import com.amazonaws.services.s3.model.HeadBucketRequest;
import com.amazonaws.services.s3.model.HeadBucketResult;
import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
import com.amazonaws.services.s3.model.ListObjectsRequest;
import com.amazonaws.services.s3.model.ListVersionsRequest;
import com.amazonaws.services.s3.model.MultiObjectDeleteException;
import com.amazonaws.services.s3.model.MultiObjectDeleteException.DeleteError;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PartETag;
import com.amazonaws.services.s3.model.Permission;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.PutObjectResult;
import com.amazonaws.services.s3.model.ResponseHeaderOverrides;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.S3ObjectInputStream;
import com.amazonaws.services.s3.model.S3ObjectSummary;
import com.amazonaws.services.s3.model.S3VersionSummary;
import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;
import com.amazonaws.services.s3.model.UploadPartRequest;
import com.amazonaws.services.s3.model.UploadPartResult;
import com.amazonaws.services.s3.model.VersionListing;
import com.hitachivantara.tools.BinaryUtils;
import com.hitachivantara.tools.Utils;public class HS3Example { //---1---修改成你的endpoint名称private String endpointName = "t1.hcp.content.com"; //---2---修改为你的用户名和密码(如果拿到的已经是编码后的ak,sk。不需要使用Utils.getBase64Value和Utils.getMD5Value进行编码)//private String ak = Utils.getBase64Value("admin");//private String sk = Utils.getMD5Value("P@ssw0rd"); private String ak = "YWRtaW4=";private String sk = "161ebd7d45089b3446ee4e0d86dbcf92";// configprivate ClientConfiguration myClientConfig = new ClientConfiguration();// aws client@SuppressWarnings("unused")private AmazonS3 hs3Client = null;public HS3Example() {myClientConfig.setProtocol(Protocol.HTTP); myClientConfig.setSignerOverride("S3SignerType"); hs3Client = AmazonS3ClientBuilder.standard().withClientConfiguration(myClientConfig).withPathStyleAccessEnabled(true).withEndpointConfiguration(new EndpointConfiguration(endpointName, "")).withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(ak, sk))).build(); } public static void main(String[] args) throws UnsupportedEncodingException {// 列出所有桶//new HS3Example().listBuckets();// 列出对象//new HS3Example().listObjects("n1");// 列出目录中对象// new HS3Example().listObjectsByDir("n1","09");// 列出对象，并分页// new HS3Example().listObjectsByDirPagination("hks3test","abc/",1,"");// new HS3Example().listObjectsByDirPagination("hks3test","abc/",2,"13/123/");// 对象复制// new HS3Example().copyObject("n1","123.docx","n2","123.docx");// 检测桶是否存在// new HS3Example().HeadBucketRequest("orclog");// 创建桶//new HS3Example().createBucket("n22");// 创建对象//new HS3Example().createObject("v83bniy1", "0123/dbar234.rar1", "c:\\test\\dbar.rar");/** //读取对象 try { new HS3Example().getObjByEtag("n1", "0123/123.txt", dir+"1232.txt"); } catch (IOException* e) { // TODO Auto-generated catch block e.printStackTrace(); }*/// new HS3Example().createObject("hks3test", "def/abc.txt", dir+"123.txt");// 删除对象new HS3Example().deleteObject("n99","abc.mp4");// 批量删除对象// new HS3Example().deleteObjects("hks3test","abc/def/123.zip","def/abc.txt");// 删除桶,不支持清空桶// new HS3Example().deleteBucket("b12");/*- 读取对象try {//new HS3Example().getObject("testnamespace","树状结构 - 副本.xlsx",//"C:/Users/lbin/Documents/HDSSvn/首都在线/testFile/树状结构 - 副本.xlsx");new HS3Example().getObject("n99","abc.mp4","C:/test/abc.mp4");} catch (IOException e) {// TODO Auto-generated catch blocke.printStackTrace();} */// 列出目录“北京/123/”下的对象// new HS3Example().listObjectsByDir("5dnaywg0","1480426251603/blocks/2016-11-29-21-34-30/0/");// 计算文件SHA256// msha256md5Hashes(new File("C:\\Users\\lbin\\Documents\\HDSSvn\\首都在线\\testFile\\123.xls"));// 读取文件写到本地/*- try {//new HS3Example().getObject("testnamespace","树状结构 - 副本.xlsx",//"C:/Users/lbin/Documents/HDSSvn/首都在线/testFile/树状结构 - 副本.xlsx");new HS3Example().getObject("n1","1223.xls","C:/Users/lbin/Documents/HDSSvn/首都在线/testFile/test1111rr1.xls");} catch (IOException e) {// TODO Auto-generated catch blocke.printStackTrace();}*//** try { new HS3Example().getObjectCache("n1","123.txt",* "C:/Users/lbin/Documents/HDSSvn/首都在线/testFile/0ta2xtje.xls"); } catch (IOException e) { // TODO* Auto-generated catch block e.printStackTrace(); }*/// 设置桶的ACL// new HS3Example().setBucketACLForUser("testnamespace3");// new HS3Example().setBucketACLForAuthenticatedUsers("testnamespace3");// -new HS3Example().setBucketACLForPublic("jftyhvds");// new HS3Example().setACLForBucketGroup("testnamespace3");// 将某个对象授权给某个用户/**/// 为特定用户设置对象的ACL//new HS3Example().setObjectACLForUser("n1","HCP-AW-VMDK-file1.nvram");// new HS3Example().setObjectACLForUser_v("testnamespace3","test.xls");// 为授权用户统一设置对象的ACL// new HS3Example().setObjectACLForAuthenticatedUsers("testnamespace3","test.xls");// 为匿名用户设置权限// new HS3Example().setObjectACLForPublic("n1","0qwe/113.jpg");// 得到对象的ACL// -new HS3Example().getObjectACL("jftyhvds","1223.xls");// new HS3Example().getObjectACL_v("testnamespace3","test.xls");// 将对象的所有版本授权给某个用户/** ArrayList<String> aclList = new ArrayList<String>(); aclList.add("READ"); new* HS3Example().setObjectVersionACL("testnamespace","树状结构 - 副本.xlsx",* aclList,"83813c7e-dd96-4572-8556-813b6a997ff7");*/// 得到对象所有版本的ACL// new HS3Example().getObjectVersionACL("testnamespace","树状结构 - 副本.xlsx");// 取消用户当前授权，并重新授权/** ArrayList<String> aclList = new ArrayList<String>(); aclList.add("WRITE"); new* HS3Example().clearObjectVersionACL("testnamespace","树状结构 - 副本.xlsx",aclList,* "83813c7e-dd96-4572-8556-813b6a997ff7");*/// 得到桶的的ACL// -new HS3Example().getACLFromBucket("jftyhvds");// 创建桶，并指定ACL// new HS3Example().createBucketWithACL("testnamespace3");// 得到版本状态// -new HS3Example().getBucketVersion("jftyhvds");// 开启桶的版本// new HS3Example().setVersionOnOff("testnamespace3");// 检测对象、目录是否存在//new HS3Example().checkObject("sc", "1074/61/20200319173512847_2.jpg");// new HS3Example().checkObject("zhfpwd5r", "/bj");// new HS3Example().createObject("9vx8m26g", "ExampleObject123.txt", dir+"jdk-8u151-linux-x64.tar.gz");// -new HS3Example().createObject("jftyhvds", "bj/123.xls", dir+"123.xls");// new HS3Example().createObject("testnamespace", "北京/北京-东城.txt", dir+"北京-东城.txt");// new HS3Example().createObject("testnamespace", "北京/北京-通州.txt", dir+"北京-通州.txt");// new HS3Example().createObject("testnamespace", "北京/北京-西城.txt", dir+"北京-西城.txt");// new HS3Example().createObject("testnamespace", "上海/上海-内环.txt", dir+"上海-内环.txt");// new HS3Example().createObject("testnamespace", "上海/上海-浦东.txt", dir+"上海-浦东.txt");// new HS3Example().createObject("testnamespace", "上海/上海-浦西.txt", dir+"上海-浦西.txt");// 创建目录// -new HS3Example().createDir("jftyhvds", "tjty/");// 列出根目录对象// -new HS3Example().listObjectsByDir("jftyhvds","");// new HS3Example().listObjectsByDirPagination("n1","",20,"");// 创建对象，附件元数据/** //元数据信息 ObjectMetadata metadata = new ObjectMetadata(); metadata.addUserMetadata("gss", "aa");* try { new HS3Example().createObjectAndMetadata("zhfpwd5r", "test123.txt", dir+"test.txt",metadata); }* catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); }*/// 得到对象元数据//new HS3Example().getObjectMetadata("n1", "2.jpg");// 现有对象增加元数据，或修改对象元数据/** - ObjectMetadata metadata = new ObjectMetadata();* * * //metadata.addUserMetadata("key1", u8); //String strGBK = URLEncoder.encode("中文", "UTF-8");* metadata.addUserMetadata("key1", "abc_def");* * new HS3Example().updateObjectMetadata("jftyhvds", "1223.xls",metadata);*/// 删除对象元数据// new HS3Example().deleteObjectMetadata("testnamespace3", "test.txt");// 删除对象的各版本//new HS3Example().deleteObjectByVersion("n1","tjty.txt");//new HS3Example().deleteObjectByVersionId("n1","tjty.txt","101684870967425");// 删除多个对象，不指定版本,暂不支持/** new HS3Example().deleteObjects("zhfpwd5r","0ta2xtje.xls" ,"1223.xls");*/// 列出对象各版本// http://zhfpwd5r.giccdn.com/rest/c38911a6-1259-40f4-aae0-98dfee6e3335.eml//new HS3Example().listVersions("test", "000001/Test1-2.log");// 删除空目录// new HS3Example().deleteDir("zhfpwd5r","bj/");// 读取对象，分片读取//** - try { new HS3Example().getObjectByRange("jftyhvds","test.txt",* "C:/Users/lbin/Documents/HDSSvn/首都在线/testFile/1223qwe.txt" , 0, 2); } catch (IOException e) { //* TODO Auto-generated catch block e.printStackTrace(); }*/// 分别读取每个版本，保存到本地/** try { new HS3Example().getObjectVersions("n1","123.txt",* "C:/Users/lbin/Documents/HDSSvn/首都在线/testFile/"); } catch (IOException e) { // TODO* Auto-generated catch block e.printStackTrace(); }*/// 生成预签名url,为桶n1中的对象1234.docx生成预签名// new HS3Example().generatePresignedUrl("b11", "1cc7bfb3-d204-4a95-b2d9-6f712185f112");//生成预签名写入方法/** try {new HS3Example().generatePresignedUrlPut("n1","binbin2.txt");} catch (IOException e) {// TODO Auto-generated catch blocke.printStackTrace();} *///new HS3Example().multipartUpload("n1","hive.tar.gz");}/*** 生成预签名url* * @param bucketName* @param objectKey*/public void generatePresignedUrl(String bucketName, String objectKey) {java.util.Date expiration = new java.util.Date();long milliSeconds = expiration.getTime();milliSeconds += 1000 * 60 * 60; // Add 1 hour.expiration.setTime(milliSeconds);GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, objectKey);generatePresignedUrlRequest.setMethod(HttpMethod.GET);generatePresignedUrlRequest.setExpiration(expiration);// generatePresignedUrlRequest.setContentType("text/plain");URL url = hs3Client.generatePresignedUrl(generatePresignedUrlRequest);System.out.println(url.toString()); }public void multipartUpload(String bucketName,String keyName){File file = new File("C:\\test\\apache-hive-2.1.1-bin.tar.gz");//文件总大小long contentLength = file.length();//分片大小long partSize = 5 * 1024 * 1024; InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(bucketName, keyName);InitiateMultipartUploadResult initResponse = hs3Client.initiateMultipartUpload(initRequest);List<PartETag> partETags = new ArrayList<PartETag>();long filePosition = 0;for (int i = 1; filePosition < contentLength; i++) {// Because the last part could be less than 5 MB, adjust the part size as needed.partSize = Math.min(partSize, (contentLength - filePosition));// Create the request to upload a part.UploadPartRequest uploadRequest = new UploadPartRequest().withBucketName(bucketName).withKey(keyName).withUploadId(initResponse.getUploadId()).withPartNumber(i).withFileOffset(filePosition).withFile(file).withPartSize(partSize);// Upload the part and add the response's ETag to our list.UploadPartResult uploadResult = hs3Client.uploadPart(uploadRequest);partETags.add(uploadResult.getPartETag());System.out.println("分片="+filePosition);filePosition += partSize;}// Complete the multipart upload.CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(bucketName, keyName,initResponse.getUploadId(), partETags);System.out.println("分片合并成功");hs3Client.completeMultipartUpload(compRequest);}public void generatePresignedUrlPut(String bucketName, String objectKey) throws IOException {java.util.Date expiration = new java.util.Date();long milliSeconds = expiration.getTime();milliSeconds += 1000 * 60 * 60; // Add 1 hour.expiration.setTime(milliSeconds);GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, objectKey);generatePresignedUrlRequest.setMethod(HttpMethod.PUT);generatePresignedUrlRequest.setExpiration(expiration);// generatePresignedUrlRequest.setContentType("text/plain");//生成预签名URL url = hs3Client.generatePresignedUrl(generatePresignedUrlRequest);System.out.println(url.toString());//向预签名中写入对象HttpURLConnection connection = (HttpURLConnection) url.openConnection();connection.setDoOutput(true);connection.setRequestMethod("PUT");OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());out.write("This text uploaded as an object via presigned URL.");out.close();connection.getResponseCode();System.out.println("HTTP response code: " + connection.getResponseCode());connection.disconnect();}/*** 取消用户当前授权，并重新授权* * @param bucketName* @param objectKey* @param ACLList* @param userId*/public void clearObjectVersionACL(String bucketName, String objectKey, List<String> ACLList, String userId) {// 得到现有对象的ACLAccessControlList list = hs3Client.getObjectAcl(bucketName, objectKey);// 清除所有授权信息list.getGrants().clear();// 为某个用户授权，需要指定用户的ID号CanonicalGrantee grantee = new CanonicalGrantee(userId);if (ACLList != null)// 为用户重新授权for (String acl : ACLList) {if (acl.equals("READ"))list.grantPermission(grantee, Permission.Read);else if (acl.equals("READ_ACP"))list.grantPermission(grantee, Permission.ReadAcp);else if (acl.equals("WRITE"))list.grantPermission(grantee, Permission.Write);else if (acl.equals("WRITE_ACP"))list.grantPermission(grantee, Permission.WriteAcp);else if (acl.equals("FULL_CONTROL"))list.grantPermission(grantee, Permission.FullControl);}// 得到对象的所有版本List<S3VersionSummary> versions = listVersions(bucketName, objectKey);for (S3VersionSummary v : versions) {// 为每个版本分别授权hs3Client.setObjectAcl(bucketName, objectKey, v.getVersionId(), list);}}/*** 得到对象所有版本的ACL* * @param bucketName*            桶名* @param objectKey*            对象key*/public void getObjectVersionACL(String bucketName, String objectKey) {// 得到对象的所有版本List<S3VersionSummary> versions = listVersions(bucketName, objectKey);for (S3VersionSummary v : versions) {System.out.println("version id  = " + v.getVersionId());// 得到指定版本对象的ACLAccessControlList list = hs3Client.getObjectAcl(bucketName, objectKey, v.getVersionId());// 列出授权信息Set<Grant> grants = list.getGrants();// 显示授权信息for (Grant g : grants) {System.out.println("id = " + g.getGrantee().getIdentifier());System.out.println(" permission = " + g.getPermission().toString());}}}/*** 得到对象的ACL* * @param bucketName*            桶名* @param objectKey*            对象key*/public void getObjectACL(String bucketName, String objectKey) {// 得到现有对象的ACLAccessControlList list = hs3Client.getObjectAcl(bucketName, objectKey);// 列出授权信息List<Grant> grants = list.getGrantsAsList();// 显示授权信息for (Grant g : grants) {System.out.println("id = " + g.getGrantee().getIdentifier());System.out.println(" permission = " + g.getPermission().toString());}}public void getObjectACL_v(String bucketName, String objectKey) {// 得到现有对象的ACLAccessControlList list = hs3Client.getObjectAcl(bucketName, objectKey, "94525645227841");// 列出授权信息List<Grant> grants = list.getGrantsAsList();// 显示授权信息for (Grant g : grants) {System.out.println("id = " + g.getGrantee().getIdentifier());System.out.println(" permission = " + g.getPermission().toString());}}/*** 将某个对象授权给某个用户 此方法将作用于对象的所有版本* * @param bucketName*            桶名* @param objectKey*            对象key* @param ACL*            可以设置的ACL值 • READ • READ_ACP • WRITE • WRITE_ACP • FULL_CONTROL* @param userId*            要授权的用户ID*/public void setObjectVersionACL(String bucketName, String objectKey, List<String> ACLList, String userId) {// 得到现有对象的ACLAccessControlList list = new AccessControlList();list = hs3Client.getObjectAcl(bucketName, objectKey);// 为某个用户授权，需要指定用户的ID号CanonicalGrantee grantee = new CanonicalGrantee(userId);for (String acl : ACLList) {if (acl.equals("READ"))list.grantPermission(grantee, Permission.Read);else if (acl.equals("READ_ACP"))list.grantPermission(grantee, Permission.ReadAcp);else if (acl.equals("WRITE"))list.grantPermission(grantee, Permission.Write);else if (acl.equals("WRITE_ACP"))list.grantPermission(grantee, Permission.WriteAcp);else if (acl.equals("FULL_CONTROL"))list.grantPermission(grantee, Permission.FullControl);}// 得到对象的所有版本List<S3VersionSummary> versions = listVersions(bucketName, objectKey);for (S3VersionSummary v : versions) {// 为每个版本分别授权hs3Client.setObjectAcl(bucketName, objectKey, v.getVersionId(), list);}}/*** 将某个对象授权给某个用户 此方法仅作用于对象的当前版本* * @param bucketName*            桶名* @param objectKey*            对象key* @param ACL*            可以设置的ACL值 • READ • READ_ACP • WRITE • WRITE_ACP • FULL_CONTROL* @param userId*            要授权的用户ID*/public void setObjectACLForUser(String bucketName, String objectKey) {// 得到现有对象的ACLAccessControlList list = hs3Client.getObjectAcl(bucketName, objectKey);list.getGrantsAsList().clear();/*配置用户只有读权限*/Grant grant0 = new Grant(new CanonicalGrantee("1442c70b-bc03-40f8-82e5-cc129969728f"), Permission.Read);list.getGrantsAsList().add(grant0);hs3Client.setObjectAcl(bucketName, objectKey, list);}public void setObjectACLForUser_v(String bucketName, String objectKey) {// 得到现有对象的ACLAccessControlList list = hs3Client.getObjectAcl(bucketName, objectKey);list.getGrantsAsList().clear();Grant grant0 = new Grant(new CanonicalGrantee("83813c7e-dd96-457c-8556-813b6a997ff7"), Permission.Write);list.getGrantsAsList().add(grant0);hs3Client.setObjectAcl(bucketName, objectKey, "94525645227841", list);}/*** 为授权用户设置ACL* * @param bucketName* @param objectKey*/public void setObjectACLForAuthenticatedUsers(String bucketName, String objectKey) {AccessControlList objACL = hs3Client.getObjectAcl(bucketName, objectKey);objACL.getGrantsAsList().clear();Grant g1 = new Grant(GroupGrantee.AuthenticatedUsers, Permission.FullControl);objACL.getGrantsAsList().add(g1);hs3Client.setObjectAcl(bucketName, objectKey, objACL);}/*** 读取对象，分片读取* * @param bucketName*            桶名* @param objectKey*            对象key* @param localFilePath*            本地文件路径* @param start*            起始位置* @param end*            结束位置* @throws IOException*/public void getObjectByRange(String bucketName, String objectKey, String localFilePath, long start, long end) throws IOException {GetObjectRequest req = new GetObjectRequest(bucketName, objectKey).withRange(start, end);S3Object obj = hs3Client.getObject(req);S3ObjectInputStream s3i = obj.getObjectContent();FileOutputStream localFile = new FileOutputStream(localFilePath);byte readBuffer[] = new byte[2048];int bufferSize;while (-1 != (bufferSize = s3i.read(readBuffer))) {localFile.write(readBuffer, 0, bufferSize);}localFile.close();}/*** 分别读取每个版本，保存到本地* * @param bucketName*            桶名* @param objectKey*            对象key* @param dirPath*            本地目录* @throws IOException*/public void getObjectVersions(String bucketName, String objectKey, String dirPath) throws IOException {// 得到各版本List<S3VersionSummary> versions = listVersions(bucketName, objectKey);// 分别读取每个版本，保存到本地for (S3VersionSummary v : versions) {getObjectVersion(bucketName, v.getKey(), dirPath + "/" + v.getVersionId() + "_" + v.getKey(), v.getVersionId());}}/*** 得到指定对象的版本* * @param bucketName* @param objectKey* @param localFilePath* @param versionId*            对象版本号* @throws IOException*/public void getObjectVersion(String bucketName, String objectKey, String localFilePath, String versionId) throws IOException {GetObjectRequest req = new GetObjectRequest(bucketName, objectKey).withVersionId(versionId);S3Object obj = hs3Client.getObject(req);S3ObjectInputStream s3i = obj.getObjectContent();FileOutputStream localFile = new FileOutputStream(localFilePath);byte readBuffer[] = new byte[2048];int bufferSize;while (-1 != (bufferSize = s3i.read(readBuffer))) {localFile.write(readBuffer, 0, bufferSize);}localFile.close();}public void getObjByEtag(String bucketName, String objectKey, String localFilePath) throws IOException {List<String> eTagList = new ArrayList<>();eTagList.add("3f34cb6d9bcc3f4698d68fb3b725a4b5");GetObjectRequest req = new GetObjectRequest(bucketName, objectKey);req.withMatchingETagConstraint("3f34cb6d9bcc3f4698d68b3b725a4b5");S3Object obj = hs3Client.getObject(req);// S3Object obj = hs3Client.getObject(bucketName,objectKey);S3ObjectInputStream s3i = obj.getObjectContent();FileOutputStream localFile = new FileOutputStream(localFilePath);byte readBuffer[] = new byte[1024 * 1024];int bufferSize;while (-1 != (bufferSize = s3i.read(readBuffer))) {localFile.write(readBuffer, 0, bufferSize);}localFile.close();}/*** 读取文件写到本地 在版本开启的情况下。如果不指定版本号，读取的是当前版本。* * @param bucketName*            桶名* @param objectKey*            对象key* @param localFilePath*            本地文件路径* @throws IOException*/public void getObject(String bucketName, String objectKey, String localFilePath) throws IOException {GetObjectRequest req = new GetObjectRequest(bucketName, objectKey); S3Object obj = hs3Client.getObject(req); S3ObjectInputStream s3i = obj.getObjectContent();FileOutputStream localFile = new FileOutputStream(localFilePath);byte readBuffer[] = new byte[1024 * 1024];int bufferSize;while (-1 != (bufferSize = s3i.read(readBuffer))) {localFile.write(readBuffer, 0, bufferSize);}localFile.close();s3i.close();}public void getObjectCache(String bucketName, String objectKey, String localFilePath) throws IOException {ResponseHeaderOverrides rho = new ResponseHeaderOverrides();rho.setCacheControl("max-age=604800");GetObjectRequest req = new GetObjectRequest(bucketName, objectKey).withResponseHeaders(rho);S3Object obj = hs3Client.getObject(req);// S3Object obj = hs3Client.getObject(bucketName,objectKey);S3ObjectInputStream s3i = obj.getObjectContent();FileOutputStream localFile = new FileOutputStream(localFilePath);byte readBuffer[] = new byte[1024 * 1024];int bufferSize;while (-1 != (bufferSize = s3i.read(readBuffer))) {localFile.write(readBuffer, 0, bufferSize);}localFile.close();}/*** 对象复制 1、可以同一个租户下的不同桶中复制 2、可以在同一个桶中复制 3、用户需要有源桶的read权，有目标桶的write权；* 4、如果版本是开启的，不指定版本号，默认复制当前版本；如果指定版本，复制指定的版本号 5、默认情况下源对象的“用户自定义元数据”会被一并复制到目标对象。* 当然，在复制的时候你也可以指定一个新的用户自定义元数据，详见（updateObjectMetadata）* 或指定一个空的用户自定义元数据，这样就会将用户自定义元数据删除掉，详见（deleteObjectMetadata）* * @param sourceBucketName* @param sourceKey* @param destinationBucketName* @param destinationKey*/public void copyObject(String sourceBucketName, String sourceKey, String destinationBucketName, String destinationKey) {CopyObjectRequest request = new CopyObjectRequest(sourceBucketName, sourceKey, destinationBucketName, destinationKey).withSourceBucketName(sourceBucketName).withSourceKey(sourceKey);try {CopyObjectResult result = hs3Client.copyObject(request);System.out.println(result.getETag());} catch (com.amazonaws.services.s3.model.AmazonS3Exception e) {e.printStackTrace();}}/*** 删除空目录* * @param bucketName* @param dir*/public void deleteDir(String bucketName, String dir) {hs3Client.deleteObject(bucketName, dir);}/*** 删除对象,如果版本功能开启。 则并不会真正删除* * @param bucketName* @param objectKey*/public void deleteObject(String bucketName, String objectKey) {DeleteObjectRequest req = new DeleteObjectRequest(bucketName, objectKey);hs3Client.deleteObject(req);}/*** 删除对象的各版本* * @param bucketName* @param objectKey*/public void deleteObjectByVersion(String bucketName, String objectKey) {// 得到对象的各版本List<S3VersionSummary> list = listVersions(bucketName, objectKey);for (S3VersionSummary sv : list) {// 一个个删除DeleteVersionRequest req = new DeleteVersionRequest(bucketName, objectKey, sv.getVersionId());hs3Client.deleteVersion(req);}}/*** 按版本号删除* @param bucketName* @param objectKey* @param versionId*/public void deleteObjectByVersionId(String bucketName, String objectKey,String versionId) {DeleteVersionRequest req = new DeleteVersionRequest(bucketName, objectKey, versionId);hs3Client.deleteVersion(req);}/*** 列出对象各版本* * @param bucketName*            桶名* @param objectKey*            对象key*/public List<S3VersionSummary> listVersions(String bucketName, String objectKey) {ListVersionsRequest req = new ListVersionsRequest().withBucketName(bucketName).withPrefix(objectKey);VersionListing vl = hs3Client.listVersions(req);List<S3VersionSummary> list = vl.getVersionSummaries();for (S3VersionSummary sv : list) {System.out.println(sv.getVersionId());}return list;}/*** 删除多个对象，不指定版本 经测试不支持* * @param bucketName* @param objectKey*            可以传入多个对象key*/public void deleteObjects(String bucketName, String... objectKey) {DeleteObjectsRequest req = new DeleteObjectsRequest(bucketName).withKeys(objectKey);try {DeleteObjectsResult dor = hs3Client.deleteObjects(req);} catch (MultiObjectDeleteException ex) {System.out.println(ex);for (DeleteError deleteError : ex.getErrors()) {System.out.format("Object Key: %s\t%s\t%s\n", deleteError.getKey(), deleteError.getCode(), deleteError.getMessage());}}}/*** 删除对象元数据* * @param bucketName*            桶名* @param objectKey*            对象key* @param newMetadata*            要修改的元数据*/public void deleteObjectMetadata(String bucketName, String objectKey) {// 声明元数据，但是并不赋值，就可以删除ObjectMetadata metadata = new ObjectMetadata();CopyObjectRequest request = new CopyObjectRequest(bucketName, objectKey, bucketName, objectKey).withSourceBucketName(bucketName).withSourceKey(objectKey).withNewObjectMetadata(metadata);hs3Client.copyObject(request);}/*** 修改对象元数据* * @param bucketName*            桶名* @param objectKey*            对象key* @param newMetadata*            要修改的元数据*/public void updateObjectMetadata(String bucketName, String objectKey, ObjectMetadata newMetadata) {CopyObjectRequest request = new CopyObjectRequest(bucketName, objectKey, bucketName, objectKey).withSourceBucketName(bucketName).withSourceKey(objectKey).withNewObjectMetadata(newMetadata);hs3Client.copyObject(request);}/*** 得到对象的用户自定义元数据* * @param bucketName* @param objectKey*/public void getObjectMetadata(String bucketName, String objectKey) {// GetObjectMetadataRequest req = new// GetObjectMetadataRequest(bucketName,objectKey,"94531548821057");GetObjectMetadataRequest req = new GetObjectMetadataRequest(bucketName, objectKey);ObjectMetadata metadata = hs3Client.getObjectMetadata(req);System.out.println(metadata.getContentLength());if (metadata != null) {// 得到用户自定义元数据System.out.print("得到用户自定义元数据");Map<String, String> map = metadata.getUserMetadata();for (Entry<String, String> entry : map.entrySet()) {try {String strUTF8 = URLDecoder.decode(entry.getValue(), "UTF-8");System.out.println("key = " + entry.getKey() + " |  value=" + strUTF8);} catch (UnsupportedEncodingException e) {// TODO Auto-generated catch blocke.printStackTrace();}}// 对象的系统元数据System.out.println("对象的系统元数据");System.out.println("md5 = " + metadata.getContentMD5());System.out.println("size = " + metadata.getContentLength());System.out.println("size = " + metadata.getLastModified());}}/*** 检测对象是否存在，通过listobject方法进行检测* * @param bucketName* @param objectKey*/public void checkObject(String bucketName, String objectKey) {boolean flag = hs3Client.doesObjectExist(bucketName, objectKey);System.out.println(flag);}/*** 分页* * @param bucketName*            桶名* @param dirName*            目录名* @param pageSize*            每页几条* @param marker*            翻页的标识。如果是第一页，可以不指定；如果是第二页，请指定你得到的第一页的最后一条记录值*/public void listObjectsByDirPagination(String bucketName, String dirName, int pageSize, String marker) {ListObjectsRequest req = new ListObjectsRequest().withBucketName(bucketName).withDelimiter("/").withPrefix(dirName).withMaxKeys(pageSize).withMarker(marker);ObjectListing objectListing = hs3Client.listObjects(req);for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {System.out.println(" * " + objectSummary.getKey() + " " + "(size = " + objectSummary.getSize() + ")");}for (String dir : objectListing.getCommonPrefixes()) {System.out.println(" * " + dir);}}/*** 按目录列出对象，不分页* * @param bucketName*            桶名* @param dirName*            限定的目录名*/public void listObjectsByDir(String bucketName, String dirName) {final ListObjectsRequest req = new ListObjectsRequest().withBucketName(bucketName).withDelimiter("/").withPrefix(dirName);ObjectListing objectListing = hs3Client.listObjects(req);for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {System.out.println(" * " + objectSummary.getKey() + " " + "(size = " + objectSummary.getSize() + ")");}for (String dir : objectListing.getCommonPrefixes()) {System.out.println(" * " + dir);}}/*** 列出桶内所有对象* * @param bucketName*/public void listObjects(String bucketName) {try{System.out.println("-----list object not with delimiter----");ListObjectsRequest req = new ListObjectsRequest().withBucketName(bucketName);// req.putCustomRequestHeader("x-amz-content-sha256", "1111111");//.getCustomRequestHeaders();// req.putCustomRequestHeader("date", "Thu, 07 Mar 2013 14:27:05 +0000");ObjectListing objectListing = hs3Client.listObjects(req);for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {System.out.println(" * " + objectSummary.getKey() + " " + "(size = " + objectSummary.getSize() + ")");System.out.println("-----get object's md5 not with delimiter----");GetObjectMetadataRequest rrr = new GetObjectMetadataRequest(bucketName, objectSummary.getKey());ObjectMetadata metadata = hs3Client.getObjectMetadata(rrr); System.out.println("--content md5 not with delimiter--"+metadata.getContentMD5());break;}}catch(Exception e){System.out.println(e.getStackTrace());System.out.println("----list object   with delimiter-----");ListObjectsRequest req = new ListObjectsRequest().withBucketName(bucketName).withDelimiter("/");ObjectListing objectListing = hs3Client.listObjects(req);for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {System.out.println(" * " + objectSummary.getKey() + " " + "(size = " + objectSummary.getSize() + ")");System.out.println("-----get object's md5 with delimiter----");GetObjectMetadataRequest rrr = new GetObjectMetadataRequest(bucketName, objectSummary.getKey());ObjectMetadata metadata = hs3Client.getObjectMetadata(rrr); System.out.println("--content md5 with delimiter--"+metadata.getContentMD5());break;}}}/*** 创* @param buck建目录*      * etName*            桶名* @param objectKey*            对象key*/public void createDir(String bucketName, String objectKey) {PutObjectRequest req = new PutObjectRequest(bucketName, objectKey, new ByteArrayInputStream(new byte[0]), null);hs3Client.putObject(req);}/*** 创建对象，并同时指定元数据信息* * @param bucketName* @param objectKey* @param localFilePath* @param metadata* @throws IOException*/public void createObjectAndMetadata(String bucketName, String objectKey, String localFilePath, ObjectMetadata metadata) throws IOException {File file = new File(localFilePath);FileInputStream fis = new FileInputStream(file);File scratchFile = File.createTempFile("prefix", "suffix");FileUtils.copyInputStreamToFile(fis, scratchFile);PutObjectRequest req = new PutObjectRequest(bucketName, objectKey, scratchFile);// req.getRequestClientOptions().setReadLimit(1024*1024);hs3Client.putObject(req);}/*** 创建对象* * @param bucketName*            桶名称* @param objectKey*            对象的key* @param localFilePath*            本地文件路径*/public void createObject(String bucketName, String objectKey, String localFilePath) {File file = new File(localFilePath);FileInputStream fis;try {fis = new FileInputStream(file);PutObjectRequest req = new PutObjectRequest(bucketName, objectKey, fis, null);PutObjectResult res = hs3Client.putObject(req); // 得到MD5值System.out.println(res.getContentMd5());} catch (FileNotFoundException e) {// TODO Auto-generated catch blocke.printStackTrace();}}/*** 得到桶的版本配置* * @param bucketName*/public void getBucketVersion(String bucketName) {BucketVersioningConfiguration conf = hs3Client.getBucketVersioningConfiguration(bucketName);System.out.println(conf.getStatus());}/*** 开启、关闭桶的版本功能 • Enabled — Enables versioning for the bucket • Suspended — Disables versioning for* the bucket* * @param bucketName*/public void setVersionOnOff(String bucketName) {BucketVersioningConfiguration configuration = new BucketVersioningConfiguration();configuration.setStatus(BucketVersioningConfiguration.ENABLED);SetBucketVersioningConfigurationRequest req = new SetBucketVersioningConfigurationRequest(bucketName, configuration);hs3Client.setBucketVersioningConfiguration(req);}/*** 为特定用户授权* * @param bucketName*/public void setBucketACLForUser(String bucketName) {AccessControlList bucketACL = hs3Client.getBucketAcl(bucketName);bucketACL.getGrantsAsList().clear();Grant grant0 = new Grant(new CanonicalGrantee("83813c7e-dd96-457c-8556-813b6a997ff7"), Permission.Write);bucketACL.getGrantsAsList().add(grant0);hs3Client.setBucketAcl(bucketName, bucketACL);}/*** 为认证用户授权* * @param bucketName*/public void setBucketACLForAuthenticatedUsers(String bucketName) {AccessControlList bucketACL = hs3Client.getBucketAcl(bucketName);bucketACL.getGrantsAsList().clear();Grant g1 = new Grant(GroupGrantee.AuthenticatedUsers, Permission.Write);bucketACL.getGrantsAsList().add(g1);hs3Client.setBucketAcl(bucketName, bucketACL);}/*** 为匿名用户设置对象ACL* * @param bucketName* @param objectKey*/public void setObjectACLForPublic(String bucketName, String objectKey) {AccessControlList objACL = hs3Client.getObjectAcl(bucketName, objectKey);objACL.getGrantsAsList().clear();Grant g1 = new Grant(GroupGrantee.AllUsers, Permission.Read);objACL.getGrantsAsList().add(g1);hs3Client.setObjectAcl(bucketName, objectKey, objACL);}/*** 为公共用户授权* * @param bucketName*/public void setBucketACLForPublic(String bucketName) {AccessControlList bucketACL = hs3Client.getBucketAcl(bucketName);// Grant g1 = new Grant(// GroupGrantee.AllUsers,// Permission.Read);Grant g2 = new Grant(GroupGrantee.AllUsers, Permission.Write);// bucketACL.getGrantsAsList().add(g1);bucketACL.getGrantsAsList().add(g2);hs3Client.setBucketAcl(bucketName, bucketACL);}/*** 得到桶的的权限信息* * @param bucketName*/public void getACLFromBucket(String bucketName) {AccessControlList bucketACL = hs3Client.getBucketAcl(bucketName);List<Grant> grants = bucketACL.getGrantsAsList();// .getGrants();for (Grant g : grants) {System.out.println(g.getGrantee().getIdentifier() + "|" + g.getPermission().name());}}/*** 列出所有桶 == 列出租户下的命名空间 列出当前用户的所有桶。 对于HCP来说，如果命名空间没有开启S3协议，在返回的桶列表中就没有它*/public void listBuckets() {List<Bucket> buckets = hs3Client.listBuckets();for (Bucket bucket : buckets) {System.out.println(" 名称 =  " + bucket.getName());System.out.println(" 创建时间 =  " + bucket.getCreationDate());System.out.println(" 所有者 =  " + bucket.getOwner());System.out.println(" ------------------------------------ ");}}/*** 创建桶==创建HCP的命名空间 HCP最小默认空间是50GB，必须保证空间足够，否则无法创建成功 创建成为命名空间的所有者*/public void createBucket(String bucketName) {com.amazonaws.services.s3.model.Bucket bk = hs3Client.createBucket(bucketName);}/*** 删除桶* * @param bucketName*/public void deleteBucket(String bucketName) {hs3Client.deleteBucket(bucketName);}/*** 创建桶，并同时指定ACL* * @param bucketName*/public void createBucketWithACL(String bucketName) {CreateBucketRequest createBucketRequest = new CreateBucketRequest(bucketName).withCannedAcl(CannedAccessControlList.PublicRead);Bucket bk = hs3Client.createBucket(createBucketRequest);}/*** 检测桶是否存在* * @param bucketName*/public void HeadBucketRequest(String bucketName) {com.amazonaws.services.s3.model.HeadBucketRequest req = new HeadBucketRequest(bucketName);// req.getCustomRequestHeaders().put("x-amz-content-sha256", "1234");HeadBucketResult res = hs3Client.headBucket(req);if (res != null)System.out.println("true");}}
AWS S3（对象存储）接口方法相关推荐

linux同步到对象存储,将Cpanel备份配置为S3对象存储的方法
使用Cpanel,你可以选择在本地备份网站数据和数据库,也可以选择在Amazon S3等云存储中备份,在本文中我们将配置Cpanel自动备份策略-本地和S3对象存储(在Linux下安装Minio并使用 ...
马逊s3云存储接口_当对象存储“湖”有了强一致性
从 2006年第一个云服务对象存储服务 Amazon S3 发布直到 2020年12月1日之前,S3 对象操作都是遵循 "最终一致性"原则,对象存储服务本身就是一个复杂的分布式系统 ...
AWS S3云存储服务
AWS S3云存储服务 1 S3概念及基础知识 2 S3的基本操作 3 S3数据安全 4 S3数据加密以及命令行CLI 4.1 S3加密工作原理 4.1.1 server端的加密 4.1.2 clie ...
java s3 与ceph的关系_Ceph存储与S3对象存储性能优化.pdf
Ceph存储与S3对象存储性能优化.pdf Ceph on Storage appliance Case Study and Performance for AWS S3 based object s ...
.NET Core AWS S3云存储
[导读]最近有需要用到AWS S3云存储上传附件,这里对利用.NET或.NET Core在调用SDK APi需要注意的一点小问题做个记录,或许能对后续有用到的童鞋提供一点帮助 Amazon Simpl ...
aws s3 参与s3game寻找宝藏游戏挑战学习s3对象存储
参考资料 Pirates S3game workshop http://s3game-level1.s3-website.us-east-2.amazonaws.com/level1.html htt ...
马逊s3云存储接口_利用 S3tests 测试 S3 接口兼容性
女主宣言在对象存储迭代研发.测试过程中,为了方便.准确验证 S3 接口协议兼容性,本文作者对Ceph官方采用的兼容性测试工具s3-tests进行了调研,并对其配置以及使用做出了详细介绍,相信对于s3 ...
马逊s3云存储接口_使用 Amazon S3 云服务轻松实现存储
Amazon Simple Storage Service (S3) 是一个公开的服务,Web 应用程序开发人员可以使用它存储数字资产,包括图片.视频.音乐和文档. S3 提供一个 RESTful A ...
minio对象存储原理_使用MinIO在本地运行S3对象存储
minio对象存储原理 Object storage is amazingly versatile. We normally associate it with AWS S3, but Object ...
java s3 与ceph的关系,ceph S3 对象存储的使用
ceph 对象存储兼容两种接口(协议)S3 和 swift. S3 最常用, 这里仅介绍如何连接(使用)S3存储接口. 访问 S3, 你需要 S3 的:用户密钥,包含:access_id 和 secr ...
AWS S3（对象存储）接口方法

AWS S3（对象存储）接口方法相关推荐

最新文章

热门文章
