我有java代码,它试图使用coniguration从S3初始化一个远程文件系统(这之前是在HDFS上的,我试图在不太修改代码的情况下将其移动到S3)。

fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain

fs.defaultFS=s3a://mybucket.devrun.algo-resources/

然后,在我使用的设置中

hdfsFileSystem = FileSystem.get(conf);

这将导致以下异常:

org.apache.hadoop.fs.s3a.AWSS3IOException: doesBucketExist on mybucket.devrun.algo-resources: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 1E91F85FA3751C44), S3 Extended Request ID: 5KDgH7lsaIX7l5DQcdBdUjeg/qxYgOEU4WJBOL0p090kqNNlYOAie31zuYUQw+R3LN4CvavdVJk=: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 1E91F85FA3751C44)

at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:178)

at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:282)

at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:236)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)

at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2848)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:181)

at pipeline.HdfsSyncUtils.(HdfsSyncUtils.java:32)

at pipeline.QueriesForArticles$QFAMapper.setup(QueriesForArticles.java:158)

at AWSPipeline.C2SIndexSearchingAlgo.(C2SIndexSearchingAlgo.java:41)

at AWSPipeline.ABTestMainRunner.main(ABTestMainRunner.java:27)

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 1E91F85FA3751C44), S3 Extended Request ID: 5KDgH7lsaIX7l5DQcdBdUjeg/qxYgOEU4WJBOL0p090kqNNlYOAie31zuYUQw+R3LN4CvavdVJk=

at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)

at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)

at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)

at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)

at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1107)

at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1070)

at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:276)

... 11 more

我用

hadoop-aws.jar

2.8.0

我用

awsfed

在下面创建凭据文件

~\.aws

你知道这个错误是什么意思吗?没有详细的错误消息。。。

编辑:

对于任何感兴趣的人,我解决了这个问题:

跟随

this answer

,我得出结论,这是区域相关的。我的水桶在美国东2区。我试着在另一个地区打开一个桶,结果成功了!

这可能与

seen in the docs

hadoop-aws.jar 2.8.0

)可能使用旧版本。