Unable to mount S3 buckets for which you only have read permissions

Description

When trying to mount a S3 bucket for which your credentials only have read permissions this will fail because internally it tries to write to the bucket:

1 2 alluxio fs mount -readonly /buckets/noaa s3a://noaa-nexrad-level2/2015/ ThriftIOException(message:Ufs path /2015 does not exist)

The log file shows the following:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 2017-03-13 06:31:30,035 ERROR logger.type (S3AUnderFileSystem.java:mkdirsInternal) - Failed to create directory: 2015 com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 9BC1DCB82E1EB594), S3 Extended Request ID: qCO7uhBcJ7RQAfBBrYSAwWiOL45qWPcMKz3dRgWddqGaWnlQihxCOb44l1vNqM0KDBovUqKrDw4= at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1305) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:852) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:630) at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:405) at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:367) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:318) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3787) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1399) at alluxio.underfs.s3a.S3AUnderFileSystem.mkdirsInternal(S3AUnderFileSystem.java:786) at alluxio.underfs.s3a.S3AUnderFileSystem.getFolderMetadata(S3AUnderFileSystem.java:633) at alluxio.underfs.s3a.S3AUnderFileSystem.getObjectDetails(S3AUnderFileSystem.java:654) at alluxio.underfs.s3a.S3AUnderFileSystem.exists(S3AUnderFileSystem.java:278) at alluxio.master.file.FileSystemMaster.mountInternal(FileSystemMaster.java:2133) at alluxio.master.file.FileSystemMaster.mountAndJournal(FileSystemMaster.java:2066) at alluxio.master.file.FileSystemMaster.mount(FileSystemMaster.java:2033) at alluxio.master.file.FileSystemMasterClientServiceHandler$12.call(FileSystemMasterClientServiceHandler.java:233) at alluxio.master.file.FileSystemMasterClientServiceHandler$12.call(FileSystemMasterClientServiceHandler.java:230) at alluxio.RpcUtils.call(RpcUtils.java:62) at alluxio.master.file.FileSystemMasterClientServiceHandler.mount(FileSystemMasterClientServiceHandler.java:230) at alluxio.thrift.FileSystemMasterClientService$Processor$mount.getResult(FileSystemMasterClientService.java:1611) at alluxio.thrift.FileSystemMasterClientService$Processor$mount.getResult(FileSystemMasterClientService.java:1595) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2017-03-13 06:31:30,082 WARN logger.type (RpcUtils.java:call) - I/O error when running rpc java.io.IOException: Ufs path /2015 does not exist at alluxio.master.file.FileSystemMaster.mountInternal(FileSystemMaster.java:2134) at alluxio.master.file.FileSystemMaster.mountAndJournal(FileSystemMaster.java:2066) at alluxio.master.file.FileSystemMaster.mount(FileSystemMaster.java:2033) at alluxio.master.file.FileSystemMasterClientServiceHandler$12.call(FileSystemMasterClientServiceHandler.java:233) at alluxio.master.file.FileSystemMasterClientServiceHandler$12.call(FileSystemMasterClientServiceHandler.java:230) at alluxio.RpcUtils.call(RpcUtils.java:62) at alluxio.master.file.FileSystemMasterClientServiceHandler.mount(FileSystemMasterClientServiceHandler.java:230) at alluxio.thrift.FileSystemMasterClientService$Processor$mount.getResult(FileSystemMasterClientService.java:1611) at alluxio.thrift.FileSystemMasterClientService$Processor$mount.getResult(FileSystemMasterClientService.java:1595) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

This occurs even if you are explicitly add the -readonly option to the Mount command. The underlying causes appears to be the use of dummy files by Alluxio. It is unclear to me what purpose they serve but they render it impossible to use public datasets which are read only

Environment

None

Status

Assignee

Calvin Jia

Reporter

Rob Vesse

Labels

Components

Fix versions

Affects versions

1.3.0

Priority

Critical
Configure