Add S3 proxy range reading

Description

What ?

Add support to range reading. Implementation should support W3C range header specification http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35

Why ?

For several reason we need to read an object through S3 proxy using a byte range in order to only pick a small byte range of the object.

Environment

None

Activity

Show:
Bin Fan
October 2, 2018, 10:06 PM

Thanks for proposing this feature.

could you elaborate on the Alluxio interface / API you plan to use this range reading feature? Essentially, if Alluxio adds a support to S3 proxy range reading, when and how do you interact with Alluxio?

François Autaa
October 11, 2018, 12:59 PM

Hi

I’ve build my own modification on this feature to do what I expect. I will try to make a pull request on with my modification. It works as desired and Boto3 can add range to the request made through S3 Rest proxy.

I’ve basically updated the S3 REST handler to allow a range passed has parameter into header of the query. And then I only return the bytes corresponding to this range request.

The plan is quite simple in some case we only read part of a file in the Alluxio storage through S3 Rest interface.

I will let you know.

Regards

Bin Fan
October 12, 2018, 12:00 AM

thanks for the clarification. we will be happy to see the change.

Assignee

Unassigned

Reporter

François Autaa

Labels

None

Components

Affects versions

Priority

Major
Configure