How to access S3 data from AWS?


#1

Data made available in a Requester Pays Amazon S3 bucket can still be accessed in all the normal ways. However, the requester needs to pass a flag at request time indicating they know they will be charged for the request and any data egress. More details on this concept can be found at https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html

The request mechanism can vary slightly depending on how you access the data currently. Links to some common access mechanisms are below:

A full list of documentation for the various access mechanisms can be found at https://aws.amazon.com/tools/.


Changes of the access rights to L1C bucket at AWS Public Datasets (Requester Pays)
AWS Sentinel-2 L2A
#2

Does this affect the availability of any Sentinel access with gdal vsis3 driver? It appears that there is no GET request registered at AWS…?


#3

You can still use the vsis3 driver, you’ll just need to make sure the AWS_REQUEST_PAYER is enabled as per http://www.gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsis3


#4

Thanks for your answer. Seems to work for me with Sentinel-1 and L2A data. Do you know where these requests are counted on AWS? I made a few requests with requester pays and vsis3 driver, but none of the GET or billing counter on AWS are changing…? I am absolutely new to AWS…


#5

Great to hear that it works. You should be able to see the charges in your Billing overview at https://console.aws.amazon.com/billing/home. You can go to Bills and then see the charges for Simple Storage Service. Note that this isn’t updated in real time and can take a while for numbers to show up.


#6

I’ve actually been downloading a lot from this s3 bucket via http request with wget.
Can we just register to AWS, get some credentials then pass that info through wget as authentication and go on as normal but paying out few cents a GB?


#7

Ben, that’s correct. You can see how to do this via HTTP at https://docs.aws.amazon.com/AmazonS3/latest/dev/configure-requester-pays-rest.html.


#8

Thanks, digging in it looks a bit more complicated as we need to generate a Signature Calculation.

Might just be easier to convert code to one of the S3 command line interfaces unless there is some script floating around to do this?


#9

Yeah, I agree, it’s a bit complicated and I haven’t found an easy script to do it searching previously, but don’t let that stop you from looking again.

I’ve found it very easy to use https://aws.amazon.com/cli for scripting purposes and it handles all the signing for you.


#10

OK looks like aws s3 cp command line is OK drop in for wget and for in python stuff replace urllib2 with boto3.

e.g.
see newer post…


#11

:+1:Glad this is working for you!


#12

Well it kicked in and exposed errors in my code. Correct use of boto3 is:

import boto3
import json    
s3 = boto3.resource('s3')
product_info_link = 'tiles/{z}/{b}/{s}/{d}/{p}/productInfo.json'.format(..)
obj = s3.Object('sentinel-s2-l1c', product_info_link).get(RequestPayer='requester')
tile_info = json.loads(obj['Body'].read())

And in aws cli:

aws s3 cp --request-payer requester s3://sentinel-s2-l1c/tiles/{z}/{b}/{s}/{d}/{p}/productInfo.json pi.json