How to access S3 data from AWS?

Sentinel@AWS
1

Data made available in a Requester Pays Amazon S3 bucket can still be accessed in all the normal ways. However, the requester needs to pass a flag at request time indicating they know they will be charged for the request and any data egress. More details on this concept can be found at https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html

The request mechanism can vary slightly depending on how you access the data currently. Links to some common access mechanisms are below:

A full list of documentation for the various access mechanisms can be found at https://aws.amazon.com/tools/.

2

Does this affect the availability of any Sentinel access with gdal vsis3 driver? It appears that there is no GET request registered at AWS…?

3

You can still use the vsis3 driver, you’ll just need to make sure the AWS_REQUEST_PAYER is enabled as per http://www.gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsis3

1 Like
4

Thanks for your answer. Seems to work for me with Sentinel-1 and L2A data. Do you know where these requests are counted on AWS? I made a few requests with requester pays and vsis3 driver, but none of the GET or billing counter on AWS are changing…? I am absolutely new to AWS…

5

Great to hear that it works. You should be able to see the charges in your Billing overview at https://console.aws.amazon.com/billing/home. You can go to Bills and then see the charges for Simple Storage Service. Note that this isn’t updated in real time and can take a while for numbers to show up.

6

I’ve actually been downloading a lot from this s3 bucket via http request with wget.
Can we just register to AWS, get some credentials then pass that info through wget as authentication and go on as normal but paying out few cents a GB?

7

Ben, that’s correct. You can see how to do this via HTTP at https://docs.aws.amazon.com/AmazonS3/latest/dev/configure-requester-pays-rest.html.

8

Thanks, digging in it looks a bit more complicated as we need to generate a Signature Calculation.

Might just be easier to convert code to one of the S3 command line interfaces unless there is some script floating around to do this?

9

Yeah, I agree, it’s a bit complicated and I haven’t found an easy script to do it searching previously, but don’t let that stop you from looking again.

I’ve found it very easy to use https://aws.amazon.com/cli for scripting purposes and it handles all the signing for you.

10

OK looks like aws s3 cp command line is OK drop in for wget and for in python stuff replace urllib2 with boto3.

e.g.
see newer post…

11

:+1:Glad this is working for you!

12

Well it kicked in and exposed errors in my code. Correct use of boto3 is:

import boto3
import json    
s3 = boto3.resource('s3')
product_info_link = 'tiles/{z}/{b}/{s}/{d}/{p}/productInfo.json'.format(..)
obj = s3.Object('sentinel-s2-l1c', product_info_link).get(RequestPayer='requester')
tile_info = json.loads(obj['Body'].read())

And in aws cli:

aws s3 cp --request-payer requester s3://sentinel-s2-l1c/tiles/{z}/{b}/{s}/{d}/{p}/productInfo.json pi.json

13

Hello everybody.
I have a big problem about downloading Sentinel-2 L1C tiles by Sentinel-EO Browser portal (https://apps.sentinel-hub.com/eo-browser/).
Before the requester pays changes in Amazon S3, I was downloading directly the tiles of interest by a normal web url from EO Browser, like this one: s3://sentinel-s2-l1c/tiles/32/T/QM/2018/9/11/0. I built a simple Matlab script able to link to this url and to download the image that i need.
Now, If I try to access, it asks me an ID key and Security Key using WinSCP program. I have created my own AWS profile and I have my access keys (access key ID and secret access key), but these don’t work and don’t let me to access to downloading.
I would like to have some advices and how I can solve this problem.
Thanks a lot!

14

Hi @francesco.massimetti, I think WinSCP is trying to access the bucket like you own it, which in this case, you do not. You could create a ticket to request S3 Requester Pays support in either WinSCP or Matlab. You can also use https://aws.amazon.com/cli/ to access the data using your credentials. It sounds like you’re comfortable writing scripts, so you may be able to just run a shell script to grab the data using the CLI. Hope that helps!

15

Dear jflasher,
thanks for your fast answer.
I am really sorry, but i don’t understand exactly what you mean.
You say that I can create a ticket to request S3 Requester Pays support in Matlab…How is possibile to do it? Who (about AWS S3 service) I can refer to do these questions?
Moreover, the CLI you mentioned, is it compatible with Matlab?
Thanks again.
F.

16

Hi Francesco, I meant to open a ticket with Matlab support or WinSCP support. S3 Requester Pays is documented at https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html so if it is desired that other tools support that, those tools would need to be updated.

You can see a list of all the available tools (like CLI and SDK) at https://aws.amazon.com/tools/. There is nothing specifically for Matlab, but there are a number of languages supported.

17

Sorry jflasher If I disturb you again, but…
I am simply questioning how now is it possibile to access to the S3 web url (like this one: s3://sentinel-s2-l1c/tiles/32/T/QM/2018/9/11/0, for example), after the requester pays changes. Is it possible that nobody of AWS team is able to answer to this question?
When I try do it through EO Browser looking for the Sentinel-2 tiles of my interest, it is asked to me to give the ID Key and the Security Key. I did a 12 months AWS plan exactly to have this keys but they don’t work. How i can access to this data, If I have an ongoing AWS plan and profile?
The requester pays section (https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) doesn’t answer clearly to this question. It is not clear how to access directly to url path and which keys (ID and password) you have to give.
I followed step by step the instruction reported here in the point numb. 4 (Changes of the access rights to L1C bucket at AWS Public Datasets (Requester Pays)) but these indications are not useful to understand how to access to the previous available data.
Please, anybody can help and inform all the community using Sentinel-2 through AWS services?
Thanks a lot.
F.

18

Hi Francesco, I’ll send you a direct message and we can talk more there off the main thread.

1 Like
21

Hi Ben, would you be willing to share a basic script you use to download a sentinel scene using s3 cp command and then wget?
I am getting started on requesting images from amazon through python and getting lost… Would be really glad to benefit from your experience.
Thanks,