Changes of the access rights to L1C bucket at AWS Public Datasets (Requester Pays)

#1

To all AWS Public Datasets users of S2 L1C data. This thread will be used to communicate changes happening to the S2 bucket. As we expect to gather some feedback from power users in this thread, we might change the approach a bit during the discussion in ongoing. The summarized and harmonized description will be available on this FAQ link.

End of June we will set S2 L1C bucket as “Requester pays” similarly as it is for S2 L2A and S1 GRD. This move is required to allow further increase of the shared data - for continuation of Sentinel-2 L1C coverage, to accommodate global rollout of Sentinel-2 L2A and to host European and then global Sentinel-2 analysis ready mosaics.

How might this affect your workflow:

  1. If you are using the data within AWS EU (Frankfurt) region using S3 protocol, there will be practically no change. There will be some very small additional charge incurred to your account, related to “GET, SELECT and all other Requests”, currently at 0.43$ per 1 million requests.
  2. If you are using the data within any other AWS region using S3 protocol, there will be a small data transfer charge incurred to your account - “Data transfer OUT from Amazon EC2 To Another AWS region”, currently at 0.02 $ per GB. You can avoid this by setting up part (or all) of your processing to AWS EU (Frankfurt) region.
  3. If you are using the data within AWS using HTTP protocol, you will need to sign requests (more info here). You will find examples on how to do it here. We will also upgrade our sentinelhub-py python libraries to work with S3 data soon (by mid of May latest) to make it easier for you.
  4. If you are using data outside of AWS, you should consider setting up AWS instance within EU-1 region. There is free tier available and you might be applicable for AWS research credits. Alternatively see this thread on how it is possible to access data directly over Internet. Note that there will be a small charge for this as well “Data transfer OUT from Amazon EC2 To Internet”, currently at 0.05-0.09 $ per GB

Note that whichever category you fall into above, you will need to make changes to your code to continue to access Sentinel-2 data on AWS. In most cases, it’s as simple as adding a flag to your object request.

As there are several web applications out there making use of Sentinel meta-data, we have decided to make some of these data available over HTTP permanently:

  1. L1C and L2A tiles
  1. L1C and L2A products

In case you believe we should make some other meta-data available in a similar manner, describe the need in the thread bellow.

We understand that this change might be an inconvenience to some users. However, the goal of this experiment is to discover how best to stage data for analysis and processing in the cloud. When the Sentinel Public Dataset was established two years ago, getting hold of Sentinel imagery was quite a challenge. It was distributed in unwieldy chunks and Copernicus SciHub had trouble managing the demand. And there was no other place to get it. At that time, we made a decision to make the data available as easily as possible and as openly as possible. Things are changing now, with the collaborative ground segment running, four DIAS-es coming in a few months, and data generally being more easily accessible. So we believe the time is now right to go back to original purpose – experimenting with how best to stage data for analysis in the cloud.

Best Regards,
Sinergise (Sentinel custodian at AWS Public Datasets)

How to access S-2 meta-data in AWS after bucket goes to Requester pays?
How to access S3 data from AWS?
Orbit number to add into output/ DEM year/month of acquisition
AWS Lambda with sentinelhub-py
Sentinel s2 l1c - 502 error by cloudfront - Unable to access meta-files and preview image from http
#2

Thanks for the information. Just to make sure I’m following correctly on scenario #4, after the switch to requester pays, downloading from the S2 L1C S3 bucket to locations outside of AWS entirely would incur transfer out charges at the rate of $0.05-0.09 per GB? Will there be any changes for the L1C SNS Topic for scene notifications or will that operate the same?

Thanks

#3

You are right. The main cost is related to data transfer. SNS topics will operate in the same way and there are no charges related to that as far as I know.

How to access S-2 meta-data in AWS after bucket goes to Requester pays?
#4

Thanks for the update on the provision of the Sentinel-2 public datasets and all the work you guys put in as custodians.

Alex

#5

Update about sentinelhub Python package:

Since the new version 2.0.1 users can choose to download Sentinel-2 L1C data from S3 instead of http. Instructions are written in documentation. Basically there is now an option to switch to S3 by setting:

$ sentinelhub.config --use_s3_l1c_bucket true

in command line.

Best regards,
Sinergise

How to download Sentinel-2 L1C data by sentinelhub-py after AWS goes to pay?
#6

Hi @gmilcinski, just checking if the S2 L1C bucket been has been switched to Requester pays yet? Thanks!

#7

No, not yet. We want to first ensure proper “proxy” to the meta-data described above. Planned to happen in a couple of weeks.
We will give advanced notice, at least 2 weeks, here, once the date is fixed.

#8

The date when Requester Pays is configured for Sentinel-2 L1C is set to 7th of August 2018.

For those requiring access to meta-data, this post describes how to get it over internet after above mentioned date.

#9

Thanks for update @gmilcinski

#10

This change has now been implemented.

#11

I’m not sure if the data is available on s3 bucket. I’ve been trying to use “aws s3 ls” using this command “aws s3 ls x-amz-request-payer=s3://sentinel-s2-l1c/” I get an error saying
A client error (InvalidBucketName) occurred when calling the ListObjects operation: The specified bucket is not valid.

#12

Hi,

once you set up your AWS credentials this is how you can access the bucket:

aws s3 ls s3://sentinel-s2-l1c/ --request-payer
#13

Good morning, I would like to know how I do to download S2 images, after this change I expect to return.

#14

HI @reginaldo, please check the first post in this thread, it describes how you can access the data stored on AWS.

#15

Good morning, I’m from Brazil, my resource for downloading these images is over the internet. So I would like to know what the steps are that I should take. I thank the attention.

#16

See point 4 from the first post.

#17

Hello @gmilcinski,
I don’t understand how I can simply have the access to the common bucket s3, for example, one like this s3://sentinel-s2-l1c/tiles/33/S/WC/2018/7/12/0

I download from Italy, so in the EU region (I don’t know if AWS EU or EU-1 region, if there are differences), and before the Requester Pays changes, I had access directly to the s3 web path and I got the download via Matlab script.
Could you please give me some advices about which of the different point (1,2,3,4) I have to follow?
Thanks a lot.

#18

Hi @francesco.massimetti, it is not clear from your question, whether you are doing this from within AWS (e.g. using AWS EC3 virtual machine) or from your own computer. If the latter, this falls under point 4. as it is “outside of the region”.
If you are doing it from AWS virtual machine, probably point 3. applies.

#19

Hi @gmilcinski,
No, the download it’s from my own computer or from the computer of the University where I work, through a personal and simple processing script . Anyway, is not from an AWS application or machine. So, I think my situation falls under outside the region.
Thank you!

#20

Hello, but I’m confused when creating the account and start enjoying the new method, because before I used https://search.remotepixel.ca to download the images in .jp2, I’d like to know how to proceed because I’m from Brazil and I’m confused. Thank you for your attention and I apologize for the inconvenience.