To all AWS Public Datasets users of S2 L1C data. This thread will be used to communicate changes happening to the S2 bucket. As we expect to gather some feedback from power users in this thread, we might change the approach a bit during the discussion in ongoing. The summarized and harmonized description will be available on this FAQ link.
End of June we will set S2 L1C bucket as “Requester pays” similarly as it is for S2 L2A and S1 GRD. This move is required to allow further increase of the shared data - for continuation of Sentinel-2 L1C coverage, to accommodate global rollout of Sentinel-2 L2A and to host European and then global Sentinel-2 analysis ready mosaics.
How might this affect your workflow:
If you are using the data within AWS EU (Frankfurt) region using S3 protocol, there will be practically no change. There will be some very small additional charge incurred to your account, related to “GET, SELECT and all other Requests”, currently at 0.43$ per 1 million requests.
If you are using the data within any other AWS region using S3 protocol, there will be a small data transfer charge incurred to your account - “Data transfer OUT from Amazon EC2 To Another AWS region”, currently at 0.02 $ per GB. You can avoid this by setting up part (or all) of your processing to AWS EU (Frankfurt) region.
If you are using the data within AWS using HTTP protocol, you will need to sign requests (more info here). You will find examples on how to do it here. We will also upgrade our sentinelhub-py python libraries to work with S3 data soon (by mid of May latest) to make it easier for you.
Note that whichever category you fall into above, you will need to make changes to your code to continue to access Sentinel-2 data on AWS. In most cases, it’s as simple as adding a flag to your object request.
As there are several web applications out there making use of Sentinel meta-data, we have decided to make some of these data available over HTTP permanently:
In case you believe we should make some other meta-data available in a similar manner, describe the need in the thread bellow.
We understand that this change might be an inconvenience to some users. However, the goal of this experiment is to discover how best to stage data for analysis and processing in the cloud. When the Sentinel Public Dataset was established two years ago, getting hold of Sentinel imagery was quite a challenge. It was distributed in unwieldy chunks and Copernicus SciHub had trouble managing the demand. And there was no other place to get it. At that time, we made a decision to make the data available as easily as possible and as openly as possible. Things are changing now, with the collaborative ground segment running, four DIAS-es coming in a few months, and data generally being more easily accessible. So we believe the time is now right to go back to original purpose – experimenting with how best to stage data for analysis in the cloud.
Best Regards,
Sinergise (Sentinel custodian at AWS Public Datasets)
Thanks for the information. Just to make sure I’m following correctly on scenario #4, after the switch to requester pays, downloading from the S2 L1C S3 bucket to locations outside of AWS entirely would incur transfer out charges at the rate of $0.05-0.09 per GB? Will there be any changes for the L1C SNS Topic for scene notifications or will that operate the same?
You are right. The main cost is related to data transfer. SNS topics will operate in the same way and there are no charges related to that as far as I know.
Since the new version 2.0.1 users can choose to download Sentinel-2 L1C data from S3 instead of http. Instructions are written in documentation. Basically there is now an option to switch to S3 by setting:
No, not yet. We want to first ensure proper “proxy” to the meta-data described above. Planned to happen in a couple of weeks.
We will give advanced notice, at least 2 weeks, here, once the date is fixed.
I’m not sure if the data is available on s3 bucket. I’ve been trying to use “aws s3 ls” using this command “aws s3 ls x-amz-request-payer=s3://sentinel-s2-l1c/” I get an error saying
A client error (InvalidBucketName) occurred when calling the ListObjects operation: The specified bucket is not valid.
Good morning, I’m from Brazil, my resource for downloading these images is over the internet. So I would like to know what the steps are that I should take. I thank the attention.
Hello @gmilcinski,
I don’t understand how I can simply have the access to the common bucket s3, for example, one like this s3://sentinel-s2-l1c/tiles/33/S/WC/2018/7/12/0
I download from Italy, so in the EU region (I don’t know if AWS EU or EU-1 region, if there are differences), and before the Requester Pays changes, I had access directly to the s3 web path and I got the download via Matlab script.
Could you please give me some advices about which of the different point (1,2,3,4) I have to follow?
Thanks a lot.
Hi @francesco.massimetti, it is not clear from your question, whether you are doing this from within AWS (e.g. using AWS EC3 virtual machine) or from your own computer. If the latter, this falls under point 4. as it is “outside of the region”.
If you are doing it from AWS virtual machine, probably point 3. applies.
Hi @gmilcinski,
No, the download it’s from my own computer or from the computer of the University where I work, through a personal and simple processing script . Anyway, is not from an AWS application or machine. So, I think my situation falls under outside the region.
Thank you!
Hello, but I’m confused when creating the account and start enjoying the new method, because before I used https://search.remotepixel.ca to download the images in .jp2, I’d like to know how to proceed because I’m from Brazil and I’m confused. Thank you for your attention and I apologize for the inconvenience.
