How to access S-2 meta-data in AWS after bucket goes to Requester pays?


#1

On 7th of August 2018 AWS Sentinel-2 L1C bucket will be configured as Requester pays.

To ensure continuous support of various Sentinel-2 browsers we have implemented a service, which will provide permanent access to the following meta-data:

One can also list

Update 2018-08-06 - the date was wrongly listed as 2019. We corrected it to 2018. As it was always stated that things will happen in 2018, we hope that readers got the right info.


Changes of the access rights to L1C bucket at AWS Public Datasets (Requester Pays)
Changes of the access rights to L1C bucket at AWS Public Datasets (Requester Pays)
Changes of the access rights to L1C bucket at AWS Public Datasets (Requester Pays)
#2

Thanks a lot Grega,

will this be a service free of charge?

thanks in advance,
Marcela


#3

Yes, this service is part of Register of open data (RODA), AWS’ initiative. Access does not require any authentication and there are no charges attached.


#4

@gmilcinski

In your example link above to list available products, the results are capped at 1,000 items. Reviewing the S3 docs [1], it seems like the “NextContinuationToken” value should be included as the request parameter ‘continuation-token’ in order to fetch the subsequent pages. When I tested it however, I always seem to get back the first page only. Is there something else needed to page the results or is the parameter maybe not being forwarded to S3 properly?

[1] https://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html


#5

The paging is not supported unfortunately (and it was not before) as this service was not really meant for crawling the data. If you are interested to get all the products in specific area, use OpenSearch.
http://opensearch.sentinel-hub.com/resto/api/collections/Sentinel2/describe.xml


#6

Edit: I got it now, this is just for the meta data


#7

I am new to this. You state you have implemented a service to provide access the metadata - where is this service? I can only link to the examples. I’m looking for tileInfo.jsons.


#8

Service is available over standard web-interface, you simply need to combine an URL as described in the first post in this thread:


#9

Many thanks for your great work, Grega.
I cannot access to metadata.xml - e.g. https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/16/R/BV/2018/4/16/0/metadata.xml
The error is “503 ERROR The request could not be satisfied.”
Any solution?
Thaks a lot for your kind support.
Naoto


#10

Hi Naoto,
I clicked the link you have provided and it works fine for me.
We have noticed that AWS Lambda sometimes simply does not work and we do not know, what is the reason for that. However, this happens rarely and few minutes later usually works fine.
Is the problem persisting?


#11

Thanks a lot for your prompt reply!
Unfortunately, the problem remains and no access to “metadata.xml”
Because of this problem, the download command (e.g., sentinelhub.aws --tile T54HVH 2017-04-14 -e) doesn’t work.
I can access to “tileInfo.json” and “productInfo.json” but no access to “metadata.xml” and “preview.jpg”.
I will try alternative ways of downloading data.


#12

Thanks a lot about the news.

But a have a doubt, maybe it is a problem because that changes. (This is a big question here)

Into my context, i get the access to the address in the first moment:
https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/29/S/PC/2018/8/23/0/

But in my process i need to do download file .jp2, so the complement here is:
https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/29/S/PC/2018/8/23/0/B8A.jp2

And download the file into my repository images is the end of my process, after… i will render it.

This scenario throw me a message error:

AccessDenied
Access Denied

Why this error code? I’m runnig my tests locally and i’m in in Europe region.

I appreciate any help.
Thanks,
Eder


#13

Hi Eder,
as described in the first post in this thread, only meta-data are available over this interface. B8A.jp2 is not meta-data (see a list of available files on top).
For your case you need to use signed requests as described in this post, check options 1-4.


#14

Hi,
I found that the last update of sentinelhub-py (vision = 2.4.1) on GitHub, and the config.json has changed from
"aws_base_url": "http://sentinel-s2-l1c.s3.amazonaws.com/"
to
"aws_metadata_base_url": "https://roda.sentinel-hub.com/"
and no “aws base url” anyway.

Is it means we cannot download S2 images as before by sentinelhub-py 2.4.1 because of these changes?

I have used the old version (2.0.1) to download image data (.jp2) successfully, which can replace the new version.

Regards


#15

Hi @zheng.tang ,

you are correct - older versions of sentinelhub-py are now outdated and cannot download S2 images from AWS anymore. sentinelhub-py serves only as an interface for download and the data is not available at “http://sentinel-s2-l1c.s3.amazonaws.com/” anymore. Therefore I recommend upgrading the package.

By the way, sentinelhub-py 2.4.2 has just been released with a few improvements in handling jpeg 2000 images.


#16

Hi @maleksandrov ,

That’s strange!

I am downloading images by using the old version, and the url is “http://sentinel-s2-l1c.s3.amazonaws.com/”

But I can’t download images by version 2.4.1, there’s some bugs with connecting to AWS S3 buckets, and the url is “https://roda.sentinel-hub.com”.


#17

I assume the version 2.0.1 still works because you have set parameter "use_s3_l1c_bucket": true which then downloads everything from s3 bucket instead of “http://sentinel-s2-l1c.s3.amazonaws.com/”. In newer versions those metadata files which are available from “https://roda.sentinel-hub.com” are downloaded from there, hence slightly less requests to s3.

It is possible that newer versions are causing you problems because we changed library for reading Jpeg 2000 images from OpenCV to Pillow. Even though Pillow doesn’t raise errors during installation sometimes its capabilities to read Jpeg 2000 images don’t get installed properly (especially on Windows). Hence installation from wheel is recommended.

Anyway if you will still have some problems with the newest version of sentinelhub-py please create a new thread on this forum about it or raise an issue on Github. We will be happy to help you resolve the problem.


#18

Hi @maleksandrov,

Yes, I set "use_s3_l1c_bucket" as true, and filled my aws id and key in "aws_access_key_id" and "aws_secret_access_key". Then I can download anything in S3.

Is that means I can download metadata only in “https://roda.sentinel-hub.com”? Then I need to write a new program to access S3 to download jp2 images.

Regards


#19

Everything else, jp2 images included, will still be downloaded from s3 just like before. Only those metadata files which are available at “https://roda.sentinel-hub.com” will be downloaded from there. This is all handled internally by sentinelhub-py, hence your code would still work without having to make any changes.


#20

Hi, when I try to download images, when sentinelhub.aws try to get the metadata I get the error:

503 ERROR
The request could not be satisfied.

Even if I access the example (https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/16/R/BV/2018/4/16/0/metadata.xml) directly from the browser I get the same error.