403 errors accessing OpenSearch via sentinelhub-py in AWS

Bugs & Features
1

Good morning folks,

As of this morning a process I run to get valid scenes over a given area has started failing with 403 errors from the OpenSearch API. My most recent successful run was on the 5th, so I’m not sure when in the interval this started.

The traceback:

[ERROR] DownloadFailedException: Failed to download from:
http://opensearch.sentinel-hub.com/resto/api/collections/Sentinel2/search.json?startDate=2020-10-10T00%3A00%3A00&completionDate=2023-01-09T23%3A59%3A59&box=-64.81561785%2C-8.22966416%2C-63.81704292%2C-7.23368282&maxRecords=500&index=1
with HTTPError:
403 Client Error: Forbidden for url: http://opensearch.sentinel-hub.com/resto/api/collections/Sentinel2/search.json?startDate=2020-10-10T00%3A00%3A00&completionDate=2023-01-09T23%3A59%3A59&box=-64.81561785%2C-8.22966416%2C-63.81704292%2C-7.23368282&maxRecords=500&index=1
Server response: ""
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 244, in lambda_handler
    main(granule, ignore_prior_ll=ignore_prior_ll, early_date=early_date,
  File "/var/task/lambda_function.py", line 215, in main
    imagery_dict = query(granule, minx, miny, maxx, maxy, early_date, latest_used,
  File "/var/task/lambda_function.py", line 142, in query
    for tile_info in get_area_info(search_bbox, search_time_interval):
  File "/opt/python/sentinelhub/opensearch.py", line 172, in search_iter
    response = client.get_json(url)
  File "/opt/python/sentinelhub/download/client.py", line 212, in get_json
    return self._single_download(request, decode_data=True)
  File "/opt/python/sentinelhub/download/client.py", line 120, in _single_download
    response_content = self._execute_download(request)
  File "/opt/python/sentinelhub/download/handlers.py", line 44, in new_download_func
    return download_func(self, request)
  File "/opt/python/sentinelhub/download/handlers.py", line 27, in new_download_func
    raise DownloadFailedException(_create_download_failed_message(exception, request.url)) from exception

And the offending line of code:
for tile_info in get_area_info(search_bbox, search_time_interval):
(search_bbox and search_time_interval are calculated earlier in the code)

The strange thing is, I can copy the URL that failed and paste it into my browser and get a JSON listing no problem. The failure is occurring in an AWS Lambda function, so I’m wondering if that’s the source of the problem (perhaps sentinelhub is denying queries from AWS?)…but if it is, I’m not sure what to do. Any advice would be appreciated.

2

Since it might be relevant - I am not passing credentials in any way (as I understand it, OpenSearch is supposed to not require authentication, and I wasn’t prompted for credentials when connecting from my local machine)

3

Hi Chris,

could you perhaps change your process to use our (STAC compliant) Catalog API rather than Open Search? Point being that Open Search API is not one of our operational APIs, it is rather a “community effort”, and therefore not as performant.

As you are using sentinelhub-py, it should be straightforward, as that one is using Catalog API by default.

I will try to find out what is the specific problem you stumbled upon in Open Search. But as said, I would strongly recommend to go to Catalog API instead. It is faster, operationally maintained and always in sync.

Best,
Grega

1 Like
4

Grega,

Certainly, we can try to make that switch. Appreciate the suggestion.

Best,
Chris

5

I got some further information. It seems someone (might be your process?) was making a ton of request to Open Search - something that the service deployed there was not scaled to, and the database went down. After re-instating it they introduced a limit, to prevent the current heavy process to crash it again.

Catalog API is indeed the right option for heavy use…

6

Understood. It definitely wasn’t us specifically, I wonder if it was just someone else from AWS who was generating all the traffic. Regardless, we’ve made the switch to the STAC API and everything seems good now. Thanks for your help!

1 Like
7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.