"Out of retries" HTTP 500 error


#1

Hi, I’m using Python and sentinelhub-py version 2.4.1 to fetch the full Sentinel2 image history of approx. 300 agricultural fields. I know this will put a large load on the web-service which may return HTTP 500 errors due to the load. I have tried to handle such errors by setting the following sentinelhub-py settings:

from sentinelhub.config import SHConfig
sh_config = SHConfig()
sh_config.max_download_attempt = 5
sh_config.download_sleep_time = 60
sh_config.download_timeout_seconds = 30

However, using these settings I now get the following HTTP 500 error for multiple of my requests:

Exception: DownloadFailedException('Failed to download with HTTPError:\n500 Server Error: Internal Server Error for url: https://services.sentinel-hub.com/ogc/wms/[INSTANCE_ID]?SERVICE=wms&BBOX=507395.0%2C6085280.0%2C507610.0%2C6085775.0&FORMAT=image%2Ftiff%3Bdepth%3D32f&CRS=EPSG%3A32632&WIDTH=43&HEIGHT=99&LAYERS=S2-L2A&REQUEST=GetMap&TIME=2018-10-10T08%3A40%3A18%2F2018-10-10T12%3A40%3A18&MAXCC=5.0&Downsampling=BILINEAR&Upsampling=BILINEAR&Transparent=False&ShowLogo=False\nServer response: "Out of retries"',)

I cannot find any documentation of this “Out of retries” error. I tried to switch IP but I still get it, so I guess that the error is bound to my INSTANCE_ID.

Is there any documentation of this HTTP error and how long is my INSTANCE_ID “locked” from performing this request?

Additinal information:
It might be helpful to know I also receive other types of HTTP 500 errors, which I also cannot understand the reason of:

Regards Peter Fogh

UPDATE
After looking more at the errors, I see that all of my received errors are for requests at the date “2018-10-10T08%3A40%3A18%2F2018-10-10T12%3A40%3A18” (i.e. 2018-10-10T08:40:18/2018-10-10T12:40:18), so I suspect that SentinelHub has and internal error of processing the data from this date?


#2

Hi Peter,

The Sentinel Hub services have been recently updated and now better limit the number of requests a user can do per minute. Now we are working on updating sentinelhub Python package to better handle these changes.

For now I recommend setting parameter max_threads of DataRequest.get_data and DataRequest.save_data methods to a smaller number which will limit the number of parallel requests and therefore decrease the number of requests per minute. Now by default number of parallel requests is 5 time number of processors you are using.

We will also check the other two errors you reported. Thanks for letting us know about this. We will keep you posted about the progress.

Best regards,
Matej


#3

Hi Matej

Okay. Looking forward to the better error messages by sentinel-py :smile:

We have already implemented the max_threads restriction to 1 thread per process, as we normally run with 20 processes in parallel, and we get no errors when fetching data before the 2018-10-10. So, I guess we got the “Out of tries” error because the data from the 2018-10-10 has some other error (like the “java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException”), which results in multiple attempts to fetch the data and finally resulting in the “Out of tries” error.


#4

Update:
We examined all 3 errors and it turns out they are all caused by some internal problem in Sentinel Hub services. You are right, the problem seems to be limited to 2018-10-10 and L2A data in that area. Our guys in core service team are already working on fixing it.


#5

The issue related to L2A products was due to processing baseline change (https://cophub.copernicus.eu/news/News00235) and should be fixed.