I have in the past two weeks noticed that requests for time ranges of more than a few months of sentinel 2 data take much longer than they used to, regardless of whether requesting a small field parcel or a larger region, and whether directly using the Sentinel hub library or using eo-learn
I looked into the inner-workings and find that most of the individual requests (for a single image on a single day) are really fast (less than a second - and are run in parallel), while others take hundreds of seconds. These requests are fast when executed alone. These requests should bring me nowhere near my rate limits and there is no evidence of rate limited requests in the dashboard.
The effect is seen regardless of the spatial location or extent, and regardless of the time range requested as long as it is over a few months.
I have never experienced this problem before and have not changed or updated any of my software (i.e. sentinel hub library), and it occurs on different computers with different operating systems, python versions and library versions.
I hope this can be resolved ass soon as possible because this delay makes my normal workflow impossible.
Can you provide more details about your requests, so that we can analyse the performance?
You are referring in the title to WCS but then you mention “time ranges of more than a few months”, which gives me thinking you are referring to Statistical API.
As many details as possible would be most appreciated, e.g. examples of the slow responses (hide the last few blocks of the instance ID).
Also timings, when you noticed these.
Few ideas:
-it might be that you run your script against a “cold” system, e.g. with not too many servers available to handle your requests, and then you hit it with 500-1.000 requests per minute. Some of these might have to wait to get a dedicated resource. Our system starts scalling up but this process takes a few minutes. It is therefore best to scale up the volume of requests to give time to system to flex.
-there will always be occasional longer-than-usual response time so it is best to take this into account in your script. E.g. if you expect the result to happen in one second, you should give up after 10 seconds and retry.
I believe (but not sure) that sentinelhub-py actually has option to configure this:
* download_timeout_seconds: Maximum number of seconds before download attempt is canceled. (default = 120 sec)
* max_download_attempts: Maximum number of download attempts from a single URL until an error will be raised. (default = 4)
Short update:
I’m using eo-learn to fill eo patches with e.g. S2L2AWCSInput()
By digging into the eo-learn and sentinelhub libraries, I’ve narrowed down what is going on.
eo-learn requires sentinelhub 3.0.0, and in 3.0.0 there is a module called sentinelhub_rate_limit.py.
For some reason, this module is returning “sleep times” of hundreds of seconds even though I have plenty of tokens. I assume there is something wrong with how this module accesses the number of tokens I have and calculates whether my requests would cause rate limited responses, and therefore whether to slow down the download requests in the first place.
As far as I can tell, it does correctly access the current available tokens on my account, but I am investigating further…
However, I have been using this exact set-up for months and this has never happened before until a few weeks ago, so I’m wondering if there is something changed on your side which the library doesn’t handle properly…?
This rate limiting package in sentinelhub-py was introduced fairly recently as we noticed people are throttled quite often (depending on their subscription), which resulted in the process not finishing successfully at all (since there is just a number of retries)
I do highly appreciate this library-side rate limiting, for that reason!
Latest: because I’m testing on small areas (agricultural fields), the processing costs should be quite small. Typically each request is only a fraction of a processing unit. So in testing, according to the dashboard, I’ve made 69 requests in the last hour, but only 3 processing units.
However, the sentinelhub_rate_limit.py code seems to calculate a processing unit cost of >30 for every request, so it very quickly things it has run out of processing unit quota (1,000 per minute). But in the dashboard, I’m clearly not actually using anywhere near that many processing units.
So now I’m trying to figure out how the expected cost in processing units is derived in the script…
Thank you for the insights @barrett. We’re checking out the problem and we’ll take this into account when releasing the new version of sentinelhub-py. The sentinelhub_rate_limit.py hasn’t been used for long so you really might have run into a case we haven’t covered/tested.
A temporary solution might be that you install an older version of sentinelhub-py (2.6.1), which doesen’t yet have the rate-limiting. Since your’re not using the new Processing API, it should work.
Latest:
The problem seems to be in sentinelhub_rate_limit.update
When the code updates the cost_per_second, it compares the remaining_units from the download header with the content of each policy bucket. But I have two separate processing unit quotas: 1k per minute, and 80k per month. The header only returns the 1k per minute remaining tokens, but compares that number to both the previous content of the 1k bucket, and the 80k bucket. When it compares to the 80k bucket, it thinks it has somehow exhausted 79k tokens since the last request, and therefore that each request must take a huge number of processing units.
If the header could return the remaining tokens of each bucket (1k and 80k), into the variable remaining_units, then the solution would be to change this line:
cost_per_second = max(bucket.count_cost_per_second(elapsed_time, remaining_units) for bucket in
self.policy_buckets if not bucket.is_request_bucket())
to something like:
cost_per_second = max(bucket.count_cost_per_second(elapsed_time, rm) for bucket, rm in zip(self.policy_buckets, remaining_units) if not bucket.is_request_bucket())
where remaining units is tuple of values for each bucket.
Just to confirm, I have no way of making the header return the remaining tokens from each of the processing unit buckets, but if I disable checking the costs per second for my 80k bucket (quick and dirty method: just adding “and bucket.capacity < 10000” to the cost_per second update line), my EO patch filling goes back to taking a few seconds (from a few hours), with each individual request taking 1-2 seconds (in parallel).
Therefore, this really does seem to be the problem.
In the newly released sentinelhub-py version 3.0.0 we completely changed the rate limit algorithm. Now it should perform much better. I wrote more info about it in this issue.
Let us know in case you experience any more problems with the download speed.