Issues using SentinelHubEvalscriptTask

Good afternoon,

I am trying to use SentinelHubEvalscriptTask to retrieve some statistics on a specific AOI. I want to evaluate the cloud cover over a specific AOI (and not just the whole EOPatch) before downloading it.
I thought of using SentinelHubEvalscriptTask ot do so, but unfortunately I did not find many examples online (here and there), so I’m kinda moving in the dark here.

I keep getting the same error when trying to run this code :

The code
evalscript = """
//VERSION=3
function setup() {
    return {
        input: [{
            bands: [
                "CLM",
                "dataMask"
                ]
            }],
            output: [
                {
                id:"data",
                bands: 1
                },
                {
                id: "dataMask",
                bands: 1
                }]
        }
    }

    function evaluatePixel(samples) {
        var sum = 0;
        for (var i = 0; i < samples.length; ++i) {
            if (samples[i].CLM === 1) {
                sum++;
            }
        }
        return [sum / samples.length];
    }
"""

cloud_over_aoi_task = SentinelHubEvalscriptTask(
            features=[(FeatureType.DATA, "CLM")],
            evalscript=evalscript,
            data_collection=DataCollection.SENTINEL2_L1C,
            resolution=resolution,
            maxcc=maxcc,
            config=config,
        )

result = cloud_over_aoi_task.execute(
            bbox = bbox_obj,
            time_interval = eopatch_date
        )

The error being :

DownloadFailedException: Failed to download from:
https://services.sentinel-hub.com/api/v1/process
with HTTPError:
400 Client Error: Bad Request for url: https://services.sentinel-hub.com/api/v1/process
Server response: "{"status": 400, "reason": "Bad Request", "message": "Output CLM requested but missing from function setup()", "code": "COMMON_BAD_PAYLOAD"}"
The detailed error
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
File ~/Documents/geocodis/dev-geocodis/lib/python3.10/site-packages/sentinelhub/download/handlers.py:40, in fail_user_errors.<locals>.new_download_func(self, request)
     39 try:
---> 40     return download_func(self, request)
     41 except requests.HTTPError as exception:

File ~/Documents/geocodis/dev-geocodis/lib/python3.10/site-packages/sentinelhub/download/sentinelhub_client.py:95, in SentinelHubDownloadClient._execute_download(self, request)
     93     continue
---> 95 response.raise_for_status()
     97 LOGGER.debug("Successful %s request to %s", request.request_type.value, request.url)

File ~/Documents/geocodis/dev-geocodis/lib/python3.10/site-packages/requests/models.py:1021, in Response.raise_for_status(self)
   1020 if http_error_msg:
-> 1021     raise HTTPError(http_error_msg, response=self)

HTTPError: 400 Client Error: Bad Request for url: https://services.sentinel-hub.com/api/v1/process

The above exception was the direct cause of the following exception:

DownloadFailedException                   Traceback (most recent call last)
Cell In[46], line 1
----> 1 result = cloud_over_aoi_task.execute(
      2             bbox = bbox_obj,
      3             time_interval = eopatch[1]
      4         )

File ~/Documents/geocodis/dev-geocodis/lib/python3.10/site-packages/eolearn/io/sentinelhub_process.py:124, in SentinelHubInputBaseTask.execute(self, eopatch, bbox, time_interval, geometry)
    122 session = None if self.session_loader is None else self.session_loader()
    123 client = SentinelHubDownloadClient(config=self.config, session=session)
--> 124 responses = client.download(requests, max_threads=self.max_threads)
    125 LOGGER.debug("Downloads complete")
    127 temporal_dim = 1 if timestamps is None else len(timestamps)

File ~/Documents/geocodis/dev-geocodis/lib/python3.10/site-packages/sentinelhub/download/sentinelhub_client.py:68, in SentinelHubDownloadClient.download(self, *args, **kwargs)
     66 self.lock = Lock()
     67 try:
---> 68     return super().download(*args, **kwargs)
     69 finally:
     70     self.lock = None

File ~/Documents/geocodis/dev-geocodis/lib/python3.10/site-packages/sentinelhub/download/client.py:103, in DownloadClient.download(self, download_requests, max_threads, decode_data, show_progress)
    101 except DownloadFailedException as download_exception:
    102     if self.raise_download_errors:
--> 103         raise download_exception
    105     warnings.warn(str(download_exception), category=SHRuntimeWarning)
    107 if progress_bar:

File ~/Documents/geocodis/dev-geocodis/lib/python3.10/site-packages/sentinelhub/download/client.py:100, in DownloadClient.download(self, download_requests, max_threads, decode_data, show_progress)
     98 for future in as_completed(download_list):
     99     try:
--> 100         results[future_order[future]] = future.result()
    101     except DownloadFailedException as download_exception:
    102         if self.raise_download_errors:

File /usr/lib/python3.10/concurrent/futures/_base.py:451, in Future.result(self, timeout)
    449     raise CancelledError()
    450 elif self._state == FINISHED:
--> 451     return self.__get_result()
    453 self._condition.wait(timeout)
    455 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:

File /usr/lib/python3.10/concurrent/futures/_base.py:403, in Future.__get_result(self)
    401 if self._exception:
    402     try:
--> 403         raise self._exception
    404     finally:
    405         # Break a reference cycle with the exception in self._exception
    406         self = None

File /usr/lib/python3.10/concurrent/futures/thread.py:58, in _WorkItem.run(self)
     55     return
     57 try:
---> 58     result = self.fn(*self.args, **self.kwargs)
     59 except BaseException as exc:
     60     self.future.set_exception(exc)

File ~/Documents/geocodis/dev-geocodis/lib/python3.10/site-packages/sentinelhub/download/client.py:116, in DownloadClient._single_download_decoded(self, request)
    114 def _single_download_decoded(self, request: DownloadRequest) -> Any:
    115     """Downloads a response and decodes it into data. By decoding a single response"""
--> 116     response = self._single_download(request)
    117     return None if response is None else response.decode()

File ~/Documents/geocodis/dev-geocodis/lib/python3.10/site-packages/sentinelhub/download/client.py:129, in DownloadClient._single_download(self, request)
    127 no_local_data = self.redownload or response_path is None or not os.path.exists(response_path)
    128 if no_local_data:
--> 129     response = self._execute_download(request)
    130 else:
    131     if not request.return_data or response_path is None:

File ~/Documents/geocodis/dev-geocodis/lib/python3.10/site-packages/sentinelhub/download/handlers.py:67, in retry_temporary_errors.<locals>.new_download_func(self, request)
     65 for attempt_idx in range(download_attempts):
     66     try:
---> 67         return download_func(self, request)
     69     except requests.RequestException as exception:  # noqa: PERF203
     70         attempts_left = download_attempts - (attempt_idx + 1)

File ~/Documents/geocodis/dev-geocodis/lib/python3.10/site-packages/sentinelhub/download/handlers.py:46, in fail_user_errors.<locals>.new_download_func(self, request)
     41 except requests.HTTPError as exception:
     42     if (
     43         exception.response.status_code < requests.status_codes.codes.INTERNAL_SERVER_ERROR
     44         and exception.response.status_code != requests.status_codes.codes.TOO_MANY_REQUESTS
     45     ):
---> 46         raise DownloadFailedException(
     47             _create_download_failed_message(exception, request.url), request_exception=exception
     48         ) from exception
     49     raise exception from exception

DownloadFailedException: Failed to download from:
https://services.sentinel-hub.com/api/v1/process
with HTTPError:
400 Client Error: Bad Request for url: https://services.sentinel-hub.com/api/v1/process
Server response: "{"status": 400, "reason": "Bad Request", "message": "Output CLM requested but missing from function setup()", "code": "COMMON_BAD_PAYLOAD"}"

I assume it’s some issue with the input/output of the evalscript but I tried different codes and didn’t manage to run it.

I thank you in advance for your help and wish you a pleasant evening.

Tom

Hi Tom,

Looking at your code and the information you’ve shared, I think you are trying to use Statistical API in conjunction with EO-Learn? This is not something that is possible. If you are looking to generate statistics for an AOI, you should only use Statistical API separately from the EOPatch.

EDIT2 & SOLVED : I have no clue why but I ran everything again this morning and it’s working : no more shifted dates, cloud clover in accordance of what can be seen on EO Browser.

Hi William,

Thank you for your help, it makes more sense now. I have followed this example then to do so, and it seems to work just fine when I reproduce the code with your settings.
However, when I customize with my settings, it appears I have issues. Here is the code :

The code
# evalscript is the same as your example
evalscript = """
//VERSION=3
function setup() {
    return {
        input: [{
            bands: [
                "CLM",
                "dataMask"
            ]
        }],
        output: [
            {
                id: "data",
                bands: 1
            },
            {
                id: "dataMask",
                bands: 1
            }]
    }
}
function evaluatePixel(samples) {
    return {
        data: [samples.CLM],
        dataMask: [samples.dataMask]
        }
}
"""

stats_request = {
    "input": {
        "bounds": {
            "bbox": bbox,
            "properties": {
                "crs": "http://www.opengis.net/def/crs/EPSG/0/32633"
                }
            },
            "data": [
                {
                    "type": "sentinel-2-l1c", # I want L1C data (L2A in your example)
                    "dataFilter": {
                        "mosaickingOrder": "leastRecent"
                    }
                }
            ]
    },
    "aggregation": {
        "timeRange": {
            "from": start_date_str,
            "to": end_date_str
        },
        "aggregationInterval": {
            "of": "P1D"
        },
        "evalscript": evalscript,
        "resx": 10,
        "resy": 10
  }
}

headers = {
    'Content-Type': 'application/json',
    'Accept': 'application/json'
}

url = "https://services.sentinel-hub.com/api/v1/statistics"
response = oauth.request("POST", url=url, headers=headers, json=stats_request)
sh_statistics = response.json()

dates_without_clouds = [(data["interval"], int(100 * data["outputs"]["data"]['bands']['B0']['stats']['mean']) ) for data in sh_statistics["data"]]

for item in dates_without_clouds:
    print( item )

The output is :

The output
({'from': '2023-07-05T00:00:00Z', 'to': '2023-07-06T00:00:00Z'}, 100)
({'from': '2023-07-10T00:00:00Z', 'to': '2023-07-11T00:00:00Z'}, 100)
({'from': '2023-07-15T00:00:00Z', 'to': '2023-07-16T00:00:00Z'}, 100)
({'from': '2023-07-20T00:00:00Z', 'to': '2023-07-21T00:00:00Z'}, 100)
({'from': '2023-07-25T00:00:00Z', 'to': '2023-07-26T00:00:00Z'}, 100)
({'from': '2023-07-30T00:00:00Z', 'to': '2023-07-31T00:00:00Z'}, 100)

I have 2 questions about the output :

  • First, the dates don’t seem to correspond to the actual images : based on a request I made on the SH catalog, 5 images are available (4th, 9th, 14th, 19th and 29th of July) for the specific AOI (Ljubljana) , which can be confirmed with the EO Browser. However, the output gives 1 extra date and the others seem to be shifted and I don’t understand why (I understand these are statistics on a period of time and not an image, but I don’t get how the statistic can be computed if there is no image in this time period);

  • Secondly, if I understand correctly, the announced cloud cover is always 100%, while it can be seen on EOBrowser that this is not the case for July 2023. I am thinking it might come from the shifted dates, as the statistics might not be processed on actual images.

Else thank you again for your reply and documentation !

EDIT : I have used your Request Builder to try my request with the same settings and evalscript I use in my code and the results appears to be consistent with what can be observed on EO Browser so I’m even more confused about why does the code does not work.