Problem with downloading bands + cloud data

[magda.kor (Magdalena Kor)]

I am downloading San Diego region data using OAuth2Session to calculate NDVI for 2015-2017 and I realised that

→ Data is only available since 2017
→ Cloud band (CLD) seems to be working since 2019

Is it possible?

The cloud band for 2017-2019 looks like below:

Can someone confirm or deny missing data and bands?

Thanks

[william.ray]

Hi Magdalena,

Can you please confirm what data collection / sensor you are using? In addition, if you are able to share your code too, this will help us replicate your problem, Thanks!

[magda.kor (Magdalena Kor)]

Hi @william.ray
I am using SENTINEL2_L2A with input: [“B02”, “B03”, “B04”, “B08”, “B11”, “B12”, “CLD”].
I have geojson data but in general the Bounding Box Coordinates are xmin=-117.585459, ymin=32.534288, xmax=-114.462801, ymax=33.505395

evalscript_all_bands = """
    function setup() {
        return {
            input: ["B02", "B03", "B04", "B08", "B11", "B12", "CLD"],
    output: {
      bands: 7
    }
  };
}

function evaluatePixel(
  sample,
  scenes,
  inputMetadata,
  customData,
  outputMetadata
) {
  return [sample.B02, sample.B03, sample.B04, sample.B08, sample.B11, sample.B12, sample.CLD];
}
"""

def get_request(slot):
    folder = os.path.join(data_folder, str(slot[1])) 
    
    if os.path.exists(folder):
        return None
    
    request = SentinelHubRequest(
        data_folder=folder,
        evalscript=evalscript_all_bands,
        input_data=[
            SentinelHubRequest.input_data(
                data_collection=DataCollection.SENTINEL2_L2A,
                time_interval=slot,
                mosaicking_order=MosaickingOrder.LEAST_CC
            )
        ],
        responses=[SentinelHubRequest.output_response("default", MimeType.TIFF)],
        geometry=geometry,
        size=betsiboka_size,
        config=config,
    )
    
    return request

list_of_requests = [get_request(slot) for slot in slots if get_request(slot) is not None]
list_of_requests = [request.download_list[0] for request in list_of_requests]

data = SentinelHubDownloadClient(config=config).download(list_of_requests, max_threads=1)

[william.ray]

Hi Magdalena,

If the data collection that you are using is Sentinel-2 L2A then Global coverage is only limited to January 2017 and onwards to the present day as you can find in our documentation. If you require Sentinel-2 data from before then, you can use the Level-1C data collection.

Secondly, I’m unable to replicate the CLD visualisation that you showed in your original post. I requested the CLD band for the same date as you and the result is below:

Did you apply any resampling methods or output the result in a different sampleType from UINT8?

[magda.kor (Magdalena Kor)]

Hi @william.ray,

I used the same methods for all date slots.

[william.ray]

Hi Magdalena,

Thanks, we replicated the issue, and found that this only appears in the overview files (when you are visualising the data at lower resolutions). To solve this, you should explicitly request the data at a higher resolution. Instead of the size parameter you can explicitly define the resolution of your image using the xres and yres parameters in your SentinelHubRequest function. In addition, as the Cloud Probability is a percentage value, and the data is encoded in UINT8 (pixel values 0-255), it’s recommended to divide the band by 100. I’ve provided an example of this in an evalscript below for you:

//VERSION=3

function setup() {
  return {
    input: ["CLD"],
    output: { bands: 3 }
  };
}

function evaluatePixel(sample) {
  return [
    sample.CLD/100,
    sample.CLD/100,
    sample.CLD/100
  ];
}

If you’re still encountering issues, then let us know

[magda.kor (Magdalena Kor)]

Hi @william.ray,

thank you for the reply. Unfortunately I am not able to provide resolution instead of size since every higher resolution fails:

DownloadFailedException: Failed to download from:
https://services.sentinel-hub.com/api/v1/process
with HTTPError:
400 Client Error: Bad Request for url: https://services.sentinel-hub.com/api/v1/process
Server response: “{“status”: 400, “reason”: “Bad Request”, “message”: “Your request of 304632.92 meters per pixel exceeds the limit 1500.00 meters per pixel of the collection S2L2A. Please revise the resolution (or corresponding width/height) to make sure it is in supported range.”, “code”: “RENDERER_EXCEPTION”}”

def get_request(slot):
    folder = os.path.join(data_folder, str(slot[1])) 
    
    if os.path.exists(folder):
        return None
    
    request = SentinelHubRequest(
        data_folder=folder,
        evalscript=evalscript_all_bands,
        input_data=[
            SentinelHubRequest.input_data(
                data_collection=DataCollection.SENTINEL2_L2A,
                time_interval=slot,
                mosaicking_order=MosaickingOrder.LEAST_CC
            )
        ],
        responses=[SentinelHubRequest.output_response("default", MimeType.TIFF)],
        geometry=geometry,
    #    size=betsiboka_size,
        resolution=(60,60), 
        config=config,
    )
    
    return request

That’s why I was using size (2500, 617) instead.

[william.ray]

Hi Magdalena,

As you have noted the reason for the failure is because the size of the image that you are requesting is too large exceeding the 2500x2500 limit of the Process API.

For larger requests, there are three approaches that you can take:

• Use the Batch Processing API. This is only available for enterprise users. If you don’t have an enterprise account, and would like to try it out then we can organise a short trial period to test this functionality out.
• We also offer the Asynchronous Process API which allows you to process more data with a single request than Processing API. You can request areas up to 10,000x10,000 pixels. This differs from Process API in that the results are not instantly returned but delivered to object storage.
• Lastly, as you are familiar with Python, you could try using the Large Area Utilities tools in the Sentinel Hub Python package.

Hope that this information helps you out!