Always receiving all zero images through a Sentinel 5p API request

Hi all,

I am relatively new to using the Sentinelhub API and am having issues with receiving S5p NO2 data. I have a list of point coordinates which I iteratively (not so neatly) convert to a rough Bbox with a certain buffer size. Based on this BBox I should receive an image of the NO2 values in that area, but whereever I try to do this I always get only zeros, also in the dataMask. So this tells me the data is not available, but I find that hard to believe given the sheer size of the request and the time_frame which can be up to a year. Is there something wrong with my request? See the code below.
Note that I can choose between S2L1 and S5p. S2L1 works fine.

from sentinelhub import DataCollection
from sentinelhub import SHConfig, BBox, CRS, SentinelHubRequest, MimeType, bbox_to_dimensions
import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import h5py
import datetime


# Define parameters 
MAIN_FOLDER = os.getcwd()
input_path = os.path.join(MAIN_FOLDER, 'Data')
output_path = os.path.join(MAIN_FOLDER, 'Data')
if not os.path.exists(output_path):
    os.makedirs(output_path)

csv_file_name = 'df_weekly_august.csv'
satellite_type = 'S5p' # S5p
if satellite_type == 'S2A':
    time_interval = ('2021-08-01', '2021-08-31')
    buffer = 1200 # in meters

elif satellite_type == 'S5p': # Need to get data for every day within this time_interval
    time_interval = ('2021-08-01', '2021-08-31')
    buffer = 20000 # in meters

# !!! Ideally this part would create a 1200 m directly around the point instead of 
#  using coordinates to do that !!!
crs = CRS.WGS84
# Approximation WGS84
# 1 degree of latitude corresponds to 111km
# 1 degree of longitude corresponds to 73km
long_adj = (buffer/67000)/2
lat_adj = (buffer/112000)/2

resolution = 10 # 10m resolution is used in paper

# Configure login details
config = SHConfig()
config.instance_id='11d91198-MASKED'
config.sh_client_id='be4369f0-MASKED'
config.sh_client_secret='#Vtzd8)-MASKED'

if config.sh_client_id == '' or config.sh_client_secret == '':
    print("Warning! To use Sentinel Hub services, please provide the credentials (client ID and client secret).")
    

#%%     
    
# https://apps.sentinel-hub.com/requests-builder/
if satellite_type == 'S2A':
    dataCollectionType = DataCollection.SENTINEL2_L1C
    evalscript = """
    //VERSION=3
    function setup() {
        return {
            input: [{
                bands: ["B01","B02","B03","B04","B05","B06","B07","B08","B8A","B09","B10","B11","B12","CLM"],
                units: "DN"
            }],
            output: {
                bands: 14,
                sampleType: "FLOAT32"
            }
        };
    }

    function evaluatePixel(sample) {
        if (sample.CLM == 1) {
            return [sample.B01,
                    sample.B02+1000,
                    sample.B03,
                    sample.B04,
                    sample.B05,
                    sample.B06,
                    sample.B07,
                    sample.B08,
                    sample.B8A,
                    sample.B09,
                    sample.B10,
                    sample.B11,
                    sample.B12,
                    sample.CLM];
         }
        return [sample.B01,
                sample.B02,
                sample.B03,
                sample.B04,
                sample.B05,
                sample.B06,
                sample.B07,
                sample.B08,
                sample.B8A,
                sample.B09,
                sample.B10,
                sample.B11,
                sample.B12,
                sample.CLM];
    }
    

"""
    

elif satellite_type == 'S5p':
    dataCollectionType = DataCollection.SENTINEL5P
    evalscript = """
    //VERSION=3
    function setup() {
      return {
        input: [
          {
            bands: ["NO2", "dataMask"],  
          }
        ],
        output: [
          {
            bands: 2,
            sampleType: "FLOAT32",        
          },    
        ],
      };
    }
    
    
    function evaluatePixel(sample) {
        if (sample.dataMask == 1) {
        return [sample.NO2];
        }
    }
      
    """
    
else:
    raise Warning("Only S2A and S5p are currently available")


groundstation_df = pd.read_csv(os.path.join(input_path, csv_file_name), index_col=None, header=0)

sentinel_paths = [] 
for index, row in groundstation_df.iterrows():
    ID = index #row['StationLocalId']
    print(ID)
    lat, long = row['latitude'], row['longitude'] #row['SamplingPoint_Latitude'], row['SamplingPoint_Longitude']
    print(lat, long)
    
    bbox = BBox( ((long-long_adj,lat-lat_adj), (long+long_adj, lat+lat_adj)), crs)
    bbox_size = bbox_to_dimensions(bbox, resolution=resolution)    
    
    if satellite_type == 'S2A':
        request = SentinelHubRequest(
            evalscript=evalscript,
            input_data=[
                SentinelHubRequest.input_data(
                    data_collection=dataCollectionType,
                    time_interval=time_interval,
                    mosaicking_order='leastCC',
                    other_args={'processing': {'upsampling': 'BILINEAR'}},
                    
            )],
            responses=[
                SentinelHubRequest.output_response('default', MimeType.TIFF)
            ],
            bbox=bbox,
            size=bbox_size,
            config=config
        )
    elif satellite_type == 'S5p':
        request = SentinelHubRequest(
            evalscript=evalscript,
            input_data=[
                SentinelHubRequest.input_data(
                    data_collection=dataCollectionType,
                    time_interval=time_interval,
                    #mosaicking_order='mostRecent',
                    other_args={'processing': {'upsampling': 'BILINEAR', \
                                               'minQa':'75', \
                                               'timeliness': 'NRTI'}, }, # minQa 75 is used in paper
                    
            )],
            responses=[
                SentinelHubRequest.output_response('default', MimeType.TIFF)
            ],
            bbox=bbox,
            size=bbox_size,
            config=config
        )
    
    image = request.get_data()[0]
    
    if satellite_type == 'S2A':
        plt.figure()
        plt.imshow(image[:,:,[1,2,3]]/10000)
    elif satellite_type == 'S5p':
        plt.figure()
        plt.imshow(image[:,:,1])

Hi @rhwvandijk,

I edited out your credentials from your post above: with them anybody would be able to access your account and could run your available units down to 0 within minutes!

To be on the safe side, you can delete your Oauth client and re-create a new one in your dashboard: see here.

Hi again,

Would it be possible for you to share some coordinates of the area you are looking at? The reason I’m asking is because I ran your code for a test area and got some results back.

Alternatively, you can check the data in EOBrowser: it’s a useful tool to quickly browse images and see what is available. For example: https://sentinelshare.page.link/cWY7

PS: in your evalscript you are requesting 2 bands as an output but only return 1.

Thanks so much Maxim!. Definitely a rookie mistake :sweat_smile:

Ah thanks for the link! I am looking at data from The Netherlands and I think this might explain why no data is available in this timeframe (see figure). I thought the S5p would cover more ground in such a long timespan especially given its low spatial resolution.

Hey,
there is an explanation for why the coverage is so low.

Sentinel-5P data is flagged with quality values (qa_value). So, Sentinel Hub has the option that lets you define which data it should return. This can be done with minQa parameter, which is the minimum (inclusive) pixel quality to be displayed (in percent). It can have value between 0 and 100, with default values being 75 for NO2 and 50 for other products (if not set explicitly).
Example: setting minQa = 75 will only display pixels with qa_value >= 75%
Here is a description of processing options for Sentinel-5P in the documentation.

In EO Browser you can change this setting under the “Effects and advanced options”. If you set it to lower value, data for more area is available, but the “quality of that data” is worse (lower qa_values).

image

I see you already have that set in your SentinelHubRequest for Sentinel-5P. You can try with different values and see what fits you best.

Hope this helps.
Cheers.