Change in S1 SentinelHubInputTask outputs?

leatresch · December 13, 2021, 10:20am

Hi all,

My team and I set up functions in order to automatically fetch and process satellite data using sentinelhub and eopatch libraries.

For S1 data, we are using a linear workflow. The main piece is a SentinelHubInputTask. It asks for bands_feature VH, VV from DataCollection.SENTINEL1_IW (we look for data in Northern Africa), and additional_data localIncidenceAngle, scatteringArea, shadowMask data. Orthorectification is used, thus this data is available.

Until the recent update, all was fine -we could request S1 data and get what we wanted, i.e. the outputs was as follow in the patch: patch.data['GRD_data'] was made of ['VV', 'VH', 'localIncidenceAngle', 'scatteringData'] (in the workflow we move shadowMask to the mask feature, hence why it’s not here).

However, now in the 0.10.1 version, our function outputs data in a different way:

localIncidenceAngle and scatteringArea went from being in patch.data['GRD_data'] to having their own sub-features in patch.data, cf image,
patch.data['GRD_data'] only has one band, when we request both VH and VV. We have no idea what is in there. We tried comparing it to solo request (only VV or only VH), but there seems to be no ressemblance.

Here is a screen of the outputs we currently have:

I could not recreate the previous state of our S1 eopatches since going back to eolearn 0.10.0 triggers the new unit errors (for instance, the code asks for VV data in reflectance, which obviously do not work and results in a 400 error). Here is an old screen of the outputs, which shows that only GRD_data was a subfeature of the data feature and that it had a shape of 4 (for VV, VH, localIncidenceAngle and scatteringArea).

(Sorry for the shapely warning

)

We looked up the changelog in both eo-learn and sentinelhub but could not link any of the changes to S1 data. We did not edit our functions, only updated the libraries versions. We really have no idea why this change happened and could not see similar experiences on the forum or on the internet at large.
Did anyone noticed that change too? Is there any explanation on the library part?

While we could easily change our post-request code to fit the new sub-features localIncidenceAngle and scatteringArea, the GRD_data issue cannot be solved so easily as we have no idea what data is in there. We really are blind here.

Thank you for taking the time to read all that!
Kind regards,
Léa

maleksandrov · December 13, 2021, 12:27pm

Hi @leatresch,

Thanks for providing a detailed description of the problem. However, somehow I’m not able to reproduce it. Here is a minimal working example:

from sentinelhub import DataCollection, BBox, CRS
from eolearn.core import FeatureType
from eolearn.io import SentinelHubInputTask

sh_task = SentinelHubInputTask(
    bands_feature=(FeatureType.DATA, 'GRD_data'),
    additional_data=[
        (FeatureType.DATA, 'localIncidenceAngle'),
        (FeatureType.DATA, 'scatteringArea'),
        (FeatureType.MASK, 'dataMask')
    ],
    size=(50, 40),
    data_collection=DataCollection.SENTINEL1_IW,
    aux_request_args={
        'processing': {
            'orthorectify': True,
            'backCoeff': 'GAMMA0_TERRAIN'
        }
    }
)
eopatch = sh_task.execute(
    bbox=BBox((-5.05, 48.0, -5.00, 48.05), CRS.WGS84),
    time_interval=('2020-06-01', '2020-06-05')
)
print(eopatch)

which produces an output:

EOPatch(
  data: {
    GRD_data: numpy.ndarray(shape=(4, 40, 50, 2), dtype=float32)
    localIncidenceAngle: numpy.ndarray(shape=(4, 40, 50, 1), dtype=float32)
    scatteringArea: numpy.ndarray(shape=(4, 40, 50, 1), dtype=float32)
  }
  mask: {
    dataMask: numpy.ndarray(shape=(4, 40, 50, 1), dtype=bool)
  }
  scalar: {}
  label: {}
  vector: {}
  data_timeless: {}
  mask_timeless: {}
  scalar_timeless: {}
  label_timeless: {}
  vector_timeless: {}
  meta_info: {
    size_x: 50
    size_y: 40
    time_difference: datetime.timedelta(seconds=1)
    time_interval: (datetime.datetime(2020, 6, 1, 0, 0), datetime.datetime(2020, 6, 5, 23, 59, 59))
  }
  bbox: BBox(((-5.05, 48.0), (-5.0, 48.05)), crs=CRS('4326'))
  timestamp: [datetime.datetime(2020, 6, 3, 6, 32, 29), ..., datetime.datetime(2020, 6, 4, 6, 23, 33)], length=4
)

So in my case the output feature GRD_data still has 2 bands (VV and VH). Could you check if this code also works for your AOI parameters? Is it possible that a subsequent EOTask in your workflow modifies and removes 1 band from GRD_data feature?

I’m using eo-learn 0.10.1 and sentinelhub 3.4.2.

Regarding older versions of eo-learn package, you are correct that they don’t work anymore for this task. It is because of a change on Sentinel Hub service which introduced a stricter validation of units in request evalscripts.

leatresch · December 14, 2021, 9:33am

Hi @maleksandrov,
Thanks for the quick answer! So I tweaked my code slightly and it seems that the shape problem on patch.data['GRD_data'] was linked to how I moved the shadowMask from patch.data['GRD_data'] to the mask feature. Now I indeed have both VV and VH in the patch -sorry, for taking your time, fully my mistake! Indeed it was the workflow.

However, this does not explain why suddenly localIncidenceAngle, scatteringArea and shadowMask are now out of patch.data['GRD_data']. Do you have any information on that? We’ll be able to adapt our codes but honestly it’s pretty confusing to us and we would like to understand where this comes from.
Originally, all additionnal data was automatically put in patch.data['GRD_data'] for S1 (on the contrary, S2 additionnal data all had they own sub-features). It was strange but it was working and we adapted our code to it (thus why we were moving the shadowMask to the mask feature, for instance). We did not see any mention of it changing, thus the confusion.

Have a good day,
Léa

maleksandrov · December 14, 2021, 10:47am

Good to hear you were able to resolve the issue.

The change regarding localIncidenceAngle, scatteringArea, etc. is a design choice we had to make in eo-learn 0.10.1 so that SentinelHubInputTask would still make sense for all data collections. If you check any DataCollection enum you can see the following:

> from sentinelhub import DataCollection
> 
> DataCollection.SENTINEL1_IW

<DataCollection.SENTINEL1_IW: DataCollectionDefinition(
  api_id: sentinel-1-grd
  catalog_id: sentinel-1-grd
  wfs_id: DSS3
  service_url: https://services.sentinel-hub.com
  collection_type: Sentinel-1
  sensor_type: C-SAR
  processing_level: GRD
  swath_mode: IW
  polarization: DV
  resolution: HIGH
  orbit_direction: BOTH
  bands: (Band(name='VV', units=(<Unit.LINEAR_POWER: 'LINEAR_POWER'>,), output_types=(<class 'numpy.float32'>,)), Band(name='VH', units=(<Unit.LINEAR_POWER: 'LINEAR_POWER'>,), output_types=(<class 'numpy.float32'>,)))
  metabands: (Band(name='localIncidenceAngle', units=(<Unit.DN: 'DN'>,), output_types=(<class 'numpy.float32'>,)), Band(name='scatteringArea', units=(<Unit.DN: 'DN'>,), output_types=(<class 'numpy.float32'>,)), Band(name='shadowMask', units=(<Unit.DN: 'DN'>,), output_types=(<class 'bool'>,)), Band(name='dataMask', units=(<Unit.DN: 'DN'>,), output_types=(<class 'bool'>,)))
  is_timeless: False
  has_cloud_coverage: False
)>

Basically the data is split into bands and metabands. Bands are the main raster satelite images while metabands are extra channels containing meta information. Typically bands are all cast into the same numpy dtype, while each metaband can have a completely different dtype. Because of that SentinelHubInputTask now stacks together only bands into a single feature, while each metaband is put into a separate feature.

I see our change log was not very detailed about changes SentinelHubInputTask. I hope this explanation clears it up.

leatresch · December 14, 2021, 1:03pm

Hi again,
Thanks for clearing this up. Indeed it makes more sense to put metadata elsewhere than with the bands data! We understand why we got the issue in the first place.
Have a good day!
Léa

kevin.hartung · September 20, 2022, 11:30am

Hi at all,

i tried your example and got an Error.
What ist is the fault?

Thanks in advance

ziga.luksic · September 21, 2022, 9:30am

Hi, what are your sentinelhub and eolearn.io versions? To me it seems like you’re using a very old eolearn.io version, are you perhaps using conda to install eolearn?

kevin.hartung · October 5, 2022, 8:24am

You are right.I had also problems to update on windows with conda. I tried the Docker-pull-request and it works.
Thank you for your helpfull comment.

kevin.hartung · October 6, 2022, 1:33pm

Is eo-learn for Windows up to date. I have Problems with installing and updating eo-learn on windows.

ziga.luksic · October 10, 2022, 1:00pm

Does it fail if you follow the installation steps for the pypi distribution?