Error while downloading data into EOPatches

1

Hi,
I have been repeatedly receiving an error while downloading some sentinel data into EO Patches. I am attaching the error message from the report.html file from the execution report.

tarfile.ReadError: unexpected end of data

Why am I getting this error message repeatedly??

2

Please can you share the code block that you are using so we can help you debug the error.

3

I am attaching the EO Workflow I’ve designed to create the EO Patches.

workflow_nodes = linearly_connect_tasks(
    add_data, 
    ndvi, 
    ndwi, 
    rvi, 
    ireci, 
    evi, 
    arvi, 
    add_sh_validmask, 
    add_valid_count, 
    vector_import_task, 
    rasterization_task,
    save
)
workflow = EOWorkflow(workflow_nodes)
workflow.dependency_graph()


%%time

# Time interval for the SH request
time_interval = ["2021-10-20", "2022-10-22"]

# Define additional parameters of the workflow
input_node = workflow_nodes[0]
save_node = workflow_nodes[-1]
execution_args = []
for idx, bbox in enumerate(bbox_list):
    execution_args.append(
        {
            input_node: {"bbox": bbox, "time_interval": time_interval},
            save_node: {"eopatch_folder": f"EOPatch_{idx}"},
        }
    )

# Execute the workflow
executor = EOExecutor(workflow, execution_args, save_logs = True)
executor.run(workers = 1)

executor.make_report()

failed_ids = executor.get_failed_executions()
if failed_ids:
    raise RuntimeError(
        f"Execution failed EOPatches with IDs:\n{failed_ids}\n"
        f"For more info check report at {executor.get_report_path()}"
    )
4

Hi, i’ll need some more information to help you with. Could you paste the whole error from the report? I’m searching for any information on which task this could happen.

Judging by the error I assume that add_data is a download task? Do you still get the error if your workflow consists only of add_data and save?

If that is the case, could you provide the code for your download task, and a bbox for which the download fails, so I can see if I can replicate the error.

5

This error is quite random. A probable cause could be interruptions in internet connections. Apart from this i wanted to know if this could occur due to some other reason.

band_names = ["B02", "B03", "B04", "B05", "B06", "B07",  "B08", "B8A", "B11", "B12"]
add_data = SentinelHubInputTask(
    bands_feature = (FeatureType.DATA, "BANDS"),
    bands = band_names,
    resolution = 10,
    maxcc = 0.01,
    time_difference = datetime.timedelta(minutes = 120),
    data_collection = DataCollection.SENTINEL2_L2A,
    additional_data = [(FeatureType.MASK, "dataMask", "IS_DATA"), (FeatureType.MASK, "CLM"), (FeatureType.DATA, "CLP")],
    max_threads = 5,
)

bbox: BBox(((77.13141416538453, 28.77664301688424), (77.31795817619644, 28.950477980746506)), crs=CRS(‘4326’))
info: {‘parent_bbox’: BBox(((77.13141416538453, 28.77664301688424), (77.50450218700834, 29.298147908471037)), crs=CRS(‘4326’)), ‘index_x’: 0, ‘index_y’: 0}
6

I tried out the download task a couple times and it worked fine. But in all cases it took a while to download, which increases the chances that errors occur due to connection issues.

If you find a way to recreate the issue consistently I can take another look. My suggestions (based on current understanding) would be:

  • try splitting the area into smaller bounding boxes. Yours are currently about 1900 x 1700 pixels, which you could potentially split into 4. Shorter download times might mitigate connection issues.
  • similarly you could do two downloads for two separate time intervals and then temporally merge the obtained eopatches
7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.