Sentinel-hub field-delineation deployment

Hello,

I would like to run the field-delineation project. While doing so, I’m getting into the following error where converting tiff files to eopatches:

2023-05-09 11:18:46,915 rasterio._filepath ERROR File-like object not found in virtual filesystem: b’29a650e4-51a0-43dc-9539-84777b3482ef/29a650e4-51a0-43dc-9539-84777b3482ef.hdr’
2023-05-09 11:18:46,916 rasterio._env INFO GDAL signalled an error: err_no=4, msg=“`/vsipythonfilelike/29a650e4-51a0-43dc-9539-84777b3482ef/29a650e4-51a0-43dc-9539-84777b3482ef’ not recognized as a supported file format.”

rokenProcessPool Traceback (most recent call last)
Cell In[67], line 1
----> 1 convert_tiff_to_eopatches(tiffs_to_eop_config)

File ~/.local/lib/python3.10/site-packages/fd-1.0.0-py3.10.egg/fd/scripts/tiffs_to_eopatches.py:71, in convert_tiff_to_eopatches(config)
68 executor = EOExecutor(workflow, exec_args, save_logs=True, logs_folder=‘.’)
70 LOGGER.info(‘Execute conversion’)
—> 71 executor.run(workers=config[‘max_workers’])
73 executor.make_report()
74 LOGGER.info(f’Report was saved to location: {executor.get_report_filename()}')

File ~/.local/lib/python3.10/site-packages/eolearn/core/eoexecution.py:150, in EOExecutor.run(self, workers, multiprocess, return_results)
147 processing_type = ‘multithreading’
149 with pool_executor_class(max_workers=workers) as executor:
→ 150 self.execution_stats = list(tqdm(executor.map(self._execute_workflow, processing_args),
151 total=len(processing_args)))
153 self.general_stats = self._prepare_general_stats(workers, processing_type)
155 self.execution_logs = [None] * execution_num

File ~/.local/lib/python3.10/site-packages/tqdm/notebook.py:254, in tqdm_notebook.iter(self)
252 try:
253 it = super(tqdm_notebook, self).iter()
→ 254 for obj in it:
255 # return super(tqdm…) will not catch exception
256 yield obj
257 # NB: except … [ as …] breaks IPython async KeyboardInterrupt

File ~/.local/lib/python3.10/site-packages/tqdm/std.py:1178, in tqdm.iter(self)
1175 time = self._time
1177 try:
→ 1178 for obj in iterable:
1179 yield obj
1180 # Update and possibly print the progressbar.
1181 # Note: does not call self.update(1) for speed optimisation.

File /usr/lib/python3.10/concurrent/futures/process.py:570, in _chain_from_iterable_of_lists(iterable)
564 def _chain_from_iterable_of_lists(iterable):
565 “”"
566 Specialized implementation of itertools.chain.from_iterable.
567 Each item in iterable should be a list. This function is
568 careful not to keep references to yielded objects.
569 “”"
→ 570 for element in iterable:
571 element.reverse()
572 while element:

File /usr/lib/python3.10/concurrent/futures/_base.py:621, in Executor.map..result_iterator()
618 while fs:
619 # Careful not to keep a reference to the popped future
620 if timeout is None:
→ 621 yield _result_or_cancel(fs.pop())
622 else:
623 yield _result_or_cancel(fs.pop(), end_time - time.monotonic())

File /usr/lib/python3.10/concurrent/futures/_base.py:319, in _result_or_cancel(failed resolving arguments)
317 try:
318 try:
→ 319 return fut.result(timeout)
320 finally:
321 fut.cancel()

File /usr/lib/python3.10/concurrent/futures/_base.py:458, in Future.result(self, timeout)
456 raise CancelledError()
457 elif self._state == FINISHED:
→ 458 return self.__get_result()
459 else:
460 raise TimeoutError()

File /usr/lib/python3.10/concurrent/futures/_base.py:403, in Future.__get_result(self)
401 if self._exception:
402 try:
→ 403 raise self._exception
404 finally:
405 # Break a reference cycle with the exception in self._exception
406 self = None

BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Could you provide any suggestions on what might not be set up appropriately? I’ve downloaded the images with Processing API and reading them locally.

Thanks!

Hi Gabriel,

Without seeing your code that you are using, it’s difficult to help you debug it. However, reading the first part of your error message, it looks like you may need to double check the file paths you are using as it seems you might be missing a file extension.

Hi @william.ray,

Thank you for your prompt response.

The code is the official one(GitHub - sentinel-hub/field-delineation: Field delineation with Sentinel-2 data from Sentinel-Hub and a ResUnet-a architecture.), I’ve just changed to read the files locally instead of Amazon S3.

I’ve replace this:

def get_tiffs_to_eopatches_workflow(config: TiffsToEopatchConfig, delete_tiffs: bool = False) → EOWorkflow:
“”" Set up workflow to ingest tiff files into EOPatches “”"

# Set up credentials in sh config
sh_config = set_sh_config(config)

import_bands = [(ImportFromTiff((FeatureType.DATA, band),
                                **folder=f's3://{config.bucket_name}/{config.tiffs_folder}',**
                                config=sh_config), f'Import band {band}')
                for band in config.band_names]

with this

def get_tiffs_to_eopatches_workflow(config: TiffsToEopatchConfig, delete_tiffs: bool = False) → EOWorkflow:
“”" Set up workflow to ingest tiff files into EOPatches “”"

# Set up credentials in sh config
sh_config = set_sh_config(config)

import_bands = [(ImportFromTiff((FeatureType.DATA, band),
                                **folder=f'{config.tiffs_folder}',**
                                config=sh_config), f'Import band {band}')
                for band in config.band_names]

It reads the file properly, but its processing is not successful.

Here is the full EOExecutionlog:

2023-05-05 11:19:16,334 eolearn.core.eoworkflow DEBUG Computing ImportFromTiff(*[], **{‘filename’: ‘35TNK_3_4/B02.tif’})
2023-05-05 11:19:16,379 rasterio.env DEBUG Entering env context: <rasterio.env.Env object at 0x000002B64EF3B4C0>
2023-05-05 11:19:16,379 rasterio.env DEBUG Starting outermost env
2023-05-05 11:19:16,379 rasterio.env DEBUG No GDAL environment exists
2023-05-05 11:19:16,380 rasterio.env DEBUG New GDAL environment <rasterio._env.GDALEnv object at 0x000002B64EF9D9F0> created
2023-05-05 11:19:16,435 rasterio._filepath DEBUG Installing FilePath filesystem handler plugin…
2023-05-05 11:19:16,435 rasterio._env DEBUG GDAL data found in package: path=‘c:\users\gabarbu\appdata\local\anaconda3\lib\site-packages\rasterio\gdal_data’.
2023-05-05 11:19:16,436 rasterio._env DEBUG PROJ data found in package: path=‘c:\users\gabarbu\appdata\local\anaconda3\lib\site-packages\rasterio\proj_data’.
2023-05-05 11:19:16,437 rasterio._env DEBUG Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x000002B64EF9D9F0>.
2023-05-05 11:19:16,437 rasterio.env DEBUG Entered env context: <rasterio.env.Env object at 0x000002B64EF3B4C0>
2023-05-05 11:19:16,437 rasterio.io DEBUG VSI path: /vsipythonfilelike/fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d
2023-05-05 11:19:16,437 rasterio._base DEBUG Sharing flag: 0
2023-05-05 11:19:16,444 rasterio._filepath ERROR File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.aux’
2023-05-05 11:19:16,445 rasterio._filepath ERROR File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.AUX’
2023-05-05 11:19:16,445 rasterio._filepath ERROR File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.aux’
2023-05-05 11:19:16,445 rasterio._filepath ERROR File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.AUX’
2023-05-05 11:19:16,525 rasterio._base DEBUG Nodata success: 0, Nodata value: 0.000000
2023-05-05 11:19:16,525 rasterio._base DEBUG Dataset is started.
2023-05-05 11:19:16,526 rasterio.env DEBUG Exiting env context: <rasterio.env.Env object at 0x000002B64EF3B4C0>
2023-05-05 11:19:16,526 rasterio.env DEBUG Cleared existing <rasterio._env.GDALEnv object at 0x000002B64EF9D9F0> options
2023-05-05 11:19:16,526 rasterio._env DEBUG Stopped GDALEnv <rasterio._env.GDALEnv object at 0x000002B64EF9D9F0>.
2023-05-05 11:19:16,526 rasterio.env DEBUG Exiting outermost env
2023-05-05 11:19:16,526 rasterio.env DEBUG Exited env context: <rasterio.env.Env object at 0x000002B64EF3B4C0>
2023-05-05 11:19:16,526 rasterio.env DEBUG Entering env context: <rasterio.env.Env object at 0x000002B64EF3A680>
2023-05-05 11:19:16,526 rasterio.env DEBUG Starting outermost env
2023-05-05 11:19:16,526 rasterio.env DEBUG No GDAL environment exists
2023-05-05 11:19:16,527 rasterio.env DEBUG New GDAL environment <rasterio._env.GDALEnv object at 0x000002B64EF9D9F0> created
2023-05-05 11:19:16,528 rasterio._env DEBUG GDAL data found in package: path=‘c:\users\gabarbu\appdata\local\anaconda3\lib\site-packages\rasterio\gdal_data’.
2023-05-05 11:19:16,528 rasterio._env DEBUG PROJ data found in package: path=‘c:\users\gabarbu\appdata\local\anaconda3\lib\site-packages\rasterio\proj_data’.
2023-05-05 11:19:16,528 rasterio._env DEBUG Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x000002B64EF9D9F0>.
2023-05-05 11:19:16,528 rasterio.env DEBUG Entered env context: <rasterio.env.Env object at 0x000002B64EF3A680>
2023-05-05 11:19:16,529 rasterio.crs DEBUG Matched. confidence=100, c_code=b’32635’, c_name=b’EPSG’
2023-05-05 11:19:16,530 rasterio._io DEBUG Output nodata value read from file: None
2023-05-05 11:19:16,531 rasterio._io DEBUG Output nodata values: [None]
2023-05-05 11:19:16,533 rasterio._io DEBUG all_valid: True
2023-05-05 11:19:16,534 rasterio._io DEBUG mask_flags: ([<MaskFlags.all_valid: 1>],)
2023-05-05 11:19:16,538 rasterio._base DEBUG Sharing flag: 0
2023-05-05 11:19:16,540 rasterio._base DEBUG Nodata success: 1, Nodata value: 0.000000
2023-05-05 11:19:16,540 rasterio._base DEBUG Dataset is started.
2023-05-05 11:19:16,541 rasterio._io DEBUG IO window xoff=0.0 yoff=0.0 width=10980.0 height=10980.0
2023-05-05 11:19:17,909 rasterio._filepath ERROR File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.aux’
2023-05-05 11:19:17,909 rasterio._filepath ERROR File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.AUX’
2023-05-05 11:19:17,910 rasterio._filepath ERROR File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d\fce3c614-75b4-4baf-b21e-90e4e1a8030d.hdr’
2023-05-05 11:19:17,910 rasterio._filepath ERROR File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.hdr’
2023-05-05 11:19:17,910 rasterio._filepath ERROR File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.HDR’
2023-05-05 11:19:17,910 rasterio._filepath ERROR File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.hdr’
2023-05-05 11:19:17,910 rasterio._filepath ERROR File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d.HDR’
2023-05-05 11:19:17,911 rasterio._filepath ERROR File-like object not found in virtual filesystem: b’fce3c614-75b4-4baf-b21e-90e4e1a8030d\fce3c614-75b4-4baf-b21e-90e4e1a8030d.hdr’
2023-05-05 11:19:17,911 rasterio._env INFO GDAL signalled an error: err_no=4, msg=“`/vsipythonfilelike/fce3c614-75b4-4baf-b21e-90e4e1a8030d/fce3c614-75b4-4baf-b21e-90e4e1a8030d’ not recognized as a supported file format.”

Hello @gabriel, it seems that the example you are trying to run was written for a much earlier version of Eo-Learn and the repository has not been updated recently. It is not currently maintained, but there is a commercial deployment; that, if you have specific requirements regarding then you can reach out and we will be happy to assist.

If you are having trouble running the notebook, it could be due to version compatibility issues with the libraries required by the code. You can try running the code using an earlier version of EO-Learn that is compatible with this particular example.

Hi @adrian.dipaolo,

Do you refer to the EDC on demand services for the commercial deployment or is there another option too?

Hi Gabriel,

Yes we would recommend accessing our field-delineation algorithm via the EDC on demand services. You can find out more in EDC Public Collections and access this service via the EDC Browser.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.