Slovenia jupyter notebook

16

Hum :thinking:

I’ve come across an error like this before, although it has nothing to do with Sentinel Hub libraries. The reason i got the error was because of incompatibilities between libraries in my Python virtual environment.
If you are using a virtual environment (I always recommend doing so) I would try creating a new one with the necessary libraries and trying again. I don’t know if you have gdal installed but sometime it is the culprit not playing nice with other libraries.

Finally, the way I solved the problem was to use Conda, which sorted all the incompatibilities for me.

17

Hi,
Yes, it might be a good idea to create a new environment and redo all the steps. I’m trying to install on Azure’s ML.

18

Hi,
I’m trying to install the package on a workstation and I re-installed the packages. Now I’m seeing the following error:

ModuleNotFoundError Traceback (most recent call last)
in
----> 1 from sentinelhub import SHConfig
2
3 config = SHConfig()
4
5 config.instance_id = ‘xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx’

ModuleNotFoundError: No module named ‘sentinelhub’

can anybody help me?

19

Hi,

I was able to run the script until the step prior to “visualize the patches”, when execution stopped:
0%| | 0/25 [00:00<?, ?it/s]

I’m using the syntax suggested:
executor.run(workers=2, multiprocess=False)

The report log txt follows:

2021-08-04 16:30:30,959 eolearn.core.eoworkflow DEBUG    Computing SentinelHubInputTask(*(), **{'bbox': BBox(((510000.0, 5135000.0), (515000.0, 5140000.0)), crs=CRS('32633')), 'time_interval': ['2019-01-01', '2019-12-31']})
2021-08-04 16:30:30,959 sentinelhub.sentinelhub_session DEBUG    Creating a new authentication session with Sentinel Hub service
2021-08-04 16:30:30,959 requests_oauthlib.oauth2_session DEBUG    Encoding `client_id` "********-****-4d1d-b305-d12952fcd62" with `client_secret` as Basic auth credentials.
2021-08-04 16:30:30,959 requests_oauthlib.oauth2_session DEBUG    Requesting url https://services.sentinel-hub.com/oauth/token using method POST.
2021-08-04 16:30:30,959 requests_oauthlib.oauth2_session DEBUG    Supplying headers {'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'} and data {'grant_type': 'client_credentials'}
2021-08-04 16:30:30,959 requests_oauthlib.oauth2_session DEBUG    Passing through key word arguments {'timeout': None, 'auth': <requests.auth.HTTPBasicAuth object at 0x000001DDA6F10730>, 'verify': True, 'proxies': None}.
2021-08-04 16:30:30,959 urllib3.connectionpool DEBUG    Starting new HTTPS connection (1): services.sentinel-hub.com:443
2021-08-04 16:30:31,459 sentinelhub.download.handlers DEBUG    Download attempt failed: HTTPSConnectionPool(host='services.sentinel-hub.com', port=443): Max retries exceeded with url: /oauth/token (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))
3 attempts left, will retry in 5s
2021-08-04 16:30:36,473 sentinelhub.sentinelhub_session DEBUG    Creating a new authentication session with Sentinel Hub service
2021-08-04 16:30:36,473 requests_oauthlib.oauth2_session DEBUG    Encoding `client_id` "********-****-4d1d-b305-d12952fcd62" with `client_secret` as Basic auth credentials.
2021-08-04 16:30:36,473 requests_oauthlib.oauth2_session DEBUG    Requesting url https://services.sentinel-hub.com/oauth/token using method POST.
2021-08-04 16:30:36,473 requests_oauthlib.oauth2_session DEBUG    Supplying headers {'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'} and data {'grant_type': 'client_credentials'}
2021-08-04 16:30:36,473 requests_oauthlib.oauth2_session DEBUG    Passing through key word arguments {'timeout': None, 'auth': <requests.auth.HTTPBasicAuth object at 0x000001DDA6F108E0>, 'verify': True, 'proxies': None}.
2021-08-04 16:30:36,473 urllib3.connectionpool DEBUG    Starting new HTTPS connection (1): services.sentinel-hub.com:443

I’m grateful if you can help.

20

Hi @tabletccar,

Did you take a look at the file report.html? It should contain reports for all your eo patches with related error messages in case the processing fails. It can give everybody a better idea why the execution stopped.

Note: Please make sure to not post your full credentials publicly on the forum. I have edited your latest post.

21

Hi,

This error generated a folder “eoexecution-report-2021_08_08-20_05_40” and within these folder 6 files “eoexecution-1” numbered from 1 to 6 were generated. It did not generate the 25 files and the report file.

22

Hi,

Now that you have sent the report.html through by email, we can help you solve your issue. It would be much more efficient if you posted the information here in the future. For the sake of sharing the information on the forum, the error message in your report file is the following:

oauthlib.oauth2.rfc6749.errors.CustomOAuth2Error: ({'status': 400, 'reason': 'Bad Request', 'message': 'Illegal client_id', 'code': 'OAUTH_ERROR'})

This error message is quite self-explanatory: it seems that the client_id that you are using is not correct. Please see my post posted further up for explanations on how to get the correct client_id and secret.

23

Hi,

Ok! I had to generate another client_id e secret. I ran it again and when I ran the workflow it showed this message:

20% 5/25 [2:30:29<11:31:06, 2073.34s/it]

C:\ProgramData\Anaconda3\envs\mm_2\lib\site-packages\geopandas\geodataframe.py:577: RuntimeWarning: Sequential read of iterator was interrupted. Resetting iterator. This can negatively impact the performance.
for feature in features_lst:
C:\ProgramData\Anaconda3\envs\mm_2\lib\site-packages\geopandas_vectorized.py:142: DeprecationWarning: An exception was ignored while fetching the attribute __array_interface__ from an object of type ‘MultiPolygon’. With the exception of AttributeError NumPy will always raise this exception in the future. Raise this deprecation warning to see the original exception. (Warning added NumPy 1.21)
aout[:] = out

Any suggestion to improve this performance?

24

Hi,

Complementing the previous message, the file eoexecution-25 follows. The report.html file was not generated.

1 attempts left, will retry in 45s
2021-08-14 23:21:48,207 sentinelhub.sentinelhub_session DEBUG Creating a new authentication session with Sentinel Hub service
2021-08-14 23:21:48,207 requests_oauthlib.oauth2_session DEBUG Encoding client_id “XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX” with client_secret as Basic auth credentials.
2021-08-14 23:21:48,207 requests_oauthlib.oauth2_session DEBUG Requesting url https://services.sentinel-hub.com/oauth/token using method POST.
2021-08-14 23:21:48,207 requests_oauthlib.oauth2_session DEBUG Supplying headers {‘Accept’: ‘application/json’, ‘Content-Type’: ‘application/x-www-form-urlencoded;charset=UTF-8’} and data {‘grant_type’: ‘client_credentials’}
2021-08-14 23:21:48,207 requests_oauthlib.oauth2_session DEBUG Passing through key word arguments {‘timeout’: None, ‘auth’: <requests.auth.HTTPBasicAuth object at 0x000001A0D7CBED60>, ‘verify’: True, ‘proxies’: None}.
2021-08-14 23:21:48,207 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): services.sentinel-hub.com:443
2021-08-14 23:21:48,723 root DEBUG EOWorkflow execution failed

25

Hi @testagem.cgc01,

Regarding the reported geopandas warnings, I don’t know if there is anything we can do about them. They will probably have to be resolved in geopandas package itself. But for more info I suggest checking geopandas GitHub issues.

Regarding the failed executions, it seems very weird that the report.html wouldn’t be generated. This should have happened in the line:

executor.make_report()

Did you perhaps remove this line or interrupt the cell before it finished? Did perhaps Python kernel of your notebook die during the execution of this cell? (The latter would indicate that you ran out of memory.)

26

Hi,

I generated by the third the credentials to try to run the script. This log was generated. Anyone know how to solve?

2021-08-25 21:17:44,967 eolearn.core.eoworkflow DEBUG Computing SentinelHubInputTask((), **{‘bbox’: BBox(((505000.0, 5135000.0), (510000.0, 5140000.0)), crs=CRS(‘32633’)), ‘time_interval’: [‘2019-01-01’, ‘2019-12-31’]})
2021-08-25 21:17:45,030 sentinelhub.sentinelhub_session DEBUG Creating a new authentication session with Sentinel Hub service
2021-08-25 21:17:45,030 requests_oauthlib.oauth2_session DEBUG Encoding client_id “xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx” with client_secret as Basic auth credentials.
2021-08-25 21:17:45,045 requests_oauthlib.oauth2_session DEBUG Requesting url https://services.sentinel-hub.com/oauth/token using method POST.
2021-08-25 21:17:45,061 requests_oauthlib.oauth2_session DEBUG Supplying headers {‘Accept’: ‘application/json’, ‘Content-Type’: ‘application/x-www-form-urlencoded;charset=UTF-8’} and data {‘grant_type’: ‘client_credentials’}
2021-08-25 21:17:45,061 requests_oauthlib.oauth2_session DEBUG Passing through key word arguments {‘timeout’: None, ‘auth’: <requests.auth.HTTPBasicAuth object at 0x000001ACE11B7520>, ‘verify’: True, ‘proxies’: None}.
2021-08-25 21:17:45,061 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): services.sentinel-hub.com:443
2021-08-25 21:17:46,317 urllib3.connectionpool DEBUG https://services.sentinel-hub.com:443 “POST /oauth/token HTTP/1.1” 200 947
2021-08-25 21:17:46,317 requests_oauthlib.oauth2_session DEBUG Request to fetch token completed with status 200.
2021-08-25 21:17:46,317 requests_oauthlib.oauth2_session DEBUG Request url was https://services.sentinel-hub.com/oauth/token
2021-08-25 21:17:46,317 requests_oauthlib.oauth2_session DEBUG Request headers were {‘User-Agent’: ‘python-requests/2.26.0’, ‘Accept-Encoding’: ‘gzip, deflate, br’, ‘Accept’: ‘application/json’, ‘Connection’: ‘keep-alive’, ‘Content-Type’: ‘application/x-www-form-urlencoded;charset=UTF-8’, ‘Content-Length’: ‘29’, ‘Authorization’: ‘Basic YzkzMzI0MmUtYjM5MC00MjUyLTg2YjktN2ZjZmJjNTQxZjZlOilDMURXXWMoMDBlIUU/WzNsWzRqeGZpOkJ9dUVIZSpAO3lGdy5EQW0=’}
2021-08-25 21:17:46,317 requests_oauthlib.oauth2_session DEBUG Request body was grant_type=client_credentials
2021-08-25 21:17:46,317 requests_oauthlib.oauth2_session DEBUG Response headers were {‘Date’: ‘Thu, 26 Aug 2021 00:17:46 GMT’, ‘Content-Type’: ‘application/json’, ‘Content-Length’: ‘947’, ‘Connection’: ‘keep-alive’, ‘cache-control’: ‘no-cache’, ‘set-cookie’: ‘shs1=62a8c79b-6849-4fba-9879-e21c33ec6563;Version=1;Domain=services.sentinel-hub.com;Path=/oauth/;HttpOnly’, ‘access-control-allow-origin’: '’, ‘access-control-allow-headers’: ‘origin, content-type, accept, accept-crs, authorization, sh-request-id, cache-control’, ‘access-control-allow-credentials’: ‘true’, ‘access-control-allow-methods’: ‘GET, POST, PUT, DELETE, OPTIONS, HEAD, PATCH’, ‘access-control-max-age’: ‘3600’} and content {“access_token”:“eyJraWQiOiJzaCIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiJiZGEyOGE5NS05ZTEwLTRmNzctOTIzYS00NGRiYmQzMWM2MWIiLCJhdWQiOiJjOTMzMjQyZS1iMzkwLTQyNTItODZiOS03ZmNmYmM1NDFmNmUiLCJqdGkiOiJjMzFkZjM4Yi0yOGE0LTQ2ZWMtOWRlMy0xMzhlZmE5MjkwYWYiLCJleHAiOjE2Mjk5NDA2NjYsIm5hbWUiOiJNYXJjZWxvIG1hcmFuaMOjbyIsImVtYWlsIjoibW1hcmFuaGFvMjFAaG90bWFpbC5jb20iLCJnaXZlbl9uYW1lIjoiTWFyY2VsbyIsImZhbWlseV9uYW1lIjoibWFyYW5ow6NvIiwic2lkIjoiNjJhOGM3OWItNjg0OS00ZmJhLTk4NzktZTIxYzMzZWM2NTYzIiwiZGlkIjoxLCJhaWQiOiI2Y2M0MDdmZC1jODBkLTQ3MDQtYjQyMi1jZjI5NDA3NTljMjQiLCJkIjp7IjEiOnsicmEiOnsicmFnIjoxfSwidCI6MTEwMDB9fX0.hQ4KMDCIMTrvKy6C44tYxrhqs24zjpNYpEpm-Sr6tFZPB8WPKWDMeR40mLPo_21x8Q3yPGFc1o7oJYw_s0FBi9CvQRJR9ZRZuI8CVsxFTHnamKBAq1-BkZRQ4fye009QU4nchN7SsdP1ELqfslqaqguuYcVidcr4FkYPrEvqHBq_C8uOzYHH0vRXToPwWqcKJf5wodJZYX8sBin_3fOXZ4frkVQfYzD1EPzXXJmliU8Rzr22QoyQ_LQ2ugX4lSBD5O90VDVGhZZBQ556aIZuaC_xI0eknAxWDDYbYt_q7c-EJbrv4MIsoeDFOKKZE4gKO9hVEibpA96_YBLnyLqKRw”,“expires_in”:3599}.
2021-08-25 21:17:46,332 requests_oauthlib.oauth2_session DEBUG Invoking 0 token response hooks.
2021-08-25 21:17:46,332 requests_oauthlib.oauth2_session DEBUG Obtained token {‘access_token’: ‘eyJraWQiOiJzaCIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiJiZGEyOGE5NS05ZTEwLTRmNzctOTIzYS00NGRiYmQzMWM2MWIiLCJhdWQiOiJjOTMzMjQyZS1iMzkwLTQyNTItODZiOS03ZmNmYmM1NDFmNmUiLCJqdGkiOiJjMzFkZjM4Yi0yOGE0LTQ2ZWMtOWRlMy0xMzhlZmE5MjkwYWYiLCJleHAiOjE2Mjk5NDA2NjYsIm5hbWUiOiJNYXJjZWxvIG1hcmFuaMOjbyIsImVtYWlsIjoibW1hcmFuaGFvMjFAaG90bWFpbC5jb20iLCJnaXZlbl9uYW1lIjoiTWFyY2VsbyIsImZhbWlseV9uYW1lIjoibWFyYW5ow6NvIiwic2lkIjoiNjJhOGM3OWItNjg0OS00ZmJhLTk4NzktZTIxYzMzZWM2NTYzIiwiZGlkIjoxLCJhaWQiOiI2Y2M0MDdmZC1jODBkLTQ3MDQtYjQyMi1jZjI5NDA3NTljMjQiLCJkIjp7IjEiOnsicmEiOnsicmFnIjoxfSwidCI6MTEwMDB9fX0.hQ4KMDCIMTrvKy6C44tYxrhqs24zjpNYpEpm-Sr6tFZPB8WPKWDMeR40mLPo_21x8Q3yPGFc1o7oJYw_s0FBi9CvQRJR9ZRZuI8CVsxFTHnamKBAq1-BkZRQ4fye009QU4nchN7SsdP1ELqfslqaqguuYcVidcr4FkYPrEvqHBq_C8uOzYHH0vRXToPwWqcKJf5wodJZYX8sBin_3fOXZ4frkVQfYzD1EPzXXJmliU8Rzr22QoyQ_LQ2ugX4lSBD5O90VDVGhZZBQ556aIZuaC_xI0eknAxWDDYbYt_q7c-EJbrv4MIsoeDFOKKZE4gKO9hVEibpA96_YBLnyLqKRw’, ‘expires_in’: 3599, ‘expires_at’: 1629940665.3326416}.
2021-08-25 21:17:46,332 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): services.sentinel-hub.com:443
2021-08-25 21:20:04,869 sentinelhub.download.handlers DEBUG Download attempt failed: HTTPSConnectionPool(host=‘services.sentinel-hub.com’, port=443): Read timed out. (read timeout=120)
3 attempts left, will retry in 5s
2021-08-25 21:20:09,884 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): services.sentinel-hub.com:443
2021-08-25 21:20:11,165 urllib3.connectionpool DEBUG https://services.sentinel-hub.com:443 “POST /api/v1/catalog/search HTTP/1.1” 200 5193
2021-08-25 21:20:11,165 sentinelhub.download.sentinelhub_client DEBUG ThreadPoolExecutor-0_1: Successful download from https://services.sentinel-hub.com/api/v1/catalog/search
2021-08-25 21:20:11,227 eolearn.io.sentinelhub_process DEBUG Downloading 63 requests of type DataCollection.SENTINEL2_L1C

27

Besides these logs, could you also provide a stack trace of your error? If you are running the code with EOExecutor the stack trace for each EOPatch should be in report.html file once you call executor.make_report().

In case there the issue is that you are running out of memory I suggest that you try running with executor.run(workers=1, multiprocess=False). This way it will process only a single EOPatch at a time and use much less memory.

28

Hi,

Follow the last lines of report.html. I regenerated the client_id and client_secret and it keeps getting the same error.

File “C:\ProgramData\Anaconda3\envs\mm_2\lib\site-packages\sentinelhub\sentinelhub_session.py”, line 34, in init raise ValueError("Configuration parameters ‘sh_client_id’ and ‘sh_client_secret’ have to be set in order " ValueError: Configuration parameters ‘sh_client_id’ and ‘sh_client_secret’ have to be set in order to authenticate with Sentinel Hub service.

29

Hi @testagem.cgc01,

I received your report.html file. The error inside seems to be:

HTTPSConnectionPool(host='services.sentinel-hub.com', port=443): Max retries exceeded with url: /oauth/token (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))

According to this I believe the issue is with an SSL certificate on your side and has nothing to do with Sentinel Hub credentials. I don’t know much about network certificates but there are a lot of posts on Stack Overflow, such as this one, of people having the same type of error. You might have to configure some permissions on your computer or your local network.

30

less_ram
less_ram2933×1717 1.59 MB

Hi,

Is there any modification to the script that limits processing to less than 8Gb of RAM?

31

Training a ML model will take up some computing resources. Since you are in a cell that iterates over patches, you could try reducing the size of the patches when you define the grid, as shown in the example page.

The splitting choice depends on the available resources of your computer. An EOPatch with a size of has around 500 x 500 pixels at 10 meter resolution has a size of about ~1 GB.

32

I understand your explanation. I’m now trying to run on Google Colab (13 Gb RAM) and it’s showing these difficulties in the attachment:

report_html
report_html1508×1026 256 KB

colab1
colab11508×1026 170 KB

33

Hi,

You are getting the same error message as in one of your previous posts. Please refer to the response: Slovenia jupyter notebook - #9 by maxim.lamare

34

Hi,

How to solve this problem?

ImportErrorTraceback (most recent call last)
in
38 from eolearn.mask import AddValidDataMaskTask
39 from eolearn.geometry import VectorToRaster, PointSamplingTask, ErosionTask
—> 40 from eolearn.features import LinearInterpolation, SimpleFilterTask, NormalizedDifferenceIndexTask
41 from sentinelhub import UtmZoneSplitter, BBox, CRS, DataCollection, MimeType, SentinelHubRequest, SHConfig

~/miniconda3/envs/py38/lib/python3.8/site-packages/eolearn/features/init.py in
9 KrigingInterpolation, LegacyInterpolation
10 from .feature_extractor import FeatureExtractionTask, FeatureExtendedExtractor
—> 11 from .feature_manipulation import SimpleFilterTask, FilterTimeSeries, ValueFilloutTask
12 from .haralick import HaralickTask
13 from .radiometric_normalization import ReferenceScenes, HistogramMatching, BlueCompositing, HOTCompositing, \

~/miniconda3/envs/py38/lib/python3.8/site-packages/eolearn/features/feature_manipulation.py in
17 import numpy as np
18
—> 19 from sentinelhub.time_utils import iso_to_datetime
20 from eolearn.core import EOTask, FeatureType
21

ImportError: cannot import name ‘iso_to_datetime’ from ‘sentinelhub.time_utils’ (/home/idies/miniconda3/envs/py38/lib/python3.8/site-packages/sentinelhub/time_utils.py)

35

Hi,

This function is depreciated since the last version of sentinelhub-py and has been removed.

You can use the following functions instead, depending on what you are trying to achieve: