Internal Server Error when downloading

Dear community,

I’m getting the following error:

sentinelhub.exceptions.DownloadFailedException: Failed to download from:
https://services.sentinel-hub.com/api/v1/process
with HTTPError:
500 Server Error: Internal Server Error for url: https://services.sentinel-hub.com/api/v1/process
Server response: “{“error”:{“status”:500,“reason”:“Internal Server Error”,“message”:“Illegal request to s3://sentinel-s2-l1c-index/tiles/37/T/CL/2022/11/19/0/B02.index. HTTP Status: ‘404’”,“code”:“RENDERER_EXCEPTION”}}”

…when downloading DataCollection.SENTINEL2_L1C using eo-learn/Python.

I’ve seen other posts that say, these errors go away (eventually) at some time. Nevertheless, someone might want to have a look and manually kick-off some processes to synchronize…?

Thank you & best regards!

Hello.

There was an issue with this index file and is now fixed. A product was corrupt and has been replaced with a new one, reprocessed in January
Thank you for letting us know.

Best regards,
Gregor Majcen

Thank you for the swift action! I can confirm that it’s working now.

Hi, I found a similar error when downloading SENTINEL2_L1C using Python:

DownloadFailedException: Failed to download from:
https://services.sentinel-hub.com/api/v1/process
with HTTPError:
500 Server Error: Internal Server Error for url: https://services.sentinel-hub.com/api/v1/process
Server response: "{"status": 500, "reason": "Internal Server Error", "message": "Illegal request to s3://sentinel-s2-l1c-index/tiles/33/M/WR/2022/9/12/0/B08.index. HTTP Status: '404'", "code": "RENDERER_EXCEPTION"}"

I just wanted to let you know it

Thank you & best regards!

Thanks. Can you perhaps try again, it should hopefully be fixed.

Hi,
I have the same problem when trying to use FIS (SENTINEL2_L1C).

Failed to download from:
https://services.sentinel-hub.com/ogc/fis/
with HTTPError:
500 Server Error: Internal Server Error for url: https://services.sentinel-hub.com/ogc/fis/
Server response: “Illegal request to s3://sentinel-s2-l1c-index/tiles/22/K/FF/2022/9/4/0/B08.index. HTTP Status: ‘404’”

Can you help me?
Thank you & best regards.

Similar issue. It should be fixed now.

Hello. I have the same problem.
An error has occurred while fetching images:
Illegal request to s3://sentinel-s2-l1c-index/tiles/38/S/ND/2023/4/3/0/B02.index. HTTP Status: ‘404’

Hi @bahadur ,

Could you please share your request as a curl command? You could go to Requests Builder to create your request and copy the curl curl command from the Request Preview window (Fig 1).

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
/usr/local/lib/python3.9/dist-packages/sentinelhub/download/handlers.py in new_download_func(self, request)
     63             try:
---> 64                 return download_func(self, request)
     65 

19 frames
HTTPError: 500 Server Error: Internal Server Error for url: https://services.sentinel-hub.com/api/v1/process

The above exception was the direct cause of the following exception:

HTTPError                                 Traceback (most recent call last)
HTTPError: 500 Server Error: Internal Server Error for url: https://services.sentinel-hub.com/api/v1/process

The above exception was the direct cause of the following exception:

DownloadFailedException                   Traceback (most recent call last)
/usr/local/lib/python3.9/dist-packages/sentinelhub/download/handlers.py in new_download_func(self, request)
     77                 if attempts_left <= 0:
     78                     message = _create_download_failed_message(exception, request.url)
---> 79                     raise DownloadFailedException(message, request_exception=exception) from exception
     80 
     81                 LOGGER.debug(

DownloadFailedException: Failed to download from:
https://services.sentinel-hub.com/api/v1/process
with HTTPError:
500 Server Error: Internal Server Error for url: https://services.sentinel-hub.com/api/v1/process
Server response: "{"status": 500, "reason": "Internal Server Error", "message": "java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0", "code": "RENDERER_EXCEPTION"}"

when trying to download SENTINEL2_L1C using Python using these functions:

download_data = SentinelHubEvalscriptTask(
    features=[(FeatureType.DATA, "bands"), (FeatureType.MASK, "dataMask")],
    evalscript=evalscript,
    data_collection=DataCollection.define_byoc(BYOC_COLLECTION_ID),
    resolution=45,
    config=config,
    max_threads=3,
)
eop = download_data.execute(bbox=bbox, time_interval=time_interval)

Hi @riccardo.gianfelici ,

You mentioned that you’re trying to download Sentinel-2 L1C data, but the code you shared is requesting data from a BYOC collection.

Could you please provide us the request which returned an error? The best way would be having your request in the curl format as shown in our documentation. Another way would be providing the full piece of your eo-learn code, i.e., the evalscript, bbox, time interval and the data collection (if it’s a byoc collection please provide the last 4 digits of the collection ID). Thank you.

I am getting the same Internal Server Error trying to download DataCollection.LANDSAT_OT_L2 imagery:
DownloadFailedException: Failed to download from:
https://services-uswest2.sentinel-hub.com/api/v1/process
with HTTPError:
500 Server Error: Internal Server Error for url: https://services-uswest2.sentinel-hub.com/api/v1/process
Server response: “{“status”: 500, “reason”: “Internal Server Error”, “message”: “java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException”, “code”: “RENDERER_EXCEPTION”}”

Hi @nedhorning ,

Could you please provide us the request which returned an error? The best way would be having your request in the curl format as shown in our documentation. Another way would be providing the full piece of your eo-learn code, i.e., the evalscript, bbox, time interval, etc.

Thank you.

Thank you @chung.horng for the prompt reply. Here is my code converted from a Jupyter Notebook. If there is a more appropriate way for me to upload the code, please let me know for future issues.

get_ipython().run_line_magic(‘matplotlib’, ‘inline’)
import datetime

import matplotlib.pyplot as plt
from aenum import MultiValueEnum
from matplotlib.colors import BoundaryNorm, ListedColormap

import geopandas as gpd
import math

from sentinelhub import CRS, BBox, DataCollection, SHConfig

from eolearn.core import EOWorkflow, FeatureType, LoadTask, OutputTask, SaveTask, linearly_connect_tasks, MergeFeatureTask
from eolearn.io import SentinelHubDemTask, SentinelHubEvalscriptTask, SentinelHubInputTask, ExportToTiffTask

In[2]:

In case you put the credentials into the configuration file you can leave this unchanged

CLIENT_ID = “”
CLIENT_SECRET = “”

In[3]:

config = SHConfig()

if CLIENT_ID and CLIENT_SECRET:
config.sh_client_id = CLIENT_ID
config.sh_client_secret = CLIENT_SECRET

if config.sh_client_id == “” or config.sh_client_secret == “” or config.instance_id == “”:
print(“Warning! To use Sentinel Hub services, please provide the credentials (client ID and client secret).”)

In[4]:

shapefile = “/home/nedhorning/Abe/LCL_Farm_Parcels/testParcelsUTM18N.shp”
eopatch_path = “/home/nedhorning/Abe/sentinel_hub_data/”
#eopatch_dir=“eopatch”
output_allbands_filename = “testUTM21S.tif”
output_ndvi_filename = “NDVI_UTM21S.tif”
epsg = “EPSG:32618”

time interval of downloaded data

time_interval = (“2022-07-01”, “2022-08-30”)

resolution of the request (in metres)

resolution = 30
expansion_dist = 210 # Distance in meters to expance each side of the bounding rectangle

maximal cloud coverage (based on Sentinel-2 provided tile metadata)

#maxcc = 0.1

time difference parameter (minimum allowed time difference; if two observations are closer than this,

they will be mosaicked into one observation)

time_difference = datetime.timedelta(hours=2)

In[5]:

gdf = gpd.read_file(shapefile)

In[6]:

bands_evalscript = “”"
//VERSION=3

function setup() {
  return {
    input: [{
      bands: ["B01", "B02", "B03", "B04", "B05", "B06", "B07", "B10", "BQA", "QA_RADSAT", "SR_QA_AEROSOL", "ST_QA", "ST_TRAD", "ST_URAD", "ST_DRAD", "ST_ATRAN", "ST_EMIS", "ST_EMSD", "ST_CDIST"]
    }],
    output: {
      id: "ms_data",
      bands: 19,
      sampleType: SampleType.UINT16
    }
  }
}

function evaluatePixel(sample) {
    return [10000 * sample.B01, 10000 * sample.B02, 10000 * sample.B03, 10000 * sample.B04, 10000 * sample.B05, 10000 * sample.B06, 10000 * sample.B07, sample.B10, sample.BQA, sample.QA_RADSAT, sample.SR_QA_AEROSOL, sample.ST_QA, sample.ST_TRAD, sample.ST_URAD, sample.ST_DRAD, sample.ST_ATRAN, sample.ST_EMIS, sample.ST_EMSD, sample.ST_CDIST]
}

“”"

In[7]:

indices_evalscript = “”"
//VERSION=3

function setup() {
    return {
        input: ["B04","B05"],
        output:[{
            id: "indices",
            bands: 1,
            sampleType: SampleType.INT16
        }]
    }
}

function evaluatePixel(sample) {
    let ndvi = Math.round(index(sample.B05, sample.B04) * 10000);
    return {
       indices: [ndvi]
    };
}

“”"

In[8]:

this will add two indices: ndvi, gci

add_indices = SentinelHubEvalscriptTask(
features=[(FeatureType.DATA, “indices”)],
evalscript=indices_evalscript,
data_collection=DataCollection.LANDSAT_OT_L2,
resolution=resolution,
#maxcc=maxcc,
time_difference=time_difference,
config=config,
max_threads=3,
)

In[9]:

this will add all Sentinel bands as DN (int) not reflectance (float)

add_ms_bands = SentinelHubEvalscriptTask(
features=[(FeatureType.DATA, “ms_data”)],
evalscript=bands_evalscript,
data_collection=DataCollection.LANDSAT_OT_L2,
resolution=resolution,
#maxcc=maxcc,
time_difference=time_difference,
config=config,
max_threads=3,
)

In[10]:

ecopatch_dir_list = []
for i in range(len(gdf)):
print(f"\rCalculating parcel {i+1} of {len(gdf)}", end=‘’, flush=True)
# Get the current polygon
polygon = gdf.loc[i, “geometry”]

# Extract the polygon's coordinates
x_coords, y_coords = polygon.exterior.coords.xy

# Get the filename by concatenating the "farm" and "name" attributes
eopatch_dir = gdf.loc[i, 'FARM'] + "_" + gdf.loc[i, 'Name']
ecopatch_dir_list.append(eopatch_dir)

xmin = int(min(x_coords) - expansion_dist)
xmax = int(max(x_coords) + expansion_dist)
ymin = int(min(y_coords) - expansion_dist)
ymax = int(max(y_coords) + expansion_dist)

# Adjust BBox so it is evenly divisible by resolution - if not then final resolution is constrained by BBox 
if (((ymax - ymin) % resolution) != 0):
    ymax = ymax - ((ymax - ymin) % resolution) + resolution
if (((xmax - xmin) % resolution) != 0):
    xmax = xmax - ((xmax - xmin) % resolution) + resolution

# region of interest
roi_coordinates = [xmin, ymin, xmax, ymax]  
roi_bbox = BBox(roi_coordinates, crs=CRS(epsg))

save = SaveTask(eopatch_path + "numpyLandsat/", overwrite_permission=2, compress_level=0)

output_task = OutputTask(eopatch_dir)

workflow_nodes = linearly_connect_tasks(add_indices, add_ms_bands, save, output_task)
workflow = EOWorkflow(workflow_nodes)

result = workflow.execute(
    {
        workflow_nodes[0]: {"bbox": roi_bbox, "time_interval": time_interval},
        workflow_nodes[-2]: {"eopatch_folder": eopatch_dir},
    }
)

Hi @nedhorning ,

I tested your evalscript and they both work fine to me. The only thing I change is the AOI. I also change the double quotation marks from “” to "" as the former don’t work for me.

The internal server error could happen because of temporal unavailability of the services. Could you please try your requests again? If the error persists, please provide the exact request so we can have a look into it. It would be easier for us to reproduce the error if you could generate your request using Requests Builder and copy the curl request from the Request Preview window.

Thanks, @chung.horng. I continue to get the same error. I need to familiarize myself with Requests Builder and figure out how to put the requests in a for loop, but I will send the curl request as soon as I am able.

Is it possible to create a curl request from an eo-learn script? I don’t see any way to do that but it seems like it could possibly exist.

Hi @nedhorning ,

Could you maybe send me the log of your execution? It should be created in a directory named execution-report-<timestamp>. Also, could provide me the last 4 digit of your Sentinel Hub client ID and the roi_bbox? I will try to reproduce the error. Thank you.

Thank you @chung.horng. I searched my computer for the execution-report directory but couldn’t find it. Can you tell me where I’d find that on an Ubuntu computer? Is the client ID the same as the Oauth clients number? The last four digits of that is: 5a81. The roi_bbox is: BBox(((641739.0, 4867107.0), (642849.0, 4868607.0)), crs=CRS(‘32618’))

Hi @nedhorning ,

Thank you for the information. I can now reproduce the error and I will investigate the request.