'504' error on request

Apps / Services
#1

Hi,

We have Python script and sometime we are getting 504 error, we could not identify any pattern for this issue.
Is there any limitation we should know about ?
code snippet below:

def get_sentinel1_data(dates_df,poly_path,poly_name,res):
    '''https://docs.sentinel-hub.com/api/latest/data/sentinel-1-grd/
    polygons should be in UTM'''
    
    poly,EPSG_code,sd,ed = get_metadata_for_each_polygon(poly_path,poly_name,dates_df)
    
    evalscript = """
    //VERSION=3
    function setup() {
      return {
        input: [{bands: ["VV","VH","dataMask"]}],
        output: [{id:"VV",bands:1},{id:"VH",bands:1},{id:"dataMask",bands:1}]}}
    function evaluatePixel(samples) {
        return {VV: [samples.VV],VH: [samples.VH],dataMask: [samples.dataMask]}}
    """    
    stats_request = {
      "input": {
                  "bounds": {"geometry": poly,"properties": {"crs": f"http://www.opengis.net/def/crs/EPSG/0/{EPSG_code}"}},
                  "data": [{"type": "sentinel-1-grd","dataFilter": {},"resolution":"HIGH","acquisitionMode":"IW",
                            "polarization":"DV","processing": {"orthorectify": "true","backCoeff": "GAMMA0_ELLIPSOID",
                                                               "demInstance":"COPERNICUS_30"}}]},
      "aggregation":{
                  "timeRange":{"from": f"{sd}T00:00:00Z","to": f"{ed}T00:00:00Z"},"aggregationInterval":{"of": "P1D"},
                  "evalscript": evalscript,"resx": res,"resy": res}}
    headers = {'Content-Type': 'application/json','Accept': 'application/json'}
    url = "https://services.sentinel-hub.com/api/v1/statistics"
    
    return oauth.request("POST", url=url,headers=headers,json=stats_request)
#2

Hi @org_accounts,

can you provide some more info, i.e.:

  • how long time series were you executing
  • for which AOI
  • when were you executing these requests
  • when did 504 happen (and how often)
  • which IP were you running your service from

You might want to also enable the logging, which might provide useful info.

That said, it is to be expected that there will be an occasional 504 error with a service that has to fetch data from hundreds of files, each several GB large, perform the orthorectification step, then calculate backscatter.
We suggest you retry such requests one more time. Note that the failed requests are not charged for.

Best,
Grega

#3

Hi Grega,

how long time series were you executing - 1 month
for which AOI - area of ~ 1 ha
when were you executing these requests - about 2 weeks ago
when did 504 happen (and how often) - it happened for about 2 days straight, I’ve sent the requests many times, and after these 2 days it seems to work
which IP were you running your service from - I don’t know

#4

Hi @org_accounts,

we have made some improvements to Statistical API this morning and hopefully you should see these as well.
In case you identify some further increased volume of 504 errors (sporadic one here or there is expected, as mentioned), do please let us know in more details (time, IP, findings), so that we can investigate the specifics.

Best,
Grega

#6

Hi, it just happened again:

JSONDecodeError Traceback (most recent call last)
~\Anaconda3\envs\sentinel_hub\lib\site-packages\requests\models.py in json(self, **kwargs)
909 try:
→ 910 return complexjson.loads(self.text, **kwargs)
911 except JSONDecodeError as e:

~\Anaconda3\envs\sentinel_hub\lib\json_init_.py in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
356 parse_constant is None and object_pairs_hook is None and not kw):
→ 357 return _default_decoder.decode(s)
358 if cls is None:

~\Anaconda3\envs\sentinel_hub\lib\json\decoder.py in decode(self, s, _w)
336 “”"
→ 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
338 end = _w(s, end).end()

~\Anaconda3\envs\sentinel_hub\lib\json\decoder.py in raw_decode(self, s, idx)
354 except StopIteration as err:
→ 355 raise JSONDecodeError(“Expecting value”, s, err.value) from None
356 return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

JSONDecodeError Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_10180/2413870548.py in
27
28 response = get_sentinel1_data(dates_df,poly_path,poly_name,res)
—> 29 r = response.json()
30
31 status = r[“status”]

~\Anaconda3\envs\sentinel_hub\lib\site-packages\requests\models.py in json(self, **kwargs)
915 raise RequestsJSONDecodeError(e.message)
916 else:
→ 917 raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
918
919 @property

JSONDecodeError: [Errno Expecting value]

504 Gateway Time-out

504 Gateway Time-out

: 0

IP: 172.15.4.34

#7

Hi @org_accounts,
this error log does unfortunately not provide any relevant info. Have you enabled logging as asked above?

Was there just one 504 error or a batch of them? As mentioned, there might always be a sporadic error here and there - this is normal and should be worked-around by one/two retries.

We do not see any traffic coming from IP 172.15.4.34. Might it be there is another proxy inbetween?

Best,
Grega

#8

Hi @org_accounts,

looking at the log I find only one single error. As mentioned above, there might always be a sporadic error here and there - this is normal and should be worked-around by one/two retries.

I have tried to recreate the last request, the one that failed, in Request Builder and it worked just fine, which confirms my assessment that it is just a sporadic error.
You can try the same by going to the application, then selecting “Statistical”, copy-pasting the lines below in the “Request preview” window, clicking “Parse”, then drawing the polygon over your area of interest and clicking “Send request”.

curl -X POST https://services.sentinel-hub.com/api/v1/statistics \
 -H 'Authorization: Bearer <AUTH BEARER>' \
 -H 'Accept: application/json' \
 -H 'Content-Type: application/json' \
 -d '{
  "input": {
    "bounds": {
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [
              960497.077763,
              5081847.299993
            ],
            [
              960535.876365,
              5081867.270808
            ],
            [
              960590.058908,
              5081861.241671
            ],
            [
              960600.180612,
              5081793.410511
            ],
            [
              960505.368188,
              5081787.263429
            ],
            [
              960497.077763,
              5081847.299993
            ]
          ]
        ]
      },
      "properties": {
        "crs": "http://www.opengis.net/def/crs/EPSG/0/32632"
      }
    },
    "data": [
      {
        "dataFilter": {
          "resolution": "HIGH",
          "acquisitionMode": "IW",
          "polarization": "DV"
        },
        "processing": {
          "orthorectify": true,
          "demInstance": "COPERNICUS_30"
        },
        "type": "sentinel-1-grd"
      }
    ]
  },
  "aggregation": {
    "timeRange": {
      "from": "2018-01-29T00:00:00Z",
      "to": "2018-03-03T23:59:59Z"
    },
    "aggregationInterval": {
      "of": "P1D"
    },
    "resx": "10",
    "resy": "10",
    "evalscript": "//VERSION=3\n    function setup() {\n      return {\n        input: [{bands: [\"VV\",\"VH\",\"dataMask\"]}],\n        output: [{id:\"VV\",bands:1},{id:\"VH\",bands:1},{id:\"dataMask\",bands:1}]}}\n\n    function evaluatePixel(samples) {\n        return {VV: [samples.VV],VH: [samples.VH],dataMask: [samples.dataMask]}\n}\n\n\n"
  },
  "calculations": {
    "default": {}
  }
}'

Best,
Grega

#9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.