deleted ghjkljhghjkjhjk
Hi Japheth,
Could you also share the payload of your Batch request, so we can help you find the issue?
Is this what you need? A little explanation; I’m doing this using python and what I did is I created multiple input_data objects for each dekad (10 day period), added them to a list before passing them to SentinelHubRequest. Using that evalscript works well with one input_data object but I’m not sure how to edit it to make it work with multiple input_data objects for multiple dekads.
It would be more efficient to set the payload dates to cover the full time-range for a single input and compute the dekadal results within the Evalscript. With the following function:
function evaluatePixel(samples, scenes) {}
you can access the dates of your images through the scenes object. More information can be found in the documentation page.
This being said, you can make your approach work, but you will need to identify your inputs and refer to them in the Evalscript. Below is a very basic example of how this would work.
Evalscript
//VERSION=3
function setup() {
return{
input: [
{
datasource: 'DEKAD1',
bands: ["B04", "B08", "SCL", "CLM"],
units: "DN"
},
{
datasource: 'DEKAD2',
bands: ["B04", "B08", "SCL", "CLM"],
units: "DN"
}
],
output: {
id: "default",
bands: 1,
sampleType: SampleType.FLOAT32
},
}
}
function evaluatePixel(sample) {
// Access Dekad1
let ndviD1 = (sample.DEKAD1.B08 - sample.DEKAD1.B04) / (sample.DEKAD1.B08 + sample.DEKAD1.B04)
if ([8, 9, 10].includes(sample.DEKAD1.SCL) ){
return [-10]
} else if ([1].includes(sample.DEKAD1.CLM) ){
return [-10]
} else{
return [ ndviD1 ]}
}
Here you can access your different inputs as keys of your sample parameter (you can loop over them or do whatever you need with them).
SentinelHubRequest
The input data of your sentinelhubrequest should have defined identifiers:
"data": [
{
"dataFilter": {
"timeRange": {
"from": "2022-09-01T00:00:00Z",
"to": "2022-09-10T23:59:59Z"
}
},
"type": "sentinel-2-l2a",
"id": "DEKAD1"
},
{
"dataFilter": {
"timeRange": {
"from": "2022-09-11T00:00:00Z",
"to": "2022-09-20T23:59:59Z"
}
},
"type": "sentinel-2-l2a",
"id": "DEKAD2"
}
]
Don’t hesitate if you need clarifications on a specific point in my answer.
Thanks @maxim.lamare I have tried the second suggestion using the evalscript below:
//VERSION=3
function setup() {
let inputs = []
for (let d=0; d<364; d+=10) {
inputs.push({
datasource: 'DEKAD'+d,
bands: ["B04", "B08", "SCL", "CLM"],
units: "DN"
})
}
return {
input: inputs,
output: {
id: "default",
bands: 1,
sampleType: SampleType.FLOAT32
}
}
}
function evaluatePixel(sample) {
for (const key of Object.keys(sample)) {
let ndviD = (sample[key].B08 - sample[key].B04) / (sample[key].B08 + sample[key].B04)
if ([8, 9, 10].includes(sample[key].SCL) ){
return [-10]
} else if ([1].includes(sample[key].CLM) ){
return [-10]
} else{
return [ ndviD ]
}
}
}
This runs but produces Geotiffs for one dekad and the Geotiffs have nodata values hence not very useful.
I would really love to try the first method that involves accessing the scenes in the evaluatePixel method. But even after going through the documentation I can’t figure out how to return Geotiffs for each individual dekad. Here’s how I’m accessing each individual scene:
function evaluatePixel(samples, scenes) {
for (let j=0; j<scenes.orbits.length; j++) {
for (let i=0; i<samples.length; i++){
let ndvi = (samples[i].B08 - samples[i].B04) / (samples[i].B08 + samples[i].B04)
if ([8, 9, 10].includes(samples[i].SCL) ){
return [-10]
} else if ([1].includes(samples[i].CLM) ){
return [-10]
} else{
return [ ndvi ]
}
}
}
}
My question is if I access the date using scenes.orbits[i].dateFrom, how do I use this value? Or rather how would you write your evalscript/evaluatePixel method?
I’m quite new to sentinelhub so any assistance will be highly appreciated however novice this may be. Thanks.
Hi @nyandorojapheth, is there a reason why you deleted your original posts?
Looking at your latest Evalscript, the reason that you don’t get a Geotiff per dekad is because you would need to set the outputs in the Evalscript to output multiple Geotiffs.
Instead of using return in your evaluatePixel function, ideally you would push the results in the loop to an array or object and when you have finished looping over all dekads, return each dekad to an output.
In this case, the Evalscript would look like this:
//VERSION=3
function setup() {
// Just as you set the inputs, we set the outputs
let inputs = []
let outputs = []
for (let d = 0; d <= 364; d += 10) {
inputs.push({
datasource: 'DEKAD' + d,
bands: ["B04", "B08", "SCL", "CLM"],
units: "DN"
})
outputs.push({
id: 'NDVI_DEKAD' + d,
bands: 1,
sampleType: SampleType.FLOAT32
}
)
}
return {
input: inputs,
output: outputs,
}
}
function evaluatePixel(sample) {
// Rather than return to a single output geotiff, we push results to an array
// I choose an array for the example, you could also use an Object with keys
var dekadalData = []
for (const key of Object.keys(sample)) {
let ndviD = (sample[key][0].B08 - sample[key][0].B04) / (sample[key][0].B08 + sample[key][0].B04)
if ([8, 9, 10].includes(sample[key][0].SCL)) {
dekadalData.push(-10)
} else if ([1].includes(sample[key][0].CLM)) {
dekadalData.push(-10)
} else {
dekadalData.push(ndviD)
}
}
// Build up an output object with all geotiffs
// https://docs.sentinel-hub.com/api/latest/evalscript/v3/#examples-1
var outputNdvi = {}
cnt = 0
for (let d = 0; d <= 364; d += 10) {
outputNdvi['NDVI_DEKAD' + d] = [dekadalData[cnt]]
cnt++
}
return outputNdvi
}
With the Evalscript above, you would need to build the payload with Python too:
responses=[
SentinelHubRequest.output_response('NDVI_DEKAD0', MimeType.TIFF),
SentinelHubRequest.output_response('NDVI_DEKAD10', MimeType.TIFF),
SentinelHubRequest.output_response('NDVI_DEKAD20', MimeType.TIFF),
SentinelHubRequest.output_response('NDVI_DEKAD30', MimeType.TIFF),
...
],
which I would build as:
responses = [SentinelHubRequest.output_response(f'NDVI_DEKAD{x}', MimeType.TIFF) for x in range(0, 364, 10) ]
For tips on how I would streamline the process, see my next post!
I would really love to try the first method that involves accessing the scenes in the evaluatePixel method.
I don’t know if this fits your workflow, but it would make things much easier if you return a single Geotiff with each individual band representing a Dekad.
The following Evalscript is how I would approach the problem. I think it can give you some good ideas: feel free to tweak it to match your use-case. Bear in mind that I am not a developer, so this example may be optimised from a coding point of view:
//VERSION=3
// Set start and end date
const startDate = new Date(2021, 0, 1)
const endDate = new Date(2021, 11, 31)
function setup() {
return {
input: ["B04", "B08", "SCL", "CLM"],
output: [{
id: "NDVI_DEKADS",
bands: 1,
sampleType: SampleType.FLOAT32
}
],
mosaicking: Mosaicking.ORBIT
}
}
// Sort the images in a chronological order
function preProcessScenes(collections) {
collections.scenes.orbits.sort(function (s1, s2) {
var date1 = new Date(s1.dateFrom)
var date2 = new Date(s2.dateFrom)
return date1 - date2
})
return collections
}
// Compute the number of output bands based on Dekads
const dekads = Math.floor(364 / 10)
// Update the number of output bands of your Geotiff
function updateOutput(output, collection) {
output.NDVI_DEKADS.bands = dekads
}
// Output the Dekad dates to a JSON file for monitoring
function updateOutputMetadata(scenes, inputMetadata, outputMetadata) {
let dekadDates = []
// Loop through dekads and add dates that fall within
let deakadStartDate = new Date(startDate.getTime())
let deakadEndDate = new Date(startDate.getTime())
deakadEndDate.setDate(startDate.getDate() + 9)
while (deakadEndDate < endDate) {
dekadDates.push(deakadStartDate.toString() + "-" + deakadEndDate.toString())
deakadStartDate = new Date(deakadEndDate.getTime())
deakadStartDate.setDate(deakadStartDate.getDate() + 1)
deakadEndDate = new Date(deakadStartDate.getTime());
deakadEndDate.setDate(deakadEndDate.getDate() + 9)
}
outputMetadata.userData = {
dekadDates
}
}
function evaluatePixel(samples, scenes) {
// Set dekad bounds
var deakadStartDate = new Date(startDate.getTime())
var deakadEndDate = new Date(startDate.getTime())
deakadEndDate.setDate(startDate.getDate() + 9)
var dekadData = []
// Loop through dekads
while (deakadEndDate < endDate) {
// Check scenes that fall in interval
var currentDekad = []
for (let i = 0; i < samples.length; i++) {
if (scenes[i].date > deakadStartDate && scenes[i].date < deakadEndDate) {
// Add ndvi of scenes in the dekad
// Here you would add the operations with CLM and SCL
let ndvi = (samples[i].B08 - samples[i].B04) / (samples[i].B08 + samples[i].B04)
currentDekad.push(ndvi)
}
}
// If empty return -10
if (currentDekad.length === 0) {
dekadData.push(-10)
} else if (currentDekad.length === 1) {
// If there is only 1 acquisition, return it
dekadData.push(currentDekad[0])
} else {
// If there are several dates in the dekad, you may want to average them or select the one without -10
// In this example I return the first value
dekadData.push(currentDekad[0])
}
// Update dekad
deakadStartDate = new Date(deakadEndDate.getTime())
deakadStartDate.setDate(deakadStartDate.getDate() + 1)
deakadEndDate = new Date(deakadStartDate.getTime());
deakadEndDate.setDate(deakadEndDate.getDate() + 9)
}
return {
NDVI_DEKADS: dekadData
}
}
Below is a snippet of the payload for example:
"data": [
{
"dataFilter": {
"timeRange": {
"from": "2021-01-01T00:00:00Z",
"to": "2021-12-31T23:59:59Z"
}
},
"type": "sentinel-2-l2a"
}
]
},
"output": {
"responses": [
{
"identifier": "NDVI_DEKADS",
"format": {
"type": "image/tiff"
}
},
{
"identifier": "userdata",
"format": {
"type": "application/json"
}
}
]
},
@maxim.lamare Thanks very much. I deleted the first few comments on the question bcoz of the code snippets. I hadn’t properly reviewed them to avoid sharing sensitive code.
I have another question, I decided to loop over a 5 year period and hence I have around 185 dekads. Does sentinelhub have a limitation for the number of bands returned per tile? From my knowledge geotiffs can hold hundreds of bands at a go.
The maximum number of bands returned by Sentinel Hub depends on the size (width*height) of your raster (and therefore on the Batch grid option + resolution that you selected), as well as sampleType.
If you are working at 10m resolution, with NDVI in Float32, you might be pushing the limits a bit too much. It would probably be wiser to split the requests into yearly Batch requests in a loop, changing the year variable programmatically.
Thanks for your response. The bounding box polygon is 10X10km so I think the width/height should be 1000 by 1000 pixels at 10m resolution per pixel. I have tried for the 5 year period with Float32 and it has failed with no error. I think because of the issue you’ve noted.
Is it possible for it to work if I use UINT16 instead? Or which sample type can work with 185 bands. The ideal case for us is to have all the bands in one Geotiff.
Did you run a Process API request or a Batch API request? And how did you run the request: in Python, with Request Builder, as a CURL request…? There is a way of getting an error message for all failed requests, but it depends on your request type.
You could try with UINT16, but I am not 100% sure it will work: you can only test to be sure. To get NDVI in UINT16 please refer to my blog post under the section Example 2: Indices.
I ran a batch api request in python. I used requests.get(https://services.sentinel-hub.com/api/v1/batch/process/{request_id}) on that endpoint to check status after every few minutes. Status is shown if I convert the response using: response.json() and print the output.
The status was processing for the first 35 minutes or so and then it changed to ‘Failed’. I tried to look for the error key in the json but couldn’t find it.
Is there an endpoint for checking errors?
How many parallel batch api requests can be made at a go? I’m thinking of doing multiple requests for each year. This also needs to be done for multiple regions so it would be good to know how many parallel batch api requests can be made at a go.
Is there an endpoint for checking errors?
There is no specific endpoint for checking the errors, but you can get to the error message by looking up the class attribute of your Batch Request:
batch_request = batch.get_request("<batch-ID>")
print(batch_request.error)
This should help in most cases.
How many parallel batch api requests can be made at a go?
You are not limited in the number of parallel Batch requests that you can make, as it is an asynchronous service. However, you have to be aware that depending on the load on our services, the requests may be queued and only start processing later.
Great! All is clear now. Thank you very much.
It seems like the size parameter that is supported when making process API requests isn’t supported in batch API. In process API one would do:
SentinelHubRequest(
evalscript=evalscript,
input_data=[
SentinelHubRequest.input_data(
data_collection=datasource,
time_interval=(from_date, to_date),
mosaicking_order='leastCC'
)
],
responses=[
SentinelHubRequest.output_response('default', MimeType.TIFF)
],
bbox=betsiboka_bbox,
size=bbox_to_dimensions(bbox, resolution=10),
config=config
)
But in batch API that would create an error:
"{"error":{"status":400,"reason":"Bad Request","message":"Request's field processRequest.output must not specify width, height, nor resolution.","code":"COMMON_BAD_PAYLOAD"}}"
So one has to add an extra step for clipping the output tiles to his/her area of interest?
Indeed, in Batch you specify your area of interest only. The resolution of your output is set with the tiling grid: https://docs.sentinel-hub.com/api/latest/api/batch/#tiling-grids.
Since this API is designed for large areas, the service divides the area of interest into tiles and processes each tile separately. You end up with the tiles in your object storage. You are right in that clipping the output to your AOI is a post-processing step.
Sure, that’s makes a lot of sense. Thanks.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.