I currently subscribe to the following SNS Topic to trigger my data pipeline.
However, what I’ve noticed is when we read the generated metadata/info link immediately, every hour, 50-150 of the 1200~1600 messages do not have the tileDataGeometry available therefore my lambda fails. In order to avoid this I have setup a delay on our SQS queue to wait to ingest the metadata 10 minutes from SNS topic, and on failure we wait another 30 minutes and we have not had an error for the past two weeks.
I was wondering if there’s any information regarding this delay, if it was intentional, how long does it usually take to populate, and if the SNS message can be sent once the data can be populated. And I was wondering how others may have handled this if they are consuming similar data.
I am extracting data from these links. https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/39/X/WD/2019/10/10/0/tileInfo.json