We are trying to search for available dates from the Catalog API and consume data from the Statistical API as soon as possible.
What is the best approach to know when there are new dates?
Looking through the forum, we realized that there’s an SNS topic we can suscribe to arn:aws:sns:eu-central-1:214830741341:SentinelS2L2A from the dataset Sentinel-2 - Registry of Open Data on AWS.
After we receive that notification, are new dates already ready in the Catalog API and Statistical API to be processed? If not, is there a way for us to know when that new date is ready in the Catalog and Statistical APIs or at least how much time we need to wait? Or maybe is there a better approach for what we are trying to accomplish?
Also, how many times is this notification fired/ is data added? Trying to understand if that would fire my SQS/Lambda too many times.
We do currently not have a dedicated notification service (it is in plans, but not short-term), so your proposed approach makes sense. It should not take more than a few minutes, after the product becomes available on S3 and SNS is triggered, to be available in Sentinel Hub’s catalogue as well (and at the same time, through StatAPI).
What do you mean by SNS should be triggered only once per product? Is it only once per day per product? I’m interested in S2-L2A, so does that mean I’m gonna get one notification per day for that product?
Just want to be sure that we are not gonna be triggering a lambda or SQS and then querying Sentinel API’s too many times per day
SNS notification comes as soon as the individual product is published on S3. So cca 12.000 times a day. You then need to parse the notification to see if it is relevant for you, i.e. if it is in the area of interest, and act upon it. Each notification carries information from productInfo.json (example).