Make consumer retry earlier init_times if the current one i now available

[AUdaltsova]

When UKV service is severely delayed, sometimes it has already started returning earlier init-times, but we still wouldn't download them because the consumer is looking for the latest one, which is still delayed. For example:

• The consumer has downloaded the 6 am init-time
• UKV started having issues after that, and 9 am init-time came in at 5 pm. Later init-time started coming in after that, but the service is still delayed several hours for "current" init-time
• Consumer continuously fails, as by now it's trying to download the 21:00 run which is delayed, and is not looking for the 9:00, 12:00, 15:00 etc that are now available
• Day Ahead model starts running out of UKV data at the tail end and can't rerun, intraday forecast is running on data 12h+ old
  (This is what happened yesterday)

One thing I thought could solve this, is if after not being able to find the current init-time the consumer would look for the one before, and the one before, and so on, until it gets to the last one it has downloaded

[peterdudfield]

If we want to add something like this, when we are looking back in time, It would be good to make sure we dont redownload data that we have already downloaded

[AUdaltsova]

Yeah I agree, that's what I meant by "until it gets to the last one it has downloaded"! So in my mind it would run the job as before, and if there is no data at source, it would check if the previous one was already downloaded and if not, retry that, and so on

[peterdudfield]

I think @devsjc is looking into this, and possibly a solution could be

• change the run time to be a env var
• if we rerun a ariflow task at an earlier time, it will use the correct time for that run, rather than now