Data not converted to csv for `deisn` on 2025-7-30

[bart1]

There seems to be plenty data here (I just looked at the presence of files): https://aloftdata.eu/browse/?prefix=baltrad/hdf5/deisn/2025/07/30
However I can't find the corresponding csv here: https://aloftdata.eu/browse/?prefix=baltrad/daily/deisn/2025/

I would expect a file called deisn_vpts_20250730.csv

[bart1]

This also seem to cause an error in getRad

require(getRad)
#> Loading required package: getRad
require(dplyr)
get_vpts("deisn", as.Date("2025-7-1")+27:30)
#> Error in `get_vpts()`:
#> ℹ In index: 1.
#> Caused by error in `httr2::req_perform_parallel()`:
#> ! HTTP 404 Not Found.
get_vpts_coverage("baltrad")|> filter(radar=="deisn", date %in%(as.Date("2025-7-1")+27:30))
#> # A tibble: 4 × 5
#>   source  radar date       directory                     file_count
#>   <chr>   <chr> <date>     <chr>                              <dbl>
#> 1 baltrad deisn 2025-07-28 baltrad/hdf5/deisn/2025/07/28        288
#> 2 baltrad deisn 2025-07-29 baltrad/hdf5/deisn/2025/07/29        288
#> 3 baltrad deisn 2025-07-30 baltrad/hdf5/deisn/2025/07/30        288
#> 4 baltrad deisn 2025-07-31 baltrad/hdf5/deisn/2025/07/31        288

[bart1]

A quick look into the monthly csv shows the data is also missing there

[bart1]

demem and nlhrw seem to have the same issue so maybe it is a more general issue?

[PietrH]

That vpts should have been generated by the workflow, not by the manual rerun done by me on HPC. I don't seem to have gotten an error message for that day.

I did for 20250731:

CLI routine 'vph5_to_vpts --modified-days-ago 2' failed raising error: '<class 'FileNotFoundError'>: inbo-aloft-prod-eu-west-1-inventory/aloftdata/aloftdata-hdf5-files-inventory/2025-07-31T01-00Z/manifest.json'.

But that may be down to the error messaging?

I have 2 observations:

• We could generate this vpts file, put it on aloft and that would fix it
• getRad should be able to fail on this elegantly

What is going wrong in getRad? The file is missing in coverage, so it shouldn't attempt to fetch it?

[bart1]

What is going wrong in getRad? The file is missing in coverage, so it shouldn't attempt to fetch it?

Hmm I think the file is present in coverage tibble (see reprex) so I guess it attempts to fetch it?

Let me make in issue for getRad for elegant failing

[PietrH]

The coverage is created based on a file listing of the files on the server, at least that's how I understand it. If the file is in the coverage, it at least was at some point also stored on the server.

Or have I got that all wrong?

[bart1]

I think the files are still in the bucket (see hdf5 prefix) but apparently somewhere in the process the conversion failed meaning the daily csv's have not been created?

[PietrH]

Of course, I was confused there for a moment. You are absolutely right.

It seems that the conversion went wrong but at least I didn't get a message. I think I should have.