Integrate data from the dark ecology dataset?

[bart1]

I think it would be nice to also include this data in the get_vpts function?

In a quick glance it looks like the data is packaged yearly? I wonder would there be a way to also access smaller subsets? For example monthly or per radar?

@peterdesmet or @dsheldon ?

https://github.com/darkecology/darkeco-dataset
https://zenodo.org/records/18436879

[dsheldon]

Yes, it would be great to include Dark Ecology data in get_vpts. The data is packaged yearly but extracts into individual csv files, one for each vertical profile, so these could be served directly or repackaged into smaller compressed subsets.

Another question is where to host the files. Did you have something in mind or would we need to set this up? Both AWS and UMass servers are possibilities but there is some cost and/or maintenance overhead.

[bart1]

Thanks for your feedback. I think for dynamic work the current setup would require very large downloads. Say some one wants too look into fall migration along a coast in that case you would need to download a full year of data (~10 GB) while your probably only interested in ~1.5% of the data (10/150 radars, 3/12 months). I currently do not have the facilities to set this up (indeed maintenance overhead). For aloft we use daily csv's but I think monthly files per radar should also work fine (~15 MB gz compressed). I see zenodo is limited to a 100 files so that is not an option for monthly files per radar.

An alternative would be for the time being to support reading files users have downloaded before ( see local directory option for https://aloftdata.github.io/getRad/reference/get_vpts.html#arg-source ).

[peterdesmet]

An alternative would be for the time being to support reading files users have downloaded before

I think that would be valuable!

[bart1]

I think that would be valuable!

@peterdesmet Do you see a way to support reading data directly or should we leave that for later?

[peterdesmet]

Do you see a way to support reading data directly

No, not without setting up new infrastructure. That's why I think we should focus on making get_vpts() understand the dark ecology directory structure and csv format of local files.

So the use case to support would be: you downloaded and unzipped profiles_2015.tar.bz2 from https://zenodo.org/records/18436879. You now want to use get_vpts() to read a subset of that local dataset and get it as a bioRad vpts object that you can pass to plot function.

[bart1]

Yes indeed maybe we can actually leave the files compressed, I will explore a bit.

[dsheldon]

Just a reminder that the bioRad function read_cajun already knows how to read a single Dark Ecology csv file (cajun refers to a previous codename for the project). So the remaining work would be to make it understand the directory structure. (We use the same directory organization as the publicly available volume scan data on AWS.)

Also, I do have a copy of all of the unzipped csv files on AWS. I would consider opening it up for download but couldn't commit to supporting it indefinitely. Maybe we can start with the version that reads local files and at some point upgrade to reading from AWS, as much of the logic would be the same.

[bart1]

Thanks for the hint I was already experimenting but not using the US data frequently I was not aware of read_cajun. I will make a quick comparison checking out the speed for reading larger sections of data. Lets for now start with the local read and then we can see what version of online data could work. My expectations is that if you read single vp's from an AWS bucket might be quite slow for downloading longer time series. Now a single radar for a month with read_cajun or vroom takes about 15 to 30 seconds.

[peterdesmet]

Approach to look for local data:

1. Create a list of directories based on user provided radar, year, month, day
2. Get the list of files from those directories. This is needed since we can't predict the file names.
3. Filter file list further on hour, minutes, seconds.
4. Read all retained files.

[bart1]

I just discussed with @PietrH here are some example call's. radar en date_range function as normally. We would add a path argument

# loads data in the aloft data structure (because baltrad is the default)
get_vpts(radar, date_range, path = "/home/user/local_folder/")
# Same result
get_vpts(
  radar,
  date_range,
  source = "baltrad",
  path = "/home/user/local_folder/"
)
# As Uva follows the same structure it would result in the same result?
get_vpts(
  radar,
  date_range,
  source = "uva",
  path = "/home/user/local_folder/"
)
# If for some reason the local file organization is different (e.g. such as on zenodo), use an option (the default option is the aloft online format?)
withr::with_options(
  list(
    "getRad.vpts_local_path_format_aloft" = "{substr(radar, 1,2)}/{radar}/{year}/{radar}_vpts_{year}{month}.csv.gz"
  ),
  get_vpts(
    radar,
    date_range,
    source = "baltrad",
    path = "/home/user/local_folder/"
  )
)

### DARK ECOLOGY ###
# For reading dark ecology data (this will probably use `read_cajun`)
get_vpts(
  radar,
  date_range,
  source = "dark_ecology",
  path = "/home/user/local_folder/"
)
# And also there there is an option to change the path format
withr::with_options(
  list(
    "getRad.vpts_local_path_format_dark_ecology" = "{radar}/{year}{month}{day}/"
  ),
  get_vpts(
    radar,
    date_range,
    source = "baltrad",
    path = "/home/user/local_folder/"
  )
)
# Open question is if for dark_ecology we just read all files and the filter 
# the vpts afterwards. As long as the folders are daily I think this is reasonable 
# as you will only read a few extra files (max a day at start and end) and avoids specifying the file format.

# The following calls should fail as they are currently not implemented:

get_vpts(
  radar,
  date_range,
  source = "rmi",
  path = "/home/user/local_folder/"
)
get_vpts(
  radar,
  date_range,
  source = "dark_ecology"
)

This changes the behavior of get_vpts a bit for local files but that is not yet in a released version so I think it is fine.