Split upload: archive contents don't match the documented design (duplicated timelapse, missing CALSTARS)

[Cybis320]

With upload_split: true, the night is uploaded as two archives, and the config documents the intent (.config:401-409):

If upload_split is true, two archives will be created. One archive, suffix _imgdata, contains the images. The other archive, suffix _metadata will only contain the data derived from the images, including the timelapse. The metadata archive is much smaller than the imagedata archive, and can be extracted very quickly.

The actual archive contents diverge from this in both directions:

1. Heavyweight derived products are duplicated into _imgdata — including the timelapse, so stations upload it twice every night.
2. _imgdata is missing the one small file that would make it self-contained for calibration work: CALSTARS.

Cause

In RMS/ArchiveDetections.py, the split predicate is a filename test:

imgdata_set = (set([item for item in file_list if item.startswith("FF") and item.endswith(".fits")]) |
               set([item for item in file_list if item.startswith("FR") and item.endswith(".bin")]))
metadata_set = set([item for item in file_list if item not in imgdata_set])

...but both archiveDir() calls then receive the full extra_files list, and Misc.archiveDir() copies those files into each archive directory. extra_files (assembled in RMS/Reprocess.py:536-576) is not just the small calibration files — it includes the timelapse.mp4, the FT timestamps tarball, flux products, and sky-model .npz files.

Meanwhile CALSTARS and FTPdetectinfo are not in extra_files; they ride in the generic night file list as TXT files and therefore land in _metadata only.

Observed contents of a real production archive

CA000Q_20260604_013355_478445_imgdata.tar.bz2 (station on upload_mode with restricted FF upload) contains:

8x  FF_*.fits                          <- the actual image payload
7x  FR_*.bin
    .config                            (extra_files duplication - good)
    mask.bmp                           (good)
    platepar_cmn2010.cal               (good)
    platepars_all_recalibrated.json    (good)
    platepars_flux_recalibrated.json
    CA000Q_..._timelapse.mp4           <- documented as _metadata content; duplicated
    CA000Q_20260603-154_FT.tar.bz2     <- duplicated
    flux_*.json (x3)                   <- duplicated
    CA000Q-{25,70,100}km.kml           <- duplicated
    CA000Q_..._observation_summary.{json,txt}
    CA000Q_..._config_audit_report.txt
    CA000Q_..._ff_intervals.png

Not present: CALSTARS_*.txt, FTPdetectinfo_*.txt.

Why it matters

Wasted bandwidth. The timelapse is typically the second-largest file of the night. Duplicating it (plus FT tar and npz products) into _imgdata doubles their upload cost across the fleet, on the archive that is already the big one — directly against the stated purpose of the split.

_imgdata is not self-contained for calibration. Downloading _imgdata is the natural thing to do to open a night in SkyFit2 — and it opens fine (images, config, platepar, mask are all there). But without CALSTARS the detected-star are missing, and the user has to go back for _metadata — for ~1 MB of compressed text missing from a multi-hundred-MB archive.

[g7gpr]

Duplicating it (plus FT tar and npz products) into _imgdata doubles their upload cost across the fleet, on the archive that is already the big one — directly against the stated purpose of the split.

Where was this stated please?

[Cybis320]

You're right - I didn't read carefully. The config states the purpose as server-side extraction speed, and the duplication doesn't undermine that.

The two underlying observations still stand, though: (1) the timelapse, FT tarball, and flux products are uploaded twice per night per station — is that duplication intentional? And (2) CALSTARS isn't in _imgdata, so it's not self-contained for SkyFit2 calibration work - which I think is inconvenient.

[g7gpr]

Given that

1. Some files are already duplicated
2. There is a genuine good reason to duplicate CALSTARS into imgdata
3. imgdata is not routinely extracted, so will not have any impact on the weblog processing time
4. CALSTARS will have a proportionally small effect on the size of imgdata archive

I think this is a good idea; it's a few lines of code to implement it. @dvida, your thoughts?

[dvida]

Hi guys - I always considered the two archives to travel together to make a whole, and I always download both locally from the server. Instead of duplicating data (the CALSTARS file in particular can be quite large), why don't we modify tools/skyfit to look for the equivalent archive/directory pair when seraching for additional data?

[Cybis320]

If size is now the issue, then the architecture is a bit off. The calstars is quite smaller than the meteor mp4 which is duplicated. It doesn't make sense to me that we would duplicate mp4 but not calstars. Also, it'd add unnecessary complexity to skyfit. In addition, it requires the user to download and unpack two archives. I'd say the most sensible thing to do from here is to not upload the mp4 twice and keep calstars with the imgdata.