From bfd9ec1e02057c9748b10040d238be641bef740a Mon Sep 17 00:00:00 2001 From: Tharindu Jayasinghe Date: Mon, 29 Jun 2026 21:54:44 -0700 Subject: [PATCH 1/2] Add survey-aware light-curve column auto-detection Expand ColumnMap's detection lists to cover the standard light-curve products of the major time-domain surveys, so a downloaded table needs no column hints: ASAS-SN, ASAS-3, ATLAS, CRTS/CSS, ZTF, Pan-STARRS, LSST, TESS, Kepler, Gaia, MACHO. Headers added from each survey's authoritative format docs: magerr (ZTF/CRTS), flux_error (Gaia), dm/uJy/duJy (ATLAS), psfFlux/psfFluxErr/obsTime/filterID (Pan-STARRS), midpointMjdTai/band + DP0.2 midPointTai/psFlux/filterName (LSST), MAG_0/MER_0 (ASAS-3), filtercode/hmjd (ZTF), bkjd (Kepler), KSPSAP_FLUX (TESS QLP). Error resolution is now domain- and measurement-aware: the error is derived from the resolved value column first (PDCSAP_FLUX -> PDCSAP_FLUX_ERR, uJy -> duJy), then the domain-appropriate list, then neutral spellings. This pairs a flux value with a flux error and a magnitude value with a magnitude error even when a table carries both (ATLAS m/dm + uJy/duJy), and avoids the ATLAS F-filter vs flux-f collision. Corrected/detrended fluxes (PDCSAP, KSPSAP) are preferred over raw SAP; infer_domain recognizes flux-unit names (uJy/mJy/nmgy). Docs: a "Supported surveys" table in the light-curves guide, incl. headerless OGLE and positional reading. Tests: a parametrized check over 13 real survey header sets. Verified: pytest 137 passed / 2 skipped, ruff + mypy clean, docs -W build clean. Co-Authored-By: Claude Opus 4.8 --- CHANGELOG.md | 6 +- docs/guide/light-curves.md | 117 +++++++++++++++++++++++++++++++++-- src/cuperiod/core/columns.py | 87 +++++++++++++++++++++++--- tests/test_columns.py | 70 +++++++++++++++++++++ 4 files changed, 265 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 018e38e..db60052 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,8 +24,10 @@ First public release. `cuperiod.batch_periodograms` (CPU process pool or GPU, resumable Parquet/CSV output). - Multi-band joint modelling for GLS, BLS, and MHAOV. - Frictionless inputs: arrays, dicts, pandas/astropy/pyarrow tables, and - CSV/ECSV/FITS/Parquet files, with case-insensitive column auto-detection (`ColumnMap`) - and automatic magnitude/flux handling. + CSV/ECSV/FITS/Parquet files, with case-insensitive, **survey-aware** column + auto-detection (`ColumnMap` — ASAS-SN, ASAS-3, ATLAS, CRTS/CSS, ZTF, Pan-STARRS, LSST, + TESS, Kepler, Gaia, MACHO) and domain-aware error pairing, plus automatic + magnitude/flux handling. - N-best-period peak finding with alias- and harmonic-aware selection. - `cuperiod` command-line interface: `run`, `batch`, `methods`, `gpu-info`, `grid-info`. - Per-method settings models with `CUPERIOD__` environment overrides, and diff --git a/docs/guide/light-curves.md b/docs/guide/light-curves.md index 7e98e2d..f622f25 100644 --- a/docs/guide/light-curves.md +++ b/docs/guide/light-curves.md @@ -62,11 +62,20 @@ Astronomers' tables carry the same three quantities under wildly different names detects the common spellings case-insensitively, most-specific first, so a table with both `BJD_TDB` and `JD` picks the corrected time. The detection lists are: -- **Time** — `bjd_tdb`, `bjd`, `hjd`, `btjd`, `mjd`, `jd`, `time`, `date`, `t` -- **Magnitude** — `mag`, `magnitude`, `vmag`, `gmag`, `rmag`, `phot_mag`, `m` -- **Flux** — `pdcsap_flux`, `sap_flux`, `norm_flux`, `rel_flux`, `flux`, `fnu`, `f` -- **Error** — `mag_err`, `e_mag`, `dmag`, `flux_err`, `err`, `sigma`, … (mag & flux spellings) -- **Band** — `band`, `filter`, `phot_filter`, `passband`, `fid` +- **Time** — `bjd_tdb`, `bjd`, `hjd`, `btjd`, `bkjd`, `mjd`, `hmjd`, `midpointMjdTai`, + `obsTime`, `jd`, `time`, `date`, `t` +- **Magnitude** — `mag`, `magnitude`, `vmag`/`gmag`/`rmag`/`bmag`/`imag`, `psfMag`, + `MAG_0`, `m` +- **Flux** — `PDCSAP_FLUX`, `KSPSAP_FLUX`, `SAP_FLUX`, `psfFlux`, `flux`, `uJy`, `mJy`, … + (corrected/detrended fluxes are preferred over raw) +- **Error** — resolved to match the value's measurement and domain: a flux value pairs + with `flux_err`/`flux_error`/`psfFluxErr`/`duJy`, a magnitude value with + `mag_err`/`magerr`/`dm`/`MER_0`, then the neutral `err`/`error`/`sigma` +- **Band** — `band`, `filter`, `filtercode`, `filterID`, `phot_filter`, `passband`, `fid` + +The error is matched to the **same measurement** as the value, so a table that carries +both (e.g. ATLAS's `m`/`dm` and `uJy`/`duJy`) pairs `uJy` with `duJy`, never with the +magnitude error. When detection isn't enough — ambiguous names, or a column you want to force — pin it with {class}`~cuperiod.ColumnMap`: @@ -82,6 +91,104 @@ Any field left `None` is auto-detected; a non-`None` field is honored verbatim ( raises {exc}`~cuperiod.ColumnResolutionError` if that column is absent). Only `time` and `value` are required; `error` and `band` are optional. +## Supported surveys + +The detection lists cover the standard light-curve products of the major time-domain +surveys, so a downloaded table usually needs **no column hints** — `cup.periodogram(df, +"GLS")` just works: + +```{list-table} +:header-rows: 1 +:widths: 18 16 22 22 16 + +* - Survey + - Time + - Value + - Error + - Band +* - ASAS-SN (Sky Patrol) + - `jd` / `hjd` + - `mag` / `flux` + - `mag_err` / `flux_err` + - `phot_filter` +* - ASAS-3 + - `HJD` + - `MAG_0`..`MAG_4` + - `MER_0`..`MER_4` + - — +* - ATLAS + - `MJD` + - `uJy` (or `m`) + - `duJy` (or `dm`) + - `F` ¹ +* - CRTS / CSS + - `MJD` + - `Mag` + - `Magerr` + - — +* - ZTF + - `mjd` / `HMJD` + - `mag` + - `magerr` + - `filtercode` / `filterID` +* - Pan-STARRS (PS1) + - `obsTime` + - `psfFlux` + - `psfFluxErr` + - `filterID` +* - LSST / Rubin + - `midpointMjdTai` ² + - `psfFlux` ² + - `psfFluxErr` ² + - `band` +* - TESS (SPOC / QLP) + - `TIME` + - `PDCSAP_FLUX` / `KSPSAP_FLUX` + - `*_FLUX_ERR` + - — +* - Kepler (SPOC) + - `TIME` + - `PDCSAP_FLUX` + - `PDCSAP_FLUX_ERR` + - — +* - Gaia (DR3 epoch) + - `time` + - `flux` (or `mag`) + - `flux_error` + - `band` +* - MACHO + - `time` / `mjd` + - `mag` (red/blue) + - `error` + - red / blue ³ +* - OGLE + - `HJD` + - magnitude + - error + - — (per file) +``` + +¹ ATLAS's single-character `F` filter isn't auto-detected (to avoid clashing with a flux +`f`); pass `ColumnMap(band="F")` if you need it for multi-band. The value defaults to the +flux (`uJy`); pin `ColumnMap(value="m")` for magnitudes. ² LSST DP0.2 uses the older +`midPointTai` / `psFlux` / `psFluxErr` / `filterName`, which are also detected. ³ MACHO's +raw per-column header spellings are not consistently documented; if a file uses +non-standard names, pass an explicit {class}`~cuperiod.ColumnMap`. + +:::{note} +**Headerless files** — OGLE `.dat` photometry (and some raw ASAS-3 dumps) have no column +names to detect. Read them positionally instead, e.g. with +[`numpy.loadtxt`](https://numpy.org/doc/stable/reference/generated/numpy.loadtxt.html): + +```python +import numpy as np +from cuperiod import LightCurve + +hjd, mag, err = np.loadtxt("OGLE-LMC-CEP-0001.dat", usecols=(0, 1, 2), unpack=True) +pg = cup.periodogram(LightCurve.from_arrays(hjd, mag, err), "GLS") +``` +::: + ## Magnitude vs flux: the `Domain` A value column is either a **magnitude** or a **flux**. cuPeriod infers which from the diff --git a/src/cuperiod/core/columns.py b/src/cuperiod/core/columns.py index a44ebde..fe6f2f7 100644 --- a/src/cuperiod/core/columns.py +++ b/src/cuperiod/core/columns.py @@ -10,6 +10,15 @@ The brightness *domain* (magnitude vs flux) is inferred from the matched value column when possible (``flux`` → flux, ``mag`` → magnitude) and can always be set explicitly. Methods that are naturally flux-based (BLS, TLS) convert internally. + +The built-in detection lists cover the standard light-curve products of the major +time-domain surveys — ASAS-SN, ASAS-3, ATLAS, CRTS/CSS, ZTF, Pan-STARRS, TESS, Kepler, +Gaia, LSST, and MACHO — so a downloaded table usually needs no column hints. Error +resolution is *domain-aware*: a flux value is paired with a flux error and a magnitude +value with a magnitude error, which disambiguates surveys (e.g. ATLAS) that expose both +``m``/``dm`` and ``uJy``/``duJy`` in one table. Headerless products (OGLE ``.dat``) have +no column names to detect — pass the data as a ``(time, value, error)`` tuple or with an +explicit :class:`ColumnMap`. """ from __future__ import annotations @@ -30,61 +39,110 @@ class Domain(StrEnum): #: Time-column name candidates, most-specific first. Barycentric/heliocentric #: corrected times precede plain JD so a table carrying both picks the better one. +#: Covers JD/HJD/BJD (ASAS, ASAS-SN, OGLE, Gaia), MJD (ATLAS, CRTS, ZTF, Pan-STARRS), +#: BTJD (TESS), BKJD (Kepler), ``midpointMjdTai`` (LSST), and ``obsTime`` (Pan-STARRS). TIME_NAMES: Final[tuple[str, ...]] = ( "bjd_tdb", "bjd", "hjd", "btjd", + "bkjd", "mjd", + "hmjd", + "midpointmjdtai", + "midpointtai", + "obstime", "jd", "time", "date", "t", ) -#: Magnitude value-column candidates. +#: Magnitude value-column candidates. Includes ASAS-3 ``MAG_0`` (first aperture), +#: ``psfMag`` (Pan-STARRS/LSST), and ``Rmag``/``Bmag`` (MACHO red/blue). MAG_NAMES: Final[tuple[str, ...]] = ( "mag", "magnitude", "vmag", "gmag", "rmag", + "bmag", + "imag", + "psfmag", "phot_mag", + "mag_0", "m", ) -#: Flux value-column candidates. +#: Flux value-column candidates. Includes ``PDCSAP_FLUX``/``SAP_FLUX``/``KSPSAP_FLUX`` +#: (TESS/Kepler), ``psfFlux`` (Pan-STARRS/LSST), and ``uJy``/``mJy`` (ATLAS, ASAS-SN). +#: ``uJy`` precedes the bare ``f`` so ATLAS's flux column wins over its ``F`` filter. FLUX_NAMES: Final[tuple[str, ...]] = ( "pdcsap_flux", + "kspsap_flux", "sap_flux", + "psfflux", + "psflux", + "psfdiffflux", "norm_flux", "rel_flux", "flux", + "ujy", + "mjy", "fnu", "f", ) -#: Error-column candidates (both magnitude and flux error spellings). -ERR_NAMES: Final[tuple[str, ...]] = ( +#: Magnitude-error candidates (used when the value column is a magnitude): ``magerr`` +#: (ZTF, CRTS, ASAS-SN), ``dm`` (ATLAS), ``MER_0`` (ASAS-3), ``e_mag``, ``dmag``, ... +MAG_ERR_NAMES: Final[tuple[str, ...]] = ( "mag_err", "magnitude_err", "mag_error", + "magerr", "e_mag", "merr", "dmag", + "dm", + "mer_0", +) + +#: Flux-error candidates (used when the value column is a flux): ``flux_error`` (Gaia), +#: ``*SAP_FLUX_ERR`` (TESS/Kepler), ``psfFluxErr`` (Pan-STARRS/LSST), ``duJy`` (ATLAS). +FLUX_ERR_NAMES: Final[tuple[str, ...]] = ( "flux_err", + "flux_error", "sap_flux_err", "pdcsap_flux_err", + "kspsap_flux_err", + "psfflux_err", + "psffluxerr", + "psfluxerr", + "psfdiffflux_err", + "psfdifffluxerr", "e_flux", + "dujy", +) + +#: Domain-neutral error spellings, tried after the domain-specific lists. +GENERIC_ERR_NAMES: Final[tuple[str, ...]] = ( "err", "error", "sigma", ) -#: Band/filter-column candidates (used only by multi-band ingestion). +#: All error spellings. Resolution is domain-aware (see :meth:`ColumnMap.resolve`): a +#: flux value pairs with a flux error and a magnitude value with a magnitude error. +ERR_NAMES: Final[tuple[str, ...]] = MAG_ERR_NAMES + FLUX_ERR_NAMES + GENERIC_ERR_NAMES + +#: Band/filter-column candidates (used only by multi-band ingestion): ``filtercode`` +#: (ZTF), ``filterID`` (Pan-STARRS), ``band`` (Gaia/LSST), ``fid`` (ZTF alerts). BAND_NAMES: Final[tuple[str, ...]] = ( "band", "filter", + "filtercode", + "filterid", + "filtername", "phot_filter", "passband", "fid", @@ -139,9 +197,10 @@ def infer_domain(value_column: str) -> Domain | None: explicit domain). """ low = value_column.lower() - if any(tok in low for tok in ("flux", "fnu")): + flux_tokens = ("flux", "fnu", "ujy", "mjy", "njy", "nmgy", "jansky") + if any(tok in low for tok in flux_tokens): return Domain.FLUX - if "mag" in low or low in {"m", "vmag", "gmag", "rmag"}: + if "mag" in low or low in {"m", "vmag", "gmag", "rmag", "bmag", "imag"}: return Domain.MAGNITUDE return None @@ -204,7 +263,6 @@ def resolve( value = _match( "value", self.value, value_candidates, available, lower_to_actual ) - error = _match("error", self.error, ERR_NAMES, available, lower_to_actual) band = _match("band", self.band, BAND_NAMES, available, lower_to_actual) if time is None: @@ -220,6 +278,16 @@ def resolve( f"{available}" ) resolved_domain = domain or infer_domain(value) or Domain.MAGNITUDE + # Resolve the error for the *same* measurement as the value: first try names + # derived from the value column (so ``PDCSAP_FLUX`` pairs with + # ``PDCSAP_FLUX_ERR`` and ``uJy`` with ``duJy``), then the spellings for the + # value's domain, then domain-neutral spellings. This keeps a flux value with a + # flux error and a magnitude value with a magnitude error. + v = value.lower() + derived = (f"{v}_err", f"{v}err", f"{v}_error", f"{v}error", f"e_{v}", f"d{v}") + domain_err = FLUX_ERR_NAMES if resolved_domain is Domain.FLUX else MAG_ERR_NAMES + err_candidates = derived + domain_err + GENERIC_ERR_NAMES + error = _match("error", self.error, err_candidates, available, lower_to_actual) return ResolvedColumns( time=time, value=value, error=error, band=band, domain=resolved_domain ) @@ -228,7 +296,10 @@ def resolve( __all__ = [ "BAND_NAMES", "ERR_NAMES", + "FLUX_ERR_NAMES", "FLUX_NAMES", + "GENERIC_ERR_NAMES", + "MAG_ERR_NAMES", "MAG_NAMES", "TIME_NAMES", "ColumnMap", diff --git a/tests/test_columns.py b/tests/test_columns.py index 85940da..97d8834 100644 --- a/tests/test_columns.py +++ b/tests/test_columns.py @@ -61,3 +61,73 @@ def test_dict_of_arrays_resolves() -> None: data = {"jd": np.arange(3.0), "mag": np.ones(3), "mag_err": np.ones(3)} resolved = ColumnMap().resolve(list(data)) assert (resolved.time, resolved.value, resolved.error) == ("jd", "mag", "mag_err") + + +# Real light-curve headers from each survey's standard data product: +# (id, columns, expected time, value, error, band). +_SURVEY_HEADERS = [ + ("asas-sn-v2", + ["asas_sn_id", "jd", "flux", "flux_err", "mag", "mag_err", "camera", + "quality", "phot_filter"], + "jd", "flux", "flux_err", "phot_filter"), + ("asas-3", + ["HJD", "MAG_0", "MAG_1", "MAG_2", "MAG_3", "MAG_4", + "MER_0", "MER_1", "MER_2", "MER_3", "MER_4", "GRADE"], + "HJD", "MAG_0", "MER_0", None), + ("atlas", # mixed mag (m/dm) + flux (uJy/duJy) + a non-error `err` flag column + ["MJD", "m", "dm", "uJy", "duJy", "F", "err", "chi/N"], + "MJD", "uJy", "duJy", None), + ("crts", + ["MasterID", "Mag", "Magerr", "RA", "Dec", "MJD"], + "MJD", "Mag", "Magerr", None), + ("ztf-api", + ["oid", "mjd", "mag", "magerr", "catflags", "filtercode", "fid"], + "mjd", "mag", "magerr", "filtercode"), + ("ztf-bulk", + ["ObjectID", "HMJD", "mag", "magerr", "catflags", "FilterID"], + "HMJD", "mag", "magerr", "FilterID"), + ("lsst", + ["band", "midpointMjdTai", "psfFlux", "psfFluxErr"], + "midpointMjdTai", "psfFlux", "psfFluxErr", "band"), + ("lsst-dp0.2", + ["filterName", "midPointTai", "psFlux", "psFluxErr"], + "midPointTai", "psFlux", "psFluxErr", "filterName"), + ("panstarrs", + ["objID", "obsTime", "psfFlux", "psfFluxErr", "filterID"], + "obsTime", "psfFlux", "psfFluxErr", "filterID"), + ("tess-spoc", # PDCSAP (corrected) must pair with PDCSAP_FLUX_ERR, not SAP + ["TIME", "SAP_FLUX", "SAP_FLUX_ERR", "PDCSAP_FLUX", "PDCSAP_FLUX_ERR", "QUALITY"], + "TIME", "PDCSAP_FLUX", "PDCSAP_FLUX_ERR", None), + ("tess-qlp", # prefer the detrended KSPSAP flux + ["TIME", "SAP_FLUX", "KSPSAP_FLUX", "KSPSAP_FLUX_ERR", "QUALITY"], + "TIME", "KSPSAP_FLUX", "KSPSAP_FLUX_ERR", None), + ("kepler", + ["TIME", "SAP_FLUX", "SAP_FLUX_ERR", "PDCSAP_FLUX", "PDCSAP_FLUX_ERR", + "SAP_QUALITY"], + "TIME", "PDCSAP_FLUX", "PDCSAP_FLUX_ERR", None), + ("gaia", + ["source_id", "band", "time", "mag", "flux", "flux_error", "flux_over_error"], + "time", "flux", "flux_error", "band"), +] + + +@pytest.mark.parametrize( + ("cols", "time", "value", "error", "band"), + [c[1:] for c in _SURVEY_HEADERS], + ids=[c[0] for c in _SURVEY_HEADERS], +) +def test_survey_headers_autodetect( + cols: list[str], time: str, value: str, error: str, band: str | None +) -> None: + resolved = ColumnMap().resolve(cols) + assert (resolved.time, resolved.value, resolved.error, resolved.band) == ( + time, value, error, band, + ) + + +def test_domain_aware_error_pairs_with_value() -> None: + # A magnitude value pairs with the magnitude error, a flux value with the flux + # error, even when both are present (the ATLAS m/dm + uJy/duJy case). + cols = ["MJD", "m", "dm", "uJy", "duJy"] + assert ColumnMap(value="m").resolve(cols).error == "dm" + assert ColumnMap(value="uJy").resolve(cols).error == "duJy" From d83332f465d22bc17917619a8b2a258b91e8cd15 Mon Sep 17 00:00:00 2001 From: Tharindu Jayasinghe Date: Mon, 29 Jun 2026 21:58:13 -0700 Subject: [PATCH 2/2] Reword the column auto-detection intro sentence --- docs/guide/light-curves.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/guide/light-curves.md b/docs/guide/light-curves.md index f622f25..cb68eb4 100644 --- a/docs/guide/light-curves.md +++ b/docs/guide/light-curves.md @@ -58,9 +58,10 @@ The originating path is recorded in `lc.meta["source"]`. ## Column auto-detection with `ColumnMap` -Astronomers' tables carry the same three quantities under wildly different names. cuPeriod -detects the common spellings case-insensitively, most-specific first, so a table with both -`BJD_TDB` and `JD` picks the corrected time. The detection lists are: +Light curves will always carry temporal and magnitude/flux measurements, with optional +magnitude/flux errors. cuPeriod detects the common spellings case-insensitively, +most-specific first, so a table with both `BJD_TDB` and `JD` picks the corrected time. +The detection lists are: - **Time** — `bjd_tdb`, `bjd`, `hjd`, `btjd`, `bkjd`, `mjd`, `hmjd`, `midpointMjdTai`, `obsTime`, `jd`, `time`, `date`, `t`