Skip to content

fix(manta-spot,manta-server,manta-cli): MAN-136 — populate dxDxcc and make unresolved geography explicit - #140

Open
catalyst-cloud-connector[bot] wants to merge 1 commit into
mainfrom
MAN-136
Open

fix(manta-spot,manta-server,manta-cli): MAN-136 — populate dxDxcc and make unresolved geography explicit#140
catalyst-cloud-connector[bot] wants to merge 1 commit into
mainfrom
MAN-136

Conversation

@catalyst-cloud-connector

@catalyst-cloud-connector catalyst-cloud-connector Bot commented Sep 7, 2026

Copy link
Copy Markdown

Summary

manta's :7301 JSON Lines stream emitted "dxDxcc": null / "deDxcc": null
on every spot, including ones whose callsign cty.dat resolves cleanly.
dispensa's spots.v1.schema.json declares dxDxcc required and
non-nullable, so cqdx's own ingest would reject the batch on that field
alone — the exact failure this ticket names. Separately, when a callsign
genuinely can't be resolved (reachable in production via MAN-28's Watch List
allowlist, which bypasses cty.is_allocated()), the geography fields fell
back to a plain empty string / 0 — indistinguishable on the wire from real,
resolved data.

This implements broad-review decision D10
(docs/DECISIONS/2026-09-06-broad-review-decisions.md): vendor a small ADIF
DXCC entity-number table alongside cty.dat and resolve it in
manta_spot::cty, rather than waiting on a dispensa contract change.

What changed

manta-spot — new crates/manta-spot/data/dxcc.tsv (346 rows, ~6.9 KB),
derived from AD1C's cty.csv (scripts/gen-dxcc-table.sh, new, regenerable)
and keyed on cty.dat's primary prefix, not entity name — measured
346/346 matches on prefix vs. 343/346 on name (three AD1C entity-name
renames in a six-week window; ADIF 3.1.7's own release notes independently
renamed five other entities). cty::Entry gains dxcc: Option<u16>,
resolved once per entity during Table::parse via a new
Table::parse_with_dxcc that the public parse signature delegates to
(zero call-site churn across manta-cli/manta-server/existing tests).
Starred subentities correctly inherit their parent's DXCC number (e.g.
Sicily's IT9 prefix → Italy's 248).

manta-serverSpotMessage::dx_dxcc/de_dxcc change from
Option<i64> to i64, making null unrepresentable at the type level.
Three new exported sentinel constants (spot_message.rs):
UNKNOWN_DXCC = -1, UNKNOWN_CONTINENT = "", UNKNOWN_CQ_ZONE = 0.
UNKNOWN_DXCC is deliberately not 0 — ADIF defines entity code 0 as
"None: the contacted station is known to not be within a DXCC entity" (a
specific, false claim to make about a call manta simply couldn't resolve).
UNKNOWN_CONTINENT/UNKNOWN_CQ_ZONE keep their pre-existing values
(unchanged wire bytes for that half) but are now named, documented, tested
and counted instead of an unlabeled unwrap_or_default(). dxLat/dxLon
stay Option<f64> and stay null when unresolved — the one contract-legal
"unknown geography" signal already available to consumers with no schema
change. The stale in-code comment claiming dxDxcc/deDxcc were nullable
on dispensa's contract (which contradicted three independent broad-review
readings of the actual schema) is removed and replaced with the accurate
contract description, cited against the new decision record.

manta-server metricsmanta_spots_unresolved_geography_total
(AtomicU64, renders at 0 before any spot fires, per the "no silent
loss anywhere in the pipeline" invariant in ARCHITECTURE.md §8).

manta-cli — counted once per spot at publish time (main.rs, next to
record_spot()), not inside SpotMessage::from_spot (which runs once per
connected client, not once per spot — counting there would scale with
client count). The increment condition checks whether the resolved cty
entry actually carries a DXCC number (cty.lookup(call).and_then(|e| e.dxcc).is_none()), not merely whether the callsign resolved at all — a
strict superset that also catches cty.dat/dxcc.tsv drift (geography
resolves but the entity has no DXCC row), so that state can't silently ship
dxDxcc: -1 with the counter still at zero. The station's own
station_callsign side is checked too, since it's operator config and not
guaranteed cty-resolvable; that check is precomputed once at server startup
rather than re-looked-up per spot, since config can't change mid-process.

Docs — new
docs/DECISIONS/2026-09-07-man136-dxcc-and-unknown-geography-sentinels.md
records the contract, the prefix-vs-name measurement, each sentinel's
rationale (including why ADIF 0 isn't reused), what consumers should key on,
and the still-open cross-repo item (dispensa Q-0028, open since
2026-07-14) as a follow-up rather than a blocker. ARCHITECTURE.md §6.2/§7/§8
and both wiki/pages/spot-output-contract.md /
wiki/pages/spot-validation.md updated to match (the wiki previously said
the dispensa schema was "not yet frozen" — no longer accurate; it also didn't
mention the MAN-28 allowlist as the mechanism that makes an unresolvable
call reachable in production).

Not touched: the telnet/RBN surface (:7300) never constructs a
SpotMessage, so it's unaffected. dxGrid/deGrid stay nullcty.dat
carries no grid data. No dispensa schema was vendored or negotiated; per D10
this ships from manta's side without needing cqdx/dispensa to act first.

Why this shape

  • Vendoring dxcc.tsv rather than switching to AD1C's cty.csv
    wholesale
    keeps the existing, battle-tested cty.dat parser (exact-call
    aliases, per-alias zone overrides, portable-suffix matching) untouched —
    only the previously-discarded primary-prefix field gets a second, narrow
    use.
  • Keying on primary prefix, not entity name, was decided by measurement
    during implementation (not by preference): a six-week version gap between
    the vendored cty.dat and a freshly-fetched cty.csv had already
    renamed 3 of 346 entities but changed zero prefixes.
  • The source of the numbers is AD1C's own cty.csv (same author, same
    license posture as cty.dat, and its column 3 is the ADIF DXCC entity
    code) rather than a hand-curated scrape of ADIF's published enumeration —
    keeps the vendored table reproducible from one documented script instead
    of an unverifiable manual transcription.
  • Both MAN-136 scenarios resolve the same underlying bug class (an
    implicit, indistinguishable-from-real fallback on a required field) with
    one consistent sentinel pattern, and this also closes the geography-null
    finding already on file in MAN-45 — that ticket's fix was never present in
    this repository's history, so its design (constant names, counter shape,
    call-site placement) was reproduced here verbatim rather than assumed
    landed, avoiding a future merge conflict if MAN-45 lands separately.

Testing

  • cargo test -p manta-spot -p manta-server -p manta-cli: 315 passed, 0
    failed (net new/changed tests across all three crates; one previously
    green test that pinned the old null-on-every-spot behavior,
    dxcc_entity_numbers_are_null_not_fabricated, was deleted and replaced by
    an assertion of the correct resolved number rather than the absence of
    one).
  • cargo test -p manta-engine -p manta-dsp -p manta-decode -p manta-input -p manta-testkit -p manta-soak-harness:
    163 passed, 0 failed (unaffected crates, run as a regression check).
  • New data-drift tripwire,
    every_entity_in_the_vendored_cty_dat_resolves_an_adif_dxcc_number, fails
    loudly if a future cty.dat refresh adds an entity dxcc.tsv doesn't
    cover, instead of silently downgrading that entity's spots to the unknown
    sentinel forever.
  • json_stream_acceptance.rs's socket-level test now asserts a non-null
    dxDxcc over a real TCP connection — the closest in-repo proxy for "a
    cqdx-shaped consumer accepts this batch."
  • crates/manta-spot/data/dxcc.tsv was verified byte-identical to a live
    re-run of the generator against upstream cty.csv (fetched two days after
    the file was committed) — the committed table is genuinely reproducible,
    not hand-edited.
  • cargo clippy --workspace --all-targets -- -D warnings and
    cargo fmt --all --check: both clean.
  • Not run in this environment (recorded, not claimed): validating an emitted
    line against dispensa's actual spots.v1.schema.json (no dispensa
    checkout available here — the schema isn't vendored in this repo by
    design) and a live daemon/socket manual check (no audio/SDR hardware in
    this container). The full cargo test --workspace in one invocation
    cannot complete in this container's constrained disk (rust-lld runs out
    of space linking manta-engine's test binaries); coverage was obtained
    instead via two cargo clean-separated batches, both green, covering
    every crate — 478 tests total, 0 failed, 6 pre-existing (unrelated)
    #[ignore]s.

Known follow-ups (not blocking, carried forward against the new decision record)

  • UNKNOWN_CONTINENT/UNKNOWN_CQ_ZONE's values were never checked against
    dispensa's actual per-field schema constraints (only dxDxcc's
    nullability was confirmed via three independent broad-review readings) —
    unverifiable in this repo since the schema isn't vendored here; this
    branch does not change those two fields' wire values, only names/tests/
    counts them.
  • parse_dxcc_table's dedup silently keeps the first row on a duplicate
    join key; there are zero duplicates in the table today, but nothing
    currently guards against a future star-stripped prefix collision.
  • An unresolvable station_callsign (operator misconfiguration) currently
    drives the same counter as a genuine unresolvable dx callsign
    (data-quality signal); a startup warning would let an operator tell those
    apart at a glance.

01d1ea1 feat: MAN-136 — cqdx should be able to ingest manta's spots without a required field being null
559d820 feat: MAN-136 — cqdx should be able to ingest manta's spots without a required field being null

Catalyst-Replay-Squash: 49f05a4
@catalyst-cloud-connector catalyst-cloud-connector Bot changed the title feat: MAN-136 — cqdx should be able to ingest manta's spots without a required field being null fix(manta-spot,manta-server,manta-cli): MAN-136 — populate dxDxcc and make unresolved geography explicit Sep 8, 2026
@catalyst-cloud-connector
catalyst-cloud-connector Bot marked this pull request as ready for review September 8, 2026 00:55
@mergify

mergify Bot commented Sep 8, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants