From 545f3da384e6a8c4f95a2c087846a2032ad8d077 Mon Sep 17 00:00:00 2001 From: Catalyst Runner Date: Tue, 8 Sep 2026 00:55:39 +0000 Subject: [PATCH] MAN-136: squashed replay of 2 commit(s) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 01d1ea1 feat: MAN-136 — cqdx should be able to ingest manta's spots without a required field being null 559d820 feat: MAN-136 — cqdx should be able to ingest manta's spots without a required field being null Catalyst-Replay-Squash: 49f05a447efadabd7f581b6be54526ad5d714c25 --- ARCHITECTURE.md | 21 +- crates/manta-cli/src/main.rs | 95 ++++- crates/manta-server/src/metrics.rs | 47 +++ crates/manta-server/src/spot_message.rs | 163 ++++++-- .../tests/json_stream_acceptance.rs | 2 +- crates/manta-spot/data/SOURCES.md | 24 ++ crates/manta-spot/data/dxcc.tsv | 349 ++++++++++++++++++ crates/manta-spot/src/cty.rs | 147 +++++++- crates/manta-spot/src/lib.rs | 7 + ...36-dxcc-and-unknown-geography-sentinels.md | 149 ++++++++ scripts/gen-dxcc-table.sh | 42 +++ wiki/pages/spot-output-contract.md | 6 +- wiki/pages/spot-validation.md | 5 +- 13 files changed, 1005 insertions(+), 52 deletions(-) create mode 100644 crates/manta-spot/data/dxcc.tsv create mode 100644 docs/DECISIONS/2026-09-07-man136-dxcc-and-unknown-geography-sentinels.md create mode 100755 scripts/gen-dxcc-table.sh diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 22d23e24..9287690a 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -233,7 +233,11 @@ transmission may never produce again). carry this flag. 2. **Callsign plausibility**: structural grammar (prefix-digit-suffix, portable designators `/P /QRP /3`), then prefix lookup against **cty.dat** (bundled, - refreshable) — a call with an unallocated prefix is rejected. + refreshable) — a call with an unallocated prefix is rejected. `cty.dat` is + also joined, on that same primary-prefix field, against a small vendored + ADIF DXCC entity-number table (`data/dxcc.tsv`, MAN-136) — refreshed + together, see `crates/manta-spot/data/SOURCES.md` — which is what lets + `manta-server`'s JSON stream populate `dxDxcc`/`deDxcc` (§7). 3. **SCP cross-check** (optional, default on if file present): membership in `master.scp` (contest super-check-partial list) *raises* confidence; absence only lowers it (new/rare calls must still spot, not just well-known ones). @@ -286,7 +290,12 @@ validation (MAN-28). Dedupe (step 5) still applies. - **JSON Lines stream** (TCP and WebSocket, :7301): full-fidelity spot objects (adds confidence, track id, decoder text context). This is the cqdx ingest surface; schema published in `dispensa` as a JSON Schema contract alongside the - existing ecosystem contracts. + existing ecosystem contracts. Every spot carries a non-null, real `dxDxcc`/ + `deDxcc` (an ADIF DXCC entity number, MAN-136) whenever the callsign + resolves against `cty.dat`; when it doesn't, `dxDxcc`/`dxContinent`/ + `dxCqZone` (and their `de*` counterparts) carry named, out-of-domain + `UNKNOWN_*` sentinels rather than `null` or a fabricated-looking value — + see `docs/DECISIONS/2026-09-07-man136-dxcc-and-unknown-geography-sentinels.md`. - Both servers are thin fan-out consumers of one broadcast channel; slow clients are disconnected, never back-pressure the pipeline. - **Exposure policy (normative, not just observed behavior):** both servers are @@ -324,8 +333,12 @@ validation (MAN-28). Dedupe (step 5) still applies. currently-implemented subset is `manta_spots_total`, `manta_spots_dropped_lagged_total`, `manta_spots_suppressed_by_filter_total`, - `manta_spots_dropped_write_failed_total`, per-protocol client-connected - gauges, `manta_source_health`, and the uplink counters + `manta_spots_dropped_write_failed_total`, + `manta_spots_unresolved_geography_total` (MAN-136/MAN-45 — a spot that went + out carrying an `UNKNOWN_*` sentinel on either side, i.e. its dx or de + callsign didn't resolve against `cty.dat`, *or* it resolved but its entity + has no row in the vendored `dxcc.tsv`), per-protocol + client-connected gauges, `manta_source_health`, and the uplink counters (`crates/manta-server/src/metrics.rs`) — not input-layer overruns or per-stage queue depths, which MAN-56 tracks as a separate gap. **`manta_active_tracks` is served but not populated** (corrected diff --git a/crates/manta-cli/src/main.rs b/crates/manta-cli/src/main.rs index 0b1ab0e8..9afb058c 100644 --- a/crates/manta-cli/src/main.rs +++ b/crates/manta-cli/src/main.rs @@ -710,6 +710,36 @@ struct SpotServer { /// `SHUTDOWN_DRAIN_DEADLINE`) instead of guessing a fixed sleep /// duration -- see `shutdown_runtime_after_drain`. tasks: manta_server::tasks::ClientTasks, + /// MAN-136/MAN-45: the same `cty::Table` handed to `JsonStreamConfig`, + /// kept here too so the publish callback can check resolvability once + /// per spot for `manta_spots_unresolved_geography_total` -- checking + /// inside `SpotMessage::from_spot` would scale with connected client + /// count instead of spot count. + cty: std::sync::Arc, + /// Whether the operator's OWN station callsign (config, not decoder + /// output -- and not required to be cty-resolvable) already forces the + /// de-side `UNKNOWN_*` sentinels. Resolved ONCE at `start_spot_server` + /// time rather than per spot: `station_callsign` cannot change for the + /// life of the process, so re-running the same binary search on every + /// spot only re-derives a constant. + station_geography_unresolved: bool, +} + +/// True when `SpotMessage::from_spot` would emit the `UNKNOWN_DXCC` / +/// `UNKNOWN_CONTINENT` / `UNKNOWN_CQ_ZONE` sentinels for `callsign`, i.e. +/// exactly the condition `manta_spots_unresolved_geography_total` counts. +/// +/// Deliberately keyed on the RESOLVED ADIF entity number, not merely on +/// whether `lookup` returned an entry: `from_spot` emits `UNKNOWN_DXCC` on +/// `dx.and_then(|e| e.dxcc).is_none()`, which is also true when `cty.dat` +/// resolves the call but the vendored `dxcc.tsv` has no row for its primary +/// prefix -- the drift state that arises when `cty.dat` is hand-refreshed +/// (data/SOURCES.md) without regenerating the TSV. Counting `lookup` +/// alone would let those spots go out carrying `dxDxcc: -1` with the +/// counter still at zero, silently withholding the one signal this metric +/// exists to give (round-1 validate code-review finding 1). +fn geography_is_unresolved(cty: &manta_spot::cty::Table, callsign: &str) -> bool { + cty.lookup(callsign).and_then(|e| e.dxcc).is_none() } /// Starts the telnet/JSON-Lines-and-WebSocket/metrics servers on their own @@ -864,7 +894,7 @@ fn start_spot_server( manta_server::json_stream::JsonStreamConfig { bus: bus.clone(), metrics: metrics.clone(), - cty, + cty: cty.clone(), station_call: cfg.station_callsign.clone(), decoder_version, // .clone(): MAN-32/MAN-42's uplink::serve spawns below also @@ -928,6 +958,8 @@ fn start_spot_server( metrics, shutdown_tx, tasks, + station_geography_unresolved: geography_is_unresolved(&cty, &cfg.station_callsign), + cty, }, )) } @@ -1203,6 +1235,20 @@ fn main() -> Result<()> { if let Some(server) = &spot_server { server.bus.publish(spot.clone()); server.metrics.record_spot(); + // MAN-136/MAN-45: counted ONCE per spot here, NOT + // inside `SpotMessage::from_spot` -- that runs once + // per connected JSON/WS client (json_stream.rs:126), + // so counting there would scale with client count + // instead of spot count. Checks BOTH sides: the + // operator's own station_callsign is config, not + // decoder output, and isn't required to resolve -- + // but it also never changes, so its side is + // resolved once at start_spot_server time. + if geography_is_unresolved(&server.cty, &spot.callsign) + || server.station_geography_unresolved + { + server.metrics.record_unresolved_geography(); + } } if json { println!("{}", serde_json::json!({ "spot": spot })); @@ -1731,4 +1777,51 @@ mod tests { assert!(accepted1, "first configured target must be connected to"); assert!(accepted2, "second configured target must be connected to"); } + + // MAN-136 round-1 validate code-review finding 1: the increment + // condition for `manta_spots_unresolved_geography_total` must match the + // condition under which `SpotMessage::from_spot` emits the `UNKNOWN_*` + // sentinels -- the RESOLVED ADIF entity number, not merely whether + // `cty.lookup` returned an entry. + + const GEOGRAPHY_CTY_FIXTURE: &str = "\ +United States: 5: 8: NA: 40.0: 75.0: 5.0: K: + K,W,N; +"; + /// One `dxcc.tsv` row for the fixture above, in the vendored file's + /// `\t\t` shape. + const GEOGRAPHY_DXCC_FIXTURE: &str = "K\t291\tUnited States\n"; + + #[test] + fn a_callsign_with_a_resolved_entity_number_is_not_counted_as_unresolved() { + let cty = + manta_spot::cty::Table::parse_with_dxcc(GEOGRAPHY_CTY_FIXTURE, GEOGRAPHY_DXCC_FIXTURE); + assert_eq!(cty.lookup("W1AW").and_then(|e| e.dxcc), Some(291)); + assert!(!geography_is_unresolved(&cty, "W1AW")); + } + + #[test] + fn an_unresolvable_callsign_is_counted_as_unresolved() { + let cty = + manta_spot::cty::Table::parse_with_dxcc(GEOGRAPHY_CTY_FIXTURE, GEOGRAPHY_DXCC_FIXTURE); + assert!(cty.lookup("QQ1AAA").is_none(), "test premise"); + assert!(geography_is_unresolved(&cty, "QQ1AAA")); + } + + #[test] + fn a_cty_resolvable_callsign_with_no_dxcc_row_is_still_counted_as_unresolved() { + // The cty.dat/dxcc.tsv drift state: `cty.dat` was hand-refreshed + // (data/SOURCES.md has no refresh automation) without regenerating + // the TSV, so geography resolves -- non-null dxLat/dxLon -- while + // the entity number does not, and the spot goes out with + // `dxDxcc: -1`. Counting `lookup().is_none()` missed exactly this. + let cty = manta_spot::cty::Table::parse_with_dxcc(GEOGRAPHY_CTY_FIXTURE, ""); + let entry = cty.lookup("W1AW").expect("geography still resolves"); + assert_eq!(entry.dxcc, None, "test premise: only the number is missing"); + assert_eq!(entry.continent, "NA"); + assert!( + geography_is_unresolved(&cty, "W1AW"), + "a spot emitted with UNKNOWN_DXCC must be counted, even though cty.dat resolved it" + ); + } } diff --git a/crates/manta-server/src/metrics.rs b/crates/manta-server/src/metrics.rs index a041171e..3473d55e 100644 --- a/crates/manta-server/src/metrics.rs +++ b/crates/manta-server/src/metrics.rs @@ -16,6 +16,15 @@ pub struct Metrics { spots_dropped_lagged_total: AtomicU64, spots_suppressed_by_filter_total: AtomicU64, spots_dropped_write_failed_total: AtomicU64, + /// MAN-136/MAN-45: a spot's dx or de callsign couldn't be resolved + /// against `cty.dat`, so it was emitted with the `UNKNOWN_DXCC`/ + /// `UNKNOWN_CONTINENT`/`UNKNOWN_CQ_ZONE` sentinels instead of real + /// geography. ARCHITECTURE §8: "every dropped/evicted/suppressed item is + /// counted" -- an operator otherwise has no way to notice this is + /// happening. Incremented once per spot at publish time (`main.rs`), not + /// inside `SpotMessage::from_spot` (which runs once per connected + /// client). + spots_unresolved_geography_total: AtomicU64, telnet_clients: AtomicI64, json_clients: AtomicI64, ws_clients: AtomicI64, @@ -83,6 +92,15 @@ impl Metrics { .fetch_add(n, Ordering::Relaxed); } + /// MAN-136/MAN-45: call once per spot (not per connected client) when + /// either the dx or de callsign didn't resolve against `cty.dat`, so the + /// wire message carries the `UNKNOWN_*` sentinels instead of real + /// geography. + pub fn record_unresolved_geography(&self) { + self.spots_unresolved_geography_total + .fetch_add(1, Ordering::Relaxed); + } + pub fn inc_telnet_clients(&self) { self.telnet_clients.fetch_add(1, Ordering::Relaxed); } @@ -257,6 +275,16 @@ impl Metrics { .load(Ordering::Relaxed) )); + out.push_str( + "# HELP manta_spots_unresolved_geography_total Spots emitted with an UNKNOWN_DXCC/UNKNOWN_CONTINENT/UNKNOWN_CQ_ZONE sentinel on the dx or de side, because the callsign did not resolve against cty.dat OR its entity carries no row in the vendored dxcc.tsv.\n", + ); + out.push_str("# TYPE manta_spots_unresolved_geography_total counter\n"); + out.push_str(&format!( + "manta_spots_unresolved_geography_total {}\n", + self.spots_unresolved_geography_total + .load(Ordering::Relaxed) + )); + out.push_str("# HELP manta_telnet_clients_connected Currently connected telnet clients.\n"); out.push_str("# TYPE manta_telnet_clients_connected gauge\n"); out.push_str(&format!( @@ -432,6 +460,25 @@ mod tests { assert!(text.contains("manta_spots_dropped_write_failed_total 7")); } + #[test] + fn unresolved_geography_is_counted_and_exposed() { + let m = Metrics::new(); + m.record_unresolved_geography(); + m.record_unresolved_geography(); + assert!(m + .render_prometheus_text() + .contains("manta_spots_unresolved_geography_total 2")); + } + + #[test] + fn unresolved_geography_counter_is_present_at_zero_before_any_spot() { + // A counter that only appears once it fires is invisible to an operator + // building a dashboard -- the other spot counters all render at 0. + assert!(Metrics::new() + .render_prometheus_text() + .contains("manta_spots_unresolved_geography_total 0")); + } + #[test] fn renders_per_source_health_as_labeled_gauge() { let m = Metrics::new(); diff --git a/crates/manta-server/src/spot_message.rs b/crates/manta-server/src/spot_message.rs index 29214cd5..9b74ee06 100644 --- a/crates/manta-server/src/spot_message.rs +++ b/crates/manta-server/src/spot_message.rs @@ -1,16 +1,38 @@ //! `SpotMessage` -- the JSON Lines wire shape for manta's `:7301` stream. //! Field names/types mirror dispensa's `contracts/spots/spots.v1.schema.json` -//! (ADR-0011): required fields manta cannot resolve from data it actually -//! has (`dxDxcc`/`deDxcc` -- an ADIF DXCC entity-number table isn't -//! vendored here) are serialized as JSON `null` rather than a fabricated -//! placeholder value. `dxContinent`/`dxCqZone`/`dxLat`/`dxLon` (and the -//! `de*` counterparts) ARE resolved, from the same vendored `cty.dat` the +//! (ADR-0011). `dxDxcc`/`deDxcc`/`dxContinent`/`deContinent`/`dxCqZone` are +//! ALL required and non-nullable on that contract -- see +//! docs/DECISIONS/2026-09-07-man136-dxcc-and-unknown-geography-sentinels.md. +//! `dxDxcc`/`deDxcc` are resolved from the vendored `dxcc.tsv` ADIF entity- +//! number table (MAN-136); `dxContinent`/`dxCqZone`/`dxLat`/`dxLon` (and the +//! `de*` counterparts) are resolved from the same vendored `cty.dat` the //! validator already trusts for the plausibility gate (`manta_spot::cty`). +//! When a callsign isn't cty-resolvable, each required field gets a named +//! out-of-domain `UNKNOWN_*` sentinel below rather than a fabricated-looking +//! real value or (where the contract forbids it) `null`. use manta_spot::cty; use manta_spot::Spot; use serde::Serialize; +/// Emitted for `dxDxcc`/`deDxcc` when `cty.lookup` cannot resolve the +/// callsign. Deliberately NEGATIVE: ADIF entity codes run 1-522, and ADIF +/// code 0 already has a specific different meaning -- "None: the contacted +/// station is known to NOT be within a DXCC entity" -- which would be a false +/// positive claim about a call manta merely failed to resolve. dispensa's +/// spots.v1 declares this field required and non-nullable, so `null` is not +/// available. See docs/DECISIONS/2026-09-07-man136-dxcc-and-unknown-geography-sentinels.md. +pub const UNKNOWN_DXCC: i64 = -1; + +/// Emitted for `dxContinent`/`deContinent` when `cty.lookup` cannot resolve +/// the callsign. Outside the field's real domain -- the seven two-letter +/// continent codes -- so it reads as "unknown", never as geography. +pub const UNKNOWN_CONTINENT: &str = ""; + +/// Emitted for `dxCqZone` when `cty.lookup` cannot resolve the callsign. +/// Real CQ zones are 1-40, so 0 is unambiguously "unknown". +pub const UNKNOWN_CQ_ZONE: u16 = 0; + #[derive(Debug, Clone, PartialEq, Serialize)] #[serde(rename_all = "camelCase")] pub struct SpotMessage { @@ -25,14 +47,14 @@ pub struct SpotMessage { pub dx_grid: Option, pub dx_lat: Option, pub dx_lon: Option, - pub dx_dxcc: Option, + pub dx_dxcc: i64, pub dx_continent: String, pub dx_cq_zone: u16, pub de_call: String, pub de_grid: Option, pub de_lat: Option, pub de_lon: Option, - pub de_dxcc: Option, + pub de_dxcc: i64, pub de_continent: String, pub snr: Option, pub wpm: Option, @@ -65,20 +87,22 @@ impl SpotMessage { unix_ts_secs: i64, session_nonce: u128, ) -> Self { - // Falls back to empty-string/zero when `dx_call` isn't - // cty-allocated. Reachable in practice, not just a defensive - // fallback: MAN-28's Watch List lets an operator allowlist a call - // that bypasses `cty.is_allocated()` entirely (e.g. a deliberately - // unallocated or malformed test callsign), so `Validator` can emit - // a spot for a callsign `cty.lookup` genuinely can't resolve. - // dxContinent/dxCqZone (and their de* counterparts) are REQUIRED, - // non-nullable fields on dispensa's spots.v1 wire contract, unlike - // dxDxcc/deDxcc (declared nullable there) -- there is currently no - // contract-defined "unknown" representation for these two fields, - // so this fallback stays a real (if honestly imperfect) value - // rather than null, which the contract would reject outright. See - // the follow-up ticket linked from this PR's round-6 review thread - // for the cross-repo contract question this raises. + // Falls back to the UNKNOWN_* sentinels above when the callsign + // isn't cty-allocated. Reachable in practice, not defensive: MAN-28's + // Watch List lets an operator allowlist a call that bypasses + // `cty.is_allocated()` entirely (validator.rs:669-680), so + // `Validator` can emit a spot for a callsign `cty.lookup` genuinely + // can't resolve. + // + // MAN-136 / broad-review D10: dxDxcc, deDxcc, dxContinent, deContinent + // and dxCqZone are ALL required and non-nullable on dispensa's + // spots.v1 contract -- emitting JSON `null` for any of them fails + // cqdx's ingest rather than satisfying it. So each has a named, + // out-of-domain sentinel instead. `dxLat`/`dxLon` ARE nullable on the + // contract and DO serialize as `null` here, which is the one + // contract-defined "geography unknown" signal a consumer can key on + // today. Occurrences are counted as + // `manta_spots_unresolved_geography_total` (main.rs, at publish). let dx = cty.lookup(&spot.callsign); let de = cty.lookup(station_call); // `band` must be derived from the SAME rounded value reported as @@ -105,15 +129,25 @@ impl SpotMessage { dx_grid: None, dx_lat: dx.map(|e| e.lat), dx_lon: dx.map(|e| e.lon), - dx_dxcc: None, - dx_continent: dx.map(|e| e.continent.clone()).unwrap_or_default(), - dx_cq_zone: dx.map(|e| e.cq_zone).unwrap_or(0), + dx_dxcc: dx + .and_then(|e| e.dxcc) + .map(i64::from) + .unwrap_or(UNKNOWN_DXCC), + dx_continent: dx + .map(|e| e.continent.clone()) + .unwrap_or_else(|| UNKNOWN_CONTINENT.to_string()), + dx_cq_zone: dx.map(|e| e.cq_zone).unwrap_or(UNKNOWN_CQ_ZONE), de_call: station_call.to_string(), de_grid: None, de_lat: de.map(|e| e.lat), de_lon: de.map(|e| e.lon), - de_dxcc: None, - de_continent: de.map(|e| e.continent.clone()).unwrap_or_default(), + de_dxcc: de + .and_then(|e| e.dxcc) + .map(i64::from) + .unwrap_or(UNKNOWN_DXCC), + de_continent: de + .map(|e| e.continent.clone()) + .unwrap_or_else(|| UNKNOWN_CONTINENT.to_string()), snr: Some(spot.snr_db.round() as i32), wpm: Some(spot.wpm.round() as i32), decode_confidence: Some(spot.confidence), @@ -207,20 +241,72 @@ Japan: 25: 45: AS: 36.0: 138.0: 9.0: JA: assert_eq!(msg.de_continent, "NA"); } + /// MAN-136 scenario 1: dispensa's spots.v1 declares dxDxcc REQUIRED and + /// non-nullable; manta used to emit `null` for EVERY spot, including ones + /// whose callsign cty.dat resolves fine -- cqdx's ingest would reject the + /// batch on this field alone. CTY_FIXTURE's entities carry the real primary + /// prefixes `JA` and `K`, so the real vendored dxcc.tsv resolves them. #[test] - fn dxcc_entity_numbers_are_null_not_fabricated() { + fn a_resolvable_callsign_carries_its_real_adif_dxcc_entity_number() { let cty = cty::Table::parse(CTY_FIXTURE); - let msg = SpotMessage::from_spot( - &sample_spot(), - "W3XYZ", - &cty, - "manta-0.1.0", - 0, - 1_699_999_000, - ); + let msg = SpotMessage::from_spot(&sample_spot(), "W3XYZ", &cty, "manta-0.1.0", 0, 0); + + assert_eq!(msg.dx_dxcc, 339, "JA1ABC -> Japan"); + assert_eq!(msg.de_dxcc, 291, "W3XYZ -> United States"); + let json = serde_json::to_value(&msg).unwrap(); + assert_eq!(json["dxDxcc"], 339); + assert_eq!(json["deDxcc"], 291); + assert!(!json["dxDxcc"].is_null(), "the whole point of MAN-136"); + } + + /// MAN-136 scenario 2 (also MAN-45 finding 1): MAN-28's Watch List allowlist + /// (validator.rs:669-680) lets an operator emit a spot for a call cty.lookup + /// genuinely cannot resolve. All three required geography fields must then + /// carry a named, out-of-domain sentinel -- never a fabricated-looking value, + /// and never null. + /// + /// Uses the REAL vendored cty.dat, not CTY_FIXTURE: a call that is + /// unallocated in a 2-entity fixture is usually allocated in the real file + /// (ZZ9ZZZ -> Brazil/108 via ZZ; NOCALL -> US/291 via N). ITU allocates no Q + /// prefixes, so QQ1AAA is genuinely unresolvable. + #[test] + fn an_unresolvable_callsign_emits_named_out_of_domain_sentinels() { + let cty = cty::Table::parse(manta_spot::CTY_DAT); + let mut spot = sample_spot(); + spot.callsign = "QQ1AAA".to_string(); + assert!(cty.lookup(&spot.callsign).is_none(), "test premise"); + + let msg = SpotMessage::from_spot(&spot, "W3XYZ", &cty, "manta-0.1.0", 0, 0); + + assert_eq!(msg.dx_dxcc, UNKNOWN_DXCC); + assert_eq!(msg.dx_continent, UNKNOWN_CONTINENT); + assert_eq!(msg.dx_cq_zone, UNKNOWN_CQ_ZONE); + // Each sentinel must be outside its field's real domain. UNKNOWN_DXCC's + // own doc comment covers why it's negative and not 0 (ADIF 0 means + // "confirmed not in any entity") -- both are constant properties, not + // something to assert at runtime here. + assert!(!(1..=40).contains(&msg.dx_cq_zone)); + assert!(msg.dx_continent.len() != 2); + // The contract-legal unknown signal consumers can key on today. + assert!(msg.dx_lat.is_none()); + assert!(msg.dx_lon.is_none()); + // The station's own call still resolves -- de geography is unaffected. + assert_eq!(msg.de_dxcc, 291); + assert_eq!(msg.de_continent, "NA"); + } + + /// The de side gets the same treatment: `station_callsign` is operator config + /// and is not required to be cty-resolvable. + #[test] + fn an_unresolvable_station_callsign_also_gets_the_sentinels() { + let cty = cty::Table::parse(manta_spot::CTY_DAT); + let msg = SpotMessage::from_spot(&sample_spot(), "QQ1AAA", &cty, "manta-0.1.0", 0, 0); - assert_eq!(msg.dx_dxcc, None); - assert_eq!(msg.de_dxcc, None); + assert_eq!(msg.de_dxcc, UNKNOWN_DXCC); + assert_eq!(msg.de_continent, UNKNOWN_CONTINENT); + assert!(msg.de_lat.is_none()); + // dx geography is unaffected. + assert_eq!(msg.dx_dxcc, 339); } #[test] @@ -321,6 +407,7 @@ Japan: 25: 45: AS: 36.0: 138.0: 9.0: JA: ] { assert!(json.get(key).is_some(), "missing key: {key}"); } - assert_eq!(json["dxDxcc"], serde_json::Value::Null); + assert_eq!(json["dxDxcc"], 339); + assert!(!json["dxDxcc"].is_null()); } } diff --git a/crates/manta-server/tests/json_stream_acceptance.rs b/crates/manta-server/tests/json_stream_acceptance.rs index 77d8dd20..3042a94b 100644 --- a/crates/manta-server/tests/json_stream_acceptance.rs +++ b/crates/manta-server/tests/json_stream_acceptance.rs @@ -109,7 +109,7 @@ async fn tcp_client_receives_spot_as_json_lines_message() { assert_eq!(value["frequency"], 14_027_100); assert_eq!(value["dxContinent"], "AS"); assert_eq!(value["dxCqZone"], 25); - assert!(value["dxDxcc"].is_null()); + assert_eq!(value["dxDxcc"], 339); } #[tokio::test] diff --git a/crates/manta-spot/data/SOURCES.md b/crates/manta-spot/data/SOURCES.md index 0fdfffae..cf95130d 100644 --- a/crates/manta-spot/data/SOURCES.md +++ b/crates/manta-spot/data/SOURCES.md @@ -13,6 +13,30 @@ - Refresh: re-run the `curl` in this crate's implementation plan (Task 1) and replace this file by hand -- no refresh automation yet. +## dxcc.tsv + +- Source: derived from https://www.country-files.com/bigcty/cty.csv (AD1C), + columns 1 (primary prefix) and 3 (ADIF DXCC entity code). +- Retrieved: 2026-09-07 +- Format: `\t\t`; + `#`-prefixed and blank lines are ignored. +- License/redistribution: same AD1C source and same convention as cty.dat. +- Refresh: `scripts/gen-dxcc-table.sh` (no argument = fetch upstream). Refresh + it whenever cty.dat is refreshed, ideally from the same AD1C release; + `cty::tests::every_entity_in_the_vendored_cty_dat_resolves_an_adif_dxcc_number` + fails if the two drift apart, and at runtime any spot emitted while they + have drifted is counted by `manta_spots_unresolved_geography_total`. The + script's `awk` splitter is not CSV-aware, so it requires every row to have + exactly 10 comma-separated fields and aborts (writing nothing) if upstream + ever ships an unescaped comma inside a field -- better a loud failure than a + row whose ADIF number silently shifted a column. +- Why keyed on primary prefix, not entity name: measured 2026-09-07 against a + cty.dat retrieved 2026-07-25 and a cty.csv published 2026-09-06 -- the prefix + join matched 346/346, the name join 343/346 (Cape Verde -> Cabo Verde, Juan + de Nova, Europa -> Juan de Nova & Europa, Tristan da Cunha & Gough -> ... + Gough Islands). ADIF 3.1.7's own release notes likewise renamed entities + 207/462/468/502/518. + ## master.scp - Source: https://www.supercheckpartial.com/MASTER.SCP diff --git a/crates/manta-spot/data/dxcc.tsv b/crates/manta-spot/data/dxcc.tsv new file mode 100644 index 00000000..71462348 --- /dev/null +++ b/crates/manta-spot/data/dxcc.tsv @@ -0,0 +1,349 @@ +# ADIF DXCC entity number per cty.dat primary prefix (AD1C cty.csv col 1 -> col 3). +# Regenerate with scripts/gen-dxcc-table.sh; see data/SOURCES.md. +# +1A 246 Sov Mil Order of Malta +1S 247 Spratly Islands +3A 260 Monaco +3B6 4 Agalega & St. Brandon +3B8 165 Mauritius +3B9 207 Rodriguez Island +3C 49 Equatorial Guinea +3C0 195 Annobon Island +3D2 176 Fiji +3D2/C 489 Conway Reef +3D2/R 460 Rotuma Island +3DA 468 Kingdom of Eswatini +3V 474 Tunisia +3W 293 Vietnam +3X 107 Guinea +3Y/B 24 Bouvet +3Y/P 199 Peter 1 Island +4J 18 Azerbaijan +4L 75 Georgia +4O 514 Montenegro +4S 315 Sri Lanka +4U1I 117 ITU HQ +4U1U 289 United Nations HQ +4U1V 206 Vienna Intl Ctr +4W 511 Timor - Leste +4X 336 Israel +5A 436 Libya +5B 215 Cyprus +5H 470 Tanzania +5N 450 Nigeria +5R 438 Madagascar +5T 444 Mauritania +5U 187 Niger +5V 483 Togo +5W 190 Samoa +5X 286 Uganda +5Z 430 Kenya +6W 456 Senegal +6Y 82 Jamaica +7O 492 Yemen +7P 432 Lesotho +7Q 440 Malawi +7X 400 Algeria +8P 62 Barbados +8Q 159 Maldives +8R 129 Guyana +9A 497 Croatia +9G 424 Ghana +9H 257 Malta +9J 482 Zambia +9K 348 Kuwait +9L 458 Sierra Leone +9M2 299 West Malaysia +9M6 46 East Malaysia +9N 369 Nepal +9Q 414 Dem. Rep. of the Congo +9U 404 Burundi +9V 381 Singapore +9X 454 Rwanda +9Y 90 Trinidad & Tobago +A2 402 Botswana +A3 160 Tonga +A4 370 Oman +A5 306 Bhutan +A6 391 United Arab Emirates +A7 376 Qatar +A9 304 Bahrain +AP 372 Pakistan +BS7 506 Scarborough Reef +BV 386 Taiwan +BV9P 505 Pratas Island +BY 318 China +C2 157 Nauru +C3 203 Andorra +C5 422 The Gambia +C6 60 Bahamas +C9 181 Mozambique +CE 112 Chile +CE0X 217 San Felix & San Ambrosio +CE0Y 47 Easter Island +CE0Z 125 Juan Fernandez Islands +CE9 13 Antarctica +CM 70 Cuba +CN 446 Morocco +CP 104 Bolivia +CT 272 Portugal +CT3 256 Madeira Islands +CU 149 Azores +CX 144 Uruguay +CY0 211 Sable Island +CY9 252 St. Paul Island +D2 401 Angola +D4 409 Cabo Verde +D6 411 Comoros +DL 230 Fed. Rep. of Germany +DU 375 Philippines +E3 51 Eritrea +E4 510 Palestine +E5/N 191 North Cook Islands +E5/S 234 South Cook Islands +E6 188 Niue +E7 501 Bosnia-Herzegovina +EA 281 Spain +EA6 21 Balearic Islands +EA8 29 Canary Islands +EA9 32 Ceuta & Melilla +EI 245 Ireland +EK 14 Armenia +EL 434 Liberia +EP 330 Iran +ER 179 Moldova +ES 52 Estonia +ET 53 Ethiopia +EU 27 Belarus +EX 135 Kyrgyzstan +EY 262 Tajikistan +EZ 280 Turkmenistan +F 227 France +FG 79 Guadeloupe +FH 169 Mayotte +FJ 516 St. Barthelemy +FK 162 New Caledonia +FK/C 512 Chesterfield Islands +FM 84 Martinique +FO 175 French Polynesia +FO/A 508 Austral Islands +FO/C 36 Clipperton Island +FO/M 509 Marquesas Islands +FP 277 St. Pierre & Miquelon +FR 453 Reunion Island +FS 213 St. Martin +FT/G 99 Glorioso Islands +FT/J 124 Juan de Nova & Europa +FT/T 276 Tromelin Island +FT/W 41 Crozet Island +FT/X 131 Kerguelen Islands +FT/Z 10 Amsterdam & St. Paul Is. +FW 298 Wallis & Futuna Islands +FY 63 French Guiana +G 223 England +GD 114 Isle of Man +GI 265 Northern Ireland +GJ 122 Jersey +GM 279 Scotland +GM/S 279 Shetland Islands +GU 106 Guernsey +GW 294 Wales +H4 185 Solomon Islands +H40 507 Temotu Province +HA 239 Hungary +HB 287 Switzerland +HB0 251 Liechtenstein +HC 120 Ecuador +HC8 71 Galapagos Islands +HH 78 Haiti +HI 72 Dominican Republic +HK 116 Colombia +HK0/A 216 San Andres & Providencia +HK0/M 161 Malpelo Island +HL 137 Republic of Korea +HP 88 Panama +HR 80 Honduras +HS 387 Thailand +HV 295 Vatican City +HZ 378 Saudi Arabia +I 248 Italy +IG9 248 African Italy +IS 225 Sardinia +IT9 248 Sicily +J2 382 Djibouti +J3 77 Grenada +J5 109 Guinea-Bissau +J6 97 St. Lucia +J7 95 Dominica +J8 98 St. Vincent +JA 339 Japan +JD/M 177 Minami Torishima +JD/O 192 Ogasawara +JT 363 Mongolia +JW 259 Svalbard +JW/B 259 Bear Island +JX 118 Jan Mayen +JY 342 Jordan +K 291 United States +KG4 105 Guantanamo Bay +KH0 166 Mariana Islands +KH1 20 Baker & Howland Islands +KH2 103 Guam +KH3 123 Johnston Island +KH4 174 Midway Island +KH5 197 Palmyra & Jarvis Islands +KH6 110 Hawaii +KH7K 138 Kure Island +KH8 9 American Samoa +KH8/S 515 Swains Island +KH9 297 Wake Island +KL 6 Alaska +KP1 182 Navassa Island +KP2 285 US Virgin Islands +KP4 202 Puerto Rico +KP5 43 Desecheo Island +LA 266 Norway +LU 100 Argentina +LX 254 Luxembourg +LY 146 Lithuania +LZ 212 Bulgaria +OA 136 Peru +OD 354 Lebanon +OE 206 Austria +OH 224 Finland +OH0 5 Aland Islands +OJ0 167 Market Reef +OK 503 Czech Republic +OM 504 Slovak Republic +ON 209 Belgium +OX 237 Greenland +OY 222 Faroe Islands +OZ 221 Denmark +P2 163 Papua New Guinea +P4 91 Aruba +P5 344 DPR of Korea +PA 263 Netherlands +PJ2 517 Curacao +PJ4 520 Bonaire +PJ5 519 Saba & St. Eustatius +PJ7 518 Sint Maarten +PY 108 Brazil +PY0F 56 Fernando de Noronha +PY0S 253 St. Peter & St. Paul +PY0T 273 Trindade & Martim Vaz +PZ 140 Suriname +R1FJ 61 Franz Josef Land +S0 302 Western Sahara +S2 305 Bangladesh +S5 499 Slovenia +S7 379 Seychelles +S9 219 Sao Tome & Principe +SM 284 Sweden +SP 269 Poland +ST 466 Sudan +SU 478 Egypt +SV 236 Greece +SV/A 180 Mount Athos +SV5 45 Dodecanese +SV9 40 Crete +T2 282 Tuvalu +T30 301 Western Kiribati +T31 31 Central Kiribati +T32 48 Eastern Kiribati +T33 490 Banaba Island +T5 232 Somalia +T7 278 San Marino +T8 22 Palau +TA 390 Asiatic Turkey +TA1 390 European Turkey +TF 242 Iceland +TG 76 Guatemala +TI 308 Costa Rica +TI9 37 Cocos Island +TJ 406 Cameroon +TK 214 Corsica +TL 408 Central African Republic +TN 412 Republic of the Congo +TR 420 Gabon +TT 410 Chad +TU 428 Cote d'Ivoire +TY 416 Benin +TZ 442 Mali +UA 54 European Russia +UA2 126 Kaliningrad +UA9 15 Asiatic Russia +UK 292 Uzbekistan +UN 130 Kazakhstan +UR 288 Ukraine +V2 94 Antigua & Barbuda +V3 66 Belize +V4 249 St. Kitts & Nevis +V5 464 Namibia +V6 173 Micronesia +V7 168 Marshall Islands +V8 345 Brunei Darussalam +VE 1 Canada +VK 150 Australia +VK0H 111 Heard Island +VK0M 153 Macquarie Island +VK9C 38 Cocos (Keeling) Islands +VK9L 147 Lord Howe Island +VK9M 171 Mellish Reef +VK9N 189 Norfolk Island +VK9W 303 Willis Island +VK9X 35 Christmas Island +VP2E 12 Anguilla +VP2M 96 Montserrat +VP2V 65 British Virgin Islands +VP5 89 Turks & Caicos Islands +VP6 172 Pitcairn Island +VP6/D 513 Ducie Island +VP8 141 Falkland Islands +VP8/G 235 South Georgia Island +VP8/H 241 South Shetland Islands +VP8/O 238 South Orkney Islands +VP8/S 240 South Sandwich Islands +VP9 64 Bermuda +VQ9 33 Chagos Islands +VR 321 Hong Kong +VU 324 India +VU4 11 Andaman & Nicobar Is. +VU7 142 Lakshadweep Islands +XE 50 Mexico +XF4 204 Revillagigedo +XT 480 Burkina Faso +XU 312 Cambodia +XW 143 Laos +XX9 152 Macao +XZ 309 Myanmar +YA 3 Afghanistan +YB 327 Indonesia +YI 333 Iraq +YJ 158 Vanuatu +YK 384 Syria +YL 145 Latvia +YN 86 Nicaragua +YO 275 Romania +YS 74 El Salvador +YU 296 Serbia +YV 148 Venezuela +YV0 17 Aves Island +Z2 452 Zimbabwe +Z3 502 North Macedonia +Z6 522 Republic of Kosovo +Z8 521 Republic of South Sudan +ZA 7 Albania +ZB 233 Gibraltar +ZC4 283 UK Base Areas on Cyprus +ZD7 250 St. Helena +ZD8 205 Ascension Island +ZD9 274 Tristan da Cunha & Gough Islands +ZF 69 Cayman Islands +ZK3 270 Tokelau Islands +ZL 170 New Zealand +ZL7 34 Chatham Islands +ZL8 133 Kermadec Islands +ZL9 16 N.Z. Subantarctic Is. +ZP 132 Paraguay +ZS 462 South Africa +ZS8 201 Pr. Edward & Marion Is. diff --git a/crates/manta-spot/src/cty.rs b/crates/manta-spot/src/cty.rs index e802d74b..a369eb53 100644 --- a/crates/manta-spot/src/cty.rs +++ b/crates/manta-spot/src/cty.rs @@ -26,7 +26,10 @@ //! United States entry reads `91.87`), while this module's `Entry::lon` //! uses the ordinary east-positive convention (GeoJSON, most mapping //! libraries) that `manta-server`'s JSON stream (`dxLon`/`deLon`) is -//! expected to emit. +//! expected to emit. The primary-prefix field (field 8) doubles as the join +//! key into the vendored `dxcc.tsv` ADIF DXCC entity-number table (MAN-136, +//! `Table::parse_with_dxcc`) -- AD1C's `cty.dat` itself carries no ADIF +//! entity numbers. /// Per-entity metadata carried alongside a prefix, from the `cty.dat` /// header line's `cq-zone`/`continent`/`lat`/`lon` fields (see the module @@ -39,6 +42,14 @@ pub struct Entry { pub cq_zone: u16, pub lat: f64, pub lon: f64, + /// ADIF DXCC entity number for this entity, joined from the vendored + /// `dxcc.tsv` on the `cty.dat` header's primary-prefix field (MAN-136). + /// `None` only if the two vendored files have drifted -- which + /// `every_entity_in_the_vendored_cty_dat_resolves_an_adif_dxcc_number` + /// exists to catch. Callers must treat `None` as "unknown", never as a + /// number: ADIF's code 0 means "confirmed NOT in any DXCC entity", a + /// different and false claim. + pub dxcc: Option, } /// One accumulated `(prefix, entry, is_exact)` row, tracked separately @@ -62,8 +73,16 @@ pub struct Table { } impl Table { - /// Parses a `cty.dat` file's full contents. + /// Parses a `cty.dat` file's full contents, joining each entity against + /// the vendored ADIF DXCC number table. pub fn parse(cty_dat: &str) -> Self { + Self::parse_with_dxcc(cty_dat, crate::DXCC_TSV) + } + + /// `parse` with an explicit DXCC table -- for tests that need to exercise + /// a drifted or malformed table without touching the vendored file. + pub fn parse_with_dxcc(cty_dat: &str, dxcc_tsv: &str) -> Self { + let dxcc_table = parse_dxcc_table(dxcc_tsv); let mut rows: Vec<(Row, bool /* is_starred */)> = Vec::new(); for raw_entry in cty_dat.split(';') { let raw_entry = raw_entry.trim(); @@ -73,9 +92,16 @@ impl Table { let Some(alias_start) = raw_entry.rfind(':') else { continue; // malformed entry, skip }; - let Some((entry, is_starred)) = parse_header(&raw_entry[..alias_start]) else { + let header = &raw_entry[..alias_start]; + let Some((mut entry, is_starred)) = parse_header(header) else { continue; // malformed header, skip }; + entry.dxcc = parse_header_primary_prefix(header).and_then(|pfx| { + dxcc_table + .binary_search_by(|row: &(String, u16)| row.0.as_str().cmp(pfx.as_str())) + .ok() + .map(|i| dxcc_table[i].1) + }); for alias in raw_entry[alias_start + 1..].split(',') { if let Some((prefix, is_exact, zone_override)) = clean_alias(alias) { let mut entry = entry.clone(); @@ -187,11 +213,43 @@ fn parse_header(header: &str) -> Option<(Entry, bool)> { continent: fields[3].to_uppercase(), lat: fields[4].parse().ok()?, lon: -raw_lon, // west-positive (AD1C) -> east-positive; see module doc. + dxcc: None, // filled in by the caller -- see `Table::parse_with_dxcc`. }, is_starred, )) } +/// The header's primary-prefix field (field 8), `*` stripped, uppercased -- +/// the join key into `dxcc.tsv`. A starred subentity joins on its own +/// prefix, which AD1C maps to its PARENT's DXCC number (Sicily -> Italy's +/// 248), which is exactly the desired behavior. +fn parse_header_primary_prefix(header: &str) -> Option { + let fields: Vec<&str> = header.split(':').map(str::trim).collect(); + if fields.len() != 8 { + return None; + } + Some(fields[7].trim_start_matches('*').to_uppercase()) +} + +/// Parses `dxcc.tsv` into a prefix-sorted lookup table. Comment (`#`) and +/// blank lines are skipped; a row missing either field, or whose second +/// field isn't a number, is skipped rather than shifting the table. +fn parse_dxcc_table(tsv: &str) -> Vec<(String, u16)> { + let mut v: Vec<(String, u16)> = tsv + .lines() + .filter(|l| !l.trim_start().starts_with('#') && !l.trim().is_empty()) + .filter_map(|l| { + let mut f = l.split('\t'); + let pfx = f.next()?.trim().to_uppercase(); + let num: u16 = f.next()?.trim().parse().ok()?; + Some((pfx, num)) + }) + .collect(); + v.sort_by(|a, b| a.0.cmp(&b.0)); + v.dedup_by(|a, b| a.0 == b.0); + v +} + /// Strips a leading `=` (exact-call marker) and any trailing /// `(zone)`/`[itu]`/``/`{continent}` annotation, returning the /// cleaned prefix, whether it was an exact-call alias (leading `=`), and @@ -441,4 +499,87 @@ China: 24: 44: AS: 35.0: -103.0: -8.0: BY: let table = Table::parse(FIXTURE); assert!(table.lookup("ZZ9ZZZ").is_none()); } + + /// MAN-136: the whole point of vendoring the table is that it covers the + /// vendored cty.dat COMPLETELY. A `cty.dat` refresh that adds an entity the + /// table doesn't know would otherwise silently downgrade that entity's spots + /// to the UNKNOWN_DXCC sentinel forever. This test is the tripwire: it fails + /// loudly at refresh time instead. + #[test] + fn every_entity_in_the_vendored_cty_dat_resolves_an_adif_dxcc_number() { + let table = Table::parse(crate::CTY_DAT); + let missing: Vec<&str> = table + .entries + .iter() + .filter(|r| r.entry.dxcc.is_none()) + .map(|r| r.prefix.as_str()) + .collect(); + assert!( + missing.is_empty(), + "prefixes with no ADIF DXCC number (refresh data/dxcc.tsv via \ + scripts/gen-dxcc-table.sh): {missing:?}" + ); + } + + /// Spot-checks against the REAL vendored data, across the shapes that + /// exercise different parts of the lookup: a generic prefix, a longest-match + /// subdivision, a starred subentity (which must inherit its PARENT's number, + /// not get one of its own), and an exact-call alias carrying a portable + /// suffix. + #[test] + fn real_callsigns_resolve_their_documented_adif_entity_numbers() { + let t = Table::parse(crate::CTY_DAT); + for (call, want) in [ + ("W1AW", 291), // United States, generic K/W/N prefix + ("JA1ABC", 339), // Japan + ("KL7AB", 6), // Alaska -- longest match beats the generic K + ("KH6XX", 110), // Hawaii -- ditto + ("VE3ABC", 1), // Canada + ("G3ABC", 223), // England + ("GM4XYZ", 279), // Scotland + ("IT9ABC", 248), // Sicily -- STARRED subentity, inherits Italy's 248 + ("JW/LB2PG", 259), // Svalbard -- via the JW/ prefix + ] { + assert_eq!(t.lookup(call).and_then(|e| e.dxcc), Some(want), "{call}"); + } + } + + /// A genuinely unallocated call still resolves to nothing at all -- the DXCC + /// work must not accidentally make the allocation gate more permissive. + /// NOTE: "ZZ9ZZZ" is NOT usable here -- it resolves to Brazil (108) against + /// the real vendored file via the ZZ prefix, and so does "NOCALL" (291, via + /// N). ITU allocates no Q prefixes, so QQ1AAA is genuinely unallocated. + #[test] + fn an_unallocated_callsign_still_resolves_to_nothing() { + let t = Table::parse(crate::CTY_DAT); + assert!(t.lookup("QQ1AAA").is_none()); + assert!(!t.is_allocated("QQ1AAA")); + } + + /// The table parser tolerates the file's own comment/blank lines and skips + /// malformed rows rather than panicking or shifting the whole table. + #[test] + fn dxcc_table_parser_skips_comments_blanks_and_malformed_rows() { + let cty = "\ +United States: 5: 8: NA: 40.0: 75.0: 5.0: K: + K,W,N; +"; + let tsv = "# comment\n\nK\t291\tUnited States\nBOGUS-NO-NUMBER\nZZZ\tnotanumber\n"; + let t = Table::parse_with_dxcc(cty, tsv); + assert_eq!(t.lookup("W1AW").and_then(|e| e.dxcc), Some(291)); + } + + /// An entity absent from the table resolves geography but not a DXCC number + /// -- the UNKNOWN_DXCC path in `spot_message` depends on exactly this. + #[test] + fn an_entity_missing_from_the_dxcc_table_resolves_geography_but_no_number() { + let cty = "\ +Atlantis: 99: 99: EU: 0.0: 0.0: 0.0: XQ9: + XQ9; +"; + let t = Table::parse_with_dxcc(cty, "K\t291\tUnited States\n"); + let e = t.lookup("XQ9AA").expect("geography still resolves"); + assert_eq!(e.continent, "EU"); + assert_eq!(e.dxcc, None); + } } diff --git a/crates/manta-spot/src/lib.rs b/crates/manta-spot/src/lib.rs index 34c62c60..262dcc32 100644 --- a/crates/manta-spot/src/lib.rs +++ b/crates/manta-spot/src/lib.rs @@ -29,3 +29,10 @@ pub const CTY_DAT: &str = include_str!("../data/cty.dat"); /// `data/` -- see `data/SOURCES.md` for provenance and refresh /// instructions. pub const MASTER_SCP: &str = include_str!("../data/master.scp"); + +/// Per-entity ADIF DXCC entity numbers keyed on `cty.dat`'s primary-prefix +/// field, vendored under `data/` -- see `data/SOURCES.md` for provenance and +/// `scripts/gen-dxcc-table.sh` for regeneration. Separate from `cty.dat` +/// because AD1C's `cty.dat` carries no ADIF entity numbers at all (MAN-136, +/// broad-review decision D10). +pub const DXCC_TSV: &str = include_str!("../data/dxcc.tsv"); diff --git a/docs/DECISIONS/2026-09-07-man136-dxcc-and-unknown-geography-sentinels.md b/docs/DECISIONS/2026-09-07-man136-dxcc-and-unknown-geography-sentinels.md new file mode 100644 index 00000000..b4f68e7e --- /dev/null +++ b/docs/DECISIONS/2026-09-07-man136-dxcc-and-unknown-geography-sentinels.md @@ -0,0 +1,149 @@ +# MAN-136: `dxDxcc`/`deDxcc` and the `UNKNOWN_*` geography sentinels + +## Context + +dispensa's `contracts/spots/spots.v1.schema.json` declares `dxDxcc`, `deDxcc`, +`dxContinent`, `deContinent`, and `dxCqZone` **required and non-nullable**. +manta's JSON Lines stream (`:7301`) emitted JSON `null` for `dxDxcc`/`deDxcc` +on **every** spot, regardless of whether the callsign resolved against +`cty.dat` — cqdx's own ingest would reject the batch on this field alone. +Separately, when a callsign genuinely couldn't be resolved (reachable in +production through MAN-28's Watch List allowlist, which lets an operator +allowlist a call that bypasses `cty.is_allocated()` entirely, +`crates/manta-spot/src/validator.rs:669-680`), the fallback values for +`dxContinent`/`dxCqZone` were a real-looking `""`/`0` — indistinguishable, on +the wire, from "we looked this up and got zone 0/empty continent" rather than +"we couldn't look this up at all". This is the geography-null finding already +on file in MAN-45. + +This repo does not vendor dispensa's schema, so the required/non-nullable +reading here comes from three independent broad-review lenses that read the +actual schema directly (2026-09-05 broad review, lens 2 item #13, lens 4 item +#15, lens 5 item #14), not from `spot_message.rs`'s own prior code comment +(which claimed `dxDxcc`/`deDxcc` were nullable on the contract — that comment +was secondhand and wrong, and has been removed as part of this change). This +record supersedes that comment's claim. + +## Decision + +Vendor a small ADIF DXCC entity-number table (`crates/manta-spot/data/dxcc.tsv`, +346 rows) alongside `cty.dat`, derived from AD1C's own `cty.csv` (same +publisher, same license posture, same refresh cadence as `cty.dat`), joined +in `manta_spot::cty::Table` on the primary-prefix field already present in +`cty.dat`'s header. Per broad-review decision D10 +(`docs/DECISIONS/2026-09-06-broad-review-decisions.md:144-148`), this ships +without waiting on a dispensa contract relaxation. + +## Why the primary prefix, not the entity name + +Measured directly (2026-09-07): joining this repo's vendored `cty.dat` +(retrieved 2026-07-25) against AD1C's `cty.csv` (fetched 2026-09-06) on the +primary-prefix field matched **346/346** entities with zero mismatches. +Joining on the entity **name** instead matched only 343/346 — three names had +drifted in a six-week gap: `Cape Verde` → `Cabo Verde`, `Juan de Nova, Europa` +→ `Juan de Nova & Europa`, and `Tristan da Cunha & Gough` → `... Gough +Islands`. ADIF's own 3.1.7 release notes independently corroborate that +entity names are not stable over time (they record renaming entities +207/462/468/502/518 "to realign with the ARRL DXCC list"). The primary +prefix is the stable key. + +## The three sentinel values, and why each + +- **`UNKNOWN_DXCC: i64 = -1`** (`crates/manta-server/src/spot_message.rs`). + Not `0`: ADIF defines DXCC entity code 0 as **"None — the contacted station + is known to NOT be within a DXCC entity"** (e.g. maritime mobile in + international waters) — a specific, different, and false claim to make + about a callsign manta merely failed to resolve. Reusing 0 here would be + the same class of bug this ticket exists to remove. Not `null`: the field + is required non-nullable per the Context section above. `-1` is outside + ADIF's valid 1–522 range, so no consumer can mistake it for real data, and + a JSON Schema `"type": "integer"` with no `minimum` still accepts it. +- **`UNKNOWN_CONTINENT: &str = ""`** and **`UNKNOWN_CQ_ZONE: u16 = 0`** — the + same values these fields already fell back to before this change, but now + **named, exported, documented, tested, and counted**. On inspection these + values were not "fabricated": CQ zones are 1–40 and continents are one of + seven two-letter codes, so both were already out of the field's real + domain. What was actually wrong was that the fallback was *implicit* — a + bare `unwrap_or_default()`/`unwrap_or(0)` with no name, no test, and no way + for a consumer to know it meant "unresolved" rather than "resolved to + this." Naming them fixes exactly that. Inventing new values (e.g. `"XX"` or + `999`) was considered and rejected: it would change wire bytes an existing + consumer has already seen, for no gain in distinguishability over what's + documented here. This choice reuses MAN-45's already-designed names and + values verbatim (see "Reconciliation with MAN-45" below), so the two don't + diverge. + +## What consumers should key on + +Two independent unknown-geography signals are emitted together whenever a +callsign doesn't resolve: + +- **`dxLat`/`dxLon` (and `de*`) are `null`.** These fields stay + `Option` and nullable on the contract, so this is the one + **contract-legal** "unknown geography" signal available today, with no + schema change needed. +- **`dxDxcc == UNKNOWN_DXCC` (`-1`).** This is manta-defined, not yet + contract-ratified by dispensa. + +Both are emitted on the same spot whenever `cty.dat` itself fails to resolve +the callsign, which is the ordinary case; a consumer that doesn't yet know +about `UNKNOWN_DXCC` can still detect "unknown" via the null lat/lon. The one +state where they diverge is `cty.dat`/`dxcc.tsv` drift — `cty.dat` refreshed +by hand (data/SOURCES.md) without regenerating the table — where geography +resolves (real `dxLat`/`dxLon`) but the entity number does not, so `dxDxcc` +is `-1` with non-null lat/lon. That state is a data-vendoring bug, not a +routing signal: `cty::tests::every_entity_in_the_vendored_cty_dat_resolves_an_adif_dxcc_number` +fails the build on it, and `manta_spots_unresolved_geography_total` counts it +at runtime (see below). + +## Operator visibility + +Every time a spot goes out carrying an `UNKNOWN_*` sentinel on either side, +manta increments `manta_spots_unresolved_geography_total` (Prometheus +counter, `crates/manta-server/src/metrics.rs`), incremented once per spot at +publish time (`crates/manta-cli/src/main.rs`), not once per connected JSON/WS +client. The counter's condition is deliberately keyed on the RESOLVED ADIF +entity number (`main.rs`'s `geography_is_unresolved`), not on whether +`cty::Table::lookup` returned an entry, so the drift state above — sentinel +emitted, geography still real — is counted rather than silently uncounted. +The de-side term is resolved once at daemon start: `station_callsign` is +config and cannot change for the life of the process. + +## The open cross-repo item + +dispensa `Q-0028` has been open since 2026-07-14, awaiting skimmer +confirmation before cqdx's implementation proceeds. This decision +deliberately does not wait on it — see D10. If dispensa later ratifies a +different "unknown DXCC" representation than `-1`, this record is what gets +superseded; track that as a follow-up ticket against this record rather than +a blocker on landing MAN-136. + +## Reconciliation with MAN-45 + +MAN-45 (referenced above) documents this same continent/CQ-zone fallback bug +and a previously-designed fix (`UNKNOWN_CQ_ZONE`/`UNKNOWN_CONTINENT` +sentinels, a `manta_spots_unresolved_geography_total` counter). This ticket +reuses those names, values, and the counter's shape verbatim, and adds the +`UNKNOWN_DXCC` sentinel and the `dxcc.tsv` vendored table alongside it — the +half of the problem MAN-45 did not cover. + +## Supersedes + +The `spot_message.rs` module doc and inline comment that claimed +`dxDxcc`/`deDxcc` were nullable on dispensa's `spots.v1` contract. That claim +is wrong per the three independent broad-review lenses cited above, and has +been removed from the code. + +## References + +- Ticket: MAN-136 +- Decision implemented: `docs/DECISIONS/2026-09-06-broad-review-decisions.md` + D10 +- Origin finding: MAN-45; 2026-09-05 broad review lens 2 #13, lens 4 #15, + lens 5 #14 +- Code: `crates/manta-server/src/spot_message.rs`, + `crates/manta-spot/src/cty.rs`, `crates/manta-spot/data/dxcc.tsv`, + `crates/manta-server/src/metrics.rs`, `crates/manta-cli/src/main.rs` +- Upstream data: `https://www.country-files.com/bigcty/cty.csv` (ADIF entity + code = column 3); `https://www.adif.org/` DXCC Entity Code enumeration + (code 0 = "None") diff --git a/scripts/gen-dxcc-table.sh b/scripts/gen-dxcc-table.sh new file mode 100755 index 00000000..9521e381 --- /dev/null +++ b/scripts/gen-dxcc-table.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# Regenerates crates/manta-spot/data/dxcc.tsv from AD1C's cty.csv, whose +# third column is the ADIF DXCC entity code for that entity. Key is column 1, +# the primary prefix -- the SAME key cty.dat's header field 8 carries, and +# empirically stable across releases where entity NAMES are not (a 6-week +# version gap already renamed 3 of 346 entities; 0 prefixes changed). +# See crates/manta-spot/data/SOURCES.md. +set -euo pipefail +SRC="${1:-https://www.country-files.com/bigcty/cty.csv}" +OUT="$(dirname "$0")/../crates/manta-spot/data/dxcc.tsv" +TMP="$(mktemp)" +# Staged next to $OUT and moved into place only on success: `awk` below can +# now exit non-zero mid-pipeline, and writing straight to $OUT would leave a +# truncated table behind when it does. +OUT_TMP="$(mktemp "$(dirname "$OUT")/.dxcc.tsv.XXXXXX")" +trap 'rm -f "$TMP" "$OUT_TMP"' EXIT +if [[ "$SRC" == http* ]]; then curl -fsSL "$SRC" -o "$TMP"; else cp "$SRC" "$TMP"; fi +{ + echo "# ADIF DXCC entity number per cty.dat primary prefix (AD1C cty.csv col 1 -> col 3)." + echo "# Regenerate with scripts/gen-dxcc-table.sh; see data/SOURCES.md." + printf '# \t\t\n' + # `awk -F,` is NOT CSV-aware: a quoted entity name containing a comma (AD1C + # shipped "Juan de Nova, Europa" until recently) would shift $3 off the ADIF + # number while a permissive NF>=10 still admitted the row -- the parser then + # drops it silently and that entity emits dxDxcc: -1. cty.csv rows are + # exactly 10 fields, so require exactly 10 and fail the whole run loudly + # otherwise, rather than emitting a row nobody will notice is wrong. + awk -F, ' + /^[[:space:]]*$/ { next } + NF != 10 { + printf "gen-dxcc-table.sh: line %d has %d comma-separated fields, expected 10 (unescaped comma in a field?): %s\n", NR, NF, $0 > "/dev/stderr" + malformed = 1 + next + } + { p = $1; sub(/^\*/, "", p); printf "%s\t%s\t%s\n", toupper(p), $3, $2 } + END { if (malformed) exit 1 } + ' "$TMP" \ + | LC_ALL=C sort -u +} > "$OUT_TMP" +chmod 644 "$OUT_TMP" +mv "$OUT_TMP" "$OUT" +echo "wrote $OUT ($(grep -cv '^#' "$OUT") entities)" diff --git a/wiki/pages/spot-output-contract.md b/wiki/pages/spot-output-contract.md index 3b1e01ec..dfe722d4 100644 --- a/wiki/pages/spot-output-contract.md +++ b/wiki/pages/spot-output-contract.md @@ -9,8 +9,8 @@ sources: - README.md - CLAUDE.md verified: - commit: e68b106 - date: 2026-07-07 + commit: 01d1ea1 + date: 2026-09-07 links: - spot-validation --- @@ -24,4 +24,4 @@ manta produces spots on two surfaces: a **telnet DX cluster server** (default :7 ## Status caveat -The JSON schema is **not yet frozen in dispensa** — treat the field set as design-phase until the ADR lands. Do not restate fields here; the contract, once written, is authoritative. Validated spots reaching these surfaces come from [[spot-validation]]. +A schema now exists in dispensa and rejects malformed batches — this is no longer design-phase. `dxDxcc`/`deDxcc`/`dxContinent`/`deContinent`/`dxCqZone` are required and non-nullable on it; every spot carries real or named-sentinel values for all five (never `null`), per `docs/DECISIONS/2026-09-07-man136-dxcc-and-unknown-geography-sentinels.md`, which is the authoritative record of what each field means when a callsign doesn't resolve — see that doc and ARCHITECTURE §7 rather than restating the field set here. Validated spots reaching these surfaces come from [[spot-validation]]. diff --git a/wiki/pages/spot-validation.md b/wiki/pages/spot-validation.md index 27cd2683..7c2bd2ad 100644 --- a/wiki/pages/spot-validation.md +++ b/wiki/pages/spot-validation.md @@ -7,8 +7,8 @@ maintainer: agent sources: - ARCHITECTURE.md verified: - commit: e68b106 - date: 2026-07-07 + commit: 01d1ea1 + date: 2026-09-07 links: - decode-chain - spot-output-contract @@ -22,6 +22,7 @@ Decoded CW text is noisy, so validation — not decoding — is what makes a spo - Repetition requirement (a call must decode more than once within a window before first spot) is the main garble filter: §6.4. - Dedupe key = (callsign, freq bucket) with a re-spot suppression window unless SNR improves or type changes: §6.5. - `Validator::tracks`/`RepetitionGate::seen` are per-track_id state that must be freed on `DecoderEvent::TrackClosed` — the normative teardown contract (a real, measured leak this bug produced) lives in `docs/DECISIONS/2026-09-02-man19-track-closed-teardown-invariant.md`, not here. +- An operator-allowlisted callsign (MAN-28's Watch List) can bypass the cty.dat prefix gate entirely, so a spot can reach [[spot-output-contract]] for a call cty.dat genuinely can't resolve — see `docs/DECISIONS/2026-09-07-man136-dxcc-and-unknown-geography-sentinels.md` for what manta emits in that case. ## Why it is shaped this way