Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 35 additions & 12 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -313,11 +313,21 @@ validation (MAN-28). Dedupe (step 5) still applies.
`bounded_io` read rejections, malformed-WS-frame disconnects, and
rejected metrics-endpoint requests are all logged (plain `fmt` output,
`RUST_LOG`-controlled, default `info`) to give an operator a durable
record to reconstruct an abuse incident after the fact. Still
aspirational: `manta-input`/`manta-engine` carry no logging of their
own yet (decode-pipeline internals, not the network-facing surface
MAN-59 scoped to), and `manta --status` hitting a local control socket
for live stats is similarly not yet implemented. Prometheus text
record to reconstruct an abuse incident after the fact. **The daemon
also logs its own liveness** — landed 2026-09-07 (MAN-122,
`docs/DECISIONS/2026-09-07-man122-operator-liveness-logging.md`): one
startup banner (version, source, sample rate, dial frequency, station
callsign, real bound telnet/JSON/metrics addresses) logged once every
bind succeeds and before any listener task is spawned, plus a
rate-limited periodic status line (`manta_server::status`,
`status_interval_secs` in `[server]`, default 60 s, `0` disables) naming
active track count, spots/min, connected client count, and uplink
connection state. Still aspirational: `manta-input`'s and
`manta-engine`'s own internals carry no logging of their own yet
(decode-pipeline internals, not the network-facing surface MAN-59
scoped to, nor the daemon-lifecycle surface MAN-122 scoped to), and
`manta --status` hitting a local control socket for live stats is
similarly not yet implemented (MAN-44). Prometheus text
endpoint (feature `metrics`): input overruns, active tracks, evictions,
decode rate, spots/min, per-stage queue depths, spot confidence
histogram — also aspirational for several of these fields; the
Expand All @@ -328,13 +338,26 @@ validation (MAN-28). Dedupe (step 5) still applies.
gauges, `manta_source_health`, and the uplink counters
(`crates/manta-server/src/metrics.rs`) — not input-layer overruns or
per-stage queue depths, which MAN-56 tracks as a separate gap.
**`manta_active_tracks` is served but not populated** (corrected
2026-09-03, review round 4): the field/gauge exists in `Metrics`, but
`set_active_tracks`'s only non-test call site is absent — `main.rs`'s
own comment says the engine exposes no hook for it yet — so every
production daemon run reports a constant `0`, not a real track count.
Listed separately from the "currently-implemented" set above so an
operator doesn't read a served-but-frozen placeholder as live data.
**`manta_active_tracks` is now populated** (corrected 2026-09-07,
MAN-122): the count comes from `TrackManager`'s own lifecycle, not from
the decode event stream. `TrackManager::decoding_track_count()` reports
how many tracks are currently promoted and holding a leased decoder
(`Active` or `Hang`); `manta_engine::listen_with_track_count()` hands
that number to `main.rs` after every processed batch (suppressing
repeats), which publishes it via `set_active_tracks`. So
`manta_active_tracks` and the status line's `tracks=` field both report
a real, moving count instead of the previous permanent `0`. An
event-derived count was tried first and rejected in review: a track
`TrackManager` has promoted but whose demodulator has not latched emits
no events at all — `TrackDecoder` withholds `TrackMeta` until
`snr_2500_db()` is `Some` — and such a track can stay ACTIVE until the
~30 s `gc_hops` silent GC, so a weak or unmodulated signal that real
decoders are working on would have reported `tracks=0`. The gauge is
driven back to `0` at end of stream, so it doesn't stay stuck at the
last live value after EOF or an SDR disconnect. Note this is
deliberately *not* `TrackManager::active_track_count()`, which also
counts unconfirmed CANDIDATEs (noise-blip rise crossings that lease no
decoder) and keeps its own meaning for `soak_metrics`.
**`manta_source_health` is one-sided** (corrected 2026-09-03, review
round 7, filed as **MAN-64**): the only production call site
(`main.rs:1082`) ever sets it `true`; nothing transitions it to `false`
Expand Down
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions crates/manta-cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,4 @@ coppa-audio = { workspace = true }
serde_json = { workspace = true }
manta-testkit = { workspace = true }
tempfile = { workspace = true }
hound = { workspace = true }
101 changes: 88 additions & 13 deletions crates/manta-cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -773,9 +773,17 @@ fn shutdown_runtime_after_drain(
rt.shutdown_timeout(std::time::Duration::from_secs(2));
}

/// What the MAN-122 startup banner names about the live source; carried as
/// one struct so `start_spot_server`'s argument list stays at four.
struct SourceInfo<'a> {
name: &'a str,
sample_rate_hz: f64,
dial_freq_hz: f64,
}

fn start_spot_server(
config_path: &std::path::Path,
sample_rate_hz: f64,
source: SourceInfo<'_>,
epoch: std::time::SystemTime,
session_nonce: u128,
) -> Result<(tokio::runtime::Runtime, SpotServer)> {
Expand Down Expand Up @@ -812,7 +820,7 @@ fn start_spot_server(
let cfg = file.server;

let bus = std::sync::Arc::new(manta_server::bus::SpotBus::new(
sample_rate_hz,
source.sample_rate_hz,
epoch,
session_nonce,
));
Expand All @@ -831,6 +839,26 @@ fn start_spot_server(
let metrics_listener =
tokio::net::TcpListener::bind((cfg.bind_addr.as_str(), cfg.metrics_port)).await?;

// MAN-122 scenario 1. Emitted after every bind succeeds (so a bind
// failure never produces a "ready" line) but BEFORE any listener task is
// spawned -- on a multi-thread runtime a spawned accept loop can admit a
// client immediately, and its per-connection line would otherwise be able
// to land ahead of the banner. `local_addr()`, not the configured port,
// so a `*_port = 0` (ephemeral) config still names the real address.
tracing::info!(
"{}",
manta_server::status::format_startup_banner(&manta_server::status::StartupInfo {
Comment on lines +848 to +850

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Delay the ready banner until pipeline startup succeeds

When a replay contains less than the two-second calibration window, or a live source fails during its initial reads, this banner has already declared the daemon ready even though listen_with_track_count subsequently fails during calibration and the process exits without ever decoding. Binding the network sockets is therefore not sufficient evidence of readiness; emit the ready banner only after engine initialization and calibration succeed, or label this event as listeners-bound and emit a separate readiness event later.

Useful? React with 👍 / 👎.

version: env!("CARGO_PKG_VERSION"),
source: source.name,
sample_rate_hz: source.sample_rate_hz,
dial_freq_hz: source.dial_freq_hz,
station_callsign: &cfg.station_callsign,
telnet_addr: telnet_listener.local_addr()?,
json_addr: json_listener.local_addr()?,
metrics_addr: metrics_listener.local_addr()?,
})
);

let telnet_ip_command_limiter = manta_server::rate_limit::IpRateLimiter::new_with_override(
manta_server::telnet::MAX_TELNET_COMMANDS,
manta_server::telnet::COMMAND_RATE_WINDOW,
Expand Down Expand Up @@ -908,6 +936,7 @@ fn start_spot_server(
// Vec is empty). Each task owns its own SpotBus subscription and
// backoff state, so one target being down never affects another's
// delivery or retry timing.
let enabled_uplinks = rbn_uplink_cfgs.iter().filter(|u| u.enabled).count();
for uplink_cfg in rbn_uplink_cfgs {
tokio::spawn(manta_server::uplink::serve(
uplink_cfg,
Expand All @@ -918,6 +947,17 @@ fn start_spot_server(
));
}

// MAN-122 scenario 2.
manta_server::status::spawn_status_line(
metrics.clone(),
cfg.status_interval_secs
.map_or(manta_server::status::DEFAULT_STATUS_INTERVAL, |secs| {
std::time::Duration::from_secs(secs)
}),
enabled_uplinks,
shutdown_rx.clone(),
);

anyhow::Ok(())
})?;

Expand Down Expand Up @@ -1129,13 +1169,21 @@ fn main() -> Result<()> {
.as_nanos(),
};

let (rt, server) =
start_spot_server(&path, src.sample_rate(), epoch, session_nonce)?;
let (rt, server) = start_spot_server(
&path,
SourceInfo {
name: source_name,
sample_rate_hz: src.sample_rate(),
dial_freq_hz: src.center_freq_hz(),
},
epoch,
session_nonce,
)?;
// Real, if coarse, health signal: this source opened
// and is running. `active_tracks` has no equivalent
// hook yet -- manta-engine exposes no live track-count
// API for `listen()`'s callbacks to read, so it stays
// at Metrics::default()'s 0 until that surface exists.
// and is running. `active_tracks` is populated below,
// from `manta_engine::listen_with_track_count`'s
// observer, which reports `TrackManager`'s own
// promoted-track count (MAN-122).
//
// MAN-55: for a source where `open()` succeeding
// doesn't confirm a live device (HPSDR's UDP
Expand Down Expand Up @@ -1170,16 +1218,16 @@ fn main() -> Result<()> {
ctrlc::set_handler(move || {
stop_handler.store(true, std::sync::atomic::Ordering::Relaxed);
})?;
let listen_result = manta_engine::listen(
let listen_result = manta_engine::listen_with_track_count(
src,
&cfg,
stop,
|ev| {
use manta_decode::events::DecoderEvent;
if json {
println!("{}", serde_json::to_string(ev).unwrap());
return;
}
use manta_decode::events::DecoderEvent;
use std::io::Write as _;
match ev {
DecoderEvent::CharDecoded { glyph, .. } => {
Expand Down Expand Up @@ -1218,6 +1266,21 @@ fn main() -> Result<()> {
spot.confidence
);
},
// MAN-122 review round 1: the live track gauge comes from
// `TrackManager`'s own lifecycle -- how many tracks are
// promoted and holding a decoder right now -- not from the
// `DecoderEvent` stream above. A promoted track whose
// demodulator has not latched emits nothing for up to the
// ~30 s silent-GC window, so an event-derived count reports
// `tracks=0` on a node that is genuinely decoding weak
// signals. `listen_with_track_count` already suppresses
// repeats, so this is one relaxed atomic store per real
// change.
|n_tracks| {
if let Some(server) = &spot_server {
server.metrics.set_active_tracks(n_tracks as u64);
}
},
);

// Run the same server-shutdown sequence on BOTH the success and
Expand Down Expand Up @@ -1599,7 +1662,11 @@ mod tests {

let (rt, _server) = start_spot_server(
cfg_file.path(),
96_000.0,
SourceInfo {
name: "file",
sample_rate_hz: 96_000.0,
dial_freq_hz: 14_000_000.0,
},
std::time::SystemTime::UNIX_EPOCH,
0,
)
Expand Down Expand Up @@ -1649,7 +1716,11 @@ mod tests {

let (rt, _server) = start_spot_server(
cfg_file.path(),
96_000.0,
SourceInfo {
name: "file",
sample_rate_hz: 96_000.0,
dial_freq_hz: 14_000_000.0,
},
std::time::SystemTime::UNIX_EPOCH,
0,
)
Expand Down Expand Up @@ -1708,7 +1779,11 @@ mod tests {

let (rt, _server) = start_spot_server(
cfg_file.path(),
96_000.0,
SourceInfo {
name: "file",
sample_rate_hz: 96_000.0,
dial_freq_hz: 14_000_000.0,
},
std::time::SystemTime::UNIX_EPOCH,
0,
)
Expand Down
Loading
Loading