Engine-mode follow-ups: witness-level mask schema, registration-seam delta (arity/sorts/artifact hashes), frozen entry point

[levi770]

Follow-ups from the downstream integration mapping of #154 (engine-mode training). All three surfaced while wiring the merged training surface to our acceptance harness; none is a defect in #154 — they are the named remainder between "training side real" and "production bridge serviceable." Ordered by design leverage:

1. Abstention/mask schema for the Phase-2 credit kernel (highest leverage — asking while the kernel is still being scoped). The current surface has no per-witness abstain/mask channel: the network is a plain softmax scorer, and the only "abstention" lives in the selector (refusing a rule) — a different axis. A calibrated typer as the neural predicate needs a witness-level outcome channel distinguishing scored from abstained from masked, with masked contributing exactly zero credit (never coerced to false — that distinction is load-bearing for our downstream Belnap semantics). Our grounding-row schema is a ready spec: neural_outcome / contribution_state / mask_reason per witness row. Happy to share the full field contract.

2. Registration-seam delta (tiny surface, spec exists). register_network (neural.rs:178) unchanged in #154. The bridge needs three additions: arity + arg_sorts (typed signature so enumeration can sort-match candidate slots) and artifact_hashes (identity law: a co-adapted network is a NEW rule — new hash ⇒ new identity downstream). Our test-side stub models the exact extended signature and doubles as the spec.

3. FROZEN-mode entry point. Engine-mode currently always trains (kfold builds a fresh network per fold — correct for selection, but the antithesis of a frozen detector). We need an entry point that runs the same credit/holdout machinery with the network's parameters frozen (no optimizer step, artifact state bit-identical after the run) — both as a production serving mode and as the control arm of our detached/frozen acceptance gate.

cc @niveousdragon — same review channel as #152/#154; we can supply the field-level specs for any of the three on request.

[levi770]

Note: these three asks were also posted in parallel (same team, same minute) as a comment on PR 154 - identical content, a coordination artifact. Treat this issue as the canonical tracking thread; the PR 154 comment defers here.

[levi770]

Field contract: witness-level grounding row (per ask #1)

Sourced from our bridge-acceptance harness (bridge_stub_network.py:145) and the ledger-export tests (test_bridge_ledgers.py:322). This is the executable spec behind the mask surface we proposed for the Phase-2 credit kernel.

Row schema (one row per joined ground witness binding)

field	type	domain	semantics
pair_key	str	"arg0:arg1"	joined entity-pair identifier for the witness binding
pred_id	int	catalog id	relation the witness grounds (resolved to pred_key via catalog)
fact_id	int	runtime id	the fact this witness row grounds
neural_outcome	enum|null	ACCEPT | REJECT | ABSTAIN | None	three-valued neural verdict; None = row is purely classical (no neural evaluation, e.g. ordinary-relation cover)
contribution_state	enum	ACTIVE | MASKED	whether this row contributes to loss/credit
mask_reason	str|null	None | "calibration_abstained" | "quarantined" | "rolled_back"	WHY the row was masked; None iff contribution_state == ACTIVE
credit	signed float	ℝ	credit weight; exactly 0.0 iff MASKED; signed (positive=ACCEPT, negative=REJECT)

Load-bearing invariants (contract, not commentary)

1. Masked ≠ false. A masked row contributes exactly zero to loss and gradient — never coerced to a false label, never assigned negative credit. This is the distinction ask feat(ilp): Tensorized Differentiable ILP engine #1 restates. Our kill criterion 2 (masked-must-not-equal-false) is validated by an explicit wrong-bridge fixture (coerce_abstain_to_false=True) that re-labels ABSTAIN as REJECT with negative credit — the arbiter rejects exactly that shape (test_bridge_kill_criteria.py:210).

2. Three-valued outcome ≠ boolean. ABSTAIN is a distinct epistemic state ("outside calibration domain"), not a fallback between accept and reject. neural_outcome=None on classical rows distinguishes "not neural's domain" from "neural abstained" — these route to different downstream ledgers and must not be collapsed.

3. mask_reason is structural, not informational. The three live values distinguish three different lifecycle paths:

   • calibration_abstained — neural-side, pre-calibration (typer declared this pair outside its calibrated domain before any runtime event)
   • quarantined — runtime-side, pending adjudication (fact temporarily held)
   • rolled_back — runtime-side, post-adjudication reverted (fact was committed, then undone)
     Different reasons route to different Belnap quadrants downstream; collapsing them loses the provenance axis our epistemic ledger requires.

4. credit is the gradient-scaling path. Active rows contribute weight * (score - target)² to loss; gradient flows through the scaling coefficient. Masked rows have no loss term. This is why the mask cannot be approximated by weight=0 on a boolean label — the target itself is undefined for ABSTAIN.

Proposed Phase-2 engine-side delta (NOT yet implemented or authorized — for discussion)

The Phase-2 credit kernel would need, per witness row, the engine to return either {outcome: ACCEPT|REJECT, target: float, weight: float} for active rows or {outcome: ABSTAIN, reason: str} for masked rows. The current surface returns only the former shape (softmax score + boolean cover); there is no field to carry ABSTAIN or mask_reason. A proposed minimal extension: CandidateSpec/credit_nll gains an optional witness_mask: Optional[Tensor] (boolean per-row, True=masked) plus an optional witness_outcome: Tensor over {ACCEPT, REJECT, ABSTAIN} codes. Default behavior unchanged: all rows ACTIVE, no change to existing accepted-shape candidates. Raising as a design point for Phase-2 scoping — not a commitment, happy to refine.

Happy to share specs for asks #2 and #3 on request.

[levi770]

Status update from the consumer side: asks 2 and 3 are delivered and merged via #156 (main @ 0b41dae98). We verified in the merged code that both review blockers were closed — the len(is_positive) == len(facts) typed refusal (silent length-1 broadcast) and the eval-mode typed refusal (BatchNorm/dropout mutation under no_grad). Thanks @niveousdragon for the fast, red-first turnaround on both.

Ask 1 (per-witness mask/abstain channel) remains the only open item — Phase-2 kernel design as scoped, with the field contract we posted above (KC2 appendix) still standing as our consumer-side proposal for the state channel layout. We're keeping this issue open to track it; asks 2/3 can be considered closed here.

On our side, adoption of the merged surface is proceeding as a pinned integration decision (commit-pinned dependency, adapter contract between our pair-relation typer artifacts and your event-level detector interface, compatibility tests) — we'll route any interface questions that surface from that work into this thread.

[niveousdragon]

Ask 1 status: the witness-level mask channel is implemented on the Python surface per your field contract (branch feat/witness-mask-channel, 7 commits, PR to follow shortly). Field mapping against your row schema, stated explicitly so nothing is trusted implicitly:

your field	our surface
contribution_state (ACTIVE/MASKED)	witness_mask: bool Tensor [num_events, n_labels] (True = MASKED), accepted by enumerate_specs / train_engine_mode / kfold_select / frozen_select; default None = all ACTIVE, byte-identical to today (pinned by test)
credit == 0.0 iff MASKED	masked rows are physically excluded from the noisy-OR index — exactly-zero credit AND gradient (a test pins grad == 0.0 on the masked row, not "small")
masked ≠ false (your KC2)	a fact whose candidate has masked witnesses and OR_active < 0.5 is UNCERTAIN (the interval [OR_active, 1] crosses the decision threshold): excluded from holdout accuracy — neither for nor against — and counted against a per-candidate coverage; OR_active ≥ 0.5 stays certain-true by monotonicity. A coverage gate (min_coverage=0.5) drops low-coverage candidates before the fit gate with a named abstention. We mirrored your coerce_abstain_to_false fixture as a test: coercion gates the true rule out, masking selects it — the decisions diverge, pinned
neural_outcome=None on classical rows	relational candidates carry no mask state (masked_any=None) — "not neural's domain" stays distinct from "neural abstained"
mask_reason	deliberately NOT carried on this surface — the boolean channel + coverage accounting live here, the reason taxonomy and Belnap routing stay in your ledger. Reasons reach the engine in Phase 2 if the kernel carries the state channel per your proposed layout

Training semantics (stated limitation): the NLL trains on the lower bound OR_active — masked rows contribute no loss term row-wise, but full interval-aware training is a kernel-phase item, documented as such in the docstrings.

Verified on an A40: the full engine-mode suite is 45 passed including the mask-channel tests, identifiability 16 passed. We also ran the scale probe that Phase-2 kernel scoping needed: at 10^4 events the torch-side credit math is NOT the wall (median step 57 ms) — engine program compilation is (249 s at 10^4, and at 10^5 compile alone exceeded a 40-minute budget; k-fold recompiles per fold, multiplying it). The Phase-2 scoping doc will carry those numbers; the practical consequence is that the kernel work packages as (a) the (float value, witness state) credit layout per your schema, plus (b) a bulk fact-ingestion path so 10^5-event programs stop paying text-parse compilation — happy to have your read on priority between the two.

[levi770]

@niveousdragon @levi770 — status and a finalization ask, now that #157 is reviewed.

Two of the three follow-ups from this issue are delivered:

• Ask feat(ilp): Tensorized Differentiable ILP engine #1 — witness-level mask/abstain channel → feat(ilp): witness-level mask/abstain channel per the #155 field contract #157 (reviewed: the masked ≠ false law holds verbatim — masked witnesses are physically excluded from the noisy-OR index, exactly-zero credit, byte-invariant to the masked row's probability; LGTM).
• Ask v0.4.0-beta → GA: Deferred Items #3 — FROZEN-mode entry point → feat(ilp): typed registry for neural relations and a frozen-detector entry point #156 (frozen_select, params bit-identical after the run).

That leaves ask #2 — the registration-seam delta — as the last remaining piece for the production bridge to be serviceable end-to-end. Requesting its finalization.

register_network (neural.rs:178) still needs the three additions from the original ask:

• arity + arg_sorts — the typed signature so enumeration can sort-match candidate slots. Our registration-mapping registers each promoted relation as NeuralRelationSpec(arity=2, arg_sorts=<catalog head/tail sort ids>); today those are validated adapter-side because the seat doesn't yet carry them.
• artifact_hashes — the identity law: a co-adapted network is a NEW rule ⇒ new hash ⇒ new downstream identity. On our side a retrain mints a new artifact_hash = a new rule_ref; we need the seat to carry the hash so the identity is upstream-anchored, not just adapter-asserted.

Our test-side stub already models the exact extended signature and doubles as the spec — happy to supply the field-level contract on request.

With ask #2 in, the loop closes: register typed → enumerate with sort-matched slots → frozen_select with the #157 mask channel, and our acceptance harness can run the full detached/frozen gate against a real registered surface instead of the stub. Asks #1 and #3 landing cleanly (the mask channel especially — it carries the masked≠false distinction our downstream Belnap semantics is load-bearing on) is what makes finalizing #2 the whole remaining gap. Thanks.

[niveousdragon]

Ask #2 finalization is in progress on our side (branch to follow): register_network gains arity / arg_sorts / artifact_hash with the arity validated against the program's own nn declarations (the engine already retains predicate_arity per declaration — the seat validates against the program, not the adapter), sorts and hash carried opaquely, plus a network_metadata(name) getter returning both the registered metadata and the actual declarations so your registration-mapping can verify against the real surface. Please post the test-side stub's field-level contract here — we will validate the delta against it before the PR opens.

[levi770]

@niveousdragon

Field-level contract of our test-side stub, as requested (source of truth: tests/bridge_stub_network.py plus the production seam that replaced it in-place; every line below is law-reviewed and verified against code on our side).

Entry fields (what our harness reads back):
name: str, module: torch.nn.Module, optimizer: Optimizer | None, scheduler: Any, batching: int, k: int, det: bool, cache: bool, cache_size: int, plus the bridge delta: arity: int, arg_sorts: tuple[int, ...], artifact_hashes: Mapping[str, str] (seam-side form; see identity note).

Registration signature:
register_network(name, module, optimizer, scheduler, batching, k, det, cache, cache_size, *, arity, arg_sorts, artifact_hashes) — the three bridge fields keyword-only.

Refuse conditions (typed errors, identical in stub and production seam):

1. duplicate name (single registry by construction);
2. len(arg_sorts) != arity;
3. any non-int element in arg_sorts, with bool explicitly excluded (subclass of int, not a valid sort id) — we recommend mirroring this trap on your side.

Identity note (convergence): at your boundary the consumer supplies a single opaque artifact hash; the hash is an identity key — a new hash means a new rule identity; upstream stores it opaque and stable, and never inspects it. Our seam-side mapping reduces to that single anchor at the boundary. Your finalization already carries singular — converged.

Arity (locus): arity=2 is the consumer's declaration commitment (we declare binary predicates in the program), not a seam constant. Our structural len(arg_sorts) == arity check at the adapter is the cheap form; your validation against the program's own nn declarations is strictly stronger and composes on top — we recommend keeping both layers.

network_metadata(name) getter: welcome — returning the registered metadata plus the actual declarations gives our registration mapping exactly the verification surface it needs.

Adjacent contract (one line): witness/abstain semantics are already fixed by the #155 field contract as merged in #157 (witness_mask: masked marks exclusion, never a false label); your register_network delta and the mask channel read as one coherent seam contract — this registration contract adds no new witness semantics.

[niveousdragon]

Ask #2 finalization delivered: #160 — converged with your field-level contract before opening: arg_sorts are int sort ids with the bool trap mirrored (refused before extraction, since pyo3 silently coerces True to 1), the three bridge kwargs are keyword-only, artifact_hash is the single opaque identity anchor (re-registration with a new hash stays legal — that is your retrain path), and the two-layer arity validation is kept as you recommended: your structural check adapter-side, our check against the program's own nn declarations engine-side. network_metadata(name) returns both the registered metadata and the actual declarations for your registration-mapping to verify against. GPU-verified on this head: 23/23 registration tests on an A40. Duplicate-name refusal deliberately stays on your adapter (single-registry construction is yours; engine-side re-registration is the identity-law retrain path). Review welcome — same channel as #156/#157.

[levi770]

Closing: all three asks from this issue are delivered and validated.

• Ask 1 / masked-witness semantics — delivered in feat(ilp): typed registry for neural relations and a frozen-detector entry point #156 (merged): masked rows are physically excluded from the noisy-OR index, never coerced false.
• Ask 3 / witness-mask plumbing — delivered in feat(ilp): witness-level mask/abstain channel per the #155 field contract #157 (merged), with the byte-invariance probe class from that review now landed as a test.
• Ask 2 / registration-seam finalization — delivered in feat(neural): registration-seam delta — typed arity/arg_sorts validated against the program, opaque artifact identity, queryable network_metadata #160 and validated by both reviewers:
  • engine/seam review: PASS, no changes requested (comment) — signature/refuse set, two-layer arity validation against nn/4 declarations, opaque singular artifact_hash with re-registration-as-retrain, duplicate-refusal boundary on the consumer adapter as published;
  • contract-fidelity review: PASS, no changes requested (comment) — every published contract term verbatim, independently executed: local wheel build from the PR head, 12/12 tests, six adversarial probes green (float refusal, re-registration hash roundtrip, opacity, negative ids, dict isolation, bool-arity observation noted as optional hardening).

Merging #160 is the remaining mechanical step on the PR itself; the contract work this issue tracked is complete. Thanks @niveousdragon for a convergence that required no changes on either side of the seam.