Skip to content

R_eff: weakest-link cascade zeroes well-evidenced artifacts (ancestor zero overrides local evidence) #392

Description

@explosivebit

Summary

forgeplan's R_eff scoring propagates a zero from a weak ancestor down the
based_on / informs chain, overriding all local evidence
on the downstream
artifact. The result: artifacts that are fully and correctly evidenced
(self_score = 1.0, multiple "Supports" evidence items each scoring 1.0) report
r_eff = 0, making them indistinguishable from genuinely unevidenced
artifacts. In a 309-artifact tree this produces ~52 false-zero artifacts and is
the main driver behind 185 weakest_link_unresolvable anomalies (>60% of the
tree).

Verified on forgeplan CLI/MCP (scores read via forgeplan_score).

Reproducer

# The cascade origin — a leaf design NOTE with NO evidence of its own.
forgeplan_score NOTE-003
#   self_score = 0.0
#   evidence   = []                      ← genuine zero (no local evidence)
#   r_eff      = 0.0
#   grade      = C

# Direct child — perfectly self-evidenced, yet zeroed by the ancestor.
forgeplan_score PRD-018
#   self_score = 1.0
#   evidence   = [EVID-027 (1.0, Supports),
#                 EVID-028 (1.0, Supports),
#                 EVID-033 (1.0, Supports)]
#   weakest_link = NOTE-003
#   r_eff      = 0.0                      ← WRONG: local evidence is perfect
#   grade      = B

# Two hops down — same story, zeroed transitively.
forgeplan_score PRD-021
#   self_score = 1.0
#   evidence   = [EVID-033 (1.0, Supports), EVID-034 (1.0, Supports)]
#   weakest_link = PRD-018
#   r_eff      = 0.0
#   grade      = B

Also verified with the same pattern (self_score 1.0, all evidence 1.0,
r_eff = 0 purely from an ancestor weakest_link): PRD-022 (3× 1.0
evidence, weakest_link = PRD-021) and PRD-023 (2× 1.0 evidence,
weakest_link = PRD-022). The chain is:

NOTE-003 (self 0, no evidence)
   └─> PRD-018 (self 1.0, 3× 1.0 evidence)  → r_eff 0
         └─> PRD-021 (self 1.0, 2× 1.0 evidence) → r_eff 0
               └─> PRD-022 (self 1.0, 3× 1.0 evidence) → r_eff 0
                     └─> PRD-023 (self 1.0, 2× 1.0 evidence) → r_eff 0

A single unevidenced leaf NOTE zeroes a 4-deep chain of otherwise-perfectly-
evidenced PRDs.

Expected behaviour

r_eff for an artifact should primarily reflect the quality of the evidence
that directly supports that artifact. An artifact with multiple
high-congruence "Supports" evidence items at 1.0 should not be reported as
r_eff = 0 solely because an ancestor it is based_on is itself weak.

Ancestor strength is a legitimate signal, but it belongs in a separate
metric (e.g. an ancestry/trust score), not as a multiplicative zero that
overrides locally-grounded evidence.

Actual behaviour

The weakest-link principle is applied across the entire ancestry chain, not
just within the artifact's own evidence set:
r_eff = min(local_evidence..., ancestor_r_eff). One unevidenced ancestor
zeroes the whole subtree regardless of how well each artifact in it is
evidenced.

Scale in a real project

  • 309 artifacts, 185 weakest_link_unresolvable anomalies (>60%).
  • ~52 of those are this bug: self_score = 1.0 + all local evidence 1.0,
    but r_eff = 0 via ancestor propagation (Class B).
  • ~95 are genuine leaf-no-evidence zeros (EVIDs / unevidenced NOTEs) — a
    separate, known concern, see forgeplan_score returns r_eff=0 for leaf EVIDs with canonical bold-pattern bodies #325; do not fix that here.
  • ~38 are detector depth-cap artifacts — filed separately (see the
    companion detector-imprecision issue).

The practical damage: forgeplan anomalies becomes un-triageable — a
maintainer sees 185 anomalies but cannot tell "genuinely unevidenced" from
"perfectly evidenced but poisoned by one weak ancestor" without manually
running forgeplan_score on every artifact.

Suggested fix direction

  1. (Recommended) Decouple local vs ancestry scoring.
    • r_eff_local = min(local evidence scores) — weakest-link within the
      artifact's own evidence only.
    • r_eff_ancestry = min(ancestor r_eff) — kept as a separate, surfaced
      metric.
    • Report r_eff = r_eff_local as the artifact's primary score; show a
      composite trust separately if desired.
  2. (Conservative) Propagate a zero only from a direct based_on parent to
    its immediate child, never transitively down the full subtree.
  3. (Minimal floor) Ancestor propagation may not reduce r_eff below the
    artifact's own self_score when the artifact has valid local evidence.

Severity

Medium-high — not a data-corruption bug (the underlying evidence links are
intact), but it makes the headline quality signal (r_eff) and the anomaly
feed misleading for the majority of a mature artifact graph.

See also

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions