Skip to content

sync.sh Phase 3 digest-cache misses cross-package deps on moved kernel symbols -> stale downstream output #606

Description

@joshsh

bin/sync.sh Phase 3 (regenerates hydra-kernel + hydra-build + hydra-rdf + hydra-pg into each host language, sync.sh:593-604) uses a digest-cache freshness check to skip regeneration when "nothing changed." That check does not model a downstream package's dependency on kernel symbol locations. So when a kernel symbol relocates, Phase 3 can short-circuit as fresh and fail to regenerate a downstream package whose generated output references that symbol — leaving stale output pointing at the symbol's old location.

Concrete failure (from #560)

dslModuleName moved from hydra.Names to hydra.Dsls (via #548/#555). dist/java/hydra-build/src/main/java/hydra/build/Routing.java referenced hydra.Names.dslModuleName and was generated 2026-07-09, before that move. Multiple subsequent scoped Java syncs did not regenerate it — Phase 1 reported "DSL -> JSON... skipped -- every input clean" (the kernel was fresh), and Phase 3 also short-circuited, even though hydra-build's Java output was stale relative to the moved kernel symbol. This surfaced only when #560 wired hydra-build into the Java build and the compile failed (cannot find symbol: dslModuleName). Force-regenerating dist/java/hydra-build (sync-packages.sh hydra-build --targets java) fixed it, confirming the artifact -- not the kernel -- was stale.

Root cause

Phase 3's freshness check keys on kernel/JSON input digests but doesn't invalidate a downstream package when a kernel symbol it references moves (the dependency edge "hydra-build's generated output <- kernel symbol locations" isn't modeled).

Impact

Any downstream package (hydra-build, hydra-rdf, hydra-pg) can silently carry stale generated output after a kernel symbol relocates -- invisible until something compiles against it. Same silent-stale class as #562 / #551.

Proposed fix directions

  • (a) include downstream packages' generated output in Phase 3's freshness digest (so a kernel change that alters emitted symbols invalidates them), or
  • (b) make Phase 3 unconditionally regenerate downstream packages when any kernel input changed (cheaper to reason about; may cost sync time), or
  • (c) a dependency-aware invalidation that models cross-package symbol references.

Relates to: #416, #559, #560, #555, #562, #551.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions