fix(sync-service): start missing dependency consumers before their parent - #4779
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4779 +/- ##
==========================================
- Coverage 60.06% 59.25% -0.82%
==========================================
Files 397 379 -18
Lines 43772 42653 -1119
Branches 12589 12333 -256
==========================================
- Hits 26290 25272 -1018
+ Misses 17401 17339 -62
+ Partials 81 42 -39
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
… when its materializer starts A dependency (subquery) shape can be registered in ShapeStatus with a completed snapshot but no running consumer: consumers for restored shapes start lazily on their first transaction, and a dependency shape orphaned by its parent's removal is restored as a plain shape on restart (prune_subquery_shapes/1 only matches dependencies through a surviving parent). A new parent shape then resolves its subquery to the orphaned handle, its dependency materializer crashes calling the missing consumer, and the parent is invalidated while the dependency stays registered, so every retry fails the same way. Have the materializer start the missing consumer via ShapeCache.start_consumer_for_handle/3, the same mechanism the transaction routing path uses, instead of assuming one is running.
…terializer `start_shape/3` assumes that the consumers of a shape's dependencies are already running, but `maybe_create_shape/2` returned an existing inner shape handle without checking that. Enforce the precondition there: when an inner shape resolves to an existing handle with no registered consumer, restore it via `restore_shape_and_dependencies/3` before the parent is started. Compared to starting the consumer from the materializer's init: - runs inside ShapeCache, so no materializer -> ShapeCache round trip - passes the inner shape `opts` through, so the restored consumer keeps `is_subquery_shape?: true` (write_unit=txn) like every other dependency consumer; the test now asserts this - a failure to start the dependency surfaces as a create error for the request instead of a materializer crash that invalidates the parent - the materializer no longer needs to change (the previous version also introduced a clause-grouping compiler warning that fails CI's --warnings-as-errors check) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Yof2snscrAFy7ToQNZgq4
alco
force-pushed
the
fix/start-dependency-consumer-on-demand
branch
from
September 1, 2026 09:15
e75eb2a to
5ada318
Compare
✅ Deploy Preview for electric-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
alco
added a commit
that referenced
this pull request
Sep 1, 2026
…n writes (#4780) ## Background Shape consumers without their own subquery dependencies start with `write_unit: :txn_fragment`. Consumers known to be inner shapes start with `write_unit: :txn`, because dependency materializers and outer-shape evaluation require transaction-atomic updates. The startup hint is not sufficient when an already-running consumer is later adopted as a dependency. This can happen in two pre-existing sequences: - a standalone shape is requested directly and later reused as a parent's subquery dependency; - after restart, transaction routing lazily recovers an orphaned dependency as a standalone shape before a new parent reuses it. In both cases the dependency materializer can subscribe to a consumer that remains in fragment-write mode. ## Fix Treat materializer subscription as the authoritative signal that a consumer is an inner shape and promote it to whole-transaction writes. If no transaction is in progress, promotion is immediate. If the subscription arrives during a fragmented transaction, that transaction finishes in fragment mode and promotion occurs at its commit boundary. This avoids mixing already-written fragments with a buffered tail of the same transaction. The existing `is_subquery_shape?` startup option remains a fast path for consumers known to be dependencies when they start. ## Tests - verifies an already-running standalone consumer is reused by a parent and promoted from `:txn_fragment` to `:txn`; - verifies subscription during an in-flight fragmented transaction defers promotion until commit; - covers the immediate and deferred state transitions directly; - full relevant files: `110 passed, 1 excluded` across ShapeCache, consumer state, and consumer tests. ## Stack This PR is stacked on #4779 and should be reviewed against `fix/start-dependency-consumer-on-demand` until that PR merges. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
This PR has been released! 🚀 The following packages include changes from this PR:
Thanks for contributing to Electric! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is based on the code from #4774 and supersedes it.
Problem
A subquery dependency can remain registered in ShapeStatus with a completed snapshot but no running consumer. When a new parent resolves its subquery to that existing handle, parent startup assumes the dependency consumer already exists. Its materializer fails to subscribe, the parent is invalidated, and every retry repeats the same failure because the dependency remains registered.
One way to reach this state is for a dependency to become orphaned before restart. The restart pruning pass only recognizes dependencies through surviving parents, while restored standalone shapes start consumers lazily when transaction routing first needs them.
Solution
When ShapeCache resolves an existing shape while creating an inner shape, ensure its consumer is running before starting the parent. Missing consumers are restored through the same dependency-aware path used by transaction routing, while preserving
is_subquery_shape?: trueso the recovered dependency buffers complete transactions.The shared restoration helper also removes duplicated consumer-starting logic.
Test
The regression test constructs the registered/completed-snapshot/no-consumer state, requests a new parent, and verifies that:
write_unit: :txn.Focused verification:
mix test test/electric/shape_cache_test.exs:1081.Follow-up
A pre-existing edge case remains when an already-running standalone consumer is later reused as a dependency: its write mode must be promoted at a transaction boundary. That behavior is addressed in #4780.