[FEATURE] Route validated expectations onward into checkpoint orchestration - #12068
Draft
joshua-stauffer wants to merge 28 commits into
Draft
[FEATURE] Route validated expectations onward into checkpoint orchestration#12068joshua-stauffer wants to merge 28 commits into
joshua-stauffer wants to merge 28 commits into
Conversation
The schema generator's expectation list is curated: a generated JSON schema is only complete if the expectation's class defines the catalog metadata block (short description, data quality issues, supported data sources). Five registered expectations define no such block, so emitting schemas for them would pass off structurally incomplete files as real ones. Name them explicitly instead, so a completeness check can tell a known, documented gap apart from an accidental one. Also drop a duplicated entry from the generator's list; it produced no second file and only invited confusion about which expectations are covered.
Consumers that want to know what Great Expectations can do currently have to re-parse every schema file to find out. Emit two generated indexes alongside the schemas instead, from the same generation step, so they cannot drift apart. The datasource index maps each schema file to its exact add_or_update_* factory method. Most types snake-case cleanly from the class name but six of twenty-six do not, so the mapping is read from the live type registry at generation time and frozen into shipped data. Because the method name is reconstructed from the registered type name, each one is checked against the real factory surface, so a change to the naming convention fails here rather than shipping an index pointing at methods that do not exist. The expectation index maps each type to its schema file and catalog metadata, and names the registered expectations that have no curated metadata block under documented_absent, so a completeness check can tell a known gap from an accidental one. Both emitters return their serialized content rather than writing it, so a drift check can regenerate in memory and compare against the checked-in file without duplicating the extraction logic it is meant to guard.
A generated file that nothing checks will silently rot. Add assertions that regenerate each index in memory and compare it byte-for-byte to the checked-in copy, that every datasource index entry names a factory method which actually exists, and that every shipped expectation is either cataloged or explicitly recorded as absent - with the catalog and the absence list required to be disjoint, so an entry that gains a schema later cannot linger in both. The completeness check compares against expectations whose implementation lives under the great_expectations package rather than the whole registry. The registry is process-global and test modules register expectations at import time, so an unfiltered comparison would depend on which other test modules happened to be collected. The datasource index guard lives in its own module because the existing schema test file is skipped on every supported Python, so an assertion added there could never run. Hoist the generator's expectation list to a module-level constant so a drift check can regenerate the index without restating the list, which would defeat the purpose of the check.
The expectation and datasource schema catalogs describe what this package can do, and tooling is expected to read them from an installed package rather than from a source checkout. Until now no JSON shipped at all, so those catalogs existed only for people working from a clone. Scope the globs to the two schema trees rather than matching JSON anywhere under the package, so an unrelated JSON file added elsewhere later does not silently become part of the distribution.
Resolving the expectation classes by name makes their type opaque to the checker, so the attribute-defined suppression on the schema call is now unused and fails the type check under warn-unused-ignores.
Three reference documents an agent consults while operating a session: finding or announcing the working context, writing an in-memory session out to a real project, and interpreting slow or failing data operations. Each procedure was executed against a live context rather than written from the API surface, which changed several of them: Neither an unusable GX_HOME nor a discarded cloud configuration produces any signal at the context entry point - both look identical to having no project - so the agent checks the environment itself instead of waiting for an error that never arrives. Adding a data source under a name that already exists replaces it and drops every asset on it, so the write-out procedure fetches first and adds only what is missing, and says plainly what the replacing call would destroy. A time budget is a check-in, not a cancellation: the query is already running on the platform's own compute and keeps costing whether or not the client waits, so the operation is polled on a worker and the user decides whether to keep waiting or narrow the data. Scope reduction leads with a narrower batch definition, because assets describe what data a project works against and batch definitions are how it gets sliced. A row limit through a query asset is offered only when no column supports partitioning, and only as a temporary exploration step.
The entry document carries the guided flow from an empty session to a batch definition that has been proven to read data. Two behaviors it encodes are not obvious from the API surface: Retrieving a batch does not prove a batch definition works. Batch retrieval is lazy for SQL query assets, so a query over a nonexistent table returns a batch and only fails when something reads through it. The flow therefore ends at a head probe, not at retrieval. Replacing a data source drops every asset on it. Reusing an existing data source to add one asset would silently destroy the user's other assets, so the flow fetches first and falls back to the replacing factory only when nothing is there to reuse. Assets and batch definitions have no update-in-place factory at all and raise on a duplicate name, so they reuse or delete-then-recreate. The catalog reference derives every configurable type, its factory name, and its asset and batch-definition surface from the shipped schema index at runtime. The factory naming is irregular in six of twenty-six cases, and a hand-maintained list would go stale the first time a type is added.
The entry document turns checks a user describes in words into expectations drawn from the shipped catalog, runs them, and reports each result. Four behaviors it encodes are not evident from the API. A suite must be registered before expectations are added to it. Expectations added to an unregistered suite validate normally and are then silently lost, because nothing is wired to a store. The handle the store returns is the one to build on. Replacing a suite discards its contents. The upsert factory saves the incoming suite over the stored one rather than merging, so passing a fresh empty suite under an existing name zeroes it with no error. The flow fetches an existing suite and reuses it, and treats replacement as a separate deliberate act. Per-expectation results do not come back in the order they were added. Pairing results to expectations by position appears to work on small suites and silently mislabels larger ones, so results are paired by their own configuration. An empty result means a metric errored rather than data failing, but only when the expectation also did not succeed: a passing expectation can carry an empty result too. The two cases are reported differently, because a broken column is a configuration problem and a failed expectation is a finding about the data. Suite names containing dots fan the store out into nested directories, so they are prohibited.
Agent platforms discover a skill by parsing its frontmatter and following its relative references, so a skill that drifts out of the format stops being found rather than failing loudly. These tests hold the shipped skills to the format: frontmatter parses, the declared name matches the directory, the description stays within limits, and every reference resolves to a real file no more than one directory below the skill root. The three session references are carried by both skills, because a skill directory must be self-contained. They are compared byte for byte, and the failure names the copy direction so drift is a one-command fix rather than a merge. Each check ships with a companion test that builds the violation it is meant to catch and asserts the check reports it. A check that cannot fail is worse than no check, because it reads as coverage. Discovery scans for entry documents rather than naming the skills, and asserts it found some, so an empty scan fails instead of passing quietly.
Guidance that carries code is only as good as the code it carries. The blocks tagged executable now run in order against a throwaway session backed by pandas and sqlite, and the run has to reach three end states: a batch definition that reads data, a validation result carrying an entry per expectation, and a written-out directory that a fresh file-backed context loads with its batch definitions and suites intact. Every other block still has to parse. The negative paths are pinned too, because each one is a claim the guidance makes. A query asset over a missing table hands back a batch and only fails when something reads through it. An expectation whose metric errored returns an empty payload while one that genuinely failed returns a populated one -- and since an expectation that passed also returns an empty payload, both halves of that test are load-bearing. Empty tables and all-null columns produce results rather than raising, including the case where a range check passes because nulls count as missing rather than unexpected. An empty partition window fails at retrieval, before any probe. The configure snippet is exercised against a session that already holds the data source, so it fails if it is ever rewritten to replace rather than reuse. Replacing a data source drops every asset on it, and a snippet that teaches the safe form should stop compiling the moment it stops teaching it. Result ordering gets the same treatment: results are grouped by column and expectations whose metric errored are moved to the front, so an example that pairs results with expectations by position is documenting a coincidence. The text now describes the mechanism, and the tests hold it to that.
Agent platforms look for skills in fixed project directories, so the skills have to be copied out of the package and into the project. The delicate part is not the copy -- it is that the command has to stay safe to re-run against a directory that may hold the user's own work. Each installed skill carries a manifest recording the version that wrote it and a hash of what was written. That is what makes ownership decidable. A copy matching its manifest is left untouched, byte for byte. A copy that still matches its manifest but trails the package is replaced. A copy that no longer matches has been edited by the user, so it is refused and left alone until the run explicitly forces it. A directory with no manifest was never ours and is refused always, including under force -- ownership is not something a flag can assert. Ownership is settled before versions are compared, because the reverse order turns a package having moved on into a licence to overwrite work the user did by hand. Writes stage beside the destination and swap in, so an interrupted run leaves either the previous skill intact or a staging directory the next run clears -- never a half-written skill. Anything that goes wrong with one skill is recorded and reported rather than raised, so one unreadable path costs one destination instead of the whole run. The package hash covers symlinks by their target rather than by what they point at, and the walk that feeds it never descends through a linked directory. Both matter: dereferencing would copy content from outside the package into the project, and directory recursion through links varies by Python version, which would make the same tree hash differently on different interpreters. Discovery resolves the package through the import system rather than walking up from a file path, so it works the same for a wheel, an editable install, and a source checkout. Finding no skills at all is raised rather than reported: nothing to install is a packaging defect, and reporting it as a run where nothing went wrong would hide it.
The skills ship inside the package, but agent platforms look for them in the project. This is the command that bridges the two, invoked as a module rather than a console script so it needs no entry-point wiring and cannot collide with anything already on the path. Install accepts a project root, a target, symlink mode, and a force flag; list reports what is bundled and what each target currently holds, naming the version that installed it so a project left behind by an upgrade is visible before anything is changed. The report is grouped by outcome and the command succeeds only when nothing failed. Two details are less obvious than they look. Failures carry a kind rather than only a message, because the reason a destination was refused cannot be recovered afterwards by looking at it. A skill refused for local edits and a skill whose subdirectory could not be read both still exist and both still hold a valid manifest. Inferring one from the other means telling a user to hunt for a stray file when the real problem is a permission bit. Only the code that made the decision knows, so it now says. The project root defaults to the working directory lazily rather than while the arguments are being defined. Reading the working directory can fail -- it has been deleted or unmounted -- and doing it during argument definition puts that failure outside the handler that turns it into a message, so every invocation raised, including the ones that never needed a directory at all.
The installer decides when to overwrite files in a directory someone else owns, so these tests check the project on disk rather than the report the installer returns. A report can say a skill was left untouched while the bytes underneath it moved; only the filesystem settles it. Modification times and inodes are compared alongside contents, because rewriting a file with identical bytes is still a write. Every check ships with a companion that breaks the installer in the one way that check exists to catch, and asserts the check notices. A test suite that cannot fail is worse than none, because it reads as coverage. Building those companions turned up a recurring weakness in the tests themselves. A helper that sets up more than the behaviour under test requires quietly removes a dimension from every test that uses it: one helper always paired an edit with a version change, so nothing could distinguish a stale skill from an edited one, and deleting that distinction from the installer left the whole suite green. The same shape hid three more, including a re-run that asks for the other install mode and a rebuild that changes content without changing the version -- the case an editable install lives in. The helpers now take those dimensions as explicit arguments and verify them, so a run set up to prove something it does not prove reports that instead of passing. Each condition the installer tests before deciding a skill is current or unmodified is now covered by its own test, one at a time.
The skills are read from the installed package at runtime, so they have to travel with it rather than exist only in the source tree. Matched by file rather than by directory, since a package_data pattern only ever selects files and wheels cannot record an empty directory.
Packaging rules fail silently: a glob that stops matching drops files from the wheel while every source-tree test stays green. This checks the artifact users actually install, after installation, so a packaging regression fails loudly instead of shipping.
Captures why the guidance is version-matched by living inside the distribution, why the command surface is module invocation rather than a console script, and why installing copies into a project by default with an ownership manifest instead of writing over what it did not create.
The rationale is the same; the phrasing no longer dates itself against whatever the test suite happened to cover at the time it was written.
A missing-driver error names the bare distribution to install, and the guidance told the agent to relay that message close to verbatim. Following it resolves a driver version against nothing in particular and leaves the environment with no record of which backend the dependency was for. The install step now names the optional dependency group the package declares for that backend, with the group names read off the installed distribution rather than recalled — they are version-matched, and several do not match the backend's common name. Handing the command to the user rather than running it is unchanged.
Two places where the flow let something happen that nobody chose. A user who expressed no batching preference fell through to a whole- collection definition, without ever being told whether the data carried a column that could slice it. The choice now stays open until the verification probe runs, whose frame already names every column and type, and a whole-collection result is reported as something the survey found rather than something the flow assumed. The worked examples opened with a bare data-source factory call. That factory replaces rather than merges, so a copy of the example aimed at a name that already exists drops every asset attached to it, silently and with the flow reporting success. The prose caveat above the block does not survive being copied, so the reuse guard now lives in the blocks themselves. The asset and batch-definition calls stay bare: those refuse a duplicate name rather than destroying anything.
Writing an in-memory session out to a project creates a directory on the user's disk, and `get_context(mode="file", project_root_dir=...)` creates it silently: no refusal, no prompt, and a return value identical whether it opened a project or brought one into existence. The guidance already said to ask first, but said it as prose above a complete, runnable procedure whose only trace of the requirement was the word "confirmed" inside a placeholder string. Followed as written, the whole flow — preflight, configure, verify, write-out — assembles into one program, and a gate that lives between two snippets in one program is not a gate at all: there is no point in the run where an answer could arrive. The confirmation is not declined, it never occurs. Nothing downstream can catch it. Once the directory exists, discovery finds it on every later turn and truthfully reports an existing project at that path, exactly as it would report one the user made themselves. The call site is the only place this is catchable, so the requirement now lives there: write-out opens with the two things the user has to have said, states that a request to "add a data source" is neither of them, and marks the write-out as its own run beginning from a reply. Preflight's never-scaffold rule no longer carves out the write-out procedure as an exception — it is the same rule with its condition met, and being inside that document is not what makes the call legitimate. The accompanying test holds the snippets to it: no shipped snippet may call `get_context(mode="file", ...)` with a directory it chose or with no directory at all. Both shapes read as ordinary code, and an example path is the part of a snippet a reader is least likely to revisit, because it already looks filled in. A placeholder cannot be run as it stands, which is what keeps the confirmation in front of the call rather than behind it.
…checkpoints The write-out procedure previously stopped at expectation suites, so a session that had gone on to build a checkpoint could not be persisted in full. The two new steps run after suites because a validation definition that references a suite not yet persisted in the target is rejected. Checkpoints are rebuilt from the objects just added to the target rather than carried over from the session: the freshness check resolves handles through the process-global active context, so any handle belonging to a different context than the active one is treated as stale. Post-run actions carry over on the rebuilt checkpoint. The procedure reports three outcomes rather than two. An object already present under the requested name is left untouched and reported as found, not written, because the object the user would then run is the one that was already there. A checkpoint can be written while grouping a validation definition that was found, so that pairing is disclosed and the name resolved with the user before the checkpoint is run.
…eckpoint Continues the guided path: a session that has connected data and written expectations can now bind those pairs into validation definitions, group them into a named checkpoint with post-run actions, and verify it with a single run. Notes on the shape of the guidance: Checkpoints are added explicitly before the verification run. A run will persist an unsaved checkpoint whose children are saved, so relying on that works silently and teaches nothing; the explicit add is stated as a rule. Reuse is fetch-first throughout. Upserting a checkpoint replaces its validation-definition membership and its actions wholesale, and passing a fresh suite object under an existing name replaces that suite's contents. Validation definitions in the store are not deleted. Whole replacement is the only sanctioned use of an upsert. The catalog documents the eight attachable actions. Four accept environment variable templates in their credential fields; the other four declare those fields as plain strings, which Great Expectations never substitutes, so a template is stored and used literally and fails at run time rather than at setup. Making such a field functional requires a real credential in checkpoints/<name>.json, which the scaffolded .gitignore does not exclude. The skill declines to write one there and says so, rather than offering an opt-in that would put a secret into tracked project config. Data Docs output is offered in any session type. Editing a project's configuration to restore a disabled site is gated on the user agreeing first.
Validation results were being presented as a terminal state: the flow ended with a saved suite and a printed report, leaving no path to a check that survives the session. The ending now keeps the honest statement that results are not persisted, then offers the checkpoint skill as the next step. The offer is a routing statement, not an obligation -- declining is an explicit end state, and the write-out offer for in-memory sessions still comes first. The closing scope list drops orchestration, which is no longer out-of-family; only rendering results elsewhere remains outside the flow.
Three contracts rested on markdown that nothing checked, each failing silently. A skill shipping without one of the shared session references would go unnoticed: the byte-equality check compares copies that exist, so a missing one has nothing to compare against. A post-run action added to the package would leave the documented action catalog quietly incomplete. And a skill named in prose as a hand-off target could be misspelled or go stale, routing a user nowhere. Adds a carriage check, an action-catalog drift check pinned to the shipped registry and scoped to this package's own registrations, and a check that skill names written in prose resolve to real bundled directories. Reference extraction now covers fenced content, where a wrong path previously passed unseen. The bundled-skill floor rises to three. Each check is paired with a test that introduces the corresponding violation into a throwaway copy and asserts the check reports it.
The write-out procedure looked its target suite up as
suites.get("my_suite"), but nothing ever persists a suite under that
name. Unlike the data source, asset and batch definition -- which the
procedure creates under the very names it later fetches -- the suite
arrives from the session already named, so the literal only matched by
coincidence.
The failure was silent rather than loud: the surrounding loop records a
failed step and continues, so a user following the procedure ended up
with a project missing both its validation definition and its checkpoint,
with no error raised. Fetching by the session suite's own name is the
only lookup that holds regardless of what the suite is called.
The checkpoint guidance was prose that nothing executed. These tests run it: bind, group, add explicitly, run, and assert per-validation-definition and per-expectation entries; a fresh context reruns the persisted, written-out checkpoint unmodified; and the handoff run snippet executes as a subprocess with its exit code asserted for both a passing and a failing checkpoint. Five traps are pinned exactly as observed, so guidance that drifts from behavior turns a test red rather than misleading a reader: the destructive add_or_update cascade in all three of its directions, including the fetch-first path that makes the documented rule safe; the ordering failure behind referencing an unpersisted suite; an ephemeral run producing local Data Docs URLs; the silent no-op under a null sites entry and its consent-gated recovery; and the mixed temporal-source failure mode. The round trip executes the shipped write-out text itself rather than a copy of it, so the procedure cannot regress while the test keeps passing.
The installed-artifacts guard asserted an exact set of two skills, so the checkpoint skill had to join it: that exact-set assertion is what catches a packaging glob which fails to pick up a new skill directory, and a wheel that ships a skill without its entry document installs nothing at all while exiting zero. The guard's own prose and one function name encoded the count as well, which would have gone stale again at the next skill. Both now describe the bundled skills without numbering them. The nearby references to the two catalog indexes are a different count and stay as they are. The skills command help now mentions checkpoint orchestration alongside data sources and expectations.
Two references named step 5 as where the explicit checkpoint add happens. Step 5 explains why step 4 adds explicitly; the add itself is in step 4. An agent following the numbering to find the call would land on prose instead of code.
Base automatically changed from
f/agent-skills/gate-write-out-scaffolding
to
develop
August 17, 2026 10:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacks on #12063. Extends the bundled agent-skill guidance so a session that has already connected a data source and written expectations can go one step further: binding validations into a checkpoint, adding post-run actions, and running it — with the same version-matched, testable-guidance discipline the earlier skills established.
Why
Guidance that describes a procedure without anything executing it degrades silently — the moment the underlying API shifts, the guidance is confidently wrong rather than visibly out of date, and nothing catches it. Every new procedural claim added here ships paired with a test that would fail if the claim stopped being true, and the installed-artifacts guard is updated to scale to an arbitrary number of bundled skills instead of re-encoding a specific count that the next skill would immediately go stale.
Note on CI
The installed-artifacts guard step does not execute on this PR: under
pull_request_targetthe workflow definition is read from the base branch, and the step is additionally gated on the PR being non-draft. A skipped or absent run of that step here is not a signal either way — it will run once this PR is out of draft on top of a base branch that already carries the workflow update.Test plan
mypystatic type-checking — clean, no issues founddevelop) for unrelated changes, secrets, and internal references