Skip to content

Restructure example notebooks and add cross-table merge support - #45

Merged
Jamestth merged 9 commits into
mainfrom
feat/cross-table-merge-annotated-output
Jul 16, 2026
Merged

Restructure example notebooks and add cross-table merge support#45
Jamestth merged 9 commits into
mainfrom
feat/cross-table-merge-annotated-output

Conversation

@Jamestth

@Jamestth Jamestth commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Three related changes to the examples and annotated-output feature:

1. Example notebook restructuring

The monolithic examples/vowl_usage_patterns_demo.ipynb (91 cells, ~357 KB) grew unwieldy. Split it into four focused, self-contained notebooks, each in its own folder:

Folder Notebook Covers
1_basic_tutorial/ basic_tutorial.ipynb Setup, running a validation (pandas/Polars), understanding results (ValidationResult, annotated output)
2_multiple_sources/ multiple_sources.ipynb Validating one contract across multiple sources
3_real_databases/ real_databases.ipynb Server-side validation with Testcontainers (Postgres/MySQL/Spark/DuckDB ATTACH)
4_advanced_usage/ advanced_usage.ipynb Explicitly defined adapters (incl. PooledAdapter) and filtering rows before validation
  • The tutorial is named the Basic Tutorial (1_basic_tutorial/basic_tutorial.ipynb). Its two denser sections — explicit adapters (incl. PooledAdapter) and row filtering — moved into the new 4_advanced_usage/ notebook so the basic tutorial stays focused on the everyday workflow.
  • Each notebook resolves the shared dataset paths on its own (repo-root walk-up) and imports what it needs, so it runs top-to-bottom independently.
  • save() calls write to a per-notebook local outputs/ folder; pre-generated reference artifacts moved there via git mv (history preserved). A clean delete-and-regenerate flushed one stale orphan CSV the current multi-source run no longer produces.
  • Section numbering and titles are scoped per notebook (each starts at 1); removed leftover global "Part N"/section numbers and duplicate banners.
  • Fixed relative doc links (../docs../../docs now that notebooks sit one level deeper) and repaired 3 corrupted source cells from the original notebook.
  • Replaced non-ASCII typographic symbols (em-dashes, arrows, ellipses) with ASCII equivalents in the two new/restructured notebooks.
  • Updated links in README.md, docs/usage-patterns.md, and rewrote examples/README.md.

2. Cross-table merge for annotated output

A subquery-projected referential check now merges onto its anchor table's check_info column instead of becoming a residue (src/vowl/validation/result.py), since its failed-rows query projects only the anchor table's columns. Bare-JOIN referential checks still correctly become residues. Includes expanded tests and expected outputs, and a clarifying docs/known-issues.md update. The basic tutorial links out to the known-issues explanation rather than duplicating the mechanics.

3. Deprecate consolidated failed-rows output

get_consolidated_output_dfs() and the output_mode="failed_rows" / "both" save modes (the legacy consolidated failed-rows CSVs) are now deprecated in favour of annotated output (get_annotated_output() / output_mode="annotated").

  • get_consolidated_output_dfs() emits a DeprecationWarning and delegates to a private _get_consolidated_output_dfs(), so internal callers (save() in failed_rows/both mode) reuse the grouping without warning.
  • save() warns on the deprecated paths: the implicit default (which will flip to "annotated" in a future minor release), explicit "failed_rows", and the failed-rows half of "both". "annotated" stays silent. All warnings use stacklevel=2 so they point at the caller.
  • ValidationConfig.output_mode docstring notes the upcoming default change. No behaviour changes yet — existing calls still work, they just warn.
  • Removed the "Consolidated Failed Rows (per table)" section from the basic tutorial and the get_consolidated_output_dfs() reference from README.md, since annotated output is the recommended per-table view. docs/getting-started.md and docs/known-issues.md keep their deprecation notices for users still on the legacy accessor.
  • CHANGELOG.md gains a Deprecated entry.

Test plan

  • pytest tests/test_annotated_output.py tests/test_check_reference_variations.py tests/test_usage_patterns.py tests/test_readme_examples.py237 passed, 17 skipped (skips are Docker/Java-gated)
  • Deprecation paths covered: tests assert the DeprecationWarning fires for each deprecated entry point and that the private _get_consolidated_output_dfs() helper stays silent
  • Executed basic_tutorial.ipynb, advanced_usage.ipynb, and multiple_sources.ipynb end-to-end via nbconvert — no error cells, contiguous execution counts; outputs regenerate into the correct outputs/ folders
  • Verified all in-notebook anchor links and cross-notebook/doc references resolve; no stale references
  • Lint green (ruff check, ruff format --check, prettier --check "**/*.md")
  • real_databases.ipynb requires Docker (not run in this environment)

🤖 Generated with Claude Code

Split the monolithic examples/vowl_usage_patterns_demo.ipynb (91 cells) into
three focused, self-contained notebooks under their own folders:

- 1_core_tutorial/      setup, running a validation, understanding results
- 2_multiple_sources/   validating one contract across multiple sources
- 3_real_databases/     server-side validation with Testcontainers

Each notebook resolves the shared dataset paths on its own (repo-root walk-up),
imports what it needs, and writes generated artifacts to a local outputs/ folder.
Section numbering/titles are scoped per notebook, and doc links in README.md and
docs/usage-patterns.md are updated to the new paths.

Also lands the cross-table-merge annotated-output work: a subquery-projected
referential check now merges onto its anchor table instead of becoming a residue
(src/vowl/validation/result.py), with expanded tests and expected outputs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Jamestth
Jamestth requested a review from a team July 9, 2026 10:13
Jamestth and others added 7 commits July 15, 2026 17:07
Apply `prettier --write` to the three files the lint CI job flagged:
table column-padding and `*emphasis*` → `_emphasis_` normalization.
No content changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mark get_consolidated_output_dfs() and output_mode="failed_rows"/"both"
as deprecated, steering users to get_annotated_output() / output_mode="annotated".

- get_consolidated_output_dfs() now emits a DeprecationWarning and delegates
  to a private _get_consolidated_output_dfs() so internal callers (save() in
  failed_rows/both mode) reuse the grouping without warning.
- save() warns on the deprecated paths: implicit default (upcoming flip to
  "annotated"), explicit "failed_rows", and the failed-rows half of "both".
  "annotated" stays silent. All warnings use stacklevel=2.
- ValidationConfig.output_mode docstring notes the default will change.
- Tests assert the warnings fire and that the private helper stays silent;
  internal/golden callers switched to the private helper.
- CHANGELOG Deprecated entry; README/getting-started/known-issues updated.
- Notebook TOC anchor-link fixes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename 1_core_tutorial/core_tutorial.ipynb to
1_basic_tutorial/basic_tutorial.ipynb and move the two denser sections
(Explicitly Defined Adapter incl. PooledAdapter, and Filtering Rows
Before Validation) into a new 4_advanced_usage/advanced_usage.ipynb so
the basic tutorial stays focused on the everyday workflow.

- Renumber the basic tutorial sections and fix its Contents/anchors
- Link the "cross-table check that merges" note out to known-issues
  instead of duplicating the mechanics
- Update cross-references in README, docs, examples/README, and the
  multiple-sources / real-databases notebooks
- Replace non-ASCII typographic symbols (em-dashes, arrows, ellipses)
  with ASCII equivalents in both notebooks
- Re-execute both notebooks top-to-bottom so outputs and execution
  counts are consistent

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Clarify which non-mergeable checks produce residues vs. appear only in
summary.json, and replace em-dash asides with plainer punctuation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Delete and regenerate the outputs/ artifacts for the basic tutorial and
multiple-sources notebooks so nothing is stale. Re-executing both
notebooks top-to-bottom also removes an orphaned join-output CSV that
the current multi-source run no longer produces.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remove the "Consolidated Failed Rows (per table)" section, which
demonstrates the deprecated get_consolidated_output_dfs() accessor.
The annotated output is now the recommended per-table view. Re-execute
the notebook so execution counts stay contiguous.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remove the get_consolidated_output_dfs() row from the ValidationResult
method table and trim it from the save() deprecation note, matching its
removal from the basic tutorial. The docs (getting-started, known-issues)
keep their deprecation notices for users still on the legacy accessor.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@zendenleong zendenleong left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@Jamestth
Jamestth merged commit e539856 into main Jul 16, 2026
10 checks passed
@Jamestth
Jamestth deleted the feat/cross-table-merge-annotated-output branch July 16, 2026 13:57
@Jamestth Jamestth mentioned this pull request Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants