M8 — strategy recommender, bundle contract, end-to-end orchestrator - #12
Merged
Merged
Conversation
…hestrator
The bridge from the data layer (L1) to the agentic layers above it.
Strategy recommender (deterministic, no LLM — FR-018): derives signals from the
bundle (schema size, detected patterns, free-text density, relationship density,
KPI count, review coverage) and recommends one of plain_schema / vector /
knowledge_graph, with confidence, reasoning, and the alternatives it rejected.
Coverage-aware confidence reduces the score and emits a warning when review
coverage is low (FR-023). SME override is recorded alongside the recommendation
and preserved across re-runs (provenance flips to sme-authored).
- recommender/{signals,strategies,engine,override}.py
- RecommendationPayload (E12) + Strategy/StrategyConfidence enums; reader dispatch
- cli: `dla recommend` (--explain, --override, --reason)
- web: /recommender page + override; nav "Strategy"
Bundle contract: `dla bundle export-schema` publishes bundle-schema.json straight
from the pydantic models (config/schemas/), and `dla bundle validate` confirms
every artifact round-trips its model and runs completeness checks — a quality and
security gate (exit 4 on errors). docs/bundle-contract.md documents the layout,
common fields, provenance state machine, and the recommendation.
End-to-end orchestrator: `dla run` sequences discover → profile → readiness →
[describe → glossary when --llm] → patterns → recommend → validate. Every step is
idempotent and resumable (--from-step / --resume / --skip-step); run state is
persisted to bundle/.run_state.json; --stop-on-readiness-critical halts before
describe on critical data-quality issues.
Tests: recommender strategy-choice + determinism + override + coverage; contract
export + validation (malformed, kpi-missing-table, missing-manifest); orchestrator
planning + offline pipeline + resume + readiness-stop; web recommender + override.
144 apps/dla unit + 29 integration green; ruff + libs mypy clean.
…ical CI's ruff flagged I001 on test_bundle_contract.py where a straight `import json` preceded an alphabetically-earlier `from` import. Replacing the straight `import json` with `from json import ...` makes the stdlib block unambiguous under any isort ordering. Same change applied to validate.py and state.py to pre-empt the identical pattern.
…g the package The `bundle/` output-dir ignore rule also matched apps/dla/src/dla/bundle/, so the new contract.py and validate.py were silently untracked — missing from the repo, they broke CI (ruff could not resolve `dla.bundle.contract` as first-party, splitting the import block → I001; imports would also fail at runtime). Add a negation so the source package is always tracked, and commit the two files.
…used import, noqa) Now that contract.py and validate.py are tracked, ruff lints them for the first time: use PEP 604 `|` unions in the discriminated artifact union (UP007), drop an unused import, and remove two non-enabled BLE001 noqa directives (RUF100). Discriminated union + schema export verified unchanged.
…run, durations); docs Adds the milestone-gate coverage and docs M8 was meant to close, so the strategy recommender + bundle contract + orchestrator meet their own Definition of Done. Tests: - SC-006 / T178: recommender choice eval over 10 hand-labeled fixtures (>=8/10) + cross-fixture determinism (tests/unit, CI-gated — the recommender is deterministic, no LLM). - T182 / T183: contract-parity + schema-version pin — committed bundle-schema.json must match the models; manifest schema_version == published version; every ArtifactType has a payload or is the documented computed-only coverage_record. - T189 / T190: orchestrator resume-after-failure (completed steps not redone) and unchanged-rerun zero-diff (byte-identical, files untouched). Engine: - FR-025 / T192: per-step duration_ms in the orchestrator's structured log. - schema_version single-sourced as SCHEMA_VERSION (schema.py) = 1.0.0; manifest and published schema now pinned equal. - Exit codes reconciled to one canonical 0-7 map across cli/main.py, cli/run.py (validation 4->5, added 6 nothing-to-resume), cli/bundle.py (4->5). Docs: - apps/dla/README.md: M6-M8 flipped to Released; 8 commands + flags documented; bundle layout, project structure, and exit-code table refreshed; Snowflake claims corrected (Postgres+CSV only). - docs/operator-guide.md: pipeline, resumability matrix, exit codes, recovery. - docs/v1-deferred-scope.md: honest record of deferred/backlog/changed scope (Snowflake, SCD, description eval, INSUFFICIENT_SIGNAL, real-DB e2e, contract drifts).
…gation The earlier `!apps/dla/src/dla/bundle/**` negation was too broad and re-included __pycache__/*.pyc (8 files slipped into the previous commit). Narrow the negation to *.py and explicitly re-ignore __pycache__ under the package; untrack the .pyc.
…tion fix - Root Makefile wraps install/lint/typecheck/test/ci/run/ui/validate/schema/clean; exports PYTHONPATH so targets work locally (editable-install quirk) and in CI. - Root README: add a Developer tasks section + links to the app README, operator guide, and bundle contract. - Commit the .gitignore negation fix that was left unstaged in the prior commit (track dla.bundle *.py sources, keep __pycache__ ignored).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Final milestone. The bridge from the data layer (L1) to the agentic accelerators above it.
What's in it
Strategy recommender (deterministic — FR-018)
Derives signals from the built bundle (schema size, detected patterns, free-text density, relationship density, KPI count, review coverage) and recommends one of
plain_schema/vector/knowledge_graph— with confidence, plain-language reasoning, and the two alternatives it rejected. No LLM in the decision path, so the same bundle always yields the same recommendation.coverage_warning(FR-023).sme-authored).recommender/{signals,strategies,engine,override}.py;RecommendationPayload(E12) + enums.dla recommend(--explain,--override,--reason); web/recommenderpage + override; nav Strategy.Bundle contract (published + validated)
dla bundle export-schema→config/schemas/bundle-schema.json, generated straight from the pydantic models (can't drift).dla bundle validate→ every artifact must round-trip its model + completeness checks. Quality and security gate (exit 4 on errors; a malformed artifact never ships).docs/bundle-contract.md— layout, common fields, provenance state machine, recommendation.End-to-end orchestrator
dla runsequences discover → profile → readiness → [describe → glossary when--llm] → patterns → recommend → validate.--from-step,--resume,--skip-step; state inbundle/.run_state.json;--stop-on-readiness-criticalhalts before describe on critical DQ issues.Tests
Notes
plain_schema/vector/knowledge_graphis the hand-off signal to the layers above L1 (knowledge-graph accelerator, vector/semantic layer, assembler).recommender_*inThresholdsConfig) — recalibrate on real bundles without code changes.