-
Notifications
You must be signed in to change notification settings - Fork 140
fix(rust-core): close all Python/Rust parity gaps — SILICON audit + HYBRID/EMPIRICAL port #1355
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
tianhaox
wants to merge
11
commits into
ai-dynamo:main
Choose a base branch
from
tianhaox:audit/rust-python-parity
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
fffb681
test(rust-core): add HYBRID/EMPIRICAL parity cases + open the rust ro…
tianhaox 6b9f866
feat(rust-core): util-empirical core + database_mode/transfer_policy …
tianhaox c11b816
feat(rust-core): GEMM-family HYBRID/EMPIRICAL dispatch (reference port)
tianhaox 28582e7
feat(rust-core): communication-family HYBRID/EMPIRICAL dispatch
tianhaox ccf6489
feat(rust-core): HYBRID/EMPIRICAL dispatch for attention/MoE/MLA/DSA/…
tianhaox b72456f
feat(rust-core): MSA op (XOP transfer) + dispatch/wideep hybrid — hyb…
tianhaox 35aa053
fix(rust-core): close all SILICON-path Python/Rust parity gaps
tianhaox c1d91a1
Merge audit/rust-python-parity (#1355): SILICON gaps + HYBRID port = …
tianhaox 53f2a1a
fix(rust-core): close capability + selection-rule audit findings
tianhaox 6a9e260
chore: parity-freeze defenses — opspec coverage tripwire, rules doc, …
tianhaox bef4492
fix(rust-core): address PR #1355 review — mode-aware wideep dispatch …
tianhaox File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| --- | ||
| description: > | ||
| Parity discipline for the compiled engine: the frozen Python op/query math | ||
| is the spec; latency-affecting changes must land on both sides with an | ||
| oracle anchor in the same PR. | ||
| paths: | ||
| - "rust/aiconfigurator-core/**" | ||
| - "src/aiconfigurator/sdk/operations/**" | ||
| - "src/aiconfigurator/sdk/perf_database.py" | ||
| - "src/aiconfigurator/sdk/perf_interp/**" | ||
| - "src/aiconfigurator/sdk/engine.py" | ||
| - "src/aiconfigurator/sdk/rust_engine_step.py" | ||
| - "tests/unit/sdk/test_opspec_coverage.py" | ||
| --- | ||
|
|
||
| # Rust-Core Parity Discipline (Python-path freeze) | ||
|
|
||
| The compiled engine (`rust/aiconfigurator-core`) is at full numeric parity | ||
| with the Python engine step for the SILICON, HYBRID, and EMPIRICAL database | ||
| modes, guarded by `rust/aiconfigurator-core/parity_tests/` (engine-step, | ||
| compile-engine, and perf gates). The Python op/query math is **frozen**: it | ||
| is the reference the parity suite compares against, and it is scheduled for | ||
| retirement (see the Python-path freeze tracking issue). | ||
|
|
||
| ## The rule | ||
|
|
||
| Any PR that changes latency-affecting behavior in: | ||
|
|
||
| - `src/aiconfigurator/sdk/operations/**` (op query math, SOL formulas, | ||
| slice/kernel selection, transfer ladder), | ||
| - the query/loader layer of `src/aiconfigurator/sdk/perf_database.py`, | ||
| - `src/aiconfigurator/sdk/perf_interp/**`, | ||
|
|
||
| MUST in the same PR: | ||
|
|
||
| 1. **Mirror the change** in the corresponding `rust/aiconfigurator-core/src/` | ||
| operator/table (the layering matches: dispatch + estimators in | ||
| `operators/`, algorithm-free loaders/accessors in `perf_database/`). | ||
| 2. **Anchor it**: a Python-generated oracle in the Rust `#[cfg(test)]` module | ||
| (1e-9, `uv run python` against a `shared_layer=False` view — copy the | ||
| existing oracle-test pattern in `operators/gemm.rs`), and/or a case in | ||
| `parity_tests/test_engine_step_parity.py` when a new config class becomes | ||
| reachable. | ||
| 3. Keep the `rust-engine-step-parity` CI job green — it is the enforcement | ||
| mechanism, not this document. | ||
|
|
||
| ## Adding a new Operation | ||
|
|
||
| A new `Operation` subclass must get a `_to_opspec` branch in | ||
| `sdk/engine.py`, an `Op` variant in `operators/op.rs` (**append at the tail** | ||
| — bincode variant indices are positional; mid-enum insertion requires an | ||
| `ENGINE_SPEC_SCHEMA_VERSION` bump on BOTH sides), the `engine/spec.rs` | ||
| round-trip fixture, and a parity case. `tests/unit/sdk/test_opspec_coverage.py` | ||
| fails until the op converts or carries a justified `EXEMPT` entry. | ||
|
|
||
| ## Selection rules are parity surface too | ||
|
|
||
| Table/slice/kernel selection must match rule-for-rule, including fallback | ||
| order and tie-breaks. Python dicts iterate in file/insertion order; Rust | ||
| `BTreeMap` iterates sorted — any "first available" fallback needs a | ||
| load-order record on the Rust side (see `quants_in_load_order` / | ||
| `first_distribution` in `perf_database/{moe,wideep_moe}.rs` for the pattern). | ||
|
|
||
| ## Known intentional splits (do not "fix" without the tracking issue) | ||
|
|
||
| - SOL / SOL_FULL modes delegate to the Python step (routing gate in | ||
| `sdk/rust_engine_step.py`). | ||
| - AFD and the VL encoder phase are Python-side orchestration; their per-op | ||
| values move to Rust via the planned op-list evaluation FFI, not by porting | ||
| the orchestration. | ||
| - Rust reads parquet only (no `.txt` legacy loading) — new data drops must | ||
| ship parquet. | ||
| - Energy/power does not yet cross the FFI (rust-routed reports show 0.0W) | ||
| until the energy follow-up PR lands. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.