Skip to content

Commit 45df3b3

Browse files
committed
feat!: Rust-backed openjd.model._v1, openjd.expr, openjd.sessions._v1 via PyO3
Introduces a Rust-backed implementation of the OpenJD data model, expression language, and session runtime types, distributed alongside the existing pure-Python reference. Built as a single PyO3 extension module — `openjd._openjd_rs` — that wraps three Rust crates from the `openjd-rs` workspace (`openjd-expr`, `openjd-model`, `openjd-sessions`) and is consumed via three Python import roots: * `openjd.expr` — Rust-only. No pure-Python predecessor. * `openjd.model._v1` — Rust-backed v1, parallel to the existing Pydantic-based `openjd.model` (a.k.a. `openjd.model.v0`). * `openjd.sessions._v1` — Rust-backed v1, with the Python wrapper module living in the sibling `openjd-sessions-for-python` repository. The Python distribution still ships under the single PyPI package name `openjd-model`; no consumer of the v0 API is forced to migrate. v1 is opt-in via the explicit `_v1` import path. ``` rust-bindings/ (PyO3 crate, name=openjd-python, lib=_openjd_rs) ├── src/lib.rs #[pymodule] _openjd_rs registration ├── src/expr/ ──→ openjd.expr wraps openjd-rs::openjd-expr ├── src/model/ ──→ openjd.model._v1 wraps openjd-rs::openjd-model ├── src/sessions/ ──→ openjd.sessions._v1 (re-exported from │ openjd-sessions-for-python, │ wraps openjd-rs::openjd-sessions) └── src/bin/stub_gen.rs pyo3-stub-gen entry point ``` * The crate is named `openjd-python`; the cdylib is `_openjd_rs`. * Every `#[pyclass]` has a `Py`-prefixed Rust identifier (e.g. `PyJob`) and is registered under its public Python name via `#[pyo3(name = "...")]`. * Exception classes raised through `create_exception!` are registered via a `register_renamed_exception` helper so `__name__` / `__module__` / `__qualname__` resolve to canonical user-facing values across pickle, repr, and tracebacks. * Build infrastructure: an in-tree PEP 517 backend (`_build_backend.py`), a `maturin develop` wrapper that injects a VCS-derived version (`scripts/maturin_build.py`), and a `pyo3-stub-gen` driver (`scripts/generate_stubs.sh`) that emits `src/openjd/_openjd_rs.pyi`. * Python: 3.9+ via `abi3-py39` (set in `rust-bindings/Cargo.toml` and enforced by `pyproject.toml` `requires-python`). 23 public symbols re-exported from `src/openjd/expr/__init__.py`: * Entry points — `evaluate_expression`, `parse_expression`, `escape_format_string`. * Value/type system — `ExprType`, `TypeCode`, `ExprValue`, `RangeExpr`, `IntRange`, `FormatString`, `SymbolTable`, `ParsedExpression`, `EvalResult`. * Profile types — `ExprProfile`, `ExprRevision`, `ExprExtension`, `HostContext`. The previous `FunctionLibrary` / `FunctionSignature` / `get_default_library` surface was replaced by these profile types; the full migration story is documented in `specs/python-expr-interface.md`. * Path types — `PathFormat`, `PathMappingRule`. * Errors — `ExpressionError`, `ExpressionTypeError`, `RangeExprError`, `FormatStringValidationError`. * Constants — `DEFAULT_MEMORY_LIMIT`, `DEFAULT_OPERATION_LIMIT`. `ParsedExpression.evaluate(...)` returns an `ExprValue` (the lighter form, no metric tracking); `ParsedExpression.evaluate_with_metrics(...)` returns an `EvalResult` value-class bundling `value` / `peak_memory` / `operation_count`, mirroring the upstream Rust `EvalResult` struct. Top-level entry points: `decode_job_template{,_str}`, `decode_environment_template{,_str}`, `decode_template`, `create_job`, `preprocess_job_parameters`, `merge_job_parameter_definitions`, `evaluate_let_bindings`. Output and type pyclasses are organised under three submodules: * `.template` — template-time pyclasses returned by `decode_*_template`: `JobTemplate`, `EnvironmentTemplate`, `StepTemplate`, the typed `Job*ParameterDefinition` / `*TaskParameterDefinition` / `*UserInterface` variants, and the template-time structural types (`TemplateAction`, `TemplateEmbeddedFile`, `TemplateHostRequirements`, …) plus their unprefixed short-alias re-exports. * `.job` — job-time pyclasses returned by `create_job`: `Job`, `Step`, `StepScript`, `StepActions`, `Action`, `Environment`, `EmbeddedFile`, `JobParameter`, `HostRequirements`, `AmountRequirement`, `AttributeRequirement`, `StepParameterSpace`, `StepParameterSpaceIterator`, `StepDependencyGraph`, and the typed task-parameter pyclasses. * `.types` — cross-cutting types: `JobParameterType`, `TaskParameterType`, `DocumentType`, `ModelProfile`, `ModelExtension`, `SpecificationRevision`, `CallerLimits`, `ValidationContext`. * `.errors` — `DecodeValidationError`, `ModelValidationError`, `UnsupportedSchema`. Binding source lives in this repository under `rust-bindings/src/sessions/`, but the user-facing wrapper module ships from a parallel branch in the `openjd-sessions-for-python` repository. The two repos must change together when the binding API changes. * `specs/python-expr-interface.md`, `specs/python-model-interface.md`, `specs/python-sessions-interface.md` are the contract for each binding component — the equivalent of `public-api.md` in `openjd-rs`. * `test/openjd/expr/` and `test/openjd/model_v1/` carry component test suites covering parse/evaluate, pickle, equality / hashability, parity probes against the v0 reference, fuzz tests, and end-to-end decode/create paths. Total: 5121 passing tests, 24 platform-skipped (Windows-only), 0 xfailed, coverage 94.30%. * `skills/eval-bindings/SKILL.md` defines the report-driven evaluation workflow used to identify and resolve binding/reference parity gaps. * `reports/` holds quality-evaluation reports for each component; the current state of each report has every numbered recommendation resolved. `rust-bindings/Cargo.toml` consumes the three Rust workspace crates from crates.io at pinned versions (`openjd-expr = "0.1.1"`, `openjd-model = "0.2.0"`, `openjd-sessions = "0.2.2"`). A commented-out `[patch.crates-io]` block at the bottom of that file documents how to redirect to a sibling `~/openjd-rs` checkout when iterating across both repos. CI does not require a sibling clone. * `cargo build --all-targets` clean * `cargo clippy --all-targets -- -D warnings` clean (hard gate in CI) * `cargo test --doc` clean * `hatch run lint` clean (ruff + black + mypy on every supported Python version) * `hatch run test` 5121 passed, 24 skipped (Windows-only), 0 xfailed, coverage 94.30% — the project-wide 94% coverage gate is enforced by the test runner. * The `.github/workflows/rust_quality.yml` workflow runs build / clippy / test / doctest on `{ubuntu, windows, macos}`. BREAKING CHANGE: This commit introduces a new `_v1` API surface behind explicit `openjd.model._v1` / `openjd.sessions._v1` imports and a brand-new `openjd.expr` import root. The existing v0 (`openjd.model` / `openjd.model.v0`, Pydantic-backed) public API is untouched and remains the default. v1 itself contains many breaking changes relative to v0 — `JobParameter.type` returns the `JobParameterType` enum rather than a string, `Action.timeout` returns `Optional[FormatString]` rather than `Optional[str]`, `PathFormat` and `JobParameterType` no longer carry a `str` mixin, and the function-library surface has been replaced by `ExprProfile` / `HostContext`, among others. Each divergence is documented in the relevant `specs/python-*-interface.md` file. The `openjd.sessions._v1` surface is published from the sibling `openjd-sessions-for-python` repository. Signed-off-by: Mark <399551+mwiebe@users.noreply.github.com>
1 parent 94129dc commit 45df3b3

186 files changed

Lines changed: 42864 additions & 213 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/scripts/get_latest_changelog.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
3030
```
3131
"""
32+
3233
import re
3334

3435
h2 = r"^##\s.*$"

.github/workflows/rust_quality.yml

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
name: Rust Quality
2+
3+
# Build, lint, and test the Rust bindings crate at rust-bindings/.
4+
#
5+
# The existing Python-side code_quality workflow covers Python tests and
6+
# transitively builds the Rust extension via `maturin develop`, but does
7+
# not run `cargo test` (which covers doctests and any pure-Rust unit
8+
# tests added to the bindings crate).
9+
#
10+
# The bindings' `openjd-*` dependencies come from crates.io at the
11+
# versions pinned in `rust-bindings/Cargo.toml`. There is no sibling
12+
# repository requirement — local-development overrides are documented
13+
# in a commented-out `[patch.crates-io]` block at the bottom of that
14+
# file.
15+
16+
on:
17+
pull_request:
18+
branches: [ mainline, release, 'patch_*' ]
19+
workflow_call:
20+
inputs:
21+
branch:
22+
required: false
23+
type: string
24+
tag:
25+
required: false
26+
type: string
27+
28+
jobs:
29+
rust_quality:
30+
name: Rust (${{ matrix.os }})
31+
strategy:
32+
fail-fast: false
33+
matrix:
34+
os: [ubuntu-latest, windows-latest, macos-latest]
35+
runs-on: ${{ matrix.os }}
36+
permissions:
37+
contents: read
38+
39+
steps:
40+
- name: Checkout openjd-model-for-python
41+
uses: actions/checkout@v4
42+
with:
43+
ref: ${{ inputs.branch || inputs.tag || github.ref }}
44+
45+
- name: Install Rust toolchain
46+
uses: dtolnay/rust-toolchain@stable
47+
with:
48+
components: clippy, rustfmt
49+
50+
- name: Cache Cargo registry and build artifacts
51+
uses: actions/cache@v4
52+
with:
53+
path: |
54+
~/.cargo/registry
55+
~/.cargo/git
56+
target
57+
key: cargo-${{ matrix.os }}-${{ hashFiles('Cargo.lock', 'rust-bindings/Cargo.toml') }}
58+
restore-keys: |
59+
cargo-${{ matrix.os }}-
60+
61+
- name: cargo fmt --check
62+
run: cargo fmt --manifest-path rust-bindings/Cargo.toml --check
63+
64+
- name: cargo build
65+
run: cargo build --manifest-path rust-bindings/Cargo.toml --all-targets
66+
67+
- name: cargo clippy
68+
run: cargo clippy --manifest-path rust-bindings/Cargo.toml --all-targets -- -D warnings
69+
70+
- name: cargo test
71+
run: cargo test --manifest-path rust-bindings/Cargo.toml
72+
73+
- name: cargo test --doc
74+
# Runs separately from `cargo test` above so a docstring regression
75+
# (e.g. a Python example rustdoc tries to compile as Rust) is
76+
# clearly visible in the CI log.
77+
run: cargo test --manifest-path rust-bindings/Cargo.toml --doc

.gitignore

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
*~
22
*#
33
*.swp
4+
*.pdb
45

56
*.DS_Store
67

@@ -22,3 +23,17 @@ __pycache__/
2223
/build
2324
/dist
2425
_version.py
26+
27+
# scripts/maturin_build.py backup file
28+
/pyproject.toml.vcs-bak
29+
30+
# Rust build artifacts
31+
/target/
32+
/rust-bindings/target/
33+
Cargo.lock
34+
35+
# Compiled extension modules
36+
*.so
37+
*.pyd
38+
*.dylib
39+
.hypothesis/

AGENTS.md

Lines changed: 292 additions & 0 deletions
Large diffs are not rendered by default.

Cargo.toml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[workspace]
2+
members = ["rust-bindings"]
3+
resolver = "2"

_build_backend.py

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
"""
3+
In-tree PEP 517 build backend that wraps `maturin` to inject a
4+
VCS-derived version.
5+
6+
Why this exists: maturin reads the wheel version from
7+
`[project].version` in pyproject.toml (or `[package].version` in
8+
Cargo.toml when `dynamic = ["version"]` is set). It has no built-in
9+
mechanism to derive the version from git, the way hatchling+hatch-vcs
10+
does. This wrapper:
11+
12+
1. Computes the same VCS version as `scripts/generate_version.py`
13+
(via setuptools_scm with the same config as `[tool.hatch.version]`).
14+
2. Writes `src/openjd/model/_version.py` so the in-Python
15+
`__version__` matches the wheel.
16+
3. Patches pyproject.toml in place — replaces `dynamic = ["version"]`
17+
with a static `version = "<v>"`.
18+
4. Delegates to maturin's PEP 517 hooks.
19+
5. Restores pyproject.toml in `finally`.
20+
21+
Wired in via:
22+
[build-system]
23+
requires = ["maturin>=1.0,<2.0", "setuptools_scm"]
24+
build-backend = "_build_backend"
25+
backend-path = ["."]
26+
27+
The companion script `scripts/maturin_build.py` does the same patching
28+
around `maturin develop` / `maturin build` for users who prefer those
29+
direct commands. Either path produces wheels with the VCS version.
30+
"""
31+
32+
from __future__ import annotations
33+
34+
import contextlib
35+
import shutil
36+
import sys
37+
from pathlib import Path
38+
39+
import maturin
40+
41+
REPO_ROOT = Path(__file__).resolve().parent
42+
PYPROJECT = REPO_ROOT / "pyproject.toml"
43+
PYPROJECT_BAK = REPO_ROOT / "pyproject.toml.vcs-bak"
44+
DYNAMIC_NEEDLE = 'dynamic = ["version"]'
45+
46+
sys.path.insert(0, str(REPO_ROOT / "scripts"))
47+
from generate_version import compute_version, write_version_file # noqa: E402
48+
49+
50+
def _recover_orphan_bak() -> None:
51+
"""If a previous build died before restoring pyproject.toml, recover."""
52+
if PYPROJECT_BAK.exists():
53+
sys.stderr.write(
54+
f"WARN: {PYPROJECT_BAK.name} found from a prior interrupted build; restoring.\n"
55+
)
56+
shutil.move(str(PYPROJECT_BAK), str(PYPROJECT))
57+
58+
59+
@contextlib.contextmanager
60+
def _patched_pyproject():
61+
"""
62+
Patch [project] to have a static VCS version. Restore on exit.
63+
64+
If the version cannot be computed (e.g. building from an sdist with
65+
no .git directory), yield without patching — maturin will fall back
66+
to the static [package].version in Cargo.toml, same as today.
67+
"""
68+
_recover_orphan_bak()
69+
70+
try:
71+
version = compute_version()
72+
except SystemExit:
73+
sys.stderr.write(
74+
"WARN: setuptools_scm could not derive a version (no git? no setuptools_scm?). "
75+
"Falling back to static version from Cargo.toml.\n"
76+
)
77+
yield None
78+
return
79+
80+
write_version_file(version)
81+
original = PYPROJECT.read_text(encoding="utf-8")
82+
if DYNAMIC_NEEDLE not in original:
83+
sys.stderr.write(
84+
f"WARN: expected {DYNAMIC_NEEDLE!r} in pyproject.toml but did not find it; "
85+
"skipping version patch.\n"
86+
)
87+
yield version
88+
return
89+
90+
shutil.copy2(PYPROJECT, PYPROJECT_BAK)
91+
try:
92+
patched = original.replace(DYNAMIC_NEEDLE, f'version = "{version}"', 1)
93+
PYPROJECT.write_text(patched, encoding="utf-8")
94+
yield version
95+
finally:
96+
shutil.move(str(PYPROJECT_BAK), str(PYPROJECT))
97+
98+
99+
def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
100+
with _patched_pyproject():
101+
return maturin.build_wheel(wheel_directory, config_settings, metadata_directory)
102+
103+
104+
def build_editable(wheel_directory, config_settings=None, metadata_directory=None):
105+
with _patched_pyproject():
106+
return maturin.build_editable(wheel_directory, config_settings, metadata_directory)
107+
108+
109+
def build_sdist(sdist_directory, config_settings=None):
110+
with _patched_pyproject():
111+
return maturin.build_sdist(sdist_directory, config_settings)
112+
113+
114+
def prepare_metadata_for_build_wheel(metadata_directory, config_settings=None):
115+
with _patched_pyproject():
116+
return maturin.prepare_metadata_for_build_wheel(metadata_directory, config_settings)
117+
118+
119+
prepare_metadata_for_build_editable = prepare_metadata_for_build_wheel
120+
121+
# Pass-through hooks: no version patching needed for requirement queries.
122+
get_requires_for_build_wheel = maturin.get_requires_for_build_wheel
123+
get_requires_for_build_editable = maturin.get_requires_for_build_editable
124+
get_requires_for_build_sdist = maturin.get_requires_for_build_sdist

hatch.toml

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,21 @@
11
[envs.default]
22
pre-install-commands = [
3+
"pip install maturin setuptools_scm",
4+
"python scripts/maturin_build.py develop --manifest-path rust-bindings/Cargo.toml",
35
"pip install -r requirements-testing.txt"
46
]
57

68
[envs.default.scripts]
79
sync = "pip install -r requirements-testing.txt"
8-
test = "pytest test/ --cov-config pyproject.toml --ignore=test/openjd/model/benchmark {args}"
9-
benchmark = "pytest test/openjd/model/benchmark --no-cov {args}"
10+
# `test` runs the canonical full test suite with the 94% coverage gate
11+
# enforced. CI invokes this. For ad-hoc subset runs during local
12+
# development (e.g. `hatch run test-subset test/openjd/expr`), use
13+
# `test-subset`, which collects coverage but does not enforce the gate
14+
# (the gate only makes sense against a full-suite run that touches every
15+
# coverable source tree).
16+
test = "pytest test/ --cov-config pyproject.toml --cov-fail-under=94 --ignore=test/openjd/model_v0/benchmark --ignore=test/openjd/model_v1/benchmark {args}"
17+
test-subset = "pytest --cov-config pyproject.toml --ignore=test/openjd/model_v0/benchmark --ignore=test/openjd/model_v1/benchmark {args}"
18+
benchmark = "pytest test/openjd/model_v0/benchmark test/openjd/model_v1/benchmark --no-cov {args}"
1019
typing = "mypy {args:src test}"
1120
style = [
1221
"ruff check {args:src test}",

0 commit comments

Comments
 (0)