Skip to content

openjd-conformance.md

Latest commit

 

History

History
147 lines (123 loc) · 8.66 KB

openjd-conformance.md

File metadata and controls

147 lines (123 loc) · 8.66 KB
Raw

OpenJD Spec Conformance

This page states, in one place, exactly how far sqi's OpenJD support goes: which spec version it implements, where that's enforced in code, which extensions it understands, and — just as importantly — what it deliberately does not implement and why that is correct rather than a gap. Read this before re-auditing OpenJD conformance from scratch.

Spec version

sqi implements jobtemplate-2023-09 — the OpenJD job template schema. A template's specificationVersion field must equal that string exactly; anything else (missing, a different version string) is a 422 validation error.

sqi does not implement the standalone environment-2023-09 template type — a second, separate top-level document format in the OpenJD spec for defining reusable environments outside a job template. sqi only ever parses jobtemplate-2023-09 documents; environments are supported only as they appear embedded in a job template (jobEnvironments, stepEnvironments). This is a missing feature, not a violated rule — nothing in the spec requires an implementation to support both template types.

Where validation lives

The pipeline is parse, then validate, then expand:

  1. internal/openjd/parse.go (Parse) — decodes YAML or JSON into a *JobTemplate (internal/openjd/model.go). Parsing is strict about shape: a field that must be a scalar but arrives as a mapping or sequence is a parse error, not a silently-wrong value.
  2. internal/openjd/validate.go (Validate, ValidateWithOptions) — walks the parsed template and returns zero or more ValidationErrors, each a JSON Pointer (RFC 6901) to the offending field plus a message. Validate(t) is ValidateWithOptions(t, ValidateOptions{EnforceLimits: true}); the submission pipeline can run with EnforceLimits: false (quantitative caps such as range size relaxed) while every structural correctness check — required fields, dependency resolution, extension gating, host-requirement shape — still runs unconditionally. A structural check that only fired under EnforceLimits: true would silently vanish for any caller that flips the flag; that class of bug is exactly what Task 4 of this cycle closed.
  3. internal/openjd/expand.go — turns a validated template's parameter spaces into concrete tasks.

A 422 Unprocessable Entity from POST /api/v1/jobs carries a detail string built from ValidationErrors.Error() — one or more <pointer>: <message> entries joined with ; . See docs/openjd-submission.md for a real captured example.

Supported extensions and the vendor-prefix rule

OpenJD extensions are opt-in: a template lists the ones it needs in its top-level extensions: [...] array. sqi validates that array unconditionally (not gated by EnforceLimits) against a fixed registry (internal/openjd/extension.go, LookupExtension):

Extension Origin What it does
TASK_CHUNKING official Chunked integer task parameters (CHUNK[INT]). See docs/openjd-extensions/task-chunking.md.
REDACTED_ENV_VARS official The openjd_redacted_env stdout directive that redacts a variable's value from logs. See docs/openjd-extensions/redacted-env-vars.md.
SQI_PATH_TRANSLATION vendor Per-product path-delivery checklist (swap_in_place/translation_file/command_flags/environment/stage_locally). See docs/openjd-extensions/path-translation.md.
SQI_CHUNK_BOUNDS vendor Exposes a CHUNK[INT] chunk's first/last integer as Task.Param.<name>.Start/.End. See docs/openjd-extensions/sqi-chunk-bounds.md.

Every vendor extension (Origin: OriginVendor — one sqi defines itself, as opposed to one specified upstream by OpenJD) name must carry the SQI_ prefix. This is enforced by a test invariant (internal/openjd/extension_test.go), not just convention: it guarantees a vendor extension name can never collide with a future official OpenJD name, which are always bare identifiers (e.g. TASK_CHUNKING, not SQI_TASK_CHUNKING). If a vendor extension is later upstreamed into the OpenJD spec, the promotion path is to drop the SQI_ prefix and flip its Origin to OriginOfficial — the registry entry moves, it doesn't get a second copy.

Declaring an extension name not present in the registry is a 422 at /extensions/{i}, regardless of EnforceLimits. Declaring CHUNK[INT] without TASK_CHUNKING in extensions is likewise rejected — the extension gate and the feature it gates are checked together.

What sqi deliberately does not implement

Two things, both out of scope by design, not bugs to fix:

  • EXPR (the OpenJD Expression Language extension) — new value types (BOOL, RANGE_EXPR, LIST[...]), let on step templates, and the *_LIST userInterface.control variants all belong to EXPR and are not implemented. A template that declares extensions: [EXPR] is rejected with a 422 at /extensions/0 — unconditionally, the same as any other unregistered extension name.
  • Standalone environment-2023-09 templates — see Spec version above.

Why rejecting an unimplemented opt-in extension is correct, not a gap: OpenJD extensions exist precisely so a template can declare "I need this capability" and a conformant implementation that lacks it can say "then I can't run you" instead of guessing. Accepting a template whose syntax sqi cannot interpret — silently ignoring the parts it doesn't understand — is strictly worse than refusing it outright: the alternative is a job that appears to submit successfully and then does something other than what the template author asked for, discovered only when the render is wrong. The spec does not mandate rejection of an unimplemented extension either way; sqi chooses to reject because a loud 422 at submission time is a better failure mode than a silent misinterpretation at run time.

This is also why the README.md/ROADMAP.md claim that jobs authored for other OpenJD-compatible tools work with sqi "without reformatting" carries an explicit caveat rather than being dropped: the claim is true for the base spec and every extension sqi implements, and false only for the extensions listed above — which a template must opt into by name to hit.

Adding a deliberate divergence

If sqi ever needs to diverge from strict spec conformance on purpose — the concrete example on the table right now is restoring something like the non-standard CHIP_INPUT parameter control that Task 9 of this cycle removed — the route is the extension mechanism above, not a silent parser change:

  1. A registry entry in internal/openjd/extension.go naming the divergence, e.g. SQI_UI_CONTROLS, with Origin: OriginVendor (so the SQI_ prefix is mandatory and enforced by the test invariant).
  2. Parse/validate support in internal/openjd gated on that extension being declared — the divergent behavior only activates for a template that opts in by name.
  3. A doc under docs/openjd-extensions/, following the shape of the existing entries (motivation, schema, validation, worker behavior).
  4. Any worker-side behavior the divergence needs, under internal/worker/.

SQI_UI_CONTROLS is not built — nothing declares it, nothing depends on it. It is recorded here only as the shape a future divergence would take: opt-in, named, registered, documented — never a bare change to what the base spec's syntax means.

Verifying documentation examples against the real parser

Every OpenJD YAML/JSON example in docs/openjd-submission.md is verified by actually parsing and validating it — via openjd.Parse + openjd.Validate — not by inspection. internal/openjd's existing TestParse* suite exercises the parser broadly; when correcting a specific documented example, the fastest way to confirm it is a throwaway _test.go file in internal/openjd that parses the exact YAML/JSON block verbatim and asserts zero validation errors, run once, then deleted before committing — it is not meant to become a permanent regression test. Two real bugs surfaced this way during this conformance cycle that a read-through would have missed: the step-level dependency key is dependencies: (a list of {dependsOn: <Name>}), not a bare dependsOn: list of {stepName: <Name>} as earlier documentation showed; and an explicit-but-empty hostRequirements: {} is now rejected (only omitting the key reserves the whole machine) now that host-requirement structural checks run unconditionally.