Skip to content

AGENTS.md

Latest commit

 

History

History
719 lines (665 loc) · 44.6 KB

AGENTS.md

File metadata and controls

719 lines (665 loc) · 44.6 KB
Raw

Codex Project Guide

This repository is Open64 with active work around very-high-level WHIRL, DSL metadata, tensor descriptors, and a future Python frontend ingestion path. Use this file as the compact default context. Pull the larger design documents only when the current task needs detail.

Instruction Intake And Completion Gate

  1. Before editing, identify all applicable instruction files in the actual checkout or worktree, including ancestor and nested AGENTS.md files and any other instructions required for the task. Read each applicable file in full, from beginning to end. Summaries, search hits, header excerpts, and arbitrary line prefixes are not substitutes for complete reading. If output is truncated, continue with bounded reads until every section is covered.
  2. Apply instruction scope and precedence explicitly. Do not assume another checkout has identical guidance. Before entering a new directory or changing task scope, check for additional applicable instructions and read them in full; reread applicable instruction files when they change.
  3. Before implementation, extract mandatory requirements into an explicit compliance checklist. Cover applicable naming, formatting, architectural and function-contract comments, ownership and phase boundaries, compatibility, dependencies, option controls, validation, documentation, and artifact retention requirements. Mark a requirement not applicable only with a reason.
  4. Before declaring a milestone complete, enumerate every new or modified file in the milestone, including untracked files, headers, helpers, tests, scripts, build files, and documentation. Check each file against all applicable items in the compliance checklist and record its result. Keep unrelated pre-existing edits outside the milestone rather than changing them to satisfy this review.
  5. Passing tests, builds, automated style checks, or git diff --check does not replace the file-by-file compliance review. Automated checks may assist, but manually verify that required comments and documentation describe the actual contracts, invariants, ownership, mutation, and failure behavior.
  6. Do not declare completion while any mandatory checklist item is failed or unchecked. Resolve omissions first; if a requirement cannot be satisfied, report partial completion, the blocker, and the precise affected files.

High-Level Direction

  1. Preserve existing Open64 and WHIRL compatibility unless a task explicitly requires a versioned IR change.
  2. Binary WHIRL file backward compatibility is a standing requirement. Do not change WHIRL binary image layout, ELF section contracts, opcode encodings, type-kind encodings, or reader/writer behavior unless the task explicitly calls for a versioned IR change with a documented migration path.
  3. Binary WHIRL uses Open64's mapped-image and ELF framework. The binary writer finalizes table and tree images into their ELF sections, and the reader maps those images back into the compiler address space. Do not describe or implement this as object-stream encoding/decoding or format conversion.
  4. Treat WHIRL.pdf as the normative architectural baseline for WHIRL semantics. Before changing DSL node representation, operators, opcode encoding, kid layout, types, mappings, printing, or mapped-image finalization, review the corresponding definitions and design objectives in WHIRL.pdf.
  5. Design DSL WHIRL as an additive extension of the existing WHIRL model. Preserve established rules such as strict tree structure, direct operand kids, operator/result/descriptor semantic completeness, continuous lowering, mapping-table annotations, and standard binary/ASCII inspection. Any intentional departure must be explicitly justified, versioned when necessary, and covered by compatibility tests.
  6. Maintain enough architectural detail for the DSL extensions to become an appendix to WHIRL.pdf. The appendix should distinguish normative semantic contracts from transitional carriers and historical ABI details, and map each extension to its operator definition, node layout, allowed WHIRL levels, type rules, verifier requirements, lowering path, binary format, and ir_b2a representation.
  7. Do not copy historical physical sizes or ABI assumptions from WHIRL.pdf without checking the current Open64 implementation. Use the document for semantic intent and verify concrete layouts against the current source, target ABI, binary reader, and binary writer.
  8. Treat Python as a source-language frontend and ingestion layer only. The Open64 middle end must consume a binary WHIRL artifact without depending on the Python interpreter.
  9. Keep WHIRL table construction, symbol/type creation, TensorDescriptorIR attachment, opcode attributes, contracts, compiler metadata, mapped image finalization, and binary IR compatibility in C++.
  10. Make ingestion emit first-class DSL/common operators or stable DSL markers, not early intrinsic-call placeholders, runtime calls, or target kernels.
  11. Run a gatekeeper/verifier before lowering so malformed very-high-level WHIRL does not silently reach downstream optimization passes.
  12. Do not treat the MPL frontend path as a Very High Level WHIRL ingestion path. MPL is incomplete reference code for porting another IR roughly equivalent to Middle Level WHIRL; it has no equivalent feature set for Very High Level WHIRL. mpl2whirl remains a valid reference for baseline Open64 symbol/type tables, scope and PU ownership, mapped-image finalization, and binary reader/writer mechanics unless a specific incompatibility is demonstrated and documented.
  13. Follow the coding guidelines below for every touched file.

Commit-Ready Milestones

  1. End each completed coding milestone with focused validation, git diff --check, and a review of the exact files to be committed. Commit coherent verified work promptly; do not leave finished milestones in a mixed dirty worktree.
  2. Stage explicit paths. Preserve unrelated or pre-existing edits, and split design changes from implementation changes when that makes review clearer.
  3. Keep generated validation artifacts under their designated artifact directories with receipts and checksums. Respect .gitignore unless a milestone explicitly requires tracking those artifacts. Never claim GPU correctness or performance from compiler-only or host-only evidence.
  4. If a milestone cannot be committed, state the blocker and the precise uncommitted files before moving on to another coding milestone.

WHIRL Single Source Of Truth For CUDA

  1. Treat the selected, lowered WHIRL image as the source of truth for generated CUDA device code. Do not independently implement an optimization in a CUDA source template or infer a missing decision inside whirl2c -cuda.
  2. Capture required CUDA meaning in WHIRL before emission: symbol, type, and parameter attributes for qualifiers and address spaces; selected kernel metadata for launch and resource constraints; and explicit operations or well-defined intrinsics for async copies, barriers, warp operations, atomics, and tensor compute. Preserve legality, target, and ordering information needed by downstream consumers.
  3. The owning optimization phase performs analysis, legality, profitability, and plan selection. whirl2c -cuda validates and prints the selected semantics; it must not choose tiling, placement, scheduling, or resource budgets from source-emission heuristics. Host launch configuration and runtime setup remain explicit consumers of the same selected kernel plan.
  4. For each newly needed CUDA extension, record its WHIRL representation, producing phase, target and legality rules, emitter spelling, host-side obligation, and verification status. If WHIRL cannot carry the required meaning, extend it with the binary-compatibility and verifier discipline above before claiming whirl2c support. Diagnose unsupported semantics rather than silently dropping them.
  5. The existing independently generated .cu path may remain option-controlled as a transitional parity baseline. Compare it with WHIRL-derived output and retire each duplicate path only after source, compile, and behavior checks establish parity. Do not treat the baseline as a second authoritative IR.

Open64 Design Continuity

  1. When new DSL infrastructure overlaps an established Open64 service, first study and reuse that service's intended extension points, controls, data structures, and tests. Preparation and postprocessing for DSL semantics are acceptable; an independent parallel implementation requires a documented incompatibility and explicit design review.
  2. DSL expression simplification must use the traditional Open64 wn_simp_code.h rule engine whenever the traditional rule has equivalent semantics. Follow the established physical-WN protocol: prepare a stack-local WN view for the simplifier, discard that temporary input after the call, and retain only simplifier results allocated in the normal WN memory-pool space. Tensor descriptor, effect, ownership, and logical-opcode checks belong in preparation; DSL result symbols, metadata, lineage, and mapped-image relationships belong in postprocessing.
  3. Do not duplicate traditional constant folding, identity, reassociation, cancellation, or factorization rules in a separate DSL rule engine. Add a DSL-only rule only when no traditional rule has equivalent semantics, and document and test that distinction.
  4. Revisit this continuity principle during design reviews for new DSL services. The concrete reuse mechanism may differ by subsystem, but the review must identify the existing Open64 design being preserved or explain why it cannot be reused.
  5. When WOPT admits unlowered DSL expressions, decode the physical OPR_DSL boundary representation into a first-class logical CODEREP identity. CODEREP hashing, equality, printing, effects, and simplification must include the DSL operator, version, canonical attributes, and TensorDescriptorIR identity. Continue using WOPT's existing CODEREP instantiation of wn_simp_code.h; do not add a parallel WOPT simplifier.
  6. Assign DSL APIs by semantic applicability, not by the first domain that requested them. An API belongs in osprey/common/com when its contract is domain-neutral or it is naturally reusable by AI/tensor compiler domains, even when FHE funded or first exercised it. Move an API to an FHE-owned module only when its correctness contract depends on cryptographic schemes, ciphertext state, key requirements, bootstrap or approximation policy, encrypted-runtime ABI, or other irreducibly FHE semantics. Caller location, current usage count, and an FHE implementation milestone are not by themselves sufficient reasons to classify an API as FHE-specific.
  7. Keep common/com policy-free. It may define DSL IR records, stable IDs, constructors/interning, structural verification, generic accessors, and logical printers. Fact harvesting from a PU, candidate discovery, legality analysis, cost/profitability modeling, optimization selection, and IR transformation belong in the phase that owns the compilation scope: be/vho for WHIRL/VHO work, be/opt for CFG/SSA/CODEREP work, be/lno for canonical-loop work, and IPA for cross-PU work. Do not place an optimization decision in common/com merely because its selected result is represented by a common IR record. A domain-neutral, AI-reusable atomic mutation mechanism may remain in common/com when the owning phase supplies the complete reviewed request and retains every legality, profitability, and policy decision.
  8. Place persisted FHE source-image, planning-image, approximation, context state, materialization, encryption, key, and FHE logical-printing contracts under osprey/common/fhe. Keep their mapped-image row and binary compatibility obligations intact, but do not place those irreducibly FHE APIs back in common/com. Generic AI-reusable DSL transactions remain in common/com under rules 6 and 7.

PREOPT Canonicalization And Optimization Roles

  1. Preserve the two distinct roles of the WOPT component. PREOPT cleans up and canonicalizes WHIRL before major optimization; full WOPT performs optimization on that canonical representation. PREOPT is also the shared canonicalization service for LNO and IPA where those pipelines invoke it; do not describe or design its canonical form as WOPT-only.
  2. When introducing a new WHIRL or DSL construct, define its canonical form and identify every downstream consumer. Ensure PREOPT produces or verifies that form before WOPT, LNO, or IPA consumes it in the applicable pipeline.
  3. Optimization candidate selection should rely on canonical IR whenever possible. Do not make every optimization recognize multiple equivalent tree shapes when PREOPT can normalize them once for all downstream phases.
  4. Keep canonicalization separate from profitability and transformation. PREOPT normalizes representation and exposes optimization opportunities; WOPT, LNO, and IPA identify legal and profitable candidates and perform transformations within their respective compilation scopes.
  5. A new canonicalization rule must preserve language, tensor descriptor, effect, alias, source-position, and strict floating-point semantics. It must honor the relevant phase and simplifier controls.
  6. When WOPT, LNO, or IPA requires a new canonical property, update the PREOPT contract, phase ordering, verifier, diagnostics, and tests together. Tests must prove that equivalent input forms converge to the canonical form and that every claimed downstream consumer receives that form in its actual pipeline and compilation scope.
  7. If a construct cannot be canonicalized before a downstream optimizer, document the reason and the additional candidate-selection complexity explicitly. Treat this as an exception requiring design review, not the default implementation path.

Compilation Scope And Optimization Ownership

  1. The compiler driver establishes compilation scope and phase lifetime. A service invoked by a phase must operate within that established scope; it must not enlarge its own scope by traversing PU_Info, restoring another PU's local symbol table, or switching Current_pu on its own.
  2. Normal backend compilation is per-PU. VHO, Preopt/WOPT, LNO, and CG operate on the active PU or on an explicitly selected REGION nested within that PU. REGION is a smaller intraprocedural scope, not an interprocedural scope, and its RID/map/pool lifetime remains driver-owned.
  3. Global symbol, type, string, TCON, and managed DSL tables provide identity, lookup, and boundary evidence. Their process-wide visibility does not grant a per-PU pass authority to analyze or mutate another PU.
  4. IPL is a per-PU summary-producing phase. It may prepare facts for IPA, but it does not turn an ordinary backend pass into an interprocedural pass.
  5. Cross-PU analysis or transformation belongs to IPA and occurs only when the -ipa compilation path establishes call-graph scope. Such work must use IPA-owned call-graph traversal, summaries, and explicit PU-context services such as IPA_NODE_CONTEXT; it must not be smuggled into VHO, WOPT, LNO, CG, or a common/com utility.
  6. Every new analysis or optimization must state its scope: expression, basic block, REGION, PU, file summary, or IPA call graph. Its ownership, invalidation, rollback, diagnostics, and tests must use that same scope.
  7. A per-PU pass may validate call/formal/result contracts visible at its boundary, but it must not infer that it may rewrite the opposite side of a call edge. Coordinated caller/callee refinement requires an explicitly designed IPA pass enabled by -ipa.
  8. Test scope must match implementation scope. A PU-local change requires intraprocedural positive, negative, rollback, and boundary-validation tests; it does not require a cross-PU transformation test. Multiple-PU fixtures may still prove independent driver coverage and absence of cross-PU mutation. Require call-graph propagation or coordinated caller/callee transformation tests only for code that executes in IPA scope under -ipa.

Optimization-Level Scope

  1. Treat optimization levels as analysis and transformation scope contracts. DSL optimization must follow the same contracts as traditional Open64; a DSL operator, tensor type, REGION, or domain does not grant permission to optimize at a broader scope.
  2. -O0 performs no optimization. It may verify legality and perform only the straightforward semantic lowering required to make the program executable. It must not perform fusion, algebraic improvement, layout optimization, profitability-driven rewriting, parallelization, or other performance-oriented transformations.
  3. -O1 permits basic-block-local optimization only. Its analysis and transformations must not depend on control-flow facts outside the active basic block, REGION-local equivalent, or other explicitly local unit.
  4. -O2 permits PU-level optimization over the active procedure's control-flow graph. It may use intraprocedural data-flow, alias, SSA, PRE, and related analyses, but it must not infer or transform across a PU boundary.
  5. -O3 adds optimization around canonical loops, with emphasis on memory behavior and parallelization. Loop transformation, tiling, locality, memory-hierarchy use, vectorization, and parallel execution must consume the canonical loop and dependence contracts established by the earlier phases.
  6. -ipa explicitly expands analysis beyond one PU through the IPA-owned call graph and summaries. Cross-PU transformation remains limited, reviewed, and controlled; enabling -ipa does not authorize arbitrary whole-program mutation by PU-local phases.
  7. Every new DSL analysis or transformation must declare its minimum optimization level, maximum compilation scope, required canonical form, invalidation behavior, and controlling option. The driver and phase must leave it disabled below that level and must not silently enlarge its scope.
  8. Validation must exercise the level boundary: prove -O0 preserves the unoptimized semantic form through straightforward lowering, prove the pass runs at its declared level, and prove it neither runs nor consumes out-of-scope facts at lower levels. Cross-PU tests are required only for explicitly enabled -ipa work.

Backend Shared-Library Dependencies

  1. Do not add a new library dependency to be.so without explicit design and build approval. This includes direct linkage and source-level references that create new defined or undefined symbols in the shared library.
  2. Before approving a dependency, identify every consumer of be.so, including lw_inline, backend plugins, and standalone tools, and prove that each consumer remains link-closed on every supported build configuration.
  3. Prefer an existing Open64 service or an already compatible header-only facility when structured parsing or another utility is needed in a backend pass. Do not introduce ad hoc parsing to avoid the dependency review.
  4. Validation for an approved dependency change must rebuild be.so and its shared consumers, inspect their defined and undefined symbols, and document platform, static/shared, licensing, version, and binary-distribution impact.
  5. Frontend-only libraries and interfaces, including DSL_Builder_*, must not leak into be.so. For example, an FHE backend may parse an authenticated JSON manifest with the repository's compatible header-only RapidJSON facility, but must not add JsonCpp linkage to be.so without approval.

Coding Guidelines

Whenever any existing function's implementation or signature is modified, review its function-head comment. If the comment no longer reflects the code's actual behavior or contract, update it in the same change. This maintenance requirement applies regardless of language, filename, or when the function was created; passing tests does not excuse a stale comment.

  1. Do not introduce tab characters in any file touched in this Open64 project.
  2. Use spaces for indentation and alignment.
  3. Makefiles may use literal tabs only where required by standard make recipe conventions.
  4. When editing existing Open64 C/C++ code, match the surrounding indentation style with spaces rather than reformatting unrelated code.
  5. Builder interfaces that create a symbol must make every reasonable attempt to set its declaration source position with Set_ST_Srcpos(). When a builder creates both a defining WN and a result ST, propagate the same complete source position, including file, line, and column, to both objects.
  6. Every newly created .h and .cxx file must begin with a concise architectural comment after the copyright notice. State the file's purpose, owning component or compilation scope, important behavior or compatibility boundary, and the repository-relative path of the controlling design or plan document. A public header should identify the contract it exposes; its implementation file should identify what it deliberately does and does not change.
  7. Add short orienting comments before non-obvious algorithm boundaries such as preparation, shared-engine reuse, legality classification, transactional mutation, verification, and postprocessing. Explain the invariant or design reason rather than restating the code. During final review, verify these comments exist in every newly added C/C++ source file and still agree with the implementation and cited design document.
  8. Every function in a DSL-related .h or .cxx file, including public APIs, private/static helpers, callbacks, and commit-only services, must have a function-level contract comment. State the function's purpose, its role in the surrounding caller/callee or phase relationship, important ownership and mutation boundaries, required invariants, and failure or rollback behavior where applicable. Public declarations must describe the caller contract; definitions must describe implementation responsibilities that are not evident from the declaration. Transactional functions must identify preflight, commit, verification, and rollback roles. A brief comment is sufficient for a trivial accessor or predicate, but merely restating the function name, parameters, or return type is not. New DSL files must satisfy this rule for every function before review. When materially modifying an existing DSL function, add or correct its contract comment in the same change; do not require unrelated whole-file comment churn for a narrow fix.
  9. All new DSL-related C/C++ files and APIs, including dsl_*, opt_dsl_*, and names prefixed with DSL_ or WOPT_DSL_, must follow the established Open64 be/opt naming style. Use ALL_CAPS_WITH_UNDERSCORES for classes, structs, typedefs, enum types, and public symbolic types; _lowercase_snake_case for ordinary private data members; and lowercase_snake_case for parameters and local variables.
  10. Name member functions with an initial capital and underscore-separated words, while preserving established compiler acronyms in uppercase, for example Build_candidates(), Verify_IR(), and Compute_PRE_saves(). Do not introduce CamelCase forms such as BuildCandidates() or mixed-case acronym forms such as Dsl in new DSL code.
  11. Name trivial field accessors after the property, such as Cfg() or Descriptor(). Use Set_, Reset_, Is_, Has_, and Can_ for mutation and predicate APIs. Reserve Get_ for operations that perform a lookup, computation, copy, or output assignment rather than a direct field read.
  12. Give public free functions, globals, enum values, and macros a stable subsystem prefix such as DSL_ or WOPT_DSL_. Enum values and macros use uppercase underscore-separated names. Extern free functions follow the WOPT API convention: preserve the uppercase subsystem acronym, then use lowercase underscore-separated component and operation words, for example DSL_tensor_evolution_create() and WOPT_DSL_populate_tensor_control_snapshot(). Preserve an additional uppercase word only when it is itself an established compiler acronym, such as WN or DIVREM. Avoid new unprefixed global names and prefer inline functions over macros unless an existing Open64 protocol requires a macro.
  13. Use established optimizer abbreviations such as cr, stmt, bb, cfg, wn, phi, aux_id, kid0, and kid1 where their meaning is local and unambiguous. Use descriptive lowercase underscore-separated names for new semantic concepts. Do not use pointer-name prefixes or other Hungarian notation.
  14. Treat compact historical structures, generated interfaces, required Makefile syntax, and tightly scoped template utilities as exceptions, not precedents for new DSL naming. When an owning directory has a stricter established convention, preserve that convention and document any intentional departure during review.
  15. A struct or class member function whose body consists of a single simple executable statement, such as a return, assignment, or forwarding call, must be defined in the appropriate .h file and explicitly marked inline, not defined out of line in a .cxx file. This rule applies only to struct and class methods, not free functions; functions private to a .cxx file remain there. Count statements, not physical source lines: wrapping one statement across several lines does not exempt it. Retain the required function-contract comment and preserve linkage, ownership boundaries, and binary compatibility when relocating an existing method definition.

Development Assertions

  1. In development builds, use Open64's Is_True assertions (enabled by Is_True_On) for violated internal compiler invariants so refactor defects stop at the failing boundary rather than causing later silent failures.
  2. Include actionable subsystem, ownership, or contract context in assertion messages. Keep assertion expressions free of side effects, since optimized builds may compile them out; retain required release-build checks.
  3. Malformed source or mapped IR, unsupported user requests, and deliberately rejected API inputs must still use their documented diagnostic/failure paths. Assert internal contradictions, not the mere presence of bad input.
  4. Test-fixture positive restoration/preparation boundaries may assert their expected invariants; deliberate negative cases must remain independently exercised and must not be disabled to make development builds pass.

DSL Comment Contracts And Maintenance

  1. Apply these comment requirements to every repository .h and .cxx file whose basename contains dsl (case-insensitive), including existing files created before the requirements were introduced. File age is not an exemption.
  2. Each such file must begin with a concise architectural comment after any existing copyright notice. Describe its purpose, owning component or scope, important behavior and compatibility boundaries, and the repository-relative path of the controlling design or plan document.
  3. Every function declaration and definition must have a function-level contract comment, including private/static helpers, inline methods, constructors, callbacks, and test functions. Describe purpose, caller/callee role, ownership, mutation, required invariants, and failure or rollback behavior where relevant. Brief meaningful comments suffice for trivial functions; merely restating a function name or signature does not satisfy the requirement.
  4. Add orienting comments at non-obvious algorithm boundaries. Explain the invariant or design reason, not just the sequence of statements.
  5. Whenever an existing function's body, signature, or behavior is modified, review its existing function-head comment against the resulting code. Update that comment in the same change if it no longer reflects the implementation, including preconditions, ownership, effects, scope, ordering, return values, and failure or rollback behavior. Add a contract comment if one is missing; retain an existing comment unchanged only when it remains accurate.
  6. During compliance review, verify both comment presence and semantic accuracy. A mechanical presence check cannot certify that a comment matches the code. Report unresolved legacy omissions and stale contracts explicitly; do not label a file compliant merely because it contains some comments.

Driver And Phase Option Convention

  1. Preserve Open64's command-line option propagation model. The driver passes the user's complete option set through the compiler pipeline.
  2. Each compiler phase selects and processes only the options applicable to that phase and silently ignores options owned by other phases.
  3. A phase must not reject a command merely because unrelated Open64 options are present. New phases, including torch2whirl, must follow this convention instead of requiring the driver to remove every unrelated option.
  4. For Python frontend option processing, use clang2whirl or the existing C frontend as the reference implementation pattern for receiving the complete option set, selecting frontend-owned options, and silently ignoring options owned by other phases.

Current Architecture Target

Python model / DSL
  -> torch.export / FX / DSC capture graph
  -> open64_dsc.WhirlExportInterpreter
  -> open64_dsc._whirl native extension
  -> libopen64_whirl_builder
  -> binary very-high-level WHIRL
  -> opencc -x whirl

The Python package should coordinate capture and graph traversal. The native builder should own all WHIRL-specific construction.

DSL And Tensor Rules

  1. DSL frontends may attach structured metadata with OPR_COMMENT nodes using the reserved prefix __WHIRL_DSL__:<domain>:<feature>:<payload>.
  2. Older tools should still see legal WHIRL. DSL-aware passes must validate and consume required markers before WOPT, LNO, CG, and other canonical-WHIRL phases.
  3. Tensor identity currently stages through an existing TY_IDX carrier plus a tensor extension side table. Do not force a new WHIRL operator enum, opcode enum, or binary type-kind change unless the stage explicitly calls for it.
  4. TensorDescriptorIR carries semantic tensor value state: element type, rank, shape, layout, strides, traits, lineage, placement, and related facts.
  5. Compiler metadata carries source context, diagnostics, pass ownership, lowering hints, and profiling data. Do not mix compiler metadata into tensor type equivalence unless a later schema explicitly promotes it.
  6. Avoid nested STL containers inside WHIRL-style table records that may later need to be dumped, reloaded, compared, or persisted in a binary IR image.
  7. Domain operators must remain domain-visible until domain gatekeeper checks complete. Do not hide CNN/Transformer ABI, layout, padding, stride, mask, head-layout, or similar legality checks through premature promotion or lowering.
  8. Treat physical OPR_DSL as a private WN escape tag. DSL-aware compiler code, diagnostics, ASCII dumps, and traces must use logical operator APIs and names such as OPR_DSLADD; they must not expose or manually decode the physical escape tag or its internal record index.

IR Compatibility And Staging Rules

  1. Keep ir_a2b and ir_b2a as compatibility gates. Any staged IR change must preserve behavior for existing WHIRL, or update those tools and matching tests in the same staged change.
  2. The binary WHIRL artifact is the frontend boundary. Avoid special in-memory bypasses from Python into the compiler pipeline; artifacts should be inspectable with ir_b2a -st -src before combined driver integration.
  3. Run gatekeeper verification before canonical lowering. Missing tensor attributes, invalid operand compatibility, unknown contracts, unsupported opcodes, and domain legality failures should be diagnosed before WOPT, LNO, or CG.
  4. Public DSL opcode enum values, string names, categories, levels, traits, shape rules, effect models, and promotion states are compatibility contracts once they appear in IR files, tests, dumps, or registries.
  5. TensorDescriptorIR carries semantic tensor value state. Compiler metadata carries source context, diagnostics, pass ownership, lowering hints, profiling, and source names. Compiler metadata must not affect tensor type equivalence.
  6. Do not add KIND_TENSOR or new binary tensor descriptor sections casually. First-class tensor type or tensor descriptor binary-section changes must land with printer, ir_b2a -st -src, binary reader, binary writer, ASCII reader/printer plan, verifier, and fallback behavior.
  7. torch2whirl, Python bindings, and the native frontend bridge must not depend on backend code generation. Backend, runtime, and kernel lowering happen after gatekeeper verification.

Domain Ingestion Staging

Use this workflow when adding a model family or a new domain to openpy:

  1. Add a small, deterministic, dependency-light source model fixture.
  2. Capture the real frontend graph and produce a reviewable operator census.
  3. Classify every captured operation as a common substrate operation, domain expression, region contract, state/effect, compiler metadata, constant, or explicit unsupported case.
  4. Review and publish stable native names, versions, operands, attributes, descriptor rules, effects, gatekeeper checks, and lowering ownership in the WHIRL infrastructure plan.
  5. Implement the native builder, mapped-image, reader/writer, logical printer, gatekeeper, and lowering support required by the published contract.
  6. Migrate the frontend from census/mock handling to opaque native APIs. The frontend must not learn WN layout or private physical DSL encodings.
  7. Certify the binary artifact, external data, ir_b2a -st -src trace, and openpy -O0 path across a process boundary.

Do not allocate DSL opcodes or broaden native WHIRL contracts solely from an expected model architecture. First capture a representative source model and review its semantic operator census. Conversely, do not force a captured model into existing operators when doing so would erase domain semantics. Frontend capture discovers the requirements; common/com review remains authoritative for native contracts and binary WHIRL behavior.

Maintain a separate frontend ingestion plan for each substantial domain or model family and link it to doc/WHIRL-DSL-INFRASTRUCTURE.md. The frontend plan owns capture, mapping, artifacts, and diagnostics; the infrastructure plan owns native representation, compatibility, verification, inspection, and lowering.

Impactful Test Selection

  1. Follow doc/IMPACTFUL-TEST-STRATEGY.md for FHE and other expensive DSL validation. Select tests from the changed contract and its reverse dependencies before launching a broad suite.
  2. Do not use the largest available model as the default edit-loop test. Run focused semantic and transaction regressions first; reserve full-model execution for an explicitly invalidated whole-program boundary, final PR head, milestone closure, or scheduled certification.
  3. Before starting a test expected to exceed ten minutes, state the affected contract, why smaller evidence is insufficient, and the expected duration.
  4. Reuse expensive artifacts only when content-addressed producer, trace, and audit receipts prove all relevant inputs unchanged. Documentation, printer, and auditor changes invalidate only their respective downstream evidence.
  5. Unknown source impact broadens the selected tests. It never justifies skipping validation. A full-model pass supplements rather than replaces focused rollback, malformed-input, and semantic-oracle tests.

Reviewable Test Artifacts

  1. Human review of compiler artifacts is part of the development and validation process. Tests that produce meaningful WHIRL evidence should retain the binary .B file, ir_b2a -st -src output, requested phase .t traces, side payloads, and relevant diagnostics after the test exits.
  2. Clean a designated disposable staging directory at the start of the next run, not at the end of the current run. A completed run must leave its evidence available until it is replaced by a later run. This rule never authorizes cleaning the durable artifacts/review hierarchy.
  3. Docker tests must write reviewable artifacts through an explicit host bind mount. Do not leave the only copy in a container filesystem or a container temporary directory that disappears when Docker exits.
  4. Keep artifact families in clearly named per-test or per-stage directories so one smoke test does not erase another test's evidence.
  5. Failed runs must not leave partial output with the name of a valid .B artifact. Preserve failure logs and diagnostics when useful, while keeping artifact publication atomic.
  6. At the end of validation, report the absolute host paths of retained artifacts so reviewers can inspect them directly.
  7. Generated review artifacts are normally local build evidence. Do not add them to Git unless the task explicitly requests checked-in golden files or review fixtures.
  8. When completing a task that produces or validates a compiler trace, include a directly viewable link to the retained trace in the final report. Show a short representative excerpt or summarize the concrete evidence it contains; do not report trace-based validation as complete using only a pass/fail statement.
  9. Use both -st and -src whenever running ir_b2a for validation or human review. Preserve or mount the original source file at the pathname recorded in the binary WHIRL DST so -src can interleave source statements with the IR. If the active ir_b2a build does not yet support -src, treat that as a tooling gap to fix; do not silently omit source cross-reference evidence.
  10. The ir_b2a output must use the input .B file's stem, for example ir_b2a -st -src resnet.B resnet.T. On case-insensitive filesystems where resnet.T collides with a driver-produced resnet.t, preserve the phase trace under a descriptive non-colliding name such as resnet.vho.t before producing resnet.T.
  11. The durable human-review root is <primary-open64-root>/artifacts/review. It is intentionally ignored by Git and is distinct from disposable test output under /private/tmp or a managed worktree. Expensive milestone artifacts must be promoted there after certification; a temporary path is not the sole retained copy.
  12. Organize durable evidence as artifacts/review/<domain>/<sync>/<milestone>/<receipt-id>/, where the receipt ID is content-addressed by the primary published artifact. Retain the binary, matching trace, side payloads, source required by -src, reports, logs, SHA256SUMS, and a machine-readable receipt together.
  13. Nothing under artifacts/review may be deleted, overwritten, truncated, moved out, or pruned without the user's explicit approval naming the affected path. Age, disk usage, a new run, branch deletion, worktree cleanup, or test startup is not approval. When storage is insufficient, stop and ask; never reclaim this directory automatically.
  14. Tests and automation may create a new receipt directory and may update an index or current symlink after validation, but they must never run a recursive cleanup against artifacts/review or any ancestor containing it. Publish a receipt by no-replace atomic promotion from staging, and reject a pre-existing destination rather than replacing it.
  15. Follow doc/REVIEW-ARTIFACT-RETENTION-POLICY.md for staging, receipt contents, review indexing, approval-controlled removal, and recovery.
  16. Every coding change that adds or modifies an IR transformation must retain reviewable before-and-after WHIRL evidence. Produce <case>.before.B and <case>.after.B, reopen each independently with ir_b2a -st -src as <case>.before.T and <case>.after.T, and retain a unified diff such as <case>.before-after.diff. An in-memory phase trace alone is not a substitute for mapped binary WHIRL evidence.
  17. Capture the before image immediately before the transformation and the after image immediately after it, using the same input, options, target, compilation scope, and source mapping. If the normal driver cannot publish both boundaries, add a focused producer or reviewed checkpoint using the existing WHIRL writer rather than fabricating textual IR.
  18. The transformation diff must make intentional WN, ST, TY, TensorDescriptorIR, REGION, and managed-table changes visible while also demonstrating relevant invariants. Preserve the raw ir_b2a diff; a focused or normalized excerpt may supplement it but must not replace it.
  19. Treat unexplained diff churn as a review blocker. The validation report and pull-request summary must describe the expected semantic changes, identify important facts that remain unchanged, and link the retained before trace, after trace, and full diff. If a transformation is expected to be a no-op for a fixture, retain and report the empty diff as evidence.

Near-Term Coding Priorities

  1. Stabilize native DSL/common infrastructure before adding Python package code.
  2. Add or maintain native tests under osprey/common/com/tests for tensor types, tensor constants, common.add, common.matmul, and ir_b2a -st -src visibility.
  3. Keep the first builder API narrow. Python-facing bindings should pass opaque handles and values; C++ should create real WHIRL objects.
  4. Use existing mapped-image / ELF WHIRL mechanisms for binary artifacts before inventing any new file format.
  5. Defer combined opencc -frontend=torch2whirl ... driver integration until the binary WHIRL artifact boundary is stable and inspectable.

Debugging And Triage

When investigating runtime or optimization failures, establish a baseline and add one condition at a time:

  1. First make -O0 work, then compare -O1, -O2, -O3, and finally -ipa.
  2. If -O0 works but -O1 fails, suspect CG.
  3. If -O1 works but -O2 fails, suspect WOPT.
  4. If -O2 works but -O3 fails, suspect LNO.
  5. If -O* works but -O* -ipa fails, suspect IPA.
  6. After identifying a component, reduce by file, then procedure, then specific optimization. Use binary search where possible.
  7. Treat -OPT:wn_simplify and its abbreviation -OPT:wn_simp as the master WHIRL construction-time simplifier control. When triaging a simplifier regression, preserve separate .B files with simplification enabled and disabled, produce matching ir_b2a -st -src traces, and compare the resulting WHIRL. New DSL construction-time simplification must honor Enable_WN_Simp; a DSL-specific control may further restrict a stage but must not override the master switch.

Reference Docs

Load these only when the task needs the detail:

  1. Open64_Python_FE_Plan.md - compact roadmap for Python ingestion and native WHIRL builder work.
  2. doc/WHIRL-DSL-INFRASTRUCTURE.md - DSL marker, tensor extension, and lowering infrastructure details.
  3. doc/VHO-DSL-OPTIMIZATION-PLAN.md - optional architecture-independent VHO optimization, parallelization, Preopt/IPA, and compiler-library codesign.
  4. doc/HOW-TO-TRIAGE-RUNTIME-FAILURE-OPEN64.md - runtime failure and optimization triage method.
  5. doc/TORCH2WHIRL-WHIRL-DSL-API-CONTRACT.md - torch2whirl to common/com DSL builder API inventory and subagent/main-agent API creation protocol.
  6. doc/Open64_Domain_Specific_Compiler_IR_Design.md - large staged design; search within it instead of loading it whole.
  7. imported_docs/DSC_Master_Design_Doc_v0.9_chapter_7.md - Python DSL ingestion architecture background.
  8. osprey/clang2whirl/README.md and osprey/clang2whirl/NAMING_CONVENTION.md - use only for clang2whirl work.
  9. WHIRL.pdf - normative WHIRL architecture and semantic baseline. Consult the relevant operator, node-layout, level, type, mapping, and ASCII-format sections before designing DSL extensions. The planned DSL appendix must remain consistent with this document or explicitly document extensions.
  10. doc/REVIEW-ARTIFACT-RETENTION-POLICY.md - permanent local review-artifact location, content-addressed receipt, indexing, immutable retention, and user-approved deletion rules.

Treat MPL-related sources as reference material only. Do not use them as evidence for the Very High Level WHIRL DSL frontend or Python ingestion design.

Most *.md files under GCC config/ directories are machine-description inputs, not general project guidance.