feat(native): build reusable units in owned compiler sessions - #8980
Open
marsninja wants to merge 67 commits into
Open
feat(native): build reusable units in owned compiler sessions#8980marsninja wants to merge 67 commits into
marsninja wants to merge 67 commits into
Conversation
…i-labs#8943) The native backend no longer builds one fused LLVM module per closure with its own IR-text cache, kernel-unit concatenation, regex-ordered initializers and a process-global scope flag. The module is the native unit: every native-locked module's JIR carries its native interface (SEC_NIFACE, digest-prefixed like the type interface), its relocatable object (SEC_NOBJ) and its bitcode (SEC_NBITCODE), each stamped with the compiler digest, codegen identity and triple; SEC_DEPS records each native dependency's interface digest, so a body edit rebuilds one object and an interface edit rebuilds its dependents. Every external symbol that is not :pub is module-qualified under the unit prefix; :pub symbols keep bare names. One link plan (backends/native/link_plan.jac) produces every artifact for jac nacompile, jac build --as native, the kernel, the seal and the in-process JIT, in objects (incremental) or bitcode (whole-program) mode over one synthesized glue object (link_glue.jac). The kernel comes from resolve_kernel() (kernel_resolve.jac): explicit switch, sealed image record, source tree by plan digest with derivation on demand, or the store parser. The OSP, fmt and region kernels are ordinary native units pulled by dependency edge. Removed: the .ir_cache/.ir_meta dependency cache and no_ir_cache, DepClibMeta and the fused linker in NativeCompilePass, kernel_units.jac, shared_emit.jac, the .nademote sidecar (the verdict lives in SEC_PLACEMENT), the scope-lowering globals (CompileOptions.native_unit), toolchain_native_by_policy (placement_facts.native_unit / native_only), _kit_site_present, CompileOptions.kernel_unit, and the diagnostics only the fused linker emitted. JIR format 25; sealed manifest format 9 records a package's native units and its one linked artifact, verified at load; a sealed app with native pins ships that artifact behind a ctypes engine.
Contributor
|
Too many files changed for review (332 files, 100 file limit). Bypass the limit by tagging |
…ocess unit compiles, host-only kernel, SEC_NDEPS The first revision of jaseci-labs#8980 was written without running it. This makes it run, and records the design in docs/internals/native-units.md. - The OSP region-death hook is `weak` and recorded in the unit's interface; the glue emits its no-op fallback only when no unit supplies it. A `linkonce_odr` definition nothing in its unit called was discarded by the unit's optimizer, and the weak glue stub then replaced it, so the graph runtime kept anchors of nodes whose arena had died: the kernel parsed one file and crashed on the next. - `jac_str_new` is exported only when a unit defines its target. - MCJIT takes llvmlite's default target machine; the pic/small pair the linked artifacts use made its AArch64 stubs branch into the GOT. - Native dependency digests live in their own stamped section, SEC_NDEPS (JIR format 26). In SEC_DEPS, any bytecode compile of a unit rewrote the rows without them and the next plan recompiled the closure. - The plan walks dependencies first and compiles each stale unit in a child jac process: an isolated in-process unit compile retained ~450 MB through comptime values cached on nodes shared with the selfhost program, and a kernel build reached 47 GB. Consumers take a dependency's interface as it stands; agreement is the plan's check. - A type-only import of a native unit is a dependency edge (it contributes layouts); a typed import served from the interface catalog gets its full AST, and a direct import served that way keeps its edge. - Loading a compiler-tree module's bytecode never rebuilds an in-process engine from its native sections. - The kernel is host-only, derived in a child process, and accepted by a source key (compiler digest plus unit module keys) without a plan. A unit compile child never derives one. - Tests updated to the plan's real behaviour: the region arena joins every plan by edge, a stale record is recompiled until the plan agrees, the str_new export rule, and the source-key acceptance. Measured on the chess example: runtime and kernel parse speed at parity with the fused build; first compile of a module and the cold kernel build are slower (details in the PR).
…lies A suppressed lowering code left a function undemoted and its module with invalid IR: the demotion of a function is decided by the errors its lowering recorded, and the pass driver already applies the diagnostic policy when it delivers a pass's alerts to the program. The IR generator now records every lowering diagnostic and lets delivery suppress it.
…THONPATH The build boots its interpreter with -I and an explicit sys.path, which a child does not inherit; importing jaclang registers the .jac importer, so the package directory is all a unit or kernel child needs.
…ks in the sealed lane The precompile stage compiled every native module and then built an in-process engine for it through the link plan, which the seal never uses. The precompile program now asks for no engine and get_bytecode honours a program's engine intent. link_plan.jac aliases the imported layout digest (its method shadowed the function) and types the parsed unit module.
python -c puts the working directory first on sys.path, and a build runs from the checkout while sealing a copy of the tree, so a unit or kernel child imported the checkout's jaclang and recorded toolchain edges into the wrong tree. Children now run in safe-path mode and insert the spawning package's parent directory first.
The precompile command is the build's engine, not a sealed kit's command, and it ends its process itself.
…n hand back their products A unit's symbol prefix hashed its absolute path, so a sealed unit compiled at the build's staging path defined symbols a consumer compiled from the installed kit never referenced; the JIT resolved them to null and every OSP fixture crashed in static init under the sealed kit. The module key is now the module's path within its package root (a sealed package, a jac.toml project or the jaclang package), which agrees across the move. Detached unit compiles hand their interface, deps, object and bitcode back to the requesting process through a products blob instead of leaving them in the shared module cache, so concurrent compile identities in one test worker no longer thrash each other's entries. The kernel's handle release uses a raw pointer, since one test sets Py_DecRef.argtypes process-wide and a py_object argument was then refused for every later parse in the worker. Tests that hand-linked one unit's IR with a hand-written entry now start from the plan's whole-program IR; the move-elision count skips the unit's exported jac_retain helper; the restore-failure test patches only the loader that still exists. Refs jaseci-labs#8943
glibc exports __cxa_atexit and keeps atexit in libc_nonshared.a, so the launcher built by the plan failed at load with an undefined atexit; the glue now defines the shim for every ELF artifact that is not static against musl, as the old entry injector did. Refs jaseci-labs#8943
The sealed build's checker refused the comprehension over the merged module's functions; the trace now walks them through a helper. Refs jaseci-labs#8943
…nit children keep their diagnostics to themselves Two units exporting one :pub symbol reached LLVM's linker, which reported a multiply defined global instead of the E5026 diagnostic that names both modules; the plan now checks its units' :pub exports before anything links. A detached unit compile's stderr is captured and shown only when the child fails: a first run that derives the kernel printed every unit's seam warnings, hundreds of kilobytes that the kit action's warm step piped into grep -q, which quit on the first match and killed echo with a broken pipe. The warm steps also read their output through here-strings now. Refs jaseci-labs#8943
…re for building the compiler Every unit compile spawned a child, and a test lane with four workers held thirteen compiler processes at once, 28 GB on a 16 GB runner. A program's few units now lower in the requesting process; JAC_NATIVE_UNIT_ISOLATE asks for a child per unit, which the kernel build and the seal set for themselves, where dozens of units would otherwise pile up in one process. A root compiled from a string keeps its recorded interface (no file can be newer than it) and takes its digest as its plan key; the cache-marker test pins a module placement keeps in Python, since a module placement promotes legitimately caches its native products. Refs jaseci-labs#8943
An empty target and the host's own triple compiled the same code but keyed different identities, so an artifact built with an explicit host target could not use the units the sealed image carries. Unit children also receive the app and nogc fields the identity folds in. Refs jaseci-labs#8943
…deriving one The source key folds each unit's package identity rather than its path, and the sidecar records those identities beside the build's paths, so a kernel built at the kit's staging directory is recognized in a checkout elsewhere. A rerouted checkout that cannot accept a kernel beside its loader now looks at the kernel the running kit carries and, when its source key matches, copies it in place; only a compiler that differs from both derives. The reroute lanes spent fourteen minutes deriving a kernel the kit they run on had already built from the same sources. Refs jaseci-labs#8943
The site's game failed to build three ways once its modules became units: the RC-free audit read the merged program, where the runtime units define the reference-counting helpers whatever the application's gc setting, so it now audits the application's own unit; the host imports a platform module declares were missed because only the entry module's C declarations counted, so every unit's count; and the host called __jac_glob_init, frame and compute by bare name, so the glue defines __jac_glob_init for wasm and the root unit's module-level functions and globals take their bare names in the merged module. Refs jaseci-labs#8943
marsninja
force-pushed
the
native-unit-model
branch
from
September 6, 2026 06:32
d8ac73c to
1323fb0
Compare
…hains under the unit model A demoted module was decided native again on its retry: the pin lookups that run without a program could not see a verdict that lives in the program's registry, so every pin check now takes the program. The importer asks the artifact carrier for a module's engine, and a plan that settled by lowering the entry again as a unit had evicted the module that held it; the compile pass now shares the engine and interop tables with the carrier. A module lowered in this process enters the registry with the digests of the dependencies it was lowered against, so a root compiled from a string is never asked to lower again from a file that does not exist. A re-export module made only of imports and includes forwards its own dependencies to the importer's plan, so a star or include chain links the units behind it. nacompile's assert-no-rc audits the program's own units, as the wasm audit does; the nogc CLI target and the body-scoped clib pin follow. Refs jaseci-labs#8943
…rted names keep their modules apart A unit's interface and its dependency-digest rows named dependencies by absolute path, so a kit's sealed units pointed at the build's staging tree and any plan that reached them from the installed kit failed. They are written as package identities now and resolved beside the reading unit. Two imported modules defining one name (hashlib's private `new`, hmac's public `new`) collided in the consumer's symbol table: the second was never declared and the first's demotion marked the bare name, so a module-qualified call failed to lower. Imported functions also register under a module-qualified key, a private function of another module is neither declared nor marked, and a colliding `:pub` name is left for the plan's collision report. Inherited-method thunks keep external linkage and are recorded as exports, and an engine carries every unit's interface so tests resolve inherited and imported symbols. The dev-loop lane takes the LLVM shim from the kit's runtime site instead of a build-keyed cache, and the local forced-native helpers in the equivalence suites pin their unit. Refs jaseci-labs#8943
…ts reach; the sweep still retires the flat native directory The kernel's plan links the AST's data model and its types beside the scope: they are jaclang modules compiled as units, not strangers. A cache root from before the generation layout still holds a native directory, which the sweep retires as it did. Refs jaseci-labs#8943
…r the package sentinel Sealed JIRs carry the module cache's placement summary so a demotion verdict survives the seal, but the summary named the module by the staging directory's absolute path, and two seals of one tree packed to different bytes. The seal rebases every path under the package root to the sentinel, as it does for the comptime dependency rows. Refs jaseci-labs#8943
…ct in the internals page Refs jaseci-labs#8943
…, and the linker tests read the plan's program The whole-repo type check refused an ambient import, a loosely typed manifest record, the `root` keyword as a loop name, an untyped engine handle and a few untyped JSON reads; each is typed now. A kernel-format walk bound a symbol under its bare name without recording its owner, and the next imported module of that name declared it twice; an already-declared symbol is bound instead. Toolchain units lead the plan order whatever path the walk reached them by. In objects mode the IR dump holds every unit and the glue, and nacompile reports its object emission. The PE, wasm, region-partition and string-kernel tests read the plan's whole program or the unit's symbol rather than the old whole-module IR. Refs jaseci-labs#8943
marsninja
changed the title
marsninja
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Native modules now own reusable products in the shared JIR cache, and every native entry point uses the same dependency-aware link plan. A body edit recompiles the affected unit; an unchanged artifact returns before whole-program optimization and object emission. This implements #8943 and removes the per-unit subprocess workaround that made build-kit substantially slower.
Compiler ownership and reuse
force_target_program.finally. Bootstrap compilations retain their frontend tier through code generation, even if imports finish in between. Native coverage replays from the completed link plan, including cached units.Cache, linking, and distribution
jac check, consistent with the test worker retirement setting; explicit counts remain operator overrides. The repository no longer forces eight workers on every machine. The formatter parallelism regression explicitly requests two workers.Validation
Final CI is green at
9ebbf110d: all 26 executed jobs passed, including Everything Passed..jacignorewithout new exclusions. Runtime CI: 1,924 passed, 8 existing skips, including the worker-budget and formatter regressions.Final build-kit at
9ebbf110dpassed in 20m29s, down from the original branch’s 30m30s (about 33% faster). Payload compilation took 934.3s and sealing 36.5s. The preceding run at005b6b1d8, with identical binary-producing code, took 18m49s (853.3s compilation, 33.8s sealing). Other cleanup revisions measured 17m48s–21m11s. The historical main build was 15m05s on an earlier revision with fewer payload modules; this is not a claim of parity with that baseline. Full downstream CI passed at9ebbf110d, including the package lane after removal of the repository’s fixed eight-worker override. The worker-memory ceiling is honored through the existing shared pool budget.Design:
jac/jaclang/cli/docs/internals/native-units.md.