Centralize Jac object semantics and native compiler migration foundations

[marsninja]

Completed work

This draft moves shared compiler infrastructure toward native execution without creating a second checker. The full native type checker has not yet been linked and executed; production native scope has not been expanded to include it.

• Jac object semantics: Jac-owned field metadata, constructors, defaults/factories, inheritance, deferred fields, representation, reflection and matching; bootstrap/runtime integration; shared field semantics across checking and code generation. External Python dataclasses use one interoperability adapter. The optional Pydantic adapter consumes Jac field metadata, including recursive schemas. Main's new external-contract decoder now uses the same record adapter; Jac/Python record regressions pass.
• Shared semantic analysis: quoted annotations use Jac parsing and annotation evaluation; prelude exports and ambient imports use Unitree/ModuleFacts; builtin extensions install typed symbols without rewriting Python stubs. Class assignment, enum ancestry, generic specialization and imported declarations use canonical identity. Tuple binding targets retain inferred types, and dictionary comprehensions share clause analysis.
• Compiler structure and typing: typed symbol providers, graph/type/catalog contracts, scoped evaluator access, editor presentation separated from semantic queries, and fixed compile-time operations expressed in Jac. Native generator and primitive source audits reached zero errors at the recorded checkpoint, with substantial typing warnings still outstanding.
• Native lowering: shared function emission state and persistent lowering-failure metadata; field/storage layout separated from constructor argument order; native factories use ordinary call classification. Container/tuple coercion, optional payloads, comprehensions, properties, generic specialization, nested call binding and imported function identity have implementation and regression coverage.
• Ownership and runtime integration: shared object/container release and tracing, retain-before-release field replacement, borrowed/self-assignment and cycle regressions, tuple-pop layout preservation, headerless memory support, walker reclamation and region coverage. Materialization identifies string fields from semantic declarations and keeps erased fields opaque. ELF linking handles optional weak imports and required strong imports consistently.
• Build, cache and packaging: explicitly requested server parsing is preserved; runtime vendoring uses an independent server compilation program. One typed kernel-unit cache helper keys memory entries by compiler identity and honors fresh builds across both cache layers. JIT dependency links retain their own options so cold builds load required shared libraries. Installer release-asset detection no longer races on early-closing pipelines. The Jac chess benchmark uses the current native_build API and can rebuild runtime-unit IR without debug code generation.
• Integration with main: merged main through 7e2ddc1f8a, resolved all seven display-name argument-order conflicts consistently, and adapted the newly introduced record decoder to Jac-owned semantics. The PR is mergeable.

Validation and current CI

Current head b21e8a20c is green in CI run 34179288887, attempt 2. The aggregate Everything Passed check succeeded, with no failed or pending jobs. The PR is mergeable.

Separate CI jobs passed 1,041 static-check targets, 1,938 runtime tests (8 skipped), 1,044 compiler-toplevel tests (2 skipped), 223 equivalence tests, 1,615 native backend/tools tests (4 skipped), and 1,217 compiler-pass/JavaScript tests (5 skipped). Binary/bootstrap, ARM64, client, scale/Kubernetes, package/docs, managed Android, installers, and packaged-site smoke checks also passed. The site check covered desktop packaging, development, production and gateway browser journeys, and the ejected backend.

The separated-client assertion now checks the contract identifier introduced by main's boundary-contract work. The earlier site entry-load failure did not recur in CI or the complete local journey using the CI binary; a fresh browser also rendered after a forced Vite dependency rebuild. The first attempt reported missing Pillow members during static analysis after restoring an analysis cache. The workflow's clean retry skipped that cache and passed all 1,041 targets; the underlying cache-related cause has not been established. No type checks or browser assertions were disabled.

Focused source runs also covered field/constructor semantics, Pydantic, canonical identity, annotations, enums, native containers and ownership. The latest record-decoder tests passed for Jac and Python records. These runs are separate evidence, not one aggregate suite.

The chess benchmark completed a warmup and one measured uncached build (10.789935 seconds). This confirms the entry point works; it is not a speedup comparison or a measurement of full native type checking. The current native compiler kernel still has recorded demotions; verified evaluator IR alone does not establish full native execution.

Remaining scope

Typed object-level graph endpoint access, remaining session/I/O/catalog boundaries, dynamic compile-time values/calls/exceptions, module identity and ownership cases, strict fresh/cached lowering parity, full checker linking/execution, corpus parity and production rollout remain unfinished. The remaining compiler phases also require their own native audit. See jac/jaclang/compiler/NATIVE_MIGRATION.md and the scope tables below.

Original scope tables

The following tables are preserved verbatim from the requested checkpoint. The completed-work and validation sections above record subsequent progress.

Implemented work

Area	What is done
Jac-owned object semantics	Added shared Jac field metadata, constructor generation, defaults/factories, inherited field ordering, deferred fields, representation, matching metadata, and reflection. Integrated into bootstrap and runtime object generation.
Dataclass consumer migration	Moved internal reflection, serialization, schema, compiler, runtime, server, and related consumers to the Jac object model. Centralized interoperability with external Python dataclasses in one adapter.
Native emission state	Centralized function state restoration across functions, methods, lambdas, generators, initialization, tests, and parallel workers, including failure paths.
Lowering failure reporting	Persisted unsupported methods, erased fields, and other lowering failures in LLVM metadata. Preserved diagnostic context and added executable-build rejection paths.
Compiler source typing	Corrected numerous evaluator contracts, optional accesses, graph relationship types, substitution/interner keys, diagnostic sequences, and catalog interfaces. Removed corrected evaluator checking exemptions.
Symbol providers	Introduced a typed shared provider contract, including bootstrap support.
Evaluator lifetime	Replaced the evaluator’s Python weak-reference pass access with scoped access.
Editor/compiler separation	Moved completion presentation into the language server while retaining shared semantic queries.
Compile-time operations	Replaced fixed Python operator dispatch with shared Jac operations. Corrected when compile-time requirements are validated.
Enum/property handling	Converted property-write classification to a Jac enum and repaired regular enum method dispatch through shared receiver and argument handling.
Native strings and intrinsics	Added tuple prefix/suffix handling and corrected registered memory intrinsic widths.
Dictionary and set layouts	Fixed shared key/value layout selection, nested tuple hashing/equality, object-key layout separation, and collection value representation.
Tuple operations	Preserved fixed-position and variadic tuple types through iteration, unpacking, and iterable inference. Added tuple-to-list conversion with typed storage and ownership.
Tuple-valued dictionaries	Fixed canonical pointer storage, singleton boxing, declared element conversions—including integers to floats and optional elements—and temporary dictionary ownership.
Callable ownership	Corrected closure binding scopes, escaping method receivers, borrowed parameters, inherited receivers, and typed callable layouts.
Imported function identity	Resolved module-qualified and direct imported aliases through checked declarations, including covered default and keyword argument cases.
Properties and destruction	Fixed computed union properties on chained/temporary receivers and inherited destruction through the nearest override.
Shared platform support	Centralized directory and errno services used by native library modules. Corrected inbound C-string ownership.
Test infrastructure	Separated raw LLVM inspection from executable builds; preserved memory-profile options and used explicit Python backend comparisons where needed.
PR preparation	Created and pushed the branch, opened the draft, documented remaining scope, added release notes, and resolved the merge conflict with current main.

Remaining work, including the unfinished portions of those changes

Area	Work left	Intended implementation
Complete native object semantics	Arbitrary native field factories, remaining constructor options, and full parity with Jac object semantics.	Extend shared field contracts and callable argument binding.
Strict native policy	Enforce NativeScopeEntry.strict consistently, including cached artifacts and imported dependencies.	One verdict evaluator for build policy and cache validation.
Dependency reporting	Distinguish required native dependencies, explicit host services, and unsupported paths; explain how each blocker is reached.	Extend existing dependency and coverage infrastructure.
Reproducible audits	Resolve remaining environment/order sensitivity and rerun relevant probes under identical configurations.	Existing test/build tooling with isolated runs.
Typed OSP endpoints	Remove remaining accesses such as __jac__.target.archetype; preserve the association between edge metadata and its target.	Shared typed endpoint/peer operations with host/native parity.
Graph mutation and lifetime	Validate required clone/replace operations, persistent/transient behavior, ownership, and cross-pass identity.	Existing graph and unitree infrastructure.
Native type algorithms	Complete executable parity for MRO, generics, protocols, enum synthesis, fixed-width rules, interning, and flow narrowing. Individual repairs are implemented; the complete subsystem is unproven.	Keep the existing Jac algorithms and repair their shared dependencies.
Compiler session	Eliminate erased JacProgram access and type remaining program, options, artifact, configuration, registry, and diagnostic contracts.	Reuse the existing program, module hub, dependency graph, and caches.
Imports and resources	Complete native module/package discovery, version handling, relocatable compiler resources, and import policy.	Shared typed resolution services.
Canonical identity	Handle class/global symbol collisions and remaining decorated/generic callable identity cases across modules.	Extend canonical declaration identity across registries and emitted symbols.
Client and TypeScript checking	Complete NPM/project discovery, sibling declarations, file reading, and external-type policy dependencies.	Typed source/package services feeding the existing declaration parser.
File and encoding support	Complete required open options, read/seek behavior, error handling, encoding policies, and buffer lifetime.	Shared native standard-library support.
Stub catalogs	Replace required mmap/erased-buffer dependencies; complete bounds-checked decoding, header operations, lazy loading, and lifecycle.	One typed byte-buffer/cursor implementation and the existing catalog format.
Python syntax dependencies	Replace required Python ast use for annotations, exports, ambient imports, and fallback parsing.	A shared frontend producing existing compiler data.
Dynamic values	Complete heterogeneous container representation, membership, equality/hash, and special-method behavior.	Shared runtime type/layout descriptors and dispatch.
Compile-time evaluation	Complete dynamic positional/keyword calls, defaults, constructors, record lookup, exceptions, materialization, deferred evaluation, and cycle handling.	Existing compile-time value model plus shared callable/runtime semantics.
Remaining evaluator/checker failures	Resolve residual imports, narrowing, reflection, class/TypedDict/property synthesis, call checking, async/client behavior, and typed pass initialization.	Repairs in the existing evaluator and checker.
Full native checker execution	Link and invoke the complete checker on real programs without unexpected fallback.	Compose the existing passes over the same native semantic graph.
Remaining ownership cases	Finish nogc temporary-container transfers, nested aggregate destruction, string-copy ownership, and exceptional temporary cleanup.	Shared ownership and container machinery.
Native language/backend gaps	Complete remaining comprehension/generator semantics, callable defaults, enum conversion differences, invalid signature fallbacks, and LLVM source typing.	General backend fixes with executable regressions.
Remaining compiler phases	Complete migration of the driver/cache, other analysis and optimization passes, Python/JCIR generation, JavaScript generation, and the native generator/LLVM binding itself.	Entry-point audits followed by shared service and backend repairs.
Full semantic parity	Compare diagnostics, locations, inferred types, graph relations, and valid/invalid program behavior across backends.	Existing compiler corpus with host/native execution.
Incremental compilation	Validate cache invalidation, dependency edits, catalog changes, repeated sessions, and resource release.	Existing session-owned caches and explicit lifetimes.
Production integration	Preserve graph residency across passes, expand native scope, and validate clean bootstrap and packaged binaries.	Existing kernel, driver, packaging, and CI infrastructure.
Performance measurement	Measure cold/warm chess compiles, phase timings, memory, and a broader corpus after native checking works.	Existing native_compile_bench.jac.
CI completion	Fix current failures and run the suites currently blocked behind the compiler build.	Keep addressing failures on the draft PR.

[greptile-apps]

Too many files changed for review (302 files, 100 file limit).

Bypass the limit by tagging @greptile-apps to review.