Bytecode serialization: compile Wren source ahead of time, load and run it without the compiler (#535)

[joshgoebel]

TL;DR

This adds a minimal, version-locked bytecode serialization path to
Wren. A new wrenSerializeModule() compiles a single Wren source file into a
self-contained byte artifact, and wrenInterpretBytecode() loads and executes
that artifact in a VM — no source text, and no compiler pass, at load time.
The artifact is deliberately not a stable ABI: it embeds the exact Wren
version and the loader rejects anything that doesn't match. Single module
only; imports, bundles, and obfuscation are explicitly out of scope for this
first pass. All 867 tests pass, plain and under AddressSanitizer.

All feedback is welcome!

Dev Notes

I'm not sure if we are still interested in merging something like this, but I thought there are likely those of you here who might want to use or expand on this for your own embedded Wren purposes. Also unsure what upstreams policy is on AI assisted contributions [know that's a hot topic in some circles]... this would not have been possible without AI - I simply don't know C that well nor have the months it would have taken to pour into this to have done it on my own. I did babysit the work and read and commented on a lot of things, but still C is not my forte...

This was a cross-collab of Sonnet 5/GPT 5.5 (planning), Fable 5 (oversight/review), Kimi K2.7 Code (all the coding). Plus an early adversarial critique of the loader ticket (004) [it seemed to be the most sticky bit] by 6 different AIs (for fun and learning):

• Deepseek V4 Pro
• Fable 5
• Gemini 3.5 Flash
• GPT 5.5
• GLM 5.2
• Qwen 3.7 Max

---

Details

This is a first-pass implementation of the long-standing request in #535
("Save (and load) compiled bytecode?"), scoped as narrowly as possible.

New public API (src/include/wren.h)

// Compile source -> artifact bytes (uses a temporary internal VM).
WrenSerializeResult wrenSerializeModule(WrenConfiguration* configuration,
                                        const char* module,
                                        const char* source,
                                        bool debugInfo);
void wrenFreeSerializeResult(WrenConfiguration* configuration,
                             WrenSerializeResult result);

// Load artifact bytes into an existing VM and run them in a new fiber,
// exactly like wrenInterpret() but without source or compiler.
WrenInterpretResult wrenInterpretBytecode(WrenVM* vm, const char* module,
                                          const uint8_t* bytes, size_t length);

Plus WREN_RESULT_LOAD_ERROR and WREN_ERROR_LOAD so hosts can distinguish
"this artifact is bad/incompatible" from compile and runtime errors.

The artifact

• 8-byte header: WREN magic, one byte each of major/minor/patch version,
  one flags byte (currently just a debug-info bit).
• The compiling VM's method-name symbol table. Method call/definition opcodes
  encode method names as indices into a VM-global table, so the loader
  interns each name in the loading VM and relocates every
  CALL_*/SUPER_*/METHOD_* operand. This is what makes an artifact
  loadable into a VM whose symbol table evolved differently (e.g. one that
  already ran other code).
• The module's own user-declared top-level variable names (slot layout only —
  values are produced by executing the module, as with a normal compile).
• The compiled ObjFn tree: code, constants (null/bool/num/string/nested
  fn), and optional debug info (function names + line tables).

All multi-byte values are big-endian; doubles are serialized as IEEE-754
bits. Core-module variables (Object, System, Fn, ...) are never
stored — the loader copies them from the loading VM's own live core module,
the same way compileInModule does, so the slot layout lines up by
construction. The module name is likewise not embedded; the caller supplies
it at load time, mirroring wrenInterpret(vm, module, source).

What you can do today

• Compile a .wren file to bytes on a desktop host and run those bytes on
  another VM instance built from the same Wren version — including a
  freshly-created VM, or one that has already interpreted other code.
• Use the full language within that file: classes, inheritance, closures,
  attributes, control flow, System.print(), etc.
• Interact with the loaded module from the host as usual:
  wrenGetVariable(), wrenCall() on handles into loaded classes, writeFn
  / errorFn all behave identically to source interpretation (there is a
  test suite asserting source-vs-bytecode behavioral equivalence).
• Strip debug info (debugInfo = false) to shrink the artifact; runtime
  errors then report without function names / line numbers.

What you can't do (v1 non-goals)

• No cross-version artifacts. The loader rejects any version mismatch.
• No stable bytecode ABI promise. The format may change freely between
  Wren versions.
• No import graph. One source file, one module per artifact. No
  serialized-module imports, no multi-module bundles, no external module
  loading.
• No loading of untrusted artifacts. The loader rigorously validates
  structure (magic/version/flags, truncation at every read, arity/upvalue/
  slot limits, constant tags, symbol-table bounds, duplicate/empty variable
  names, trailing bytes) and fails cleanly with WREN_RESULT_LOAD_ERROR, but
  it does not semantically verify opcodes. Treat artifacts with the same
  trust you'd give source files you ship yourself.
• No obfuscation/encryption layer, and no standalone wrenc/wrenvm
  binaries yet — this lands as library API first; tooling can be layered on
  later.

Implementation footprint

Nearly everything lives in one new file, src/vm/wren_serialize.c
(serializer + loader, ~1100 lines). Changes to the existing VM are tiny:

• wren_vm.c: extract wrenRunClosure() (shared by wrenInterpret and the
  loader) and add a bounds guard in methodNotFound() for symbols a loaded
  artifact may reference.
• wren_compiler.c: MAX_UPVALUES/MAX_CONSTANTS moved to wren_common.h
  so the loader can validate against them, plus one bug fix (below).
• No changes to the interpreter loop, GC, object model, or bytecode opcodes.

Included bug fix: while validating the loader's GC rooting under
WREN_DEBUG_GC_STRESS + ASan, we found a pre-existing upstream
use-after-free: wrenCompile() scans its first two tokens before
initCompiler() registers the compiler with the GC, so a collection
triggered mid-scan can sweep a token's ObjString while the parser still
references it. Fixed by registering the compiler before priming the token
stream (see BUGS.md for the full investigation). This window exists on
every wrenCompile call upstream, independent of this feature.

Tests

• ~35 new C API tests across four suites: source-vs-bytecode equivalence,
  malformed-artifact rejection (including an every-byte truncation
  sweep), host API interaction, and format sanity (round-trips,
  fresh-VM loads, debug flag).
• Full suite: 867 tests / 3862 expectations pass in release and in a
  debug + AddressSanitizer build.
• Planning docs and per-ticket design records are under plan/ for
  reviewers who want the rationale behind each decision.

---

FAQ

What about ABI / bytecode format compatibility?
The format is intentionally version-locked and makes no stability promise —
the header carries the exact Wren version and the loader rejects mismatches
outright. We think that's the right trade, and for most real uses of Wren it
costs nothing: Wren is an embedded language that you compile into your
application, and your scripts ship with your application. When you release
a new build of your project against a newer Wren, you simply recompile your
.wren sources to fresh artifacts in the same build step — exactly like .o
files, which nobody expects to survive a compiler upgrade. The artifact is a
build product, not an archival format. This also answers the original concern
in #535 that a serialization feature "will put pressure on us to keep that
format stable": Lua's luac has made zero cross-version promises for decades
and that has worked fine, and here the version stamp makes misuse fail loudly
and immediately rather than mysteriously.

Doesn't the compiler's coupling to the live runtime make this impossible?
That coupling (constants stored as live heap values, method names resolved
against a VM-global symbol table, core variables injected by slot position)
was the main technical objection in #535. This PR solves it narrowly rather
than generally: constants are serialized by tagged value (only null, bools,
numbers, strings, and nested functions can appear in constant tables), the
method-name table is carried in the artifact and relocated at load time, and
core-module slots are rebuilt from the loading VM's own live core module. No
new IR, no compiler rewrite — the serializer walks the same ObjFn tree the
compiler already produces.

Is this a security boundary? Can I load bytecode from untrusted sources?
No. The loader validates structure exhaustively and fails cleanly on
malformed input, but it is not a verifying loader — hostile-but-well-formed
bytecode is out of scope (a future bytecode verifier is noted in the planning
docs). Load artifacts you built yourself.

Does this protect my source code?
It raises the bar — you don't ship source text, and you can strip names and
line info — but it is not DRM. As noted in #535, bytecode can be reverse
engineered; if you need real protection, that's a legal/contractual matter.
Obfuscation/encryption layers are explicitly out of scope for v1.

What about imports and multi-module projects?
Out of scope for v1, deliberately. One file, one module. Whole-module
granularity is also what @munificent suggested as the workable compromise in
the original thread. External module loading and bundle formats are on the
backlog and nothing in the format precludes them.

Does this break the REPL or incremental compilation?
No. The normal wrenInterpret() path is untouched; the serializer runs in
its own temporary VM and the loader is purely additive.

Does this actually help embedded/memory-constrained targets?
It removes the two costs raised in #535: the compiler pass at startup, and
holding source text (plus wrenInterpret's copy of it) in RAM. Artifacts can
additionally drop debug info. We have not yet published benchmark numbers on
a microcontroller target — that measurement work is a natural follow-up.

Is the artifact portable across platforms?
The format is fully byte-defined (big-endian integers, IEEE-754 bit-pattern
doubles, length-prefixed strings) and contains no pointers or
platform-dependent layouts, so artifacts should be portable across
architectures running the same Wren version (including NaN-tagging vs.
non-NaN-tagging builds, since values are reconstructed at load). Cross-
platform round-trips are not yet exercised in CI, so treat this as designed-
for rather than guaranteed.

Why no wrenc command-line tool?
Library API first. The compile-to-artifact and load-from-artifact halves are
each one function call, so a standalone tool (or a wren-cli subcommand) is
a thin follow-up once the API shape has soaked.

[joshgoebel]

Yes commit history is terrible, but I can clean that up if there was interest in merging.

[tccountus]

Tests failing on debian trixie, built using the makefile:

$ python3 util/test.py 
FAIL: test/api/bytecode_loader.wren
      Unexpected output on stderr:
      realloc(): invalid pointer
      Expected return code 0 and got -6. Stderr:
      realloc(): invalid pointer
      
      Missing expected output "true" on line 6.

866 tests passed. 1 tests failed.

[joshgoebel]

Tests failing on debian trixie, built using the makefile:

Architecture? Could you reproduce this inside Docker? If so I could take a look. All of my builds were on Mac OS Darwin.

[tccountus]

I have an x86_64 cpu, default compiler (gcc), just normal desktop linux

[joshgoebel]

I'll see if I can fire up an x86 container and reproduce there...

[tccountus]

I wonder: normal programs work just fine, only the tests fail

[tccountus]

were you able to reproduce the issue?

[joshgoebel]

No, a lot going on right now, might not be able to get back to this for a bit.

[joshgoebel]

Fixed the Debian/glibc realloc(): invalid pointer failure in 03d1a59.

btNewContext() returned TestContext by value after setting config.userData to its local stack address, so output callbacks later used a dangling context pointer. The helper now initializes caller-owned storage in place, and all bytecode tests use that API.

Validation: python3 util/test.py passes: 867 tests, 3862 expectations.

[joshgoebel]

@tccountus Give that a shot!

[tccountus]

Thanks, can confirm it's fixed. Before merging, I would recommend (re-)moving the ai generated files

[joshgoebel]

Before merging I would recommend (re-)moving the ai generated files

I can absolutely clean it up for sure IF merging were desirable... I posted it largely for learning and discussion. I'm not sure if this is even desired in core (though it may ultimately end up in my own wren-console project). Also might be that we'd want to improve it a bit further before merging - such as adding the ability to handle modules, import other files, etc... this was just a very tight MVP to prove the encoding/decoding all can be done properly.

Ultimately all up to @ruby0x1