Stamp parsetree v1: what remains before the AST can be frozen for PPXs

[cristianoc]

Context

parsetree0.ml is the frozen v0 AST that external PPXs receive today, and parsetree.ml has moved a long way from it. The bridge (ast_mapper_to0 / ast_mapper_from0) is best-effort and lossy by design. At some point a version of the current AST should be stamped as v1 so PPXs, or a better mechanism inspired by them, can target it directly.

This issue is the audit of what stands between the current tree and that stamp. It reviews what changed since v0, extracts the principles those changes were made under, and applies them to the rest of the surface syntax. File and line references are against the tip of the Lambda stack (#8615–#8620) as of 2026-09-04.

What has changed since v0

Area	v0	Now
Literals	Pconst_char of int, Pconst_string of string * string option	Pconst_char {source; semantic}, Pconst_string of String_literal.string_literal, new Pconst_json, Pconst_raw_source
Functions	unary nested Pexp_fun; Pexp_newtype; Pexp_function	Pexp_fun {newtypes; params; body; async}, n-ary, invariant params <> []
Application	Pexp_apply of expr * args	Pexp_apply {funct; args; partial; transformed_jsx}
Arrows	unary Ptyp_arrow	Ptyp_arrow {params: arg list; ret}
Records	(lid * expr) list	record_element list with opt; patterns add record_pat_rest
Objects	Pexp_send/new/setinstvar/override	Pexp_object_get/set/literal
New nodes	—	Pexp_await, Pexp_jsx_element and the jsx_* family, Pexp_for_of, Pexp_for_await_of, Pexp_template, Pexp_tagged_template, Pexp_break, Pexp_continue
Declarations	attributes only	pld_optional, pld_runtime_name, pcd_runtime_tag with constructor_tag
Bindings	—	pvb_constraint: value_constraint option
Externals	pval_prim: string list	pval_prim: primitive_repr option (Prim_name, Prim_ffi, Prim_inline_const)
Labels	Noloc.arg_label	arg_label carrying a loc
Removed	Pexp_lazy/poly/object, Ppat_lazy, Ptyp_class, Pstr_class, Psig_class*, Pexp_unreachable	—

Plus pc_bar on cases and Lexing.position fields on JSX nodes.

The principles those changes followed

1. Structure over attribute. What the compiler decides on becomes a field or constructor; the attribute is surface syntax, reconstructed only for printing and the v0 wire. One extraction point per fact: a smart constructor (Ast_helper.Type.field) and its inverse (Type.field_attributes), so there is never a second reader of the attribute.
2. Source and meaning, both explicit. {source; semantic} on literals, source_segments on templates, constructor_tag keeping its spelling. The boundary is stated once: source spelling is discarded after type checking.
3. ReScript's shape, not OCaml's encoding. N-ary functions, JSX and object operations as nodes, OCaml-only constructs deleted rather than kept as dummies.
4. Declaration-shaped, not string-encoded. primitive_repr instead of string list.
5. A location for everything printable. arg_label gained a loc; pld_runtime_name keeps the attribute's own loc so it reprints where it was written.
6. Bridge, don't break. Every change shipped to0/from0, a fixture under tests/syntax_tests/data/ast-mapping/, and ounit round-trips.

Audit

v1 is the parser's output

External PPXs run on bsc -bs-ast output, before jsx_v4 and before FFI digestion (cmd_ppx_apply.ml:27-33; Prim_ffi is documented as never observed by PPXs). So the stamp covers exactly what res_core.ml produces. That gives a clean scope: attributes emitted by the parser are in scope; attributes emitted by later passes are not.

A. Semantics still carried by parser-emitted attributes

res_core.ml emits ten res.* markers. Three classes:

Consumed by the type checker (real semantics as attributes, PPX-visible):

Marker	Consumer
res.inlineRecordDefinition	typedecl.ml:1534
res.patVariantSpread	variant_coercion.ml:284
res.dictPattern	dict_type_helpers.ml:37
res.braces	typecore.ml:2404 (error-message context)
res.ternary	typecore.ml:3049, next to TODO(attributes) Unify the attribute handling in the parser and rest of the compiler
res.await on module expressions	res_core.ml:6937, consumed via Ast_await from bs_builtin_ppx.ml:365-762

The last one is a gap left by the Pexp_await work: await e is a node, await M for dynamic import is still an attribute on Pmod.

Layout hints consumed only by the printer (PPX-visible): res.braces/ns.braces, res.ternary, res.iflet, res.doc. Formatting facts inside a semantic tree, the same category pc_bar was moved out of attributes into a field.

Parser desugaring with a reprint marker: res.spread and res.dictSpread. [...a, b] is desugared in the parser to List.spread([...]) (res_core.ml:4381) with the marker on the identifier so the printer can undo it. The AST does not contain the construct; a PPX sees the desugaring.

User-facing attributes in the same position @as was in before #8619: @react.component, @jsx.component, componentWithProps, consumed by jsx_v4.

B. Language features keyed by an extension name

%raw, %re, %ffi, %todo, %debugger are language features selected by a string in Pexp_extension. Pconst_raw_source and Pconst_json exist because they had no node: half a step taken.

C. Post-PPX state stored in the pre-PPX type

• Prim_ffi of {name; spec: External_ffi_types.t} in pval_prim: the FFI digestion result lives in the parsetree and is documented as never observed by PPXs. It cannot be both invisible to PPXs and part of the frozen wire.
• Pexp_apply.transformed_jsx: set by jsx_v4.ml:1295, read by translcore.ml and js_dump.ml:540. Same shape of problem on a PPX-visible node.

D. Self-containment of the type

parsetree.ml depends on Longident, Location, Lexing, String_literal, External_ffi_types, List.

• Lexing.position on 7 fields (JSX, pc_bar) against Location.t everywhere else: two position conventions on one wire.
• mutable pexp_attributes (parsetree.ml:250, "Hack: made pexp_attributes mutable for use in analysis exe"), written by analysis/src/xform.ml:273.
• pat_record_label (parsetree.ml:225): zero uses anywhere. Dead.
• Rinherit / row fields still tuple-encoded, with an in-tree TODO: switch to a record representation, and keep location (parsetree.ml:176).

E. Constructors the parser never produces

Resolved mechanically: for all 131 constructors in parsetree.ml, grep res_core.ml for the constructor or its Ast_helper builder.

• Dead: Ppat_open. Zero producers in res_core.ml or anywhere else; the only mentions are consumers in res_ast_debugger.ml, analysis/src/utils.ml, completion_front_end.ml, completion_patterns.ml. Remove.
• Everything else is reachable. Pstr_typext, Psig_typext, Pstr_recmodule, Psig_recmodule and Pexp_jsx_element are built through differently named helpers; Pct_* through Type.constructor; Prim_ffi and Prim_inline_const are digestion products by design (see C).

F. The wire

• PPXs receive v0 as output_value of Parsetree0 values with magic Caml1999M022. v1 exists only in-process; its magic ResImpl01306 has been bumped six times since 2026-08-28. Harmless while nothing external reads it, but it means there is no stable v1 yet.
• v0 encodings use fake structure a traversing PPX will act on: for..of becomes a for loop with 0..0 bounds and the iterable hidden in an attribute (ast_mapper_to0.ml:591-603); await is an attribute on the inner expression with an ordering trick to split two attribute lists (:703); object set is an application of #=; break and continue are extensions.
• Marshal ties a PPX to the compiler's OCaml version and runtime layout. The "better mechanism" question is the wire-format question, and it constrains the type: Lexing.position, loc_ghost, Longident all become wire commitments.

G. Test coverage at the bridge

10 fixtures under ast-mapping/ and 22 ounit tests, but none of the fixtures contain break, continue, a record rest pattern, a type a. constraint, dict{}, %raw, a unicode char escape, @optional, @inline, @tag, variant spread, or catch. Coverage is thinnest on exactly the newest v1-only nodes.

What stamping v1 requires

A. Apply principle 1 to the rest of the PPX-visible surface, one PR each, with the recipe from #8619 (smart constructor, _attributes inverse, ast-mapping fixture, ounit round-trip): inline record definitions, dict patterns and spread, variant spread, if let, ternary, await on module expressions, list spread, JSX component attributes, and the five extensions as nodes. The layout hints need a design decision first: a field on the node, or attributes documented as part of v1. Principle 2 argues for the field.

B. Make the type self-contained. Move External_ffi_types out of the parsetree (keep Prim_name; digestion output belongs where digestion runs, as transformed_jsx should). Lexing.position to Location.t. Drop mutable. Delete pat_record_label and Ppat_open. Record-ify row fields.

C. Decide the wire format before freezing, because it constrains the type.

D. Test to the bar of a stamp. A fixture for every v1-only node; a v1-to-wire-to-v1 identity test alongside the v0 one; a check that Ppat_open stays gone, generalized to "the parser cannot emit X".

E. Write it down. The recent nodes in parsetree.ml carry real doc comments; the older ones do not. The whole file needs that standard, plus a magic-number policy and a v0 sunset.

Suggested order: B first, since it is mechanical and shrinks the surface before anything is frozen; then A as a PR series; C in parallel as a design note; D grows with A; E last.

Out of scope for the stamp, still worth fixing

Attributes emitted after PPXs run and consumed downstream, invisible to a PPX but the same disease: res.hoistedFunction (bs_builtin_ppx.ml:68 to translcore.ml:970, lam_compile_main.ml:199), res.jsxComponentProps (jsx_v4.ml:211 to error_message_utils.ml:898), res.patFromVariantSpread and res.constructor_from_spread (internal to variant_type_spread.ml).