AE14 — Close out HTML tokenizer, tree construction, and parser-created DOM foundation

[joris97jansen]

Description

Close Milestone AE by documenting and validating the completed static HTML tokenizer, tree-construction, parser-created DOM, recovery, diagnostic, and conformance-testing foundation.

AE13 must provide an honest architectural closeout of what Borrowser can parse, how parser state and ownership are structured, which malformed inputs recover through browser-shaped rules, which specialized structures are supported, how parser-created nodes reach downstream consumers, and which parts of the broader HTML platform remain incomplete.

This issue is primarily a closeout, audit, and documentation issue. It must not become a container for substantial new parser features. Any material missing implementation discovered during closeout must be classified explicitly and either:

• treated as a blocker for a prior AE issue;

• recorded as a future issue under an appropriate milestone;

• retained as a documented limitation when it is outside the agreed Milestone AE scope.

Milestone AE must be closed as a static HTML parsing and DOM-construction foundation. It must not be described as complete HTML, complete DOM, complete forms, complete SVG/MathML, or browser-platform conformance.

The goal is that future CSS, Layout, Paint, accessibility, resource-loading, DOM, and JavaScript work can consume the parser-created document model without depending on undocumented parser shortcuts.

Requirements

Closeout audit

Perform a final audit of all completed Milestone AE issues.

The audit must cover:

• AE1 ownership boundaries;

• AE2 parser-created DOM node model;

• tokenizer preprocessing and typed token foundations;

• tokenizer state and EOF recovery support;

• deterministic parse-error reporting;

• tree-construction foundations;

• insertion-mode tracking;

• stack of open elements;

• active formatting elements;

• body-mode recovery;

• specialized table handling and foster parenting;

• select and form parser handling;

• template tree construction and template-content representation;

• namespace-aware SVG and MathML boundaries;

• parser fixtures, snapshots, parity tests, and regression harnesses.

For each area, classify support as:

• supported;

• partially supported;

• intentionally deferred;

• not applicable to Milestone AE.

Do not classify an area as supported solely because types or placeholders exist.

Base support claims on production behavior, tests, contracts, and fixture evidence.

Milestone closeout document

Add:

• docs/html5/ae13-html-parser-foundation-closeout.md

The closeout document must contain:

• milestone purpose;

• final architecture;

• ownership map;

• end-to-end parsing pipeline;

• completed support;

• partial support;

• deferred behavior;

• non-goals;

• testing and conformance evidence;

• downstream integration boundary;

• known risks and extension points;

• issue-by-issue completion summary;

• final milestone closeability decision.

The document must link to the relevant AE contracts rather than duplicating all low-level rules.

End-to-end parser architecture

Document the complete supported static parsing flow:

1. source bytes or strings enter the supported input boundary;

2. input preprocessing normalizes the supported character sequences;

3. the tokenizer consumes preprocessed input through explicit states;

4. typed tokens and tokenizer errors are emitted;

5. the tree builder dispatches tokens according to insertion mode and foreign-content context;

6. parser state tracks the stack of open elements and auxiliary structures;

7. parser-created nodes and patches are produced;

8. document mode is selected;

9. parser finalization flushes pending state and validates invariants;

10. deterministic snapshots and fixtures expose observable parser results;

11. browser/runtime and rendering consumers receive parser-created output without owning parser semantics.

Clarify which stages are streaming or chunk-aware.

Clarify which state is parser-owned and which state is merely consumed downstream.

Ownership summary

Document final subsystem ownership.

HTML/parser owns

• input preprocessing used by the HTML parser;

• tokenizer states;

• typed tokens;

• tokenizer parse errors;

• tree-construction insertion modes;

• stack of open elements;

• active formatting elements;

• table-text pending state;

• form-element pointer where implemented;

• template insertion-mode stack;

• foreign-content dispatch;

• parser-created namespace selection;

• document-mode selection;

• parser-created DOM semantics;

• parser diagnostic output;

• parser fixture semantics.

DOM or shared node model owns

• stable node identity;

• node kinds;

• parent-child relationships;

• attributes;

• namespaces;

• document-fragment or template-content representation;

• invariants required independently of rendering.

State precisely whether these types live physically in the HTML crate while remaining conceptually part of the parser-created DOM boundary.

Browser/runtime owns

• orchestration;

• materialization or consumption of parser-created output;

• form-control runtime initialization where supported;

• document lifecycle outside parsing;

• navigation and resource coordination in future milestones.

Browser/runtime must not own parser recovery or insertion decisions.

CSS, Layout, Paint, and accessibility consumers

Document that these subsystems consume the parser-created tree and must not repair malformed HTML structure independently.

Tokenizer support summary

Summarize implemented tokenizer behavior.

Include:

• input preprocessing;

• supported tokenizer states;

• supported character-reference handling;

• tag and attribute tokenization;

• comment handling;

• doctype handling;

• duplicate-attribute handling;

• self-closing state;

• raw-text, escapable-raw-text, script-data, plaintext, or other text-like states according to actual support;

• EOF recovery;

• tokenizer parse errors;

• chunked-input behavior;

• known unsupported tokenizer branches.

Do not state “HTML5 tokenizer complete” unless the implementation and conformance evidence justify that exact claim.

Provide a support table with:

• feature;

• implementation status;

• owning contract;

• primary test or fixture evidence;

• remaining limitation.

Tree-construction support summary

Summarize the implemented tree-construction model.

Include:

• initial and document-level modes;

• before-html, head, after-head, body, text, and after-body behavior according to actual support;

• explicit insertion-mode state;

• token reprocessing;

• stack of open elements;

• scope predicates;

• implied end tags;

• active formatting elements;

• reconstruction of formatting elements;

• adoption-agency behavior according to the actual supported subset;

• comments and text insertion;

• insertion-location selection;

• parser finalization;

• deterministic recovery and parse errors.

Clearly identify any standard insertion modes or algorithm branches that remain absent or partial.

Body-mode recovery summary

Document the completed AE7 behavior, including:

• paragraph auto-closing;

• unmatched paragraph end-tag handling;

• list-item recovery;

• implied end-tag generation;

• supported formatting-element reconstruction;

• supported active-formatting behavior;

• malformed-body recovery limitations.

Do not imply complete adoption-agency-algorithm support unless fully established.

Table and foster-parenting summary

Document the completed AE8 behavior.

Include:

• supported table insertion modes;

• implied table structure;

• row and cell recovery;

• pending table-character-token handling;

• foster-parent insertion-location selection;

• insertion-before behavior;

• stack clearing;

• token reprocessing;

• unsupported table parser branches.

State explicitly that Milestone AE does not implement:

• table layout;

• table sizing;

• border collapsing;

• CSS table formatting;

• table painting.

Select and form parser summary

Document the completed AE9 behavior.

Include:

• in select;

• in select in table;

• option and optgroup closure;

• table/select recovery;

• form-element pointer semantics where implemented;

• form start/end recovery;

• supported parser handling for input, textarea, button, and other covered elements;

• parser/runtime ownership boundary.

State explicitly that Milestone AE does not implement:

• form submission;

• validation;

• focus;

• selection state;

• control interaction;

• event dispatch;

• complete form-owner reassociation algorithms unless actually supported;

• accessibility form semantics.

Template summary

Document the completed AE10 behavior.

Include:

• template element parsing;

• template-content representation;

• template insertion-mode stack;

• in template;

• token reprocessing;

• active-formatting marker behavior;

• nested template handling;

• EOF recovery;

• rendering inertness boundary.

State explicitly which template behaviors remain deferred, including where applicable:

• public HTMLTemplateElement APIs;

• cloning;

• scripting integration;

• custom-element interaction;

• declarative shadow DOM;

• live DOM mutation behavior.

Do not retain “full template behavior if incomplete” as a vague gap. List the exact remaining behavior.

Foreign-content summary

Document the completed AE11 behavior.

Include:

• HTML, SVG, and MathML namespaces;

• namespace-aware element construction;

• foreign-content dispatch;

• SVG tag-name adjustment;

• supported attribute and namespaced-attribute adjustment;

• self-closing foreign elements;

• foreign end-tag behavior;

• breakout rules;

• SVG HTML integration points;

• MathML text integration points;

• unknown foreign-element fallback;

• mixed-namespace snapshots.

State explicitly that namespace-correct parsing does not imply:

• SVG rendering;

• MathML layout;

• animation;

• filters;

• geometry;

• external resource handling;

• XML parsing;

• complete namespace-aware DOM APIs.

Parser-created DOM model summary

Document the completed parser-created node model.

Include:

• supported node kinds;

• stable node identity;

• node ownership;

• parent-child invariants;

• attribute representation and ordering;

• duplicate-attribute behavior;

• namespace representation;

• document type;

• comments;

• text nodes;

• template-content/document-fragment nodes;

• parser insertion and patch semantics;

• non-renderable parser-created nodes;

• materialization boundary.

State which public DOM concepts are not yet implemented.

Document-mode summary

Document:

• supported doctype recognition;

• no-quirks mode;

• limited-quirks mode where supported;

• quirks mode;

• malformed or unsupported doctype behavior;

• how document mode is exposed downstream;

• how document mode is represented in fixtures.

Do not claim complete standard doctype classification if only a subset exists.

Parse-error and diagnostic summary

Document the final parser diagnostic model.

Include:

• preprocessing errors where applicable;

• tokenizer errors;

• tree-construction errors;

• typed error identities;

• deterministic ordering;

• source-position support and limitations;

• unsupported/deferred feature reports;

• transition traces;

• final-state invariant failures;

• distinction between normative parser errors and diagnostic-only output.

Clarify that parse errors do not necessarily abort parsing.

Snapshot and fixture summary

Document the AE12 regression surfaces.

Include:

• fixture format;

• tokenizer snapshots;

• parse-error snapshots;

• document-mode snapshots;

• tree snapshots;

• patch snapshots;

• transition traces;

• namespace output;

• template-content output;

• unsupported-feature output;

• chunked-input parity;

• final-state invariant validation;

• snapshot-update workflow;

• CI integration;

• external-suite adapter path.

State which surfaces are normative regression contracts.

Conformance evidence matrix

Add a closeout conformance matrix.

For every major parser feature, record:

• support classification;

• relevant source module;

• owning contract;

• unit-test coverage;

• fixture coverage;

• snapshot coverage;

• chunked-input parity coverage;

• known missing branches.

The matrix must include at least:

• preprocessing;

• tokenizer;

• doctypes and document mode;

• core document construction;

• body recovery;

• active formatting;

• tables;

• foster parenting;

• select;

• forms;

• templates;

• SVG;

• MathML;

• namespaces;

• parser-created DOM;

• patches;

• finalization;

• diagnostics;

• fixture harness.

Avoid using raw test counts as the primary proof of conformance.

Final invariant audit

Document and validate final parser invariants.

At minimum, cover:

• preprocessing state is finalized before tokenizer EOF completion;

• tokenizer finalization occurs once;

• pending token data is flushed;

• token reprocessing is bounded;

• stack-of-open-elements operations preserve node validity;

• active-formatting markers remain balanced for supported paths;

• pending table text is not left unflushed;

• form-element pointer does not dangle;

• template insertion-mode stack matches template context;

• every supported template has one valid contents root;

• namespaces are preserved through patches and materialization;

• parent-child relationships are acyclic and internally consistent;

• deterministic snapshots contain no unstable IDs;

• whole-input and chunked-input results match for the covered behavior.

Where invariants are executable, reference the production assertions or invariant tests.

Rendering-pipeline boundary

Document how parser-created DOM output reaches the existing browser/rendering pipeline.

Confirm that:

• HTML owns parser semantics;

• downstream consumers do not rerun or repair tree construction;

• parser-created node order is preserved;

• namespaces and template-content boundaries are preserved or intentionally skipped;

• non-renderable nodes do not accidentally become layout boxes;

• forms are consumed without moving form parsing into runtime;

• unsupported SVG/MathML rendering does not flatten parser namespaces;

• parser diagnostics remain available without becoming rendering dependencies.

Add or update smoke tests only where necessary to prove this boundary.

Do not make rendering output the normative parser conformance surface.

Feature-gap tracker update

Update:

• docs/engine-feature-gap-tracker.md

The tracker must reflect Milestone AE completion precisely.

Mark completed or narrowed gaps for:

• static HTML input preprocessing;

• typed tokenizer tokens;

• supported tokenizer states;

• deterministic parse errors;

• explicit tree-construction modes;

• stack of open elements;

• supported active formatting;

• parser-created DOM construction;

• document-mode selection;

• malformed-markup recovery;

• table parsing and foster parenting;

• select and form parser tree shape;

• template-content parsing foundation;

• namespace-aware SVG and MathML boundaries;

• deterministic parser fixtures and snapshots;

• parser-to-render ownership boundary.

Keep remaining gaps explicit.

Do not replace precise gaps with broad statements such as “HTML supported.”

Remaining HTML and DOM gaps

The closeout must retain exact future gaps where still true.

At minimum, evaluate and classify:

• tokenizer states or branches not yet supported;

• complete character-reference conformance;

• complete adoption-agency behavior;

• fragment parsing;

• scripting-enabled parser branches;

• parser pause/resume behavior;

• document.write;

• speculative parsing;

• encoding sniffing and byte decoding beyond the supported input boundary;

• full html5lib coverage;

• full WPT coverage;

• public DOM APIs;

• DOM mutation algorithms;

• ranges and selections;

• mutation observers;

• event dispatch;

• event loop and task queues;

• custom elements;

• shadow DOM;

• declarative shadow DOM if not included;

• complete template DOM APIs;

• complete form-owner algorithms;

• form submission and validation;

• accessibility tree semantics;

• SVG rendering;

• MathML layout;

• XML parsing;

• resource loading;

• navigation;

• browsing contexts;

• history;

• JavaScript execution.

Remove any listed gap that is demonstrably complete.

Do not preserve obsolete gaps merely because they appeared in an older tracker.

Future milestone recommendations

Define future extension points without implementing them.

For each recommended future area, provide:

• proposed milestone title;

• milestone description;

• one or more issue titles;

• issue descriptions;

• dependency on completed AE contracts;

• explicit boundaries.

At minimum, evaluate future work for:

DOM Core and mutation APIs

Potential scope:

• public node traversal;

• element and document APIs;

• mutation algorithms;

• attribute mutation;

• text mutation;

• document fragments;

• cloning;

• insertion and removal;

• mutation invariants.

Static document lifecycle and resource discovery

Potential scope:

• parser-discovered resources;

• base URL handling;

• stylesheet links;

• image candidates;

• preload or fetch requests;

• lifecycle states;

• parser/runtime handoff.

Forms platform behavior

Potential scope:

• form ownership beyond parser state;

• control values;

• disabled state;

• validation;

• submission;

• reset;

• interaction;

• encoding.

Accessibility semantics

Potential scope:

• semantic tree derivation;

• roles;

• names;

• states;

• form semantics;

• table semantics;

• SVG accessibility boundaries.

JavaScript and DOM bindings

Potential scope:

• bindings;

• script scheduling;

• parser-blocking scripts;

• event loop;

• document.write;

• custom elements;

• live DOM integration.

Broader parser conformance

Potential scope:

• fragment parsing;

• complete html5lib adapter;

• larger conformance imports;

• fuzzing;

• differential testing;

• remaining tokenizer and tree-builder branches.

Do not automatically create a new issue for every known platform gap. Group future work according to coherent browser-engine ownership.

Prior-issue closeability audit

Confirm whether every AE issue is closeable before closing the milestone.

For each issue, record:

• issue title;

• final status;

• contract delivered;

• implementation delivered;

• test evidence;

• documentation evidence;

• remaining work that belongs elsewhere;

• whether the GitHub issue can be closed.

If AE13 discovers that a prior AE issue does not meet its own exit criteria, Milestone AE is not closeable until that is resolved or the prior issue’s scope is explicitly corrected.

Do not hide unfinished prior-issue work inside the closeout document.

Milestone closeability statement

Conclude the closeout document with one explicit verdict:

• Closeable;

• Closeable with documented out-of-scope gaps;

• Not closeable.

A closeable verdict must explain why the remaining gaps are outside Milestone AE rather than unfinished AE requirements.

The statement must explicitly say that Milestone AE establishes a static HTML tokenizer, tree-construction, and parser-created DOM foundation—not a complete web platform.

Tests and validation

Run and document the final validation set required for Milestone AE.

At minimum, include:

• formatting checks;

• linting for affected crates;

• HTML crate unit tests;

• tokenizer tests;

• tree-builder tests;

• parser-created DOM tests;

• fixture and snapshot tests;

• chunked-input parity tests;

• invariant tests;

• patch tests;

• browser/runtime parser-consumption smoke tests;

• relevant workspace integration tests.

Run feature combinations required by the parser, snapshot, fuzzing, or invariant surfaces where those features exist.

Document any intentionally excluded slow or external suites and where they run instead.

All final validation commands and results must be recorded in the closeout document or an associated review packet.

Do not change production behavior solely to make the closeout tests easier to run.

Documentation consistency audit

Review all Milestone AE documentation for contradictions or stale claims.

At minimum, inspect:

• docs/html5/ae1-html-parser-dom-ownership-contract.md;

• docs/html5/ae2-parser-created-dom-node-model.md;

• tokenizer contracts added by earlier AE issues;

• tree-construction contracts;

• docs/html5/ae7-body-mode-recovery-contract.md;

• docs/html5/ae8-specialized-table-tree-construction-contract.md;

• docs/html5/ae9-select-form-tree-construction-contract.md;

• docs/html5/ae10-template-tree-construction-contract.md;

• docs/html5/ae11-foreign-content-tree-construction-contract.md;

• docs/html5/ae12-parser-conformance-regression-harness.md;

• docs/html5/html5-core-v0.md;

• docs/html5/spec-matrix-treebuilder.md;

• docs/html5/invariants.md;

• docs/html5/dompatch-contract.md;

• docs/html5/node-identity-contract.md;

• fixture README files;

• docs/engine-feature-gap-tracker.md.

Resolve inconsistent ownership claims, terminology, support classifications, and issue references.

Renumber stale AE9 or AE10 references that referred to the former fixture and closeout issues.

No implementation expansion

AE13 must not add major new parser algorithms.

Permitted implementation changes are limited to:

• correcting a small closeout-blocking defect;

• adding missing invariant validation;

• adding missing fixtures or snapshots for already implemented behavior;

• stabilizing debug output;

• correcting documentation or tracker inconsistencies;

• removing obsolete or contradictory code paths.

Any substantial new tokenizer state, insertion mode, DOM model, form algorithm, template feature, or foreign-content branch must be handled through the owning earlier issue or a new future issue.

Exit Criteria

• docs/html5/ae13-html-parser-foundation-closeout.md exists and contains the complete Milestone AE closeout.

• Every AE issue is audited against its own requirements and exit criteria.

• Every completed support claim is backed by implementation, contracts, and test or fixture evidence.

• The supported end-to-end parsing pipeline is documented from input preprocessing through parser-created DOM output.

• Final subsystem ownership is explicit and internally consistent.

• Tokenizer support and limitations are documented precisely.

• Tree-construction support and limitations are documented precisely.

• Body recovery, table parsing, foster parenting, select/form parsing, templates, and foreign content each have explicit final support summaries.

• Parser-created DOM node kinds, identities, attributes, namespaces, template contents, and patch semantics are documented.

• Document-mode selection and limitations are documented.

• Parse-error and diagnostic behavior is documented.

• Parser fixtures, snapshots, parity testing, invariant validation, and CI coverage are documented.

• A conformance evidence matrix maps every major parser area to its contracts and tests.

• Final parser invariants are documented and validated.

• The parser-to-browser/runtime and parser-to-rendering boundaries are documented and tested.

• docs/engine-feature-gap-tracker.md accurately narrows completed parser gaps without claiming full HTML or DOM support.

• Remaining tokenizer, parser, DOM, forms, JavaScript, rendering, resource-loading, navigation, and platform gaps are listed precisely.

• Obsolete gap statements and stale issue numbering are removed.

• Future extension points are grouped into coherent browser-engine milestones and issues.

• All required Milestone AE validation commands pass, or any failure is explicitly identified as blocking closeout.

• Documentation across all AE contracts is consistent.

• No substantial new parser feature is hidden inside the closeout issue.

• Every prior AE issue is either explicitly closeable or identified as blocking Milestone AE.

• The closeout document gives an explicit milestone verdict.

• Milestone AE can be closed as a static HTML tokenizer, tree-construction, parser-created DOM, recovery, and conformance-testing foundation.

• Milestone AE is not described as complete HTML-platform, DOM, forms, SVG/MathML rendering, JavaScript, resource-loading, navigation, event-loop, or WPT conformance.