Skip to content

Suppress EML template detection inside OCaml comments - #431

Open
MavenRain wants to merge 1 commit into
camlworks:masterfrom
MavenRain:365-eml-ocaml-comments
Open

Suppress EML template detection inside OCaml comments#431
MavenRain wants to merge 1 commit into
camlworks:masterfrom
MavenRain:365-eml-ocaml-comments

Conversation

@MavenRain

Copy link
Copy Markdown

Fixes #365.

Problem

The EML tokenizer decides line-by-line whether template text begins (indented <..., %%, or a leading %), with no awareness of OCaml lexical state. A multi-line (* *) comment whose body contains an indented HTML-like line therefore flips the scanner into template mode: the comment body is emitted as template text, and the terminating *) lands inside a generated string literal. The OCaml lexer reads string literals inside comments, so the emitted file dies with Comment not terminated:

  (*
  <p>Im gonna break everything:( </p>
  *)
#1 "main.eml.ml"
  (*
let ___eml_buffer = Buffer.create 4096 in
(Buffer.add_string ___eml_buffer "<p>Im gonna break everything:( </p>\n*)");
(Buffer.contents ___eml_buffer)

Fix

scan_code_block now advances a small OCaml lexical state machine (code_syntax in src/eml/eml.ml) over each character it consumes, and skips template detection on lines that begin inside a comment. Tracking just comment delimiters is not enough, so the machine follows the corresponding subset of OCaml's lexical conventions:

  • comments nest;
  • string literals are lexed inside comments, so a "*)" inside a string inside a comment does not close the comment;
  • a (* inside a string literal in code does not open a comment - this covers "..." with escapes, plain quoted strings {id|...|id}, and extension quoted strings {%ext|...|} / {%%ext delim|...|delim};
  • '"' is a character literal, not a string delimiter, while 'a is a type variable and x' continues an identifier, neither of which opens a character literal.

The same state also fixes the sibling of #365 for free: template detection is suppressed on lines that begin inside a multi-line string or quoted string literal, so let icon = {%html| followed by an indented <svg>...</svg> line stays one raw string instead of being split into template output. This cannot affect any currently-working file: a line that begins inside a string literal of a valid OCaml program can never have been intended as template text (an unclosed literal is not valid OCaml), and today such inputs already generate broken code.

Tracking applies only to OCaml input: Tokenizer.scan takes ?syntax (defaulting to `OCaml), and --emit-reason input is scanned exactly as before, so Reason code such as the (*) operator passed as an argument keeps working. Inside comments, quoted strings are recognized the way the OCaml comment lexer does from 4.11 onward.

With the fix, the repro above passes through as a verbatim comment. As a side effect, whole template sections (including %% lines) can now be commented out with (* *).

Testing

  • 23 new expect tests in test/expect/eml/tokens.ml (53 total): the issue repro, comment followed by a live template, nested comments, "*)" and escaped quotes inside strings inside comments, {|...|} and {%ext|...|} inside comments, a commented-out %% section, a same-line (* a *) (template detection must still fire on the next line), the false-open guards ("(*", '"', x'"', {|(*|}, {%html|(*|}, {%sql foo|(*|foo}, and {%|, whose empty attribute id opens nothing), type variables, a ( / * split across a line boundary, multi-line "..." / {|...|} / {%html|...|} literals containing template-like lines, and an OCaml/Reason pair showing Reason input is unaffected.
  • make test TEST=test/expect/eml green on OCaml 4.14.1 (the 30 pre-existing tests are byte-identical, so comment-free inputs tokenize exactly as before); new stdlib usage stays within the 4.08 floor (Seq.fold_left, String.to_seq, Option.fold).
  • End-to-end: dream_eml --stdout on the repro emits the comment verbatim, and the output compiles with ocamlc -c.
  • Mutation checks: weakening the suppression threshold, removing string-literal tracking, breaking extension-string recognition, breaking identifier-apostrophe handling, and disabling string-literal suppression each turn the new tests red.

Drafted with AI assistance; design, verification, and testing reviewed and run by me.

The EML tokenizer decided line-by-line whether template text begins,
with no awareness of OCaml lexical state. A multi-line (* *) comment
containing an indented HTML-like line therefore flipped the scanner
into template mode: the comment body was emitted as template text and
the terminating *) ended up inside a generated string literal, leaving
the output .ml with an unterminated comment (camlworks#365).

scan_code_block now advances a small OCaml lexical state machine over
each consumed character, and skips template detection on lines that
begin inside a comment. Tracking just comment delimiters is not
enough, so the machine follows the corresponding subset of OCaml's
lexical conventions: nested comments; string literals with escapes
(lexed inside comments too); character literals versus type variables
and identifier apostrophes; and quoted strings, both plain
{id|...|id} and extension {%ext|...|} forms.

The same state also suppresses template detection on lines beginning
inside a multi-line string or quoted string literal, which is the
sibling of the reported bug and cannot affect any currently-working
file.

Tracking applies only to OCaml input. Reason .eml.re input is scanned
exactly as before (Tokenizer.scan takes ?syntax, defaulting to OCaml).

As a side effect, whole template sections (including %% lines) can now
be commented out with (* *).

Fixes camlworks#365

Signed-off-by: Onyeka Obi <softwareengineerasaservant@isurvivable.cv>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Eml ocaml comments support

1 participant