Suppress EML template detection inside OCaml comments - #431
Open
MavenRain wants to merge 1 commit into
Open
Conversation
The EML tokenizer decided line-by-line whether template text begins, with no awareness of OCaml lexical state. A multi-line (* *) comment containing an indented HTML-like line therefore flipped the scanner into template mode: the comment body was emitted as template text and the terminating *) ended up inside a generated string literal, leaving the output .ml with an unterminated comment (camlworks#365). scan_code_block now advances a small OCaml lexical state machine over each consumed character, and skips template detection on lines that begin inside a comment. Tracking just comment delimiters is not enough, so the machine follows the corresponding subset of OCaml's lexical conventions: nested comments; string literals with escapes (lexed inside comments too); character literals versus type variables and identifier apostrophes; and quoted strings, both plain {id|...|id} and extension {%ext|...|} forms. The same state also suppresses template detection on lines beginning inside a multi-line string or quoted string literal, which is the sibling of the reported bug and cannot affect any currently-working file. Tracking applies only to OCaml input. Reason .eml.re input is scanned exactly as before (Tokenizer.scan takes ?syntax, defaulting to OCaml). As a side effect, whole template sections (including %% lines) can now be commented out with (* *). Fixes camlworks#365 Signed-off-by: Onyeka Obi <softwareengineerasaservant@isurvivable.cv>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #365.
Problem
The EML tokenizer decides line-by-line whether template text begins (indented
<...,%%, or a leading%), with no awareness of OCaml lexical state. A multi-line(* *)comment whose body contains an indented HTML-like line therefore flips the scanner into template mode: the comment body is emitted as template text, and the terminating*)lands inside a generated string literal. The OCaml lexer reads string literals inside comments, so the emitted file dies withComment not terminated:Fix
scan_code_blocknow advances a small OCaml lexical state machine (code_syntaxinsrc/eml/eml.ml) over each character it consumes, and skips template detection on lines that begin inside a comment. Tracking just comment delimiters is not enough, so the machine follows the corresponding subset of OCaml's lexical conventions:"*)"inside a string inside a comment does not close the comment;(*inside a string literal in code does not open a comment - this covers"..."with escapes, plain quoted strings{id|...|id}, and extension quoted strings{%ext|...|}/{%%ext delim|...|delim};'"'is a character literal, not a string delimiter, while'ais a type variable andx'continues an identifier, neither of which opens a character literal.The same state also fixes the sibling of #365 for free: template detection is suppressed on lines that begin inside a multi-line string or quoted string literal, so
let icon = {%html|followed by an indented<svg>...</svg>line stays one raw string instead of being split into template output. This cannot affect any currently-working file: a line that begins inside a string literal of a valid OCaml program can never have been intended as template text (an unclosed literal is not valid OCaml), and today such inputs already generate broken code.Tracking applies only to OCaml input:
Tokenizer.scantakes?syntax(defaulting to`OCaml), and--emit-reasoninput is scanned exactly as before, so Reason code such as the(*)operator passed as an argument keeps working. Inside comments, quoted strings are recognized the way the OCaml comment lexer does from 4.11 onward.With the fix, the repro above passes through as a verbatim comment. As a side effect, whole template sections (including
%%lines) can now be commented out with(* *).Testing
test/expect/eml/tokens.ml(53 total): the issue repro, comment followed by a live template, nested comments,"*)"and escaped quotes inside strings inside comments,{|...|}and{%ext|...|}inside comments, a commented-out%%section, a same-line(* a *)(template detection must still fire on the next line), the false-open guards ("(*",'"',x'"',{|(*|},{%html|(*|},{%sql foo|(*|foo}, and{%|, whose empty attribute id opens nothing), type variables, a(/*split across a line boundary, multi-line"..."/{|...|}/{%html|...|}literals containing template-like lines, and an OCaml/Reason pair showing Reason input is unaffected.make test TEST=test/expect/emlgreen on OCaml 4.14.1 (the 30 pre-existing tests are byte-identical, so comment-free inputs tokenize exactly as before); new stdlib usage stays within the 4.08 floor (Seq.fold_left,String.to_seq,Option.fold).dream_eml --stdouton the repro emits the comment verbatim, and the output compiles withocamlc -c.Drafted with AI assistance; design, verification, and testing reviewed and run by me.