Skip to content

makeliteral(): replace rule elements with one token (#450) (4.2.28) - #784

Merged
ddehilster merged 1 commit into
masterfrom
feature/literal-450
Oct 15, 2026
Merged

ddehilster merged 1 commit into
masterfrom
feature/literal-450

Conversation

@ddehilster

Copy link
Copy Markdown
Member

Closes #450.

@POST
  makeliteral(1, 2, "70");
@RULES
_xNIL <- 7 O @@

This turns OCR's 7O (letter O) into one num token 70:

  • later passes match it as _xNUM, with no change to the rest of the analyzer;
  • its $text is 70;
  • it keeps the offsets of the 7O it replaced.

Token type. The type is what the tokenizer would give the text: all digits → num, all whitespace → white, a single ASCII punctuation character → punct, otherwise alpha. A fourth argument forces it, for example makeliteral(1, 1, "USA", "alpha"); the accepted types are num, alpha, punct, white, ctrl, emoji and node.

How it works.

  • Pat::redLiteral reduces the range as singler() does.
  • Pat::makeLiteral then drops the new node's children and gives it the text, the type, and a new LITERAL flag. Pn::pnStr returns a literal node's own name rather than text from the input buffer, which handles the issue's note that the literal "won't match the underlying input buffer".
  • Compiled: Arun::makeliteral reduces with the compiled singler() and calls the same Pat::makeLiteral.
  • Like singlezap, it replaces the matched nodes, so it should be the last action in @POST.

Naming. It's makeliteral, not literal as first planned. parse-en-us defines its own literal(node) and calls it from about ten passes. Because a post action is resolved before user functions, a builtin literal changed parse-en-us's output; the Full English Test caught it before this PR.

Testing

  • New fixture tests/literal, with CI steps for Windows and Linux. It covers 7O → num 70 ($text, offsets), a forced type, an inferred punct, and a later pass matching the literal as _xNUM.
  • I also built the fixture as a compiled analyzer DLL; its output is identical to the interpreted run.
  • All other Windows CI steps pass locally. The -DEV Full English Test is unchanged, and all 20 analyzers (21 inputs) produce byte-identical output to 4.2.18.

🤖 Generated with Claude Code

…) (4.2.28)

    @post
      makeliteral(1, 2, "70");
    @rules
    _xNIL <- 7 O @@

turns OCR's "7O" into one num token "70": later passes match it as _xNUM, its
$text is "70", and it keeps the offsets of the "7O" it replaced. The type is
what the tokenizer would give the text (all digits num, all whitespace white,
one ASCII punctuation character punct, otherwise alpha), or a 4th argument
forces it: makeliteral(1, 1, "USA", "alpha").

Pat::redLiteral reduces the range as singler() does, then Pat::makeLiteral
drops the node's children and gives it the text, type and a new LITERAL flag;
Pn::pnStr returns a literal node's own name rather than the input buffer.
The compiled Arun::makeliteral reduces with the compiled singler() and calls
the same Pat::makeLiteral. Like singlezap, it replaces the matched nodes, so
it goes last in @post.

Named makeliteral, not literal: parse-en-us has its own literal(node), and a
post action is resolved before user functions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ddehilster
ddehilster merged commit 51b4194 into master Oct 15, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NLP++ enhancement: create a literal

1 participant