Repository navigation
makeliteral(): replace rule elements with one token (#450) (4.2.28) - #784
Merged
Merged
Conversation
…) (4.2.28)
@post
makeliteral(1, 2, "70");
@rules
_xNIL <- 7 O @@
turns OCR's "7O" into one num token "70": later passes match it as _xNUM, its
$text is "70", and it keeps the offsets of the "7O" it replaced. The type is
what the tokenizer would give the text (all digits num, all whitespace white,
one ASCII punctuation character punct, otherwise alpha), or a 4th argument
forces it: makeliteral(1, 1, "USA", "alpha").
Pat::redLiteral reduces the range as singler() does, then Pat::makeLiteral
drops the node's children and gives it the text, type and a new LITERAL flag;
Pn::pnStr returns a literal node's own name rather than the input buffer.
The compiled Arun::makeliteral reduces with the compiled singler() and calls
the same Pat::makeLiteral. Like singlezap, it replaces the matched nodes, so
it goes last in @post.
Named makeliteral, not literal: parse-en-us has its own literal(node), and a
post action is resolved before user functions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #450.
This turns OCR's
7O(letter O) into one num token70:_xNUM, with no change to the rest of the analyzer;$textis70;7Oit replaced.Token type. The type is what the tokenizer would give the text: all digits →
num, all whitespace →white, a single ASCII punctuation character →punct, otherwisealpha. A fourth argument forces it, for examplemakeliteral(1, 1, "USA", "alpha"); the accepted types arenum,alpha,punct,white,ctrl,emojiandnode.How it works.
Pat::redLiteralreduces the range assingler()does.Pat::makeLiteralthen drops the new node's children and gives it the text, the type, and a newLITERALflag.Pn::pnStrreturns a literal node's own name rather than text from the input buffer, which handles the issue's note that the literal "won't match the underlying input buffer".Arun::makeliteralreduces with the compiledsingler()and calls the samePat::makeLiteral.singlezap, it replaces the matched nodes, so it should be the last action in@POST.Naming. It's
makeliteral, notliteralas first planned. parse-en-us defines its ownliteral(node)and calls it from about ten passes. Because a post action is resolved before user functions, a builtinliteralchanged parse-en-us's output; the Full English Test caught it before this PR.Testing
tests/literal, with CI steps for Windows and Linux. It covers7O→ num70($text, offsets), a forced type, an inferredpunct, and a later pass matching the literal as_xNUM.-DEVFull English Test is unchanged, and all 20 analyzers (21 inputs) produce byte-identical output to 4.2.18.🤖 Generated with Claude Code