Fix: Left-recursive rule fails to grow when it occupies a non-final field of an enclosing sequence

[ChrisH07]

Left-recursive rule fails to grow when it occupies a non-final field of an enclosing sequence

Summary

SequenceRule has built-in support for direct left recursion (AddLeftRecursionRules,RecursiveContinuation/Continuation in AnyText.Core/Rules/SequenceRule.cs), and it works correctly for the case: a rule X: XBinary | Atom; with XBinary: left=X op=Op right=Atom;, referenced from elsewhere as a single field. But when a left-recursive rule is used as the left (non-final) operand of a different, non-recursive SequenceRule — e.g. a Comparison: left=NumericExpr operator=RelOp right=NumericExpr; where NumericExpr is itself a left-recursive additive/multiplicative ladder — growth silently fails to parse, while the exact same rule used as the right (final) operand of that same sequence works fine.

Minimal reproduction

The full grammar is in LeftRec.anytext (sibling folder) — a boolean expression ladder (ImpliesExpr → OrExpr → AndExpr → NotExpr → BoolAtom) with BoolAtom including a Comparison alternative whose operands are a numeric ladder (AdditiveExpr → MultiplicativeExpr → NumericAtom):

grammar LeftRec (lr)
root Container

Container:
  'test' value=ImpliesExpr;

ImpliesExpr returns BoolExpr:
  ImpliesBinary | OrExpr;

ImpliesBinary returns BoolBinaryExpr:
  left=ImpliesExpr operator=ImpliesOp right=OrExpr;

enum ImpliesOp returns BoolOperator:
  Implies => '->';

OrExpr returns BoolExpr:
  OrBinary | AndExpr;

OrBinary returns BoolBinaryExpr:
  left=OrExpr operator=OrOp right=AndExpr;

enum OrOp returns BoolOperator:
  Or => '|';

AndExpr returns BoolExpr:
  AndBinary | NotExpr;

AndBinary returns BoolBinaryExpr:
  left=AndExpr operator=AndOp right=NotExpr;

enum AndOp returns BoolOperator:
  And => '&';

NotExpr returns BoolExpr:
  Negation | BoolAtom;

Negation:
  '!' <nsp> inner=NotExpr;

BoolAtom returns BoolExpr:
  Comparison | BoolVarRef;

Comparison:
  left=AdditiveExpr operator=RelOp right=AdditiveExpr;

enum RelOp returns RelOperator:
  Leq => '<='
  Eq => '=';

BoolVarRef:
  name=Identifier;

AdditiveExpr returns NumericExpr:
  AdditiveBinary | MultiplicativeExpr;

AdditiveBinary returns NumericBinaryExpr:
  left=AdditiveExpr operator=AdditiveOp right=MultiplicativeExpr;

enum AdditiveOp returns NumericOperator:
  Add => '+'
  Subtract => '-';

MultiplicativeExpr returns NumericExpr:
  MultiplicativeBinary | NumericAtom;

MultiplicativeBinary returns NumericBinaryExpr:
  left=MultiplicativeExpr operator=MultiplicativeOp right=NumericAtom;

enum MultiplicativeOp returns NumericOperator:
  Multiply => '*';

NumericAtom returns NumericExpr:
  NumberLiteral | NumericVarRef;

NumberLiteral:
  value=Number;

NumericVarRef:
  name=Identifier;

terminal Number returns nmeta.Double:
  /-?[0-9]+(\.[0-9]+)?/;

terminal Identifier:
  /[a-zA-Z_][a-zA-Z0-9_]*/;

Program.cs parses four lines:

Test("test a <= 0.1");        // bare atom, no growth needed
Test("test a - b <= 0.1");    // growth needed on LEFT (non-final in Comparison's sequence)
Test("test 0.1 <= a - b");    // growth needed on RIGHT (final in Comparison's sequence)
Test("test a & b & c");       // growth needed on AndBinary's own left, final field of Container

static void Test(string text)
{
    var grammar = new LeftRecGrammar();
    var parser = grammar.CreateParser();
    var result = parser.Initialize(new[] { text });
    var ok = result != null && !parser.Context.Errors.Any();
    Console.WriteLine($"[{(ok ? "OK" : "FAIL")}] \"{text}\"");
    if (!ok)
    {
        foreach (var e in parser.Context.Errors)
        {
            Console.WriteLine($"    {e.Message} at {e.Position}");
        }
    }
}

Expected

All four lines print [OK].

Actual

[OK] "test a <= 0.1"
[FAIL] "test a - b <= 0.1"
    Unexpected content at (0,0)
[OK] "test 0.1 <= a - b"
[OK] "test a & b & c"

Only the case where the left-recursive rule occupies the non-final field of an unrelated enclosing sequence (Comparison.left) fails. The same rule as the final field of that sequence (Comparison.right) works, and an entirely different left-recursive rule used as the non-final field of its own binary rule (AndBinary.left, growing AndExpr itself) also
works.

Root cause

This bug depends on whether the left-recursive rule starts matching at the same text position as an already-active, unrelated left-recursive rule higher up the grammar.

Setup time is fine. Grammar.FindAllRecursiveRules (AnyText.Core/Grammars/Grammar.cs:84-98) runs once per rule over the whole grammar. For each rule that's left-recursive on itself (CanStartWith(this)), it computes that rule's own
Continuations via AddLeftRecursionRules. Critically, SequenceRule.AddLeftRecursionRules only walks into a nested rule if the container itself is self-recursive:

protected internal override void AddLeftRecursionRules(List<Rule> trace, List<RecursiveContinuation> continuations)
{
    if (!trace.Contains(this) && CanStartWith(this))
    {
        trace.Add(this);
        continuations.Add(new Continuation(this, trace));
        Rules[0].Rule.AddLeftRecursionRules(trace, continuations);
    }
}

Comparison (a plain SequenceRule) doesn't recurse into itself, so CanStartWith(this) is false and this method does nothing for it — the walk starting from ImpliesExpr never reaches AdditiveExpr through Comparison. That's correct and intentional: AdditiveExpr gets its own, separate FindAllRecursiveRules pass later in the same loop (since it wasn't marked IsLeftRecursive by ImpliesExpr's pass), and its Continuations are computed correctly and independently. Both rules' Continuations are correct in isolation.

Match time is where it breaks.
Matcher.MatchOrCreateMatchProcessor only
creates a new RecursionContext when there isn't already one active at the exact same position:

if (rule.IsLeftRecursive)
{
    RecursionContext oldRecursion = recursionContext;
    RecursionContext createdRecursion = null;
    if (recursionContext == null || recursionContext.Position != position)
    {
        recursionContext = new RecursionContext(position, rule.Continuations);
        createdRecursion = recursionContext;
    }
    // ... otherwise, silently reuses the existing recursionContext, whatever it is

Trace what happens parsing "test a - b <= 0.1": matching starts on ImpliesExpr at position P (right after 'test ' is consumed). ImpliesExpr is left-recursive, no active context yet, so a RecursionContext(P, ImpliesExpr.Continuations) is created. Parsing proceeds through OrExpr → AndExpr → NotExpr → BoolAtom → Comparison — none of these consume any input (they're all "try this alternative" ChoiceRules or a SequenceRule whose first field hasn't matched yet), so Comparison.left=AdditiveExpr starts matching at that exact same position P. AdditiveExpr is also left-recursive — but recursionContext != null && recursionContext.Position == P, so the condition above is false: the existing context (holding only ImpliesExpr's continuations) is reused unchanged. AdditiveExpr.Continuations — computed correctly at setup time — is never consulted for this match attempt. AdditiveExpr can match a single atom (the "seed") but has no way to grow past it, so "a - b" fails and only "a" would have been accepted.

Compare with "test 0.1 <= a - b": by the time Comparison.right=AdditiveExpr starts matching, the operator <= has already advanced the position past P, to some P2 ≠ P. Since recursionContext.Position (P) != P2, a new RecursionContext(P2, AdditiveExpr.Continuations) is correctly created, and growth works.

And AndBinary.left=AndExpr (the case that always worked): AndExpr's own continuation is ImpliesExpr's continuation for this purpose — AndExpr is reached through ImpliesExpr's own recursive structure (ImpliesExpr → OrExpr → AndExpr, all part of the same FindAllRecursiveRules walk since each of those is self-recursive), so when AndBinary needs to grow AndExpr at position P, the continuation it needs is already present in the context that was seeded from ImpliesExpr.Continuations — there's no gap to fall into.

Fix

In Matcher.MatchOrCreateMatchProcessor (AnyText.Core/Matcher.cs): when an existing RecursionContext would be reused at the same position, check whether the current rule's own continuations are already present in it. If not, build a new context with the union of both instead of reusing the old one:

if (recursionContext == null || recursionContext.Position != position)
{
    recursionContext = new RecursionContext(position, rule.Continuations);
    createdRecursion = recursionContext;
}
else if (!AllContinuationsPresent(recursionContext, rule.Continuations))
{
    recursionContext = new RecursionContext(position, MergeContinuations(recursionContext.Continuations, rule.Continuations));
    createdRecursion = recursionContext;
}

with two small helpers (AllContinuationsPresent, MergeContinuations) using reference equality over RecursiveContinuation — safe here since each continuation instance is created once at grammar-init time and consistently reused by reference.

Verification

• This repro: all four lines (including the previously-failing "test a - b <= 0.1") now print
  [OK].
• Full AnyText.Tests suite: 214/214 passing, no regressions — important given this touches core
  matching logic used by every left-recursive rule in every grammar, not just this one shape.

Workaround

Restrict the non-final (left) operand of a comparison-like rule to non-recursive alternatives (atoms, function calls, parenthesized sub-expressions) instead of the full expression ladder, while leaving the final (right) operand as the full ladder:

Comparison:
  left=NumericAtom operator=RelOp right=AdditiveExpr;

Compound left-hand operands then need an explicit wrapping paren, e.g. (a - b) <= 0.1 instead
of a - b <= 0.1.

---

Summary by MergeMonkey

• Docs Updates:
  • Fixes left-recursive rules losing their growing continuations when they start matching at a position already claimed by an unrelated left-recursive rule
• Improvements:
  • Left-recursive rules in non-final sequence fields now correctly grow when encountering partial continuation matches at already-claimed positions
• Refactors:
  • Updated changelog with bug fix description

[mergemonkeyhq]

⚡ Risk Assessment — NEEDS-TESTING

Focus areas: left-recursion continuation merging logic · sequence rule parsing behavior · RecursionContext lifecycle

Assessment: Bug fix in parser matching logic for left-recursive rules in sequences - requires testing with various left-recursive grammar patterns.

Walkthrough

When parsing a sequence containing a left-recursive rule in a non-final position, the parser creates a RecursionContext for that rule. If another unrelated left-recursive rule later claims the same starting position but with additional continuations, the new code detects that not all continuations are present and merges them into a new RecursionContext, allowing the original rule to continue growing its matches.

Changes

Files	Summary
Left-recursion continuation merging AnyText/AnyText.Core/Matcher.cs AnyText/AnyText.history	Adds logic to merge missing continuations into existing recursion context when a left-recursive rule matches at a position already claimed by another left-recursive rule, enabling proper growth behavior; updates changelog with bug fix description.

Sequence Diagram

sequenceDiagram
    participant P as Parser
    participant M as Matcher
    participant RC as RecursionContext
    participant Seq as SequenceRule
    participant LR as LeftRecursiveRule
    P->>M: MatchOrCreateMatchProcessor(rule)
    alt First encounter of left-recursive rule
        M->>RC: Create RecursionContext with initial continuations
    else Position already has recursion context
        M->>M: AllContinuationsPresent(recursionContext, rule.Continuations)
        alt Not all continuations present
            M->>RC: MergeContinuations(existing, additional)
            M->>RC: Create new RecursionContext with merged continuations
        end
    end
    M->>P: Return processor with recursion context
    P->>LR: Continue matching with full continuation set

Loading

Dig Deeper With Commands

• /review <file-path> <function-optional>
• /chat <file-path> "<question>"
• /roast <file-path>

Runs only when explicitly triggered.

[georghinkel]

Very cool, thank you 👍

As a bit of history, AnyText started computing continuations on the fly, closely to the paper on left-recursive packrat parsing. However, this turned out incompatible with incremental parsing given that the parser would not see all rules.