[jfenton]

Ran into an infinite spin using grok build on a React-to-Sveltejs port of something, and traced it to Syntect's parser entering an infinite loop on two kinds of cyclical grammar references.

Examples:

contexts:
main:
- include: loop
loop:
- include: main
- match: '.*'

Explanation: Cyclic include. main includes loop, and loop includes main, so the include stack grows forever.

and

contexts:
main:
- match: '(?=.)'
set: main
- match: '.*'

Explanation: Zero-width set. (?=.) matches without consuming a character, then set: main returns to the same context. The cursor stays on the first byte and the scope-op list grows forever.

Causal root:

MatchIter follows includes by pushing the target context and walking its patterns. Nothing checks whether that context is already on the include stack, so a context that includes itself - directly, or through another context - pushes forever. The fix skips an include whose context pointer is already on ctx_stack.

The other loop is in parse_line. A match that does not consume input is fine in isolation (a lookahead push, for example). Upstream already bails out when a non-consuming pop happens at the same stack depth as an earlier non-consuming push. A non-consuming set however, or a push/pop whose depths do not line up, misses that check.

Coverage:

Tests cover a grammar whose contexts include each other, and one whose only rule is a zero-width set back to itself. Both used to hang inside parse_line.

[yxxbc]

I can confirm this fix resolves a real-world hang, independently of the grok case.

A Svelte diff wedged a TUI (a Rust app on syntect 5.3.0 + two-face's onig dump) on a single 5-byte line, /**. /usr/bin/sample showed the main thread spinning forever:

ParseState::parse_line → MatchIter::next → RawVec::grow_one → realloc
ParseState::parse_line → context_iter  → onig_region_copy / onig_region_free

The grammar (bat's Svelte, from corneliusio/svelte-sublime) has cyclical includes:

style-common  → style-lang-attribute → style-lang-decider-type → style-common
style-common  → style-lang-attribute → style-lang-decider-lang → style-common
script-common → script-lang-attribute → script-lang-decider-lang → script-common
script-lang-decider-type → script-common → script-lang-attribute → …

Only svelte hangs — html / javascript / typescript / vue parse the same 401 lines in < 10 ms. Capping oniguruma's retry_limit_in_match does not help (down to 20 000 still spins): the loop is in the include walk, not in backtracking.

Both halves are needed. With only the MatchIter::push_include cycle check, /** still loops (that's the non-consuming-set case). Adding the non_consuming_streak guard in parse_line fixes it — the same file then parses in ≤ 20 ms.

I ported both onto the released v5.3.0 (master's parse_line now returns ParseLineOutput, so this branch can't be consumed by 5.3.0 users as-is) and verified end to end: resuming the wedged session now settles to 0 % CPU.

Would be great to get this merged — it affects every syntect consumer that highlights Svelte.

[yxxbc]

Verified against a real Svelte hang; both halves are required. See comment.

[keith-hall]

I believe this has been solved in a different way by #708, please check if that solution works for you or whether we still need this.