[oetiker]

Refs #202.

In #202, @trishume wrote:

Yah I can see this being useful definitely. I would accept a PR that added a version of the parse function that takes a timeout as a std::time::Duration that just compares the elapsed time against that Duration every Nth parsing operation / regex match (not sure what N should be, depends on how much overhead getting the current time is).

This PR adds that, plus the token count it is built on.

What it adds

• ParseState::parse_line_with_limit(line, syntax_set, limit: Option<NonZeroUsize>). Some(n) permits exactly n tokens. One more returns ParsingError::TokenLimitExceeded { limit }. None is unlimited.
• ParseState::parse_line_with_timeout(line, syntax_set, timeout: Duration). It uses the same counter and reads the clock once every 256 tokens (a private constant), so it can overshoot by up to 255 tokens. When the deadline has passed, it returns ParsingError::TimeLimitExceeded { timeout }.

The count is the primitive and the Duration is layered on top of it. A token count is deterministic, so a test can assert the exact boundary: "abcd\n" parses with a limit of 5 and fails with a limit of 4. A wall-clock limit cannot be asserted that way on a loaded machine. Callers who want reproducible behaviour, for example in their own tests, can use the count. Callers who think in time, as in #202, can use the Duration.

Compatibility

• parse_line keeps its signature. It calls the same code with no limit, so no existing caller changes behaviour.
• ParsingError is #[non_exhaustive], so the two new variants are not a breaking change.
• tests/snapshots/public-api.txt gains the two methods and the two variants, and nothing else.

Details a reviewer may want to check

• Tokens spent re-parsing buffered lines after a cross-line fail (Implement branch_point/branch/fail backtracking #614) count against the limit of the parse_line call that triggered the replay. Without this, a replay could run without a limit.
• The timeout does not stop a single regex search that takes a long time. It stops the loop between tokens. This is the same scope as the proposal in long-term feature request: ability to specify a timeout for highlighting #202.
• If an error is returned, the state is not meaningful to continue from. The rustdoc says to keep a clone from before the call if you need to resume.
• The counter is threaded through exec_pattern and handle_fail, which adds one argument to each. Both already had a clippy::too_many_arguments warning before this change.

Tests

Five new tests in parsing::parser::tests. Four use a small syntax that produces one token per character:

• a limit equal to the line's cost parses,
• a limit one below it returns TokenLimitExceeded { limit: 4 },
• None returns the same ops as parse_line,
• on a line of 512 characters, a zero timeout returns TimeLimitExceeded, a one-hour timeout parses, and Duration::MAX returns the same ops as parse_line.

The fifth uses a cross-line branch_point: line 2 fits a limit of 20 on its own, but the fail replays 100 characters of line 1, so the call returns TokenLimitExceeded.

cargo test --all-features: 194 lib (2 ignored), 8 error_handling, 1 public_api, 14 doctests pass. On master the lib count is 189.

Why now

A second project has hit a hang of this kind: mdmost, a Markdown pager, hung on two lines of JavaScript. See my comment on #706: #706 (comment). #706 fixes that specific loop. This PR gives callers a limit for loops that no guard detects yet.