fix(paddleocr_vl): make repetition truncation catch multi-line and cap-truncated output

[subin9]

PaddleOCR-VL decodes greedily with no repetition penalty (the local predictor ignores repetition_penalty/temperature/top_p, and generation_config.json sets none), so truncate_repetitive_content() is the only thing that stops a degenerate block from reaching the document. It misses two common shapes.

1. Repetition inside a multi-line block

The phrase-level check runs only when the content has no newline, and the line-level check needs line_threshold (10) near-identical lines. A block that decodes a few valid lines and then loops on the last one passes every check.

2. Output cut in the middle of a character

A generation stopped at max_new_tokens can be cut mid-character; decoding leaves U+FFFD at the end. find_repeating_suffix() anchors on the exact suffix (s.endswith(unit * min_repeats)), so that single character stops every candidate unit from matching. The check silently returns the degenerate text unchanged — on exactly the output it exists for.

Reproduction

No model needed:

from paddlex.inference.pipelines.paddleocr_vl.uilts import truncate_repetitive_content

loop = "abcdefgh " * 200

# works today
truncate_repetitive_content(loop, min_count=50)                      # -> truncated

# 1. multi-line: not truncated
truncate_repetitive_content("line one\nline two\n" + loop, min_count=50)

# 2. trailing U+FFFD: not truncated
truncate_repetitive_content(loop + "�", min_count=50)

min_count=50 is what pipeline.py passes for non-table blocks.

Fix

Run the phrase-level check per line instead of only for single-line content, and drop a trailing replacement character before anchoring. Priority 2 and 3 are unchanged.

Impact

Checked against the recognition output of a full 1649-page OmniDocBench run (78,710 blocks, min_count=50, 5000 for tables):

• 78,681 blocks byte-identical before/after — no behaviour change on normal output.
• 29 blocks newly truncated (17 unique), all degenerate: repeated -10 -10 -10 …, CH2-CH2-CH2 …, a text block looping on the same formula for 5,134 characters. Every one has a newline, so the current code leaves all of them intact.

Related user reports of the symptom: #17288, #17828, #17903 (PaddleOCR).

[CLAassistant]

All committers have signed the CLA.

[paddle-bot]

Thanks for your contribution!