[PaddleOCR-VL 1.6] Proposal: reduce eager-attention VRAM from ~8.35 GiB to ~4.3–4.6 GiB with query chunking

[DK6WT]

Summary

I investigated PaddleOCR-VL 1.6 GPU-memory usage on Windows/NVIDIA and found a way to substantially reduce both model-loading and inference VRAM requirements.

On my test system, the approximate PaddleOCR-VL workload increase was reduced from:

~8.35 GiB

to approximately:

~4.3–4.6 GiB

The measured total GPU-board peak in the final configuration was:

5504 MiB

with approximately 0.9–1.2 GiB already occupied by Windows/WDDM/background processes.

The largest inference-side improvement comes from query-chunked global eager attention.

This is not window attention: every query token still attends to the complete K/V sequence.

---

Environment

OS: Windows / NVIDIA WDDM
GPU: RTX 3090 24 GB
PaddlePaddle GPU: 3.2.2
CUDA runtime: 11.8
PaddleOCR: 3.7.0
PaddleX: 3.7.2
Model: PaddleOCR-VL 1.6 / 0.9B

---

Problem

For the PaddleOCR-VL demo image, the vision attention input was:

query shape = [1, 16, 4884, 72]
dtype       = paddle.bfloat16

The eager attention path calculates the full attention matrix:

attn_weights = paddle.matmul(
    x=query.scale(scaling),
    y=key,
    transpose_y=True,
)

attn_weights = attn_weights.cast(paddle.float32)

For this input, the FP32 [1,16,4884,4884] tensor alone is approximately 1.42 GiB.

With additional tensors, softmax intermediates and allocator workspace, this causes a much larger temporary VRAM peak.

Instrumented vision-stage measurements:

Unchunked eager attention:

start allocated: 1847.4 MiB
peak allocated:  4826.5 MiB
peak reserved:   5704.7 MiB

---

Query-chunked global attention

I changed only the query dimension to be processed in blocks:

if not module.training and attention_mask is None:
    query_chunk_size = 1024
    outputs = []

    query_length = query.shape[2]

    for start in range(0, query_length, query_chunk_size):
        end = min(start + query_chunk_size, query_length)

        query_chunk = query[:, :, start:end, :]

        attn_weights = paddle.matmul(
            x=query_chunk.scale(scaling),
            y=key,
            transpose_y=True,
        )

        attn_weights = attn_weights.cast(paddle.float32)
        attn_weights = F.softmax(attn_weights, axis=-1)
        attn_weights = attn_weights.cast(origin_dtype)

        attn_output_chunk = paddle.matmul(attn_weights, value)
        outputs.append(attn_output_chunk)

        del attn_weights
        del attn_output_chunk

    attn_output = paddle.concat(outputs, axis=2)
    attn_output = attn_output.transpose((0, 2, 1, 3))

    return attn_output, None

The K/V tensors remain complete.

Conceptually:

Q[0:1024]       -> all K/V tokens
Q[1024:2048]    -> all K/V tokens
Q[2048:3072]    -> all K/V tokens
...

Therefore the attention remains global.

With query chunking, the instrumented vision peak dropped to:

peak allocated: 2228.9 MiB
peak reserved:  2446.5 MiB

This is a reduction of approximately:

2597.6 MiB

in active vision-stage peak allocation.

---

End-to-end VRAM measurements

Stock configuration

Observed total board peak: ~9357 MiB
Background usage:          ~1006 MiB
Approx. workload increase: ~8351 MiB

Optimized, query chunk 1024

Observed total board peak: 5504 MiB
Background usage:          ~0.9–1.2 GiB
Approx. workload increase: ~4.3–4.6 GiB

The GPU was sampled with nvidia-smi approximately every 250 ms.

---

Processing-time measurements

Same local demo image, three runs each.

Unchunked

42.583 s
42.399 s
47.115 s

Average: 44.03 s

Chunk 512

46.350 s
43.526 s
45.600 s

Average: 45.16 s

Chunk 1024

41.411 s
41.986 s
42.031 s

Average: 41.81 s

I am not claiming that chunk 1024 is generally faster from this small benchmark.

The important result is that there was no obvious performance penalty while VRAM usage decreased substantially.

---

OCR output

The same image completed successfully with:

• unchunked eager attention
• query chunk 512
• query chunk 1024

I did not observe visible OCR degradation on this test image.

This is not yet a full accuracy benchmark, so I am not claiming dataset-level equivalence.

---

Other changes used in the low-memory configuration

The final test configuration also included:

1. model.eval() for inference
2. lower-memory safetensors loading through CPU/NumPy
3. avoiding the FP32 vision-module workaround on NVIDIA
4. empty_cache() after model loading and after the vision encoder

I think these should be reviewed independently upstream.

The query-chunked eager-attention change can be isolated into a small patch.

---

Possible upstream implementation

Instead of hard-coding a chunk size, PaddleX could expose an optional setting:

attention_query_chunk_size = None

For example:

None -> current behavior
512  -> lower-memory inference
1024 -> lower-memory inference

or activate chunking only for long sequences:

if (
    not module.training
    and attention_mask is None
    and query.shape[2] > threshold
):
    ...

This would leave the existing behavior unchanged by default.

---

Related PaddleOCR OOM report

There is an existing PaddleOCR Windows OOM report:

PaddlePaddle/PaddleOCR#17349

The failure there occurs during GPU model loading.

The low-memory loading changes I tested address that issue separately from the large eager-attention allocation described above.

---

Question

Would the PaddleX maintainers be interested in an isolated PR implementing optional query-chunked global eager attention for PaddleOCR-VL inference?

I have preserved the tested patch and benchmark data and can provide a minimal diff.