Fix progressive slowdown in text streaming across generations

[auroter]

Summary

Consecutive generations in the Gradio web UI get progressively slower in the text streaming (thinking/recaption) phase. The model generates tokens at a constant rate (~24 tok/s), but each yield history sends the full chatbot history to Gradio for serialization. Two compounding issues cause this:

• ~200 yields per generation: Every single text token triggers a yield history, and serialization cost grows with history size (O(tokens × images))
• PIL re-encoding on every yield: gr.Image(pil_image, type="pil") objects in history get re-encoded to PNG by Gradio on every yield — even for unchanged old messages (Gradio creates a new component instance each time in chatbot.py:557). This affects both generated images and user-uploaded images.

A 1024px PNG encode takes ~100ms. With N prior images and ~200 yields, this adds N×20 seconds of overhead per generation, making text streaming visibly degrade from fast to ~1 word/second after just 2-3 generations.

Changes (single file: app/run_chatbot.py)

• Throttle yields to 50ms intervals (~20-40 per generation instead of ~200). Text is still buffered and flushed before image events.
• Store file paths in gr.Image instead of PIL objects. Generated images are saved to temp files first; user-uploaded images are already file paths from Gradio's MultimodalTextbox, so dropping type="pil" avoids an unnecessary PIL round-trip. File path serialization is nearly free vs. re-encoding.
• Close the file descriptor from tempfile.mkstemp() to prevent fd exhaustion after many generations.

Together these reduce per-generation overhead from O(tokens × images) to roughly O(1).

Measured results

Generation	Model tok/s	Avg yield time (before fix)	Avg yield time (after fix)
1st	24.0	2ms	2ms
2nd (1 image in history)	24.6	~500ms+ (unthrottled)	72ms
3rd (2 images in history)	24.5	~1000ms+ (unthrottled)	129ms

Model speed is constant — the slowdown was entirely in the Gradio serialization layer.

Test plan

• Run 3+ consecutive text-to-image generations with single_round + think mode
• Confirm text streaming speed stays consistent across all generations
• Confirm images still display correctly in the chatbot
• Confirm --image-cache-dir still saves images correctly
• Test with unlimited context mode to verify full history still works
• Test image-to-image (user uploads an image) to confirm uploaded images display correctly

🤖 Generated with Claude Code