Skip to content

add flux-klein example top of trickle protocol#32

Open
zargarzadehm wants to merge 1 commit into
livepeer:mainfrom
zargarzadehm:add-flux-klein
Open

add flux-klein example top of trickle protocol#32
zargarzadehm wants to merge 1 commit into
livepeer:mainfrom
zargarzadehm:add-flux-klein

Conversation

@zargarzadehm

Copy link
Copy Markdown

Add flux-klein — realtime FLUX.2-klein-4B video-to-video over trickle

Adds a new example implementing the request in #1: realtime FLUX.2 [klein] img2img
over trickle on the live runner, modelled on the existing trickle/static-runner
examples.

App id livepeer-example/flux-klein
Registration static (runners.json + /health poll, like vllm)
Mode persistent
Transport trickle (video in/out + /update control)
Port 8720
Requires NVIDIA GPU, ~16 GB VRAM

What it does

Receives a live video stream, transforms every frame against a text prompt with
FLUX.2-klein-4B, and streams the result back. Prompt, noise seed, and camera
anchoring are all adjustable mid-stream over /update.

Self-contained inference. flux_klein.py wraps diffusers
(Flux2KleinPipeline) directly — it does not run Daydream Scope or any plugin
runtime. The reference implementation cited in #1,
hthillman/scope-flux-klein, is a
Scope plugin; this example borrows its recipe (the Krea-style feedback loop and
the truncated-schedule refine) and reimplements it standalone so the example stays
self-contained like the others.

The feedback loop. Klein is a 4-step model, so full inference per frame is far
too slow for video. Frame 0 runs full inference; every frame after partially
denoises the previous output blended with the incoming frame. Because
Flux2KleinPipeline has no strength parameter (it conditions on the image via
joint attention), the refine hand-rolls a truncated flow-matching schedule.

Measured throughput

NVIDIA RTX 6000 Pro, 384×384, 4 steps, feedback 0.5:

Config fps
defaults ~19
FLUX_BATCH=2 FLUX_COMPILE=max-autotune ~22.7

GET /stats reports live input/inference/output rates, frames_skipped, and the
active tuning parameters.

Files

flux_klein.py (model + feedback loop) · runner.py (aiohttp app, trickle
channels, worker) · client.py (file or webcam in, live prompt/seed/blend updates)
· Dockerfile · docker-compose.yml + docker-compose.onchain.yml ·
runners.json · pyproject.toml · .env.example · view.sh · README.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant