Skip to content

USER_GUIDE.md

Latest commit

 

History

History
237 lines (171 loc) · 9.76 KB

USER_GUIDE.md

File metadata and controls

237 lines (171 loc) · 9.76 KB
Raw

KleinFlow User Guide

This guide explains the purpose and cost of each major KleinFlow UI section. For exact JSON field names and limits, use the generated Settings Reference.

Core Components

KleinFlow core components validates the transformer, text encoder, tokenizer, VAE, and scheduler. It also reports optional model availability.

  • Qwen text encoder selects the default Ozzy 4-bit encoder or a manually installed Unsloth fallback.
  • VAE decoder selects the standard FLUX.2 VAE or optional Small Decoder.
  • Prompt token length trades prompt capacity for encoding time and memory.
  • Low-VRAM mode pre-encodes prompts and unloads Qwen before diffusion.

Use the standard VAE first. Treat alternate decoders as a visual and performance choice, not as a guaranteed quality upgrade.

Render Controls

The Render area defines seed, frames, FPS, resolution, img2img strength, motion mode, and output name. Presets auto-apply width and height, but width and height remain editable.

  • Frame zero is text-to-image.
  • Later frames use the prepared prior frame as img2img input.
  • Strength controls how far a new frame may depart from that input.
  • 2D Motion uses affine transforms; 3D Motion uses camera/depth controls.

LoRA Loader

Up to eight fixed LoRA slots can be selected from models\loras. LoRAs are compatibility-checked before loading. Multiple compatible adapters stay loaded and use independent strengths. Start with one LoRA and a moderate weight before stacking adapters.

Flux2Klein Enhancer

The Enhancer supplies prompt-side controls rather than another diffusion model:

  • Profiles apply coordinated enhancer settings.
  • Prompt embedding changes embedding emphasis and enhancement.
  • Sectioned prompt separates subject, environment, composition, lighting, style, and detail intent.
  • Detail controller shapes detail emphasis without changing the four-step scheduler.

2D Motion Schedules

  • Presets generates schedules for common moves.
  • Schedules exposes angle, zoom, translation X, and translation Y.
  • Edge Fill controls reflected or blurred content outside transformed edges.
  • Motion Preview visualizes the camera path before diffusion.

Schedule syntax accepts keyframes and supported expressions. Keep cumulative zoom and translation modest on long renders to avoid repeated edge repainting.

3D Camera And Depth

Camera motion

Camera presets and schedules control 3D translation, rotation, FOV, overscan, perspective, vanishing point, blur, and smoothing. This mode can run as a lightweight projective fake-3D effect without a depth model.

Depth-Lite 3D

Depth Anything V2 Small estimates depth at a selectable analysis resolution. Temporal fusion aligns recent maps, combines confidence, and divides the scene into depth layers for parallax. Depth can come from generated or video-reference frames. Cache intervals reduce repeated depth inference.

Depth Warp 2.0

Depth Warp 2.0 projects depth into a lightweight Gaussian/surfel representation with a native PyTorch CUDA engine. Projection density, splat behavior, recovery, hole filling, and diagnostics are configurable. It is more expensive than camera-only motion and should be introduced after a stable baseline render.

Video Reference And Hybrid

  • Profiles applies coordinated reference behavior.
  • Reference Setup selects the video, extraction, frame alignment, and range.
  • Hybrid Blend controls how reference color or image content enters frames.
  • Motion & Flow controls OpenCV/SEA-RAFT flow, cache, confidence, occlusion, scene resets, and adaptive strength.
  • Preview & Output exposes extracted frames, masks, maps, and timelines.

Precomputing reusable flow isolates learned flow work from FLUX generation and improves repeatability. Scene reset maps prevent propagation across detected cuts.

Semantic Masking

Semantic Masking uses optional Florence-2 phrase grounding to locate a named subject in generated or reference frames. KleinFlow converts the grounded box into a soft or GrabCut-refined mask, then propagates it between detection frames with SEA-RAFT or OpenCV flow.

  • Detection defines the target phrase, re-grounding interval, size limits, and missing-subject behavior.
  • Mask controls refinement, expansion, feathering, and inversion.
  • Tracking selects learned or portable optical-flow propagation.
  • Routing applies the mask to supported diffusion, init preparation, color, hybrid-reference, or Final Polish operations.
  • Preview inspects masks before committing to a long render.

Start with a concrete phrase such as the person in a red coat. Use shorter re-grounding intervals for fast subject motion and enable diagnostics while tuning mask thresholds.

Diffusion Cadence

Cadence skips selected diffusion frames and fills them by warping, trading detail renewal for speed.

  • Profiles provides tested combinations.
  • Cadence Mode selects Fixed, Adaptive, or VeloSeq Scheduled behavior.
  • Prompt Triggers forces diffusion around meaningful prompt changes.
  • Burst Frames adds temporary high-quality windows.
  • Timeline Preview shows which frames will diffuse or warp.

Use cadence 1 while tuning motion. Increase it only after the base animation is stable.

Temporal Consistency

  • Profiles replaces older one-click buttons with coordinated settings.
  • Core controls consistency mode, seed behavior, base-frame coherence, and prompt smoothing.
  • Seed Travel interpolates starting noise between seeds and can synchronize prompt-embedding travel.
  • Adaptive reacts to motion and visual change.
  • Anchor & Resets prevents accumulated drift and resets around transitions.
  • Detail Refresh periodically restores texture lost through repeated img2img.

Temporal Overlap Renderer

Overlap rendering uses neighboring context windows and reinjection to improve continuity. Staggered passes and multi-frame motion memory can add substantial render cost. The UI displays an estimated cost before generation.

Temporal Memory Pyramid

The memory pyramid stores short-, medium-, and long-term visual summaries and fuses them back into frame preparation. It is placed between overlap rendering and init-frame preparation so it can stabilize long sequences without changing the base FLUX model.

Predictive Motion Guidance

FloweR uses four completed frames to predict image-space motion, occlusion, and a candidate next frame. KleinFlow blends that prediction into img2img preparation; it does not inject unsupported latents or modify FLUX attention.

Init-Frame Preparation

  • Profiles applies complete preparation strategies.
  • Basic controls contrast and enable state.
  • Detail controls sharpening and detail recovery.
  • Noise controls noise amount, type, and content-aware behavior.
  • Adaptive adjusts preparation from frame health and motion.
  • Health prevents feedback collapse, clipping, or runaway degradation.

Color Match Options

Color matching can stabilize palette drift independently in luminance and chroma channels. Temporal EMA smoothing reduces sudden palette changes, while protection limits saturation and clipping. Use Preview to compare settings before a long render.

Final Polish

Final Polish performs an optional final Klein refinement and image-space detail pass with protection controls. It improves selected details but adds another pass and can change temporal texture. Preview it before enabling it for a long sequence.

Frame Interpolation

RIFE creates intermediate frames after generation.

  • Timing selects multiplier or target FPS.
  • VRAM controls device and precision policy.
  • Output controls frame/video retention and export behavior.

Interpolation changes playback smoothness, not source animation detail.

FlowUpscaler 2x

FlowUpscaler is an optional post-render 2x pass using the compatible checkpoint available through download_models.bat. It is installed as models\upscale\flow_upscaler.safetensors. It runs after interpolation in the current post-processing order, so the upscaler receives the final temporal frame set.

Video Export

Video Export controls H.264 CRF, encoder preset, bitrate cap, profile, and pixel format. CRF is the primary quality control: lower values retain more quality and produce larger files. The default CRF 20 is a balanced portable result.

Preset Library

Presets are read from presets\builtin and presets\user. A preset declares the settings it updates; applying it leaves unrelated settings unchanged. The source and update list are shown before application.

Audio Reactive

  • Source selects audio/video, segment start/end, setup persistence, and final audio muxing.
  • Analyze Audio runs Enhanced DSP V2, displays feature groups, caches analysis, and maps signals into animation values.
  • Generate VeloSeq creates a new audio-driven manifest.
  • Inject Into Existing VeloSeq overwrites only selected fields, preserving prompts unless prompt fields are selected.

Audio analysis uses bundled dependencies and FFmpeg. It extracts normalized energy, frequency-band, onset, transient, beat, stereo, and rhythm features.

VeloSeq Scheduling

  • Manifest edits JSON and loads examples.
  • Interpolation sets the default numeric transition curve.
  • Expressions injects formulas into selected fields.
  • Timing selects frames, seconds, or beats and configures BPM/sync offset.
  • Preview evaluates a table and curves without rendering.

Read the VeloSeq Manual before writing complex expressions.

Settings JSON

Settings can be exported as a downloadable JSON file, loaded from the editor, loaded from an uploaded file, or restored from the last saved session. Exported JSON contains generator metadata and every persisted RenderSettings field.