I-frame and random-sampling modes are not implemented

[BlohoJo]

"At inference time, SparkVSR supports flexible keyframe selection (manual specification, codec I-frame extraction, or random sampling) and a reference-free guidance mechanism that continuously balances keyframe adherence and blind restoration, ensuring robust performance even when reference keyframes are absent or imperfect."

I'm not seeing that.

It appears that only "manual" works.

If --ref_indices is left out, then it works in a very simple "automatic" kind of mode, returning only the first, middle, and last frame of the clip. (The _select_indices function in finetune/utils/ref_utils.py literally returns [0, total_frames // 2, total_frames - 1].)

Any idea as to when the much better I-frame and random sampling modes will be implemented?

Or is that never going to happen?

[BlohoJo]

Personally, I would recommend PySceneDetect to create the indices, its adaptive mode should work well with SparkVSR.

(pip install scenedetect[opencv])

Here's a sample python script for it that generates frame numbers for scene changes that can be used as indices in SparkVSR.

Usage:

• Basic: python detect_scenes.py video.mp4
• Save to file: python detect_scenes.py video.mp4 --output scenes.txt
• More sensitive (Content detector): python detect_scenes.py video.mp4 --detector content --threshold 20

#!/usr/bin/env python3
import argparse
import sys
from scenedetect import detect, AdaptiveDetector, ContentDetector


def main():
    parser = argparse.ArgumentParser(
        description="Detect scene changes in an H.264 MP4 video and output the list of frame numbers "
                    "where each new scene begins (0-based indexing)."
    )
    parser.add_argument("input_video", help="Path to the input MP4 video file")
    parser.add_argument(
        "-o", "--output",
        help="Optional text file to save the frame numbers (one per line)"
    )
    parser.add_argument(
        "--detector",
        choices=["adaptive", "content"],
        default="adaptive",
        help="Detection algorithm (adaptive is usually best; content is the classic option)"
    )
    parser.add_argument(
        "--threshold",
        type=float,
        default=27.0,
        help="Sensitivity threshold (lower = more scenes). Only used with --detector content"
    )
    parser.add_argument(
        "--min-scene-len",
        type=int,
        default=15,
        help="Minimum scene length in frames (avoids detecting tiny cuts)"
    )
    args = parser.parse_args()

    print(f"🔍 Detecting scenes in {args.input_video}... (this may take a while for long videos)")

    # Choose detector
    if args.detector == "adaptive":
        detector = AdaptiveDetector(min_scene_len=args.min_scene_len)
    else:
        detector = ContentDetector(threshold=args.threshold, min_scene_len=args.min_scene_len)

    # Run detection (show_progress gives a nice progress bar)
    scene_list = detect(
        args.input_video,
        detector,
        show_progress=True
    )

    # Extract start frame numbers for every new scene
    # scene_list = [(start_FrameTimecode, end_FrameTimecode), ...]
    scene_start_frames = [scene[0].frame_num for scene in scene_list]

    print("\n✅ Done! Frame numbers where new scenes begin:")
    print(scene_start_frames)

    # Optional: save to file
    if args.output:
        with open(args.output, "w") as f:
            for frame in scene_start_frames:
                f.write(f"{frame}\n")
        print(f"📁 Saved frame list to {args.output}")

    return scene_start_frames


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python detect_scenes.py your_video.mp4 [--output frames.txt] [--detector content] [--threshold 30]")
        sys.exit(1)
    main()

detect_scenes.py

[zhouyizhuo]

@BlohoJo @Raymond-Qiancx @Jiongze-Yu @sparkvsr I-frame and random-sampling modes are not implemented

[zhouyizhuo]

@BlohoJo Hi, did u test further? will the PySceneDetect produce better results?

[Jiongze-Yu]

@zhouyizhuo Thanks for pointing this out!

Yes, our current inference code only includes the manually selected keyframe setting. We will add support for random keyframe selection and I-frame-based selection ASAP.
We are also actively working on more advanced keyframe selection strategies to further improve the usability and performance of SparkVSR. Please stay tuned for future updates in this repo.

Also, thanks to the community developer @BlohoJo for the support and contribution!