SoundFx — automatic sound design for video edits

Drop a video in, get it back with sound effects from your own SFX library placed where the picture asks for them — whooshes on whip pans, sub impacts on flashes and zoom punches, risers into drops, glitches on stutters — with the video stream copied bit-for-bit (no re-encode).

▶ Watch the demo with sound (MP4) · 9 s car edit, 23 sounds placed automatically, video stream untouched

• Works fully offline with a rule engine (no LLM needed).
• Optional AI vision pass (any OpenAI-compatible vision endpoint, e.g. NVIDIA Build) adds foley and VFX sounds: footsteps, sparks, holograms, explosions, text pops.
• Web UI (SoundFx.bat / python app.py) and CLI.
• Consistent palettes: sounds are chosen by measured character (sub weight, attack, peak position, brightness), not by random file names.

How it works

1. Analyze — scene cuts (PySceneDetect), per-frame motion/zoom/brightness (OpenCV optical flow), beat grid and loudness (librosa). Every cut is classified: flash, blur_in, blur_out, zoom_in, zoom_out, to_light, to_dark, stutter, hard, plus the motion profile of the next shot.
2. Understand — timing rules distilled from pro references and measured edits (.claude/skills/sfx-edit/reference.md): whoosh peaks on the fastest frame and lasts as long as the move, hits land ~50 ms after the event, flash frames get a sub impact only, at most two layers per event, whooshes on ≤60 % of transitions. Optional VLM pass reads each shot for physical actions and VFX.
3. Select — the library is indexed once (library/index.json: duration, peak, attack, spectral centroid, sub ratio, flatness, sync point). Palette roles (impact_deep, impact_swell, whoosh_dark, whoosh_quick, riser, glitch, …) pick 2–3 consistent variants per role.
4. Mix — each SFX is peak-normalized, placed sample-accurately (transient on the frame), music is ducked 2–4 dB under impacts, a look-ahead limiter keeps peaks at −1 dBFS, output is normalized to −14 LUFS (or kept at the original level). FFmpeg muxes the new audio with -c:v copy; the video stream MD5 is verified to be identical.

Install

git clone https://github.com/SpicesFire/SoundFx && cd SoundFx
python -m pip install -r requirements.txt

Requirements: Python 3.11+, FFmpeg on PATH (Windows: winget install Gyan.FFmpeg). Optional GPU is only used by Whisper speech analysis.

Put your SFX packs into library/ (any folder structure; wav/mp3/ogg/flac). The packs are not part of this repo. Then index:

python sfx.py index

Use

Web UI

python app.py          # opens http://127.0.0.1:8765

Pick a video (or upload into input/), choose packs / loudness, press Analyze & place sound effects. You get a 480p preview, the full-quality output in output/, and the cue table with the reason for every sound.

CLI

python sfx.py auto input/edit.mp4 --packs jerry --lufs -14        # one shot
python sfx.py analyze input/edit.mp4                               # report + auto cue sheet only
python sfx.py apply input/edit.mp4 work/edit/cuesheet.json --preview
python sfx.py find -c whoosh -t quick --max-dur 0.6                # search the library
python sfx.py palette                                              # show the role palette
python sfx.py verify input/edit.mp4 output/edit_sfx.mp4            # video stream identical?

AI vision (optional)

set SFX_VISION_API_KEY=...                 # NVIDIA Build key or any OpenAI-compatible endpoint
set SFX_VISION_BASE_URL=https://integrate.api.nvidia.com/v1
set SFX_VISION_MODEL=meta/llama-3.2-90b-vision-instruct
python sfx.py auto input/story.mp4 --vision

Cue sheet

work/<video>/cuesheet.json is plain JSON you can edit and re-apply:

{"video": "input/edit.mp4",
 "master": {"duck": true, "limiter": true, "target_lufs": -14, "packs": ["jerry"]},
 "cues": [
  {"t": 2.65, "query": {"role": "whoosh_dark"}, "gain_db": -8, "note": "whip pan, peak on fastest frame"},
  {"t": 2.76, "query": {"role": "impact_deep"}, "gain_db": -6, "trim": [0, 1.6], "fade_out": 0.4},
  {"t": 5.33, "query": {"category": "riser", "pack": "social"}, "align": "end", "gain_db": -11},
  {"t": 1.02, "sfx": "foley_footstep_concrete_1", "gain_db": -12}
 ]}

align: sync (file's transient lands on t, default), start, end. Also fade_in, fade_out, trim, speed, pitch, duck.

Project layout

sfx/            analyze · transitions · palette · library · cuesheet · mix · vision · auto · cli
app.py, web/    Flask UI
knowledge/      timing rules, visual-effect → sound mapping, reference notes
library/        your packs (git-ignored)   work/ output/ input/  (git-ignored)

License

MIT. Sound packs are not included and remain under their own licenses.