A research handbook and practical learning guide to video summarization: classical and supervised methods, weak/semi/self-supervision, unsupervised and reinforcement learning, training-free selection, multimodal foundation models, and query or user-conditioned systems. It covers keyframes, keyshots, textual and multimodal summaries, with source-linked code, datasets and protocol-aware benchmarks.
Last literature audit: 2026-09-19. Coverage is selective and dated; the ledger and latest audit distinguish newly verified entries from inherited technical audits. Source availability does not imply successful reproduction.
Start learning: Learning path · Find a paper: Catalog · Find code: Implementations · Compare results: Benchmarks
| Axis | Examples |
|---|---|
| Supervision | Supervised, weak/semi/self-supervised, unsupervised, training-free, zero/few-shot |
| Output | Storyboard, keyshot skim, highlight, text, multimodal, timestamp/event summary |
| Setting | Generic, query-focused, personalized, multi-video, egocentric, long-form, online |
| Mechanism | Clustering, sparse/submodular/DPP, recurrent, reconstruction/GAN, RL, graph/attention, contrastive, VLM/LLM, diffusion |
These axes are independent. The four original unsupervised families remain a sub-taxonomy. Supervised methods are core reading, including methods whose frozen encoders or auxiliary contrastive losses can obscure their human-label dependence.
SumMe and TVSum support classic importance-to-skim studies; VideoXum supports visual and textual summaries; query and egocentric resources have their own annotations and metrics. MoSu adds behavior-derived multimodal importance targets, TRINITY separates Event/Emotion/Nature saliency, KnowVis generates pedagogical images, and Multi-VidSum pairs keyframes with captions. QA and grounding resources remain explicitly adjacent. See dataset cards for licensing, availability, features and unresolved fields.
A score is meaningful only with its split, training data, features, sampling, segmentation, shot values, budget solver, reference aggregation and metric. The generated benchmark catalog isolates unknown/incompatible protocols and distinguishes author-reported values from reproduced ones. It does not assert a field-wide winner.
Recent starting points include TRINITY, SGWIB, metadata-conditioned zero-shot highlight detection, KnowVis, agentic narrative summaries, multimodal sports systems, TripleSumm and CoE. The modern-method chapter explains their actual training signals and output differences; the latest audit preserves count conflicts, restricted sources and withheld numerical claims.
No model or dataset download is needed for the synthetic decoder example:
python3 examples/summary_baselines.py --demoChange shot lengths, scores and budget to see how knapsack and user-reference aggregation affect the output. Continue with the learning path and source-audited resources, including official implementations, author presentations, tutorials, feature archives and checkpoints.
CONTRIBUTING.md defines evidence, schemas and validation. Edit registries under data/, then run python3 scripts/generate_catalog.py. The documentation workflow checks schemas, generated files, Markdown, mathematics, Mermaid and tests. The weekly external-link workflow reports restricted access separately from missing links.
Use CITATION.cff to cite this handbook with the revision you used, and cite original papers/datasets for their methods and results. The repository is MIT licensed; linked code, datasets, media and model weights retain their own licenses.