Repository contains easy to use Python implementation of "Cut and paste" augmentation for object detection and instance and semantic segmentations. The main idea was taken from Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation and supplemented by the ability to add objects in 3D in the camera coordinate system using a Bird's Eye View Transformation (BEV). Optional wrappers are available for Albumentations and Torchvision.
The package is published on PyPI:
pip install cap-augmentationOptional integrations are installed as extras:
pip install "cap-augmentation[albumentations]" # CapAlbumentations wrapper
pip install "cap-augmentation[torchvision]" # CapTorchvision wrapper
pip install "cap-augmentation[histogram]" # histogram_matching=True support
pip install "cap-augmentation[viz]" # visualization helpers (matplotlib)
pip install "cap-augmentation[dataset]" # dependencies for dataset_tools/ scriptsTo install several extras at once:
pip install "cap-augmentation[albumentations,torchvision,histogram,viz]"Clone the repository and install in editable mode with the test and developer extras:
git clone https://github.com/RocketFlash/cap-augmentation.git
cd cap-augmentation
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip setuptools wheel
python -m pip install -e ".[test,dev,torchvision]"
pytest
black --check src tests dataset_tools
ruff check src tests dataset_toolsfrom cap_augmentation import (
CapAug, # core cut-and-paste augmenter
CapAugMulticlass, # combine per-class CapAug instances
CapAlbumentations, # Albumentations DualTransform wrapper
CapTorchvision, # torchvision v2-style wrapper
ImageMaskTransform, # adapter for per-object (image, mask) callables
resize_keep_ar, # aspect-ratio-preserving resize helper
seamless_blend, # Poisson (cv2.seamlessClone) blend of a PNG over a background
)The wrapper classes require their respective extras (albumentations,
torchvision).
Examples are available as notebooks:
When a badge is opened, Colab loads the notebook from GitHub. Run the first
Colab setup cell before the rest of the notebook; it clones this repository
into /content/cap-augmentation, switches the working directory there, and
installs the package with the extras used by the demos. Dataset-specific cells
still expect the corresponding generated data under data/ or a mounted
external dataset path.
from pathlib import Path
import cv2
from cap_augmentation import CapAug
# Any list of PNG paths with an alpha channel works as a source.
# Typical workflow: generate cutouts with dataset_tools/cityscapes/
# (see "Data preparation" below) and glob them:
#
# SOURCE_IMAGES = sorted(Path("data/human_dataset_filtered").glob("*.png"))
SOURCE_IMAGES = ["path/to/source1.png", "path/to/source2.png"]
image = cv2.imread("path/to/the/destination/image")
cap_aug = CapAug(
SOURCE_IMAGES,
n_objects_range=[10, 20],
h_range=[80, 120],
x_range=[500, 1500],
y_range=[600, 1000],
coords_format="xyxy", # "xyxy" | "xywh" | "yolo"
)
result_image, bboxes, semantic_mask, instance_mask = cap_aug(image)When bev_transform is set, x_range, y_range, z_range, and h_range
are interpreted in meters relative to the camera. The package projects
each object to its pixel location and scales it by perspective using the
provided calibration.
import cv2
from cap_augmentation import CapAug
from cap_augmentation.bev import BEV
SOURCE_IMAGES = ["path/to/source1.png", "path/to/source2.png"]
image = cv2.imread("path/to/the/destination/image")
# Extrinsic camera parameters (camera pose relative to the ground frame).
camera_info = {
"pitch": -2,
"yaw": 0,
"roll": 0,
"tx": 0,
"ty": 5,
"tz": 0,
"output_w": 1000, # BEV (top-down) output canvas
"output_h": 1000,
}
# Path to intrinsic camera parameters YAML. If None, the packaged default
# (src/cap_augmentation/bev/default_calibration.yaml) is used. That default
# corresponds to a 1920x1080 AXIS surveillance camera (~46° horizontal FOV)
# — it is a placeholder. Pass your own ROS-style camera_info YAML when
# working with a different camera; mismatched intrinsics will shift the
# BEV projection and skew the meters→pixels conversion.
calib_yaml_path = None
bev_transform = BEV(
camera_info=camera_info,
calib_yaml_path=calib_yaml_path,
)
cap_aug = CapAug(
SOURCE_IMAGES,
bev_transform=bev_transform,
n_objects_range=[30, 50],
h_range=[2.0, 2.5], # object heights in meters
x_range=[-25, 25], # left/right of camera axis, meters
y_range=[0, 100], # distance from camera, meters
z_range=[0, 2], # vertical offset, meters
coords_format="yolo", # "xyxy" | "xywh" | "yolo"
)
result_image, bboxes, semantic_mask, instance_mask = cap_aug(image)If you don't have a real camera calibration, BEV.from_image_shape synthesizes
intrinsics from the destination image's dimensions: principal point at the
image center, fx = fy = max(W, H) (≈ 50° horizontal FOV, a reasonable
"normal-lens" prior). Combined with BEV's built-in extrinsic defaults
(pitch=-2°, ty=5m, …), this lets you opt into BEV mode without writing
a YAML or measuring your camera.
import cv2
from cap_augmentation import CapAug
from cap_augmentation.bev import BEV
image = cv2.imread("path/to/scene.jpg")
bev_transform = BEV.from_image_shape(image.shape) # zero-config BEV
cap_aug = CapAug(
SOURCE_IMAGES,
bev_transform=bev_transform,
h_range=[2.0, 2.5], x_range=[-25, 25],
y_range=[10, 50], z_range=[0, 0],
)
result_image, bboxes, semantic_mask, instance_mask = cap_aug(image)Caveat: the synthesized intrinsics will be wrong for fisheye, wide-angle,
or strongly telephoto sensors — perspective scaling of distant objects
will be off. Pass a real ROS-style YAML via BEV(calib_yaml_path=…) when
you have one.
If you don't need perspective at all, skip BEV entirely and use the
pixel-coordinates path shown earlier (CapAug(...) with no
bev_transform=) — it cuts and pastes in image space and never touches
camera geometry.
CapAugMulticlass runs several CapAug instances (one per class) and merges
their boxes/masks, tagging each generated box with its class id (appended as
the fifth column of the output box array).
from cap_augmentation import CapAug, CapAugMulticlass
cap_augs = [
CapAug(
PEDESTRIAN_IMAGES,
n_objects_range=[5, 10],
h_range=[80, 120],
x_range=[0, 1920],
y_range=[400, 1000],
),
CapAug(
CAR_IMAGES,
n_objects_range=[2, 5],
h_range=[60, 100],
x_range=[0, 1920],
y_range=[400, 1000],
),
]
cap_multiclass = CapAugMulticlass(
cap_augs=cap_augs,
probabilities=[1.0, 0.7],
class_idxs=[1, 2],
)
result_image, boxes_with_class, semantic_mask, instance_masks = cap_multiclass(image)Install the optional Albumentations integration first:
pip install "cap-augmentation[albumentations]"import albumentations as A
from cap_augmentation import CapAlbumentations
transform = A.Compose(
[
CapAlbumentations(
p=1.0,
source_images=SOURCE_IMAGES,
n_objects_range=[10, 20],
h_range=[80, 120],
x_range=[500, 1500],
y_range=[600, 1000],
class_idx=1,
),
A.HorizontalFlip(p=0.5),
A.RandomBrightnessContrast(p=0.2),
A.RandomRain(p=1.0, blur_value=3),
],
bbox_params=A.BboxParams(format="pascal_voc"),
)Do not share one CapAlbumentations instance across concurrent threads;
Albumentations calls image, mask, and bounding-box hooks sequentially on the
same transform object.
The Torchvision integration follows the detection target style used by
torchvision.transforms.v2: images can be tensors, tv_tensors.Image, PIL
images, or numpy arrays, and targets are dictionaries with boxes, labels,
and optionally masks.
from cap_augmentation import CapTorchvision
transform = CapTorchvision(
source_images=SOURCE_IMAGES,
n_objects_range=[10, 20],
h_range=[100, 101],
x_range=[500, 1500],
y_range=[600, 1000],
class_idx=1,
)
image, target = transform(image, target)Pass an integer seed (or a numpy.random.Generator) via rng= to make a
CapAug instance deterministic without seeding global state. Two
instances built with the same seed produce bit-identical images, boxes,
and masks:
from cap_augmentation import CapAug
aug = CapAug(SOURCE_IMAGES, rng=42)If you leave rng unset, CapAug falls back to the stdlib random
module and np.random — seed both for global reproducibility.
Decoded source PNGs are cached in memory by default (one entry per
unique source path). For training loops with n_objects_range=(10, 20)
this avoids decoding the same PNG dozens of times per augmented image.
Pass cache_size=N to cap the cache, or cache_size=0 to disable it
(useful when source files are rewritten between calls).
By default, CapAug alpha-composites each pasted object using the alpha
channel of its source PNG: hard-edged masks produce crisp paste boxes,
anti-aliased masks blend smoothly into the destination.
blending_coeff adds an optional "ghost" effect: values in (0, 1)
blend the source colors with the destination colors at the given source
weight before the alpha composite, so blending_coeff=0.5 produces a
translucent paste. The default 0 (no ghost) is the most common
setting. Source PNGs missing a transparency channel trigger an
OpaqueSourceWarning because the pasted "object" then covers the full
source rectangle — usually a bug in the source list.
CapAug can also transform each pasted object before it is inserted into the
destination. New code should use the library-neutral object_transforms
argument; the older albu_transforms parameter is kept as a deprecated alias
and still accepts Albumentations callables.
histogram_matching=True additionally requires the histogram extra.
from cap_augmentation import CapAug, ImageMaskTransform
def object_transform(image, mask):
# ... transform the per-object RGB image and its alpha mask ...
return image, mask
cap_aug = CapAug(
SOURCE_IMAGES,
object_transforms=ImageMaskTransform(object_transform),
)Any PNG image with transparency is suitable as a source: the alpha channel defines the visible region, and bounding boxes are computed from it. You can generate such cutouts yourself from instance segmentation datasets. An example for Cityscapes / CityPersons is below.
The dataset_tools/ scripts are repository tools, not part of the
installed Python package. Clone the repository and install the dataset
extra to use them:
git clone https://github.com/RocketFlash/cap-augmentation.git
cd cap-augmentation
pip install -e ".[dataset]"Put Cityscapes and
CityPersons into ./data/.
Edit dataset_tools/cityscapes/config.py if needed, then run:
./dataset_tools/cityscapes/run.shThis produces the filtered cutouts in data/human_dataset_filtered/ (or the
path set in the config file).
You can also run the two steps manually. First, cut PNGs of people out of Cityscapes images using their instance masks:
python dataset_tools/cityscapes/generate_dataset.pyThen filter out cutouts that are too small or too cropped (only a small part of the body visible):
python dataset_tools/cityscapes/filter_dataset.pyThe result is available in ./data/human_dataset_filtered/ and can be passed
directly to CapAug(source_images=...).