Config driven detection pathways - #9
Merged
Merged
Conversation
Replace the hardcoded camera->detector logic (fly_camera_layout, expand_passes, assemble_skeleton, the L/R flip, front-runs-twice, and the side->slice layout) with a config-driven plan of four decoupled lists whose counts may differ: [[sources]] named footage globs (decoded once each) [[preprocessors]] named, reusable frame-op pipelines (mirror/crop/resize) [[models]] detector class + weights + input contract (a registry) [[pathways]] source -> preprocessor -> model -> (view, point) mapping A pathway maps each model channel i into points[i] of a view (-1 drops it). Visibility is now the union of those maps -- an unmapped (view, point) is NaN -- so [skeleton.visibility] and apply_visibility are removed. The L/R flip stays a detector-input step; points are mapped back into the view frame by inverting the pathway preprocessing (new FrameTransform.unmap_points), since a mirror cannot be folded into the camera model. This is consistent with the camera's (w-1)/2 principal point (a latent ~1px fix vs the old normalized 1-x convention). Detection is fully batched: each pathway's window is preprocessed in one shot, then the forward runs per-model over one time-major batch. Cameras become geometry-only views; footage globbing and image sizing move from cameras to sources. Fingerprints track the plan (sources/preprocessors/models/ pathways) instead of camera_patterns/preprocess. New modules: pose2d/models.py (registry) and pose2d/pathways.py (DetectionPlan + scatter). BREAKING: old configs and cached poses.h5/run.json snapshots must be regenerated; deeperfly init writes the explicit plan. Verified: 431 tests pass; end-to-end on examples/data matches the prior output on visibility exactly and to ~1px on coordinates (the documented convention fix). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Decouple each pathway (a named source -> preprocessor -> model run) from
where its outputs land. A new [point_sources.<view>] table, keyed by point
name, maps each tracked point to { pathway, out_channel }. Keying on
(view, point) makes every point single-sourced (a duplicate is an error),
replacing the per-pathway view/points arrays.
Renames:
- model n_channels -> n_out_channels; mapping entry heatmap -> out_channel
- source field input -> filename
- skeleton joint_names -> point_names, [skeleton.limb_joints] -> limb_points
(values may be point names or indices), [skeleton.palette] -> limb_palette,
and the *_stripe* points -> *_abdomen*
Also fix the front-view mapping (channels were out of range and the left/right
sides swapped), update both packaged configs, the tests, docs/configuration.md,
and the walkthrough's point_names references. results.py reads the old
joint_names HDF5 key as a fallback so existing result files still load.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A stage with no cached result from a previous run is running for the first time, so announcing it as a recompute (with a reason) was misleading. Snapshot which stages had a prior record before the loop and log plain "running <stage>" for the rest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The walkthrough was built on the removed per-camera "passes" model (fly_camera_layout / expand_passes / assemble_skeleton / inference.preprocess / triangulation.apply_visibility and the Skeleton visibility table). Rewrite it around the current config-driven DetectionPlan: Config.default() -> detection_plan / skeleton / camera_group, load_models, per-pathway prepare/predict_heatmaps, heatmap_to_points + map_to_view + scatter_pathway, inference.detect_sequence, then calibrate -> reconstruct_ransac -> run_from_points2d. Visibility is now intrinsic to the plan (an unwritten (view, point) stays NaN) and the front camera is just one source feeding two pathways -- the notebook unfolds exactly what detect_sequence does. Remove examples/pipeline_demo.py (also on the removed API) and its Examples entry in docs/library.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The view axis is now a first-class part of the model contract so a future cross-view model can keep V and mix across it rather than folding it away. HourglassNet.forward accepts 5D (B, V, 3, H, W) -- V views folded into the conv batch and unfolded on output -- as well as the existing 4D (N, 3, H, W) (no-view case). All predict_* return (B, V, ...) for a 5D input. detect_sequence passes the (T, Pm, 3, H, W) stack through as (B, V) chunks instead of pre-flattening, and detect_candidates_sequence wraps/strips a B=1 axis; both are numerically identical to the previous flat dispatch. On CUDA, the folded conv batch is laid out channels_last under fp16/bf16 autocast so cuDNN picks its Tensor-Core kernels (~10% on fp16/bf16 at bs=32 on an RTX 4090); the fp32 path keeps NCHW where channels_last gives no win. The model weights are also stored channels_last on CUDA. USE_CHANNELS_LAST flag allows disabling for A/B or as a cuDNN regression escape hatch. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
One pass over the repo for clarity and simplicity: - Fix bugs: wrong clone URL in CONTRIBUTING, broken pipeline.py doc link, and a stages docstring that claimed a visibility-masking step that does not exist. - Rewrite "no longer / used to be / the old X" comments to describe present behavior only, across src, docs, and the walkthrough notebook. - Drop backwards-compat cruft: the reproject/none triangulation aliases (renaming _resolve_triangulation -> _validate_triangulation), the joint_names HDF5 fallback, and the removed-config-form migration error. - De-duplicate tests: rely on conftest's cameras/fly fixtures and a single fly_masked helper in tests/helpers.py. - Tidy docs: simpler detection-plan heading, trimmed preprocessor section, and a corrected public-API list in library.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Introduce .github/workflows/ci.yml running the test matrix (py3.11-3.13) with coverage, ruff lint/format, uv.lock sync, and an advisory pyright typecheck. Enable ruff import sorting (I) and pycodestyle warnings (W), wire up pytest-cov and coverage/pyright config, and sort imports across the tree to match. Declare explicit __all__ exports for the public modules, add a tests/test_recordings.py suite covering discovery/globbing/output planning, and give [[pipeline.visualization.videos]] typo-friendly validation errors at parse time. Replace silent excepts in the torch probes with debug logging, and drop the removed dev/ benchmarks from the docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rich's stock ReprHighlighter read "0x960" out of "960x960" as a hex literal and "da:0" out of "cuda:0" as an IPv6 address, colouring partial tokens. Add a tailored RegexHighlighter that drops the ip/MAC/uuid patterns and anchors the number pattern so dimensions and device ids colour as whole tokens. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Detector confidence can now weight the DLT in all three triangulation methods (dlt/greedy/ransac) via a weighted least-squares solve: each view's rows are scaled by sqrt(confidence) in triangulate_dlt, threaded through CameraGroup.triangulate, triangulation.triangulate / triangulate_ransac, and the pipeline reconstruct* helpers. For RANSAC the weights drive the candidate fits and final refit but not the consensus vote, which stays a geometric reprojection test so a confidently-wrong detection cannot vote itself into the inlier set. Both stages are gated by a config flag at the config/pipeline seam (the low-level functions take an optional weights array, mirroring bundle_adjust): - [pipeline.triangulation].weigh_by_confidence (default false; new) - [pipeline.bundle_adjustment].weigh_by_confidence (default true; preserves existing always-weighted BA behavior) Edge cases are handled at the foundation so every caller is protected: negative and non-finite confidences clamp to zero (dropping the observation) instead of producing NaN through sqrt; an all-zero weight set falls back to uniform weights (BA warns) rather than going ill-posed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Calibration previously always subsampled to evenly spaced frames. Add a frame_sampling knob (even/confidence/coverage/diversity) alongside max_frames so BA can prefer the frames the detector is surest about, the best-conditioned (most multi-view-observed) frames, or postures that are maximally spread apart, while keeping "even" as the default. Decouple confidence-based frame selection from residual weighting: calibrate now always receives conf, with weigh_by_confidence controlling only whether residuals are scaled by sqrt(confidence). frame_sampling ="confidence" can use the confidences for picking frames even when weighting is off. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Give the two pathway helpers names that describe what they do: map_to_view -> normalized_peaks_to_original_pixels (it inverts a pathway's preprocessing to put normalized model peaks back in source pixels) and scatter_pathway -> route_channels_to_points_in_views (it routes each model output channel to its (view, point) slot). Update the inference call sites and tests accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the jaxtyping axis labels and docstring shape symbols to one consistent vocabulary across the codebase: N -> P (skeleton points), J -> C_out (detector output channels / heatmaps), Hh/Ww -> H_out/W_out (heatmap size), and the per-heatmap peak count P -> K. No behavior change; these are annotation and documentation only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add two Mermaid data-flow diagrams (raw frames -> 2D keypoints, and 2D -> 3D keypoints) with a symbol legend defining the canonical axis names (V, T, P, C_out, H_out/W_out, ...) used throughout the code, document the front-camera two-lane / left-right mirroring scheme, note the new max_frames / frame_sampling calibration knobs, and split the caching and re-run rules into their own section. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GitHub renders Mermaid in a sandbox without KaTeX, so the $$...$$ math labels parsed to empty nodes. Replace them with plain text (<br> breaks, plain subscript names matching the legend table), keeping edge structure so the linkStyle indices still line up. Fix "unobsered" typo. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The fly skeleton now lives in the [skeleton] section of the packaged default_config.toml; Skeleton.fly() loads Config.default(), and the standalone data/skeleton_fly.toml plus its drift-guard test are removed. Bundle adjustment's `keypoints` (raw indices) becomes `points_to_use` (skeleton point names), resolved to indices against the skeleton in stage_bundle_adjustment, so configs no longer hardcode index order. Pathways may omit `preprocessor` (identity frame prep), which lets the default config drop the no-op `noflip` preprocessor and use cleaner pathway names. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pipeline module, stage, and config section already used the `bundle_adjustment` name; the leftover "calibrate"/"calibration" naming was the only holdout. Rename the high-level wrapper `pipeline.calibrate` -> `bundle_adjust_cameras` (distinct from the low-level `bundle_adjustment.bundle_adjust` solver), the `run_from_points2d` params `do_calibrate`/`calibrate_kwargs` -> `do_bundle_adjust`/`bundle_adjust_kwargs`, and `_resolve_calibration_points` -> `_resolve_bundle_adjustment_points`. Update docstrings, comments, config, docs, tests, and the walkthrough notebook to match. The preprocessing distortion-coefficient error message refers to intrinsic calibration for a rotated frame (a different concept), so it was reworded rather than renamed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move each stage's parameter table out of [pipeline.<stage>] to a top-level [<stage>] table, and relocate the 2D detection plan under [pose2d]: [[preprocessors]]/[[models]]/[[pathways]] become [[pose2d.preprocessors]]/ [[pose2d.models]]/[[pose2d.pathways]], and [point_sources.<view>] becomes [pose2d.output_points.<view>]. [pipeline] now holds only the do_<stage> toggles. The [pipeline].fps knob is dropped; the playback rate is detected from the input videos and overridden via [visualization].output_fps. Also fix the triangulation defaults in the packaged template, which had drifted from the Python defaults (method "dlt" -> "ransac", weigh_by_confidence true -> false, and restore the active ransac_threshold / min_inliers lines) so the anti-drift guard passes again. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Point the architecture, comparison, and configuration docs and the example notebooks at the new top-level stage tables ([<stage>], [pose2d.*], [pose2d.output_points], [visualization.*]) and drop the [pipeline].fps note. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.