Skip to content

REPRODUCE.md

Latest commit

 

History

History
80 lines (60 loc) · 3.76 KB

REPRODUCE.md

File metadata and controls

80 lines (60 loc) · 3.76 KB
Raw

Reproducing the paper results

All commands run from the repository root with the OV-MAP uv environment active (source .venv/bin/activate). See DATA.md for the expected data layout and INSTALL.md for setup.

Prerequisites (for the full pipeline): ScanNet200 data prepared per DATA.md, including the 0.05 mesh segmentation (generate with scripts/generate_mesh_segmentation.sh or download with scripts/download_precomputed.sh --mesh-seg), and the checkpoints downloaded into weights/ (see ../weights/README.md). The fast reproduction below needs none of these.

The scripts take positional arguments: [mask_model] [depth_type] [gpu] [part] (defaults: cropformer sup 0 0). 2D masks and supplemented (raw + synthetic) depth are produced inside the build step (ovmap/build_map.py) — there is no separate mask / depth-generation script.

Fast reproduction with precomputed outputs (recommended)

The full pipeline takes several GPU-hours to build + ~1 day for CLIP feature extraction. To reproduce the ScanNet200 numbers in minutes, download the precomputed 3D build masks (~13 MB) and CLIP features (~68 MB) and run only the (cheap) final steps:

# 1. Download precomputed save_3d_mask + mask_features (defaults to jkim50104/OVMap-scannet200).
bash scripts/download_precomputed.sh

# 2. Regenerate the mask_array locally (fast; note it writes ~33 GB to disk).
bash scripts/prepare_evaluation.sh

# 3. Evaluate using the downloaded features (minutes).
cd open_query/clip_embedding
python evaluation_scannet/run_eval_close_vocab_inst_seg.py \
  --gt_dir="$(pwd)/../../data/scannetv2/preprocessed_data/scannetv2_200_openmask3d/instance_gt/validation" \
  --mask_pred_dir="$(pwd)/../../output/scannetv2/OVM_OClo3_VX0.03_IT10/mask_array/cropformer_sup_depth" \
  --mask_features_dir="$(pwd)/../../output/scannetv2/OVM_OClo3_VX0.03_IT10/mask_features/cropformer_sup_depth" \
  --evaluation_output_dir="$(pwd)/../../output/scannetv2/OVM_OClo3_VX0.03_IT10/evaluation_result.txt"

The precomputed masks already include the dominant-voting result, so this path needs neither the mesh segmentation nor the synthetic depth.

Full pipeline — ScanNet200 (Table I: AP 11.9 / AP50 17.4 / AP25 23.2)

# 1. Build 3D instance masks for all validation scenes.
#    Output: output/scannetv2/OVM_OClo3_VX0.03_IT10/save_3d_mask/cropformer_sup_depth/<scene>.pth
bash scripts/build_ovmap.sh cropformer sup 0

#    To shard across two GPUs, edit SUBPROCESS_NUM=2 in the script and run:
#    bash scripts/build_ovmap.sh cropformer sup 0 0   # GPU 0, part 0
#    bash scripts/build_ovmap.sh cropformer sup 1 1   # GPU 1, part 1

# 2. Convert 3D masks to OpenMask3D-style mask arrays.
bash scripts/prepare_evaluation.sh

# 3. Compute CLIP mask features + evaluate (writes evaluation_result.txt with AP/AP50/AP25).
bash scripts/run_scannet200_eval.sh cropformer sup 0

The overlap criterion (lo3 = large-overlap 0.3), voxel size (0.03), and image sampling (every 10th frame) are the paper defaults, set in scripts/build_ovmap.sh / scripts/run_scannet200_eval.sh.

Ablation — depth type (Table III)

Re-run the ScanNet200 flow with different depth_type values:

bash scripts/build_ovmap.sh cropformer raw 0    # raw depth       (AP 11.1)
bash scripts/build_ovmap.sh cropformer pc  0    # synthetic depth (AP 10.1)
bash scripts/build_ovmap.sh cropformer sup 0    # supplemented    (AP 11.9)

Notes

  • Reproduced numbers may differ slightly from the paper due to GPU type and inherent randomness in the pipeline.
  • The CLIP feature-computation step (run_scannet200_eval.sh) is the most time-consuming stage; sharding the build across GPUs (SUBPROCESS_NUM) speeds up step 1.