SiPhy: Single-Image Physical Property Reasoning

Hoang Le¹    Joonwoo Kwon¹    Elkhan Ismayilzada¹    Yufei Zhang²    Zijun Cui¹

¹Michigan State University    ²Independent Researcher

---

SiPhy predicts an object's physical properties from a single RGB image. This repository provides two components: the mass-prediction pipeline (below) and the finetuned VLM for material prediction (Section 7).

---

1. Installation

The pipeline uses two Python environments because SAM2 and the main pipeline have conflicting dependencies:

Env	Used for	Suggested name
Main	feature fusion, captioning, material proposal, mass prediction	siphy
SAM	SAM2 mask generation only	siphy_sam

# Main env
python -m venv ~/envs/siphy && source ~/envs/siphy/bin/activate
pip install torch torchvision open_clip_torch transformers openai \
            numpy scipy scikit-learn matplotlib opencv-python pandas \
            pillow tqdm imageio trimesh open3d

# SAM env (install SAM2 from the vendored copy)
python -m venv ~/envs/siphy_sam && source ~/envs/siphy_sam/bin/activate
pip install -e external/sam2

SAM2 weights

The SAM2 checkpoints are not shipped (they are public Meta weights). Download them into external/sam2/checkpoints/:

cd external/sam2/checkpoints && bash download_ckpts.sh && cd -

OpenAI API key

Captioning and material proposal call the OpenAI API. The key is read from a local file my_api_key.py that you create yourself (it is git-ignored and must never be committed):

# my_api_key.py  (at the repo root)
OPENAI_API_KEY = "sk-..."

Alternatively, point that file at an environment variable:

import os
OPENAI_API_KEY = os.environ["OPENAI_API_KEY"]

---

2. Data layout

The data follows the same format as NeRF2Physics. Download and set up the dataset exactly as described in their repository: https://github.com/ajzhai/NeRF2Physics

Then point data_dir in configs/abo500/general_our.json at that dataset directory (default: data/abo_500_cvl). The dataset is not distributed here.

---

3. Running the mass pipeline

The full end-to-end pipeline is wrapped in one script:

bash pipelines/mass_pipeline.sh \
    --split test \
    --main_env ~/envs/siphy/bin/activate \
    --sam_env  ~/envs/siphy_sam/bin/activate

It runs five stages (see pipelines/mass_pipeline.sh --help):

1. Feature fusion — feature_fusion.py: CLIP patch features lifted to 3D.
2. SAM inference — external/sam2/tools/abo_inference.py: per-object masks.
3. Captioning — captioning.py: object caption.
4. Material proposal — material_proposal.py: candidate materials + thickness.
5. Mass prediction — mask_material_proposal.py + predict_property.py: per-mask material confidence, then integral mass estimate + evaluation.

Run a single object first to smoke-test: add --end_idx 1.

---

4. Thickness mode

The thickness variant is selected with --mats_load_name (or via mats_load_name in configs/abo500/mass.json, which defaults to combine_detail_llm):

python predict_property.py \
    --split test \
    --general_config configs/abo500/general_our.json \
    --mass_config    configs/abo500/mass.json \
    --mask_config    configs/advance_mask.json \
    --mats_load_name combine_detail_llm      # info | combine_llm | combine_detail_llm

---

5. Configs

• configs/abo500/general_our.json — data dir, feature names, sampling stride.
• configs/abo500/mass.json — mass hyperparameters, model checkpoint, and the default thickness mode (combine_detail_llm).
• configs/advance_mask.json — SAM mask settings (advance_box_mask, postprocess_masks).

Config values override the corresponding CLI defaults.

---

6. Checkpoint

checkpoints/mlp_attention_contrastive_stride10_one_mask_combined_L2_100.0_blip2_s0_e-1_train_bs8_contrastive_20250624_045722_1.pth is the trained mass-regression head referenced by mass.json.

---

7. VLM material prediction (Octopi)

external/octopi/ holds the finetuned SiPhy VLM (an Octopi model: frozen CLIP ViT-L/14 encoder + MLP projection + Vicuna-7B with LoRA) that predicts an object's materials from its image. This release contains the prediction code only (no training or evaluation code).

Environment

python -m venv ~/envs/octopi && source ~/envs/octopi/bin/activate
pip install -r external/octopi/requirements.txt

On first run, two base models are pulled from HuggingFace automatically: openai/clip-vit-large-patch14 (frozen encoder) and lmsys/vicuna-7b-v1.5 (base LLM; the LoRA adapter is applied on top).

Checkpoint

Download the finetuned checkpoint from HuggingFace into external/octopi/checkpoints/siphy_vlm/:

hf download lexuanhoang1904/SiPhy --local-dir external/octopi/checkpoints/siphy_vlm

It contains project.pt, llm_weights/ (LoRA adapter), and tokenizer/. The visual encoder is frozen CLIP, so it is not shipped; it loads from HuggingFace at runtime.

Test data

Provide test data in Octopi format at external/octopi/dataset_<DATASET>_<MODALITY>_test/test_qa.json, where <MODALITY> is description_gpt, description, or only_img.

Run

CHECKPOINT_DIR=checkpoints/siphy_vlm DATASET=<your_dataset> MODALITY=description_gpt \
    bash external/octopi/predict_material.sh

Predictions are written to external/octopi/exps/<timestamp>_.../test_preds.json.

---

Acknowledgements

We thank the authors of these repositories for their work:

• NeRF2Physics
• Segment Anything (SAM)
• Octopi