Skip to content

preprocess.md

Latest commit

 

History

History
255 lines (186 loc) · 9.33 KB

preprocess.md

File metadata and controls

255 lines (186 loc) · 9.33 KB
Raw

Preprocessing on Artemis Data [WIP]

Steps to process raw Artemis data from scratch. All scripts referenced below are in preprocessing/ and should be run from that directory.

Every script accepts --animal and --root_path flags. The --root_path is the root Artemis data directory containing per-animal subdirectories.

Prerequisites

  • Directional -- to obtain tangent basis on the mesh
  • NeuS -- to obtain full animal geometry (included as submodule)
  • SMAL -- to obtain parametric animal model (SMALify-neuralfur, included as submodule)

Download raw data

Download the Dynamic Furry Animals (DFA) dataset from Artemis:

animal="panda"
unzip ${animal}.zip -d /path/to/raw/Artemis/

1. Prepare images, silhouettes, and cameras

Converts raw Artemis captures (images, alpha masks, camera intrinsics/extrinsics) into the format expected by NeuS. Outputs images, silhouettes, and camera matrices (projection, intrinsic, extrinsic).

python preprocessing/preprocessing_all_data.py \
  --animal panda \
  --root_path /path/to/raw/Artemis \
  --save_path /path/to/save/Artemis

This creates <save_path>/<animal>_processed/<motion_seq>/ containing:

  • images/ and silhouette/ -- input views and alpha masks
  • cameras.npz -- projection matrices (intrinsic x extrinsic x scale)
  • cameras_intr.npy, cameras_extr.npy -- separate intrinsic and extrinsic matrices

2. Reconstruct surface with NeuS

Runs NeuS to obtain an SDF-based mesh reconstruction from the multi-view images.

cd submodules/NeuS
scene_name="panda"
data_dir="/path/to/Artemis/${scene_name}_processed"
python exp_runner.py --mode train --conf ./confs/wmask_artemis.conf --case walk --dataset artemis --scene ${scene_name} --data_dir ${data_dir}
python exp_runner.py --mode validate_mesh --conf ./confs/wmask_artemis.conf --case walk --dataset artemis --scene ${scene_name} --data_dir ${data_dir} --is_continue

The reconstructed mesh is saved to submodules/NeuS/exp/<scene_name>/<case>/wmask/meshes/.

3. Prepare data in GaussianHaircut format

Converts the NeuS-processed data into the directory structure expected by GaussianHaircut: padded image filenames, hair/body masks, projection matrices, and the NeuS mesh exported as OBJ.

python preprocessing/prepare_data_in_GH_format.py \
  --animal panda \
  --root_path /path/to/Artemis \
  --neus_root_path ./submodules/NeuS/exp

4. Calculate orientation maps

Computes hair/fur orientation maps using Gabor filters. These provide directional supervision for strand optimization.

cd submodules/GaussianHaircut/src/preprocessing
DATA_PATH="/path/to/Artemis/panda_processed_GH/walk"
python calc_orientation_maps.py \
  --img_path $DATA_PATH/images_2 \
  --mask_path $DATA_PATH/masks_2/body \
  --orient_dir $DATA_PATH/orientations_2/angles \
  --conf_dir $DATA_PATH/orientations_2/vars \
  --filtered_img_dir $DATA_PATH/orientations_2/filtered_imgs \
  --vis_img_dir $DATA_PATH/orientations_2/vis_imgs

5. Fit SMAL model

Fit a SMAL parametric animal model to the reconstructed mesh using submodules/SMALify-neuralfur, our fork of SMALify with a multi-stage differentiable fitting pipeline. This produces a body model with semantic vertex groups.

Install dependencies and download the SMAL model files (my_smpl_00781_4_all.pkl, my_smpl_data_00781_4_all.pkl, symIdx.pkl) from SMAL into submodules/SMALify-neuralfur/data/SMALST/smpl_models/ -- see submodules/SMALify-neuralfur/README.md for full setup instructions.

Then run the fitter against the Step 2 NeuS mesh (a directory of .obj meshes, plus a YAML config describing the optimization stages -- see submodules/SMALify-neuralfur/fitter_3d/sample_bear/cfg.yaml for a full example):

cd submodules/SMALify-neuralfur
python fitter_3d/optimise.py \
  --mesh_dir ../NeuS/exp/<scene_name>/<case>/wmask/meshes \
  --yaml_src <path-to-config.yaml>

Each stage's optimized SMAL parameters are saved as Stage<N>.npz under the configured results directory. Convert the final stage back to an .obj mesh for the annotation transfer in Step 6:

python scripts/npz2obj.py --results_dir <out_dir> --stage Stage5

This produces <out_dir>/Stage5.obj.

6. Annotate and transfer SMAL body part annotations

6a. Annotate SMAL model in Blender [DONE and same across animals]

Open annotate_smal.blend in Blender, paint vertex groups for body parts (legs, belly, tail, ears, etc.), and export them as JSON by running ./preprocessing/save_annotations_fur_blender.py from within Blender.

6b. Transfer annotations to target mesh

Transfers vertex group annotations from the fitted SMAL model to the target mesh using nearest-neighbor matching.

python preprocessing/transfer_smal_to_neus.py \
  --animal panda \
  --root_path /path/to/Artemis \
  --annotation_json ./data/part_annotations_SMAL.json \
  --input_mesh_path furless_reshaped.obj

6c. Validate and complete annotations

Ensures all vertices have a body-part label (assigns remaining to "body"):

python preprocessing/check_fur_length_and_blender_annotations.py \
  --animal panda \
  --root_path /path/to/Artemis \
  --mode furless_reshaped

7. Annotate per-part fur properties with ChatGPT

For each new animal, we use ChatGPT to obtain three per-part annotations by sending two reference images (frontal and side views):

  1. Fur length (mapping_length) -- fur length in cm per body part
  2. Effective fur thickness (effective_fur_thickness_cm) -- how much to shrink the mesh inward to get the furless body (used in step 8)
  3. Fur growing direction (mapping_gravity) -- a 3D direction vector per body part indicating how fur grows/hangs

Additionally, ask ChatGPT for the distance between the eyeballs in cm (eye_dists_VQA), used for metric scale estimation.

See prompts.md for the exact prompts to use. Paste the results into:

  • YAML config (submodules/GaussianHaircut/src/arguments/metrical_panda_furless_15k_small.yaml): fur length (mapping_length), fur growing direction (mapping_gravity), and eye distance (eye_dists_VQA)
  • src/animal_config.py: effective fur thickness (effective_fur_thickness_cm, used in step 8 for mesh shrinkage)

See src/animal_config.py for reference values across different animals.

8. Extract furless body mesh

Shrinks the NeuS mesh inward along vertex normals based on per-part fur thickness to obtain the furless (skin) body mesh. Uses effective_fur_thickness_cm from src/animal_config.py and scales to metric space using eye distance.

python preprocessing/extract_furless_body.py \
  --animal panda \
  --root_path /path/to/Artemis \
  --eye_dist_vqa 11.0 \
  --smooth_iterations 100 \
  --voxel_resolution 256

9. Compute tangent field with Directional

9a. Clean mesh before processing

Clean the mesh to remove artifacts from marching cubes (isolated faces, degenerate geometry):

python preprocessing/fix_mesh_before_directional.py \
  --animal panda \
  --root_path /path/to/Artemis \
  --input_mesh shrunken_panda_sdf_only.obj \
  --target_faces_hr 160000 \
  --target_faces_lr 10000

9b. Compute raw tangent field with Directional

Use the Directional library to compute a tangent field on the mesh surface. Refer here and here for details. The output rawFaceField.dmat is then converted into a normalized field:

python preprocessing/save_directional_basis.py \
  --animal panda \
  --root_path /path/to/Artemis \
  --input_field rawFaceField.dmat \
  --input_mesh furless_reshaped.obj \
  --output field_furless_reshaped.npy

9c. Orient tangent field with parallel transport

python preprocessing/compute_tangent_basis.py \
  --animal panda \
  --root_path /path/to/Artemis \
  --mode furless_reshaped

Add --visualize_tan to export a PLY visualization of the tangent field.

10. Create eye landmarks and measure eye distance

Extract eye keypoints from the Artemis dataset or from SMAL model and save as eyes.ply.

python preprocessing/extract_eyes.py \
  --animal panda \
  --root_path /path/to/Artemis

11. Losses

11a. Compute SDF volume

Computes a signed distance field on a regular grid from the furless body mesh. Used as a penetration loss during fur optimization.

python preprocessing/compute_sdf.py \
  --animal panda \
  --root_path /path/to/Artemis \
  --mode furless_reshaped \
  --grid_size 32

11b. Save bald mask to exclude regions from loss calculation

Creates per-frame masks marking regions without fur (paw pads, eyes, nosetip):

python preprocessing/visibility_map.py \
  --animal panda \
  --root_path /path/to/Artemis \
  --mesh_name furless_reshaped.obj \
  --annotations_name annotations_furless_reshaped.json \
  --exclude_groups paw_pads eyes nosetip

11c. Extract UV map (optional)

python preprocessing/save_uvmap.py \
  --animal panda \
  --root_path /path/to/Artemis \
  --input furless_reshaped_uv.obj \
  --output furless_reshaped_uv.pt