This work focuses on advancing the 3D Highlighter model [1], a system capable of automatically localizing fine-grained semantic regions on 3D meshes based on textual descriptions. We extend this model to handle point clouds and real-world 3D scans, integrating the concept of affordance to identify not only parts, but also the potential uses and interactions suggested by an object's shape and characteristics.
Semantic localization, in this case, is not only about recognizing the geometry, but also about interpreting the behavior and potential for interaction with those regions — an aspect that is typical of affordance. Affordance acts as a bridge between pure 3D geometry and the functional purpose of objects in the real world.
This expansion overcomes the limitations of previous models that relied exclusively on meshes and demonstrates potential applications in areas such as structural analysis and augmented reality.
Full details and results are in PROJECT_PAPER.pdf.
3D Highlighter base model. 3D Highlighter [1] is a model that takes as input a mesh and a prompt and, due to the fundamental role of the CLIP model [4], produces a mesh with the highlighted region specified in the prompt. CLIP provides a joint vision-language embedding space that allows semantic comparisons between rendered images and textual prompts, calculating the similarity between them through their embeddings. These embeddings guide the optimization of the neural network parameters through a CLIP loss function that aligns the highlighted regions with the semantic meaning of the input prompt.
The Neural Highlighter has as input all the vertices of the mesh and, elaborating them through all its layers, gives as output a probability, for each vertex, of belonging to the specified region. Following the obtained probabilities, the mesh is colored, giving to each vertex a color that is a weighted blend of a highlight color and gray. Initially all vertices have a probability of 0.5, which makes the mesh half-highlighted; during optimization, vertices transition smoothly toward gray or highlight color, aligning themselves with the text-specified regions. The colored mesh is then rendered from multiple views using a differentiable renderer, 2D augmentations are applied, and the resulting images are compared to the prompt via the CLIP loss — with no 3D supervision required.
Mesh reconstruction from point clouds. Point clouds are representations of objects in raw data points — more flexible and accurate than meshes, and more commonly available in real-world scenarios. Since the pipeline takes meshes as input, meshes are obtained from surface reconstruction of point cloud objects. Three methods (via Open3D [3]) were compared: Alpha Shape, which excludes points that do not contribute to the underlying shape using a parameter α; Ball Pivoting, which simulates a ball pivoting around the point cloud, progressively forming triangles; and Poisson Surface Reconstruction, which uses the point cloud's surface normals to guide the reconstruction and approximate the shape of the original object.
Poisson Surface Reconstruction was chosen because it creates continuous and smooth meshes even when dealing with noisy or incomplete data, giving more robust and trustworthy results — especially where completeness and mesh continuity are crucial. It is, however, particularly sensitive to surface normals, and works best on closed surfaces: bottles, being three-dimensional objects with a continuous surface, are better suited to this technique than open-surface objects like tables or hats.
Affordance application. The concept of affordance refers to the ability of a system to recognize and interpret the potential interaction offered by an object — how it can be used or manipulated based on its physical characteristics. This project shifts the focus from semantic localization, as in the original 3D Highlighter, to object affordance, expanding its application domain to scenarios involving human-object interactions. To evaluate and test the model, 3D AffordanceNet [2] was used — a dataset for affordance estimation on 3D point clouds, consisting of 56,307 affordance annotations for 22,949 shapes covering 18 affordance classes and 23 semantic object categories.
Point clouds from 3D AffordanceNet are converted to meshes and passed through the pipeline with an affordance-oriented prompt (e.g. "A 3D render of a gray bottle with the grasped and wrapped area highlighted"). To compare predictions with the ground truth (defined on the point cloud rather than the mesh), a projection using a k-d tree associates each mesh vertex with the nearest point cloud point, before computing mIoU.
CLIP ViT-B/32 at 224×224 resolution, PyTorch, Google Colab (NVIDIA T4 GPU), ~15 min average runtime per test. ViT-B/32 was chosen over the original paper's ViT-L/14 to reduce compute time given the available resources.
| Configuration | LR | # Augmentations | # Views | Depth |
|---|---|---|---|---|
| Original (paper baseline) | 0.0001 | 5 | 5 | 4 |
| Optimized — mesh pipeline | 0.001 | 7 | 5 | 4 |
| Optimized — AffordanceNet pipeline | 0.0001 | 7 | 5 | 4 |
Hyperparameter tuning (higher learning rate, more augmentations) improved region localization on the original horse/dog test meshes compared to the paper's default configuration:
Working with the bottle object class and the wrap-grasp affordance, the model was evaluated (6 instances) and tested (6 held-out instances) against AffordanceNet ground truth, using mIoU:
Evaluation phase (mIoU per instance):
Testing phase (best configuration: LR 0.0001, 7 augmentations, 5 views, depth 4):
mIoU on the test set ranged from 0.07 to 0.34 (average ≈ 0.15). The authors note that although these values are numerically modest, the highlighted regions are qualitatively convincing — AffordanceNet's ground-truth intensities are diffuse over a wider margin than the model's more selective predictions, which penalizes mIoU without necessarily reflecting a worse result visually.
Using Polycam to scan real bottles (photogrammetry/LIDAR) and reconstructing meshes with the pipeline above, the model was applied with the same affordance prompts:
Results on the same wrap-grasp affordance across several real bottles:
Generalizing to a different, unseen affordance (openable, semantic shift from wrap-grasp) without re-tuning:
Results on real-world scans were inconsistent: the model performed well on some meshes but failed or mislocalized on others, largely attributed to lower mesh quality (scanner noise, lighting/shadow artifacts) and, for the openable affordance, to the lack of prior hyperparameter tuning for that specific class.
The model exhibits limitations when dealing with certain affordance types, where its performance is less robust. It struggles to generalize well across several affordance categories, particularly those that require complex interaction understanding. This inconsistency indicates that further improvements are needed to enhance the model's reliability and consistency in handling varied affordances and mesh inputs.
This project successfully extends the 3D Highlighter model to incorporate affordance understanding, enabling it to handle point clouds and real-world 3D scans effectively. By leveraging affordance, the model not only improves in semantic localization but also enhances its applicability in practical scenarios involving human-object interactions. The application of the model to real-world scans demonstrates its versatility and potential for wider use cases in fields such as robotics, augmented reality, and structural analysis. Future work could further explore the integration of more complex affordance categories and real-time applications, pushing the boundaries of 3D semantic localization.
├── data/ # .obj meshes used in the baseline experiments (candle, dog, horse)
├── mesh_polycam/ # .obj meshes obtained from real-world Polycam scans
├── Normalization/ # Mesh/point cloud normalization utilities
├── mesh.py # Mesh loading and handling
├── render.py # Differentiable rendering of highlighted meshes
├── load_dataset.py # AffordanceNet point cloud dataset loader
├── utils.py # Shared helper functions
├── 3DHighlighter.yml # Conda environment definition
├── notebook_base.ipynb # Phase 1 — base 3D Highlighter model, config search & hyperparameter tuning
├── mesh_reconstruction.ipynb # Phase 2 — mesh reconstruction pipeline from point cloud data
├── pc_to_mesh.ipynb # Phase 3 — point cloud (AffordanceNet) → mesh conversion
├── eval_phase_notebook.ipynb # Phase 3 — evaluation phase
├── test_phase_notebook.ipynb # Phase 3 — testing phase
├── extension_notebook.ipynb # Phase 4 — extension to real-world scans
├── PROJECT_PAPER.pdf # Full project report
└── README.md
This project contains several Jupyter notebooks for different phases and tasks of the project. Below is a list of all the .ipynb files included in this project:
- notebook_base.ipynb: Base notebook for initial setup and experiments. It refers to the part 1 of the experiments where the 3D Highlither model is used and evaluated, by searching for the best model configuration, tuning hyperparameters.
- mesh_reconstruction.ipynb: Notebook for testing the pipeline of the mesh reconstruction starting from point cloud data. It refers to the part 2 of the experiments, starting from meshes in
datafolder. - pc_to_mesh.ipynb: Notebook for converting point clouds, from AffordanceNet dataset, to mesh. It refers to the part 3 of experiments where there are two different phases:
- eval_phase_notebook.ipynb: Notebook for evaluation phase.
- test_phase_notebook.ipynb: Notebook for testing phase.
- extension_notebook.ipynb: Notebook for extending functionalities on 3D real-world scans.
The data directory contains several .obj files used in the first phase of the project:
- candle.obj
- dog.obj
- horse.obj
The mesh_polycam directory contains additional .obj files used for the extension part and they refer to the mesh obtained with the tool Polycam.
- Decatur, D., Lang, I., & Hanocka, R. 3D Highlighter: Localizing Regions on 3D Shapes via Text Descriptions. CVPR 2023. [paper] [code]
- Deng, S., Xu, X., Wu, C., Chen, K., & Jia, K. 3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding. CVPR 2021. [paper]
- Open3D Team. Surface Reconstruction — Open3D Documentation, 2018–2020.
- Radford, A. et al. Learning Transferable Visual Models From Natural Language Supervision (CLIP). 2021.
- Xie, Y. et al. Neural Fields in Visual Computing and Beyond. 2022.
This project builds on 3DHighlighter by Decatur, Lang & Hanocka — check their repository for license terms before reuse. This work is otherwise part of an academic course project at Politecnico di Torino.
Authors: Luigi Nicola Mogano, Luciana Colella, Lorenzo Greco — Politecnico di Torino