SEMASIA: A Large-Scale Dataset of Semantically Structured Latent Representations

Tip

Latent representations learned by neural networks often exhibit semantic structure, where concept similarity is reflected by geometric proximity in embedding space. However, comparing such spaces across models remains difficult: changes in architecture, pretraining data, objective, or random seed can yield embeddings with similar content but incompatible geometry. This latent space alignment problem is central to interpretability, transfer and multimodal learning, federated systems, and semantic communication; however, progress remains limited by the lack of large-scale, model-diverse, and metadata-rich benchmarks. To address this gap, we introduce SEMASIA, a large-scale collection of latent representations extracted from approximately 1,700 pretrained vision models across eight standard image-classification benchmarks. SEMASIA pairs embeddings with structured metadata describing architectures, training regimes, pretraining sources, and model scale. We demonstrate three applications of the resource. First, we analyze the conceptual organization of individual latent spaces, showing consistent prototype-like clustering and hierarchical semantic neighborhoods across models and datasets. Second, we benchmark supervised alignment mappings between latent spaces using reconstruction error and downstream task performance. Third, we perform a large-scale regression analysis of how pretraining-data complexity, specialization, transfer learning, augmentation, and model scale relate to geometric and probing properties of embeddings. By coupling representational scale with standardized metadata, SEMASIA provides a reproducible foundation for studying latent geometry, evaluating alignment methods, and developing next-generation heterogeneous and interoperable AI systems.

Two-dimensional t-SNE projection of the aimv2_1b_patch14_224.apple_pt 2048-dimensional latent space, populated with samples from seven SEMASIA benchmarks. Each benchmark forms its own cluster, but semantically overlapping concepts collapse onto shared neighborhoods regardless of source: flower images from Oxford Flowers and CIFAR-100 occupy the same region, as do large mammals and vehicles drawn from CIFAR-10, CIFAR-100, and Tiny-ImageNet (highlighted with the related origin dataset color).

Dependencies

This project uses uv for Python dependency management and just as the task runner.

Install prerequisites

Install the required tools:

• uv
• just

Follow the installation instructions from their official documentation.

Setup the development environment

From the project root, run:

just setup

The setup recipe will:

• Create the .venv virtual environment (if it does not exist)
• Install all project dependencies using uv

After the command completes, the development environment will be ready to use. 🚀

Analysis

All analyses are run via just. Configurations are managed with Hydra — any default can be overridden by appending key=value pairs to the command (e.g. just tda-compare cifar10 comparison.distance=wasserstein).

Encoding

Before running any analysis on a new dataset, encode it with all available timm models:

just timm-encode DATASET   # encode locally without pushing to HuggingFace
just push DATASET          # encode + push to HuggingFace + update readme

Statistical Metrics

Compute geometric and probing properties of latent spaces, then run regression analysis over model metadata:

just compute-metrics DATASET        # compute stat metrics for a dataset
just compute-metrics-tda DATASET    # include TDA metrics (slower)
just compute-metrics-test DATASET   # quick test run limited to 5 models

# Two-phase workflow for large runs
just compute-download               # Phase 1: download and cache all latents
just compute-compute                # Phase 2: compute metrics from cached latents

just stat                           # run regression analysis on precomputed metrics
just stat-plot                      # plot regression results

Graph Signatures

Extract KNN-graph structural metrics (cycles, Wiener index, eigengap, etc.) from latent spaces:

just tsp-extraction

Prototype Analysis

Compare prototype structures across pairs of models:

just proto-compare                             # default model pair from config
just proto-compare-dataset DATASET            # override dataset
just proto-compare-models MODEL_A MODEL_B     # override model pair

Latent Space Alignment

Evaluate alignment methods (proto, CCA, linear) that transmit embeddings from model A into model B's space, and visualize the results:

just alignment                                 # run evaluation with config defaults
just alignment-plot                            # plot accuracy/MSE vs compression ratio (k)

# PC correlation heatmap between two models on a dataset
just plot-heatmap MODEL_A MODEL_B DATASET

Prototype-based alignment using lstsq probing on the transmitted space:

just proto-alignment                           # default config
just proto-alignment-dataset DATASET          # override dataset
just proto-alignment-models MODEL_A MODEL_B   # override model pair

Notebooks

Interactive marimo notebooks for exploring the dataset and results are located in the notebooks/ directory:

Notebook	Description
semasia.py	Main dataset overview and exploration
latent_exploration.py	Visualize and explore individual latent spaces
compare_latents.py	Side-by-side comparison of latent spaces
model_semantics.py	Semantic structure of a model's latent space sampled across multiple datasets
models_exploration_analysis.py	Model registry EDA
prototype_alignment.py	Interactive prototype-based alignment walkthrough
section2.py	Experiments from Section 2 of the paper

To launch the marimo editor for all notebooks, run:

just marimo

This opens the marimo notebook editor (marimo edit .) via uv, letting you browse and run any notebook in the project. Alternatively, you can open the main notebook directly in the browser via the badge at the top of this README.

Citation

If you find this code useful for your research, please consider citing the following paper:

@misc{pandolfo2026semasialargescaledatasetsemantically,
      title={SEMASIA: A Large-Scale Dataset of Semantically Structured Latent Representations}, 
      author={Mario Edoardo Pandolfo and Enrico Grimaldi and Lorenzo Marinucci and Leonardo Di Nino and Simone Fiorellino and Sergio Barbarossa and Paolo Di Lorenzo},
      year={2026},
      eprint={2605.09485},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2605.09485}, 
}

Authors

• Mario Edoardo Pandolfo
• Enrico Grimaldi
• Lorenzo Marinucci
• Leonardo Di Nino
• Simone Fiorellino
• Sergio Barbarossa
• Paolo Di Lorenzo

Used Technologies