Overview

This library provides algorithms for creating semantic descriptions of mineral-related tables for data extraction. It also integrates with SAND to interactively curate the semantic descriptions.

Installation

MinMod KG Setup

To work with MinMod KG, we need the ta2-minmod-data and ta2-minmod-kg repositories, and the default folder structure is:

<DARPA-CRITICALMAAS-DIR>
├── data                      # for storing databases
├── ta2-minmod-data           # ta2-minmod-data repository
├── ta2-minmod-kg             # ta2-minmod-kg repository
└── ta2-table-understanding   # ta2-table-understanding repository

To setup the above structure, you can run:

git clone --depth 1 https://github.com/DARPA-CRITICALMAAS/ta2-minmod-data
git clone --depth 1 https://github.com/DARPA-CRITICALMAAS/ta2-minmod-kg
git clone --depth 1 https://github.com/DARPA-CRITICALMAAS/ta2-table-understanding
mkdir data

To make it easier to run the next commands, we will use an environment variable MINMOD_DIR to denote the <DARPA-CRITICALMAAS-DIR> folder. If you are in the <DARPA-CRITICALMAAS-DIR> directory, you can run this command to set up the environment variable:

export MINMOD_DIR=$(pwd)

Note: The folder structure is fully customizable. Please see the Configuration Section for more information.

Setup dependencies

You can setup the library and its dependencies using a virtual environment or with Docker (recommended).

We use uv as our package manager (you need to have it installed on your machine first). To install the library and its dependencies, run uv sync in the root directory of this repository. Then, you can run source .venv/bin/activate to activate the virtual environment or use uv run <command> to run the commands in the virtual environment.

cd ta2-table-understanding
python -m venv .venv
uv sync
cd ..

With the working folder structure setup, we can build the necessary databases (entities, ontology classes, and properties) by running:

export CFG_FILE=$MINMOD_DIR/ta2-minmod-kg/config.yml.template
cd ta2-table-understanding
uv run python -m tum.make_db [--project <project=minmod>]
cd ..

You need to setup the environment variables by copying the .env.template file to .env and updating the variables as needed. Remember to load them into your shell environment.

Alternatively, you can use Docker to install the library as below (remember to setup the environment variables first -- docker will pick up the .env file automatically):

cd ta2-table-understanding
bash scripts/build_docker.sh
cd ..

Usage

How to model and publish data to MinMod Knowledge Graph: https://www.youtube.com/watch?v=iAZeKYzepSg

API

Check out the demo notebook on how to use the library programmatically.

GUI

Alternatively, you can use the SAND UI to interactively load a table, create the semantic description, and extract data from the table.

To install SAND, you can run the following commands:

source .venv/bin/activate
pip install web-sand sand-drepr

1. Setup SAND (run only once): uv run python -m sand init -d $MINMOD_DIR/data/minmod/sand.db
2. Start SAND: uv run python -m sand start -d $MINMOD_DIR/data/minmod/sand.db -c $MINMOD_DIR/ta2-table-understanding/minmod.sand.yml

If you use Docker, it is already included in the Docker container. You can start with docker compose up.

Configuration

1. The working folder <DARPA-CRITICALMAAS-DIR> can be modified by setting the environment variable CRITICAL_MAAS_DIR.
2. To customize SAND, you can update the file minmod.sand.yaml