Skip to content
BranchesTags

Repository files navigation

Dedalus

DEDALUS aims to provide a next-generation platform that offers the best of both classical and quantum worlds, effectively partitioning big data algorithms into parts solvable by classical or quantum computing, combining the advantages of both technologies.

alt text

Table of Contents

Architecture

alt text

Requirements

Setup

First, install Docker by following the official Docker installation instructions.

  1. Clone the repository:

    git clone https://github.com/mlimnaios/Dedalus-VLDB-2027
cd Dedalus-VLDB-2027

  2. Build and start containers:

    Quick setup (recommended):

    # Native platform build
make build
# or Apple Silicon (arm64)
make build-arm64

    This will build the container images (including the TPC-H generator) and start all database containers.

    Apple Silicon note: The tpch-gen image now builds dbgen from source, so you should not need Buildx/QEMU. If you previously built images with an amd64-only base, remove the old image/volume before re-running:

    docker compose down -v

    Manual setup:

    • First, create the shared Docker network (only required once during initial setup):

      docker network create dedalusnet

    • Build the container images:

      # Native platform build
docker compose build pg-sampledb pg-job pg-tpch tpch-gen app
# Apple Silicon (arm64)
DOCKER_DEFAULT_PLATFORM=linux/arm64 docker compose build pg-sampledb pg-job pg-tpch tpch-gen app

    • Start the database containers:

      Note: On first installation, pg-job and pg-tpch will download/generate and populate their databases, which may take a considerable time.

      docker compose up -d pg-sampledb pg-job
chmod +x db/tpch/init/00-preprocess.sh
docker compose --env-file db/tpch/.env.sf1 -p tpch_sf1 up -d pg-tpch

      Container descriptions:

      • pg-sampledb: Sample database for testing

      • pg-job: IMDB's movie database (dataset from May 2013) (More info: https://event.cwi.nl/da/job/).

        • If initialization fails, manually run:

          make init-job

          Note: Place imdb.tgz in /db/job/ to skip re-downloading the dataset.

      • pg-tpch: Loads and hosts the TPC-H database. Waits for tpch-gen to complete before loading tables.

        • TPC-H Environment Configuration: TPC-H uses scale factor-specific .env files to configure database parameters:

          • Example: db/tpch/.env.sf1 - Configuration for scale factor 1

          • Example: db/tpch/.env.sf10 - Configuration for scale factor 10

          • These files contain:

            SF=1
PORT=5435
SHM_SIZE=128mb

        • tpch-gen: Generates TPC-H benchmark dataset for the specified SCALE_FACTOR.

          • To re-generate data (ARM64-safe):

            make tpch-gen SF=1

          • To (re)populate TPC-H and then start the DB:

            make populate-tpch SF=1

TPC-H first-run troubleshooting (macOS/ARM64)

If pg-tpch starts but tables are missing, check the container logs. A common first-run issue is a non-executable init script:

/docker-entrypoint-initdb.d/00-preprocess.sh: /bin/bash: bad interpreter: Permission denied

Fix it and re-run the TPC-H setup:

chmod +x db/tpch/init/00-preprocess.sh
docker compose --env-file db/tpch/.env.sf1 -p tpch_sf1 down -v
make tpch-gen SF=1
docker compose --env-file db/tpch/.env.sf1 -p tpch_sf1 up -d pg-tpch

Configuration

Pipeline Configuration

The pipeline config specifies a single query to run against a database with a chosen solver.

Example:

{
  "database": {
    "host": "pg-job",                           // Database host (e.g., localhost, or Docker service name)
    "port": 5432,                               // Database port
    "user": "postgres",                         // Database username
    "password": "postgres",                     // Database password
    "name": "job"                               // Database name
  },
  "solver": {
    "backend": "dwave",                         // Solver backend (dwave, qiskit, etc.)
    "solver": "simulated_annealing",            // Specific backend's solver algorithm
    "solver_opts": {"mode": "dwave"},           // Solver-specific options
    "weight_mode": "cardinality",               // Weight calculation mode for building QUBO weights
    "cardinality_type": "unfiltered",           // Cardinality type (relevant only when weight_mode=cardinality)
    "run_mode": "normal",                       // Execution mode
    "explain_format": "txt",                    // Query plan format (txt or json)
    "verbose": false,                           // Enable verbose logging intermediate results
    "export": true,                             // Export results to directory
    "normalize_weights": true,                  // Normalize weights
    "normalize_after_pruning": true             // Normalize weights after pruning
  },
  "query": {
    "sql": null,                                // Inline SQL query (alternative to file_path)
    "file_path": "db/job/queries/5rel/2c.sql",  // Path to query file
    "output_dir": "output"                      // Output directory for results
  }
}

Benchmark Configuration

The benchmark config runs multiple queries against multiple solvers and generates performance metrics (plots).

Example:

{
  "database": {                             // Database configuration
    "host": "pg-job",                       // pg-job / pg-sampledb / pg-tpch-sf1 / pg-tpch-sf10 etc..
    "port": 5432,
    "user": "postgres",
    "password": "postgres",
    "name": "job"                           // job / sampledb / tpch
  },
  "queries": [                              // List of queries to run
    "db/job/queries-best/4rel/3a.sql",
    "db/job/queries-best/5rel/2a.sql"
  ],
  "solvers": {
    "dwave": [                              // List of D-Wave solvers to test
      {
        "solver": "simulated_annealing",
        "solver_opts": {"mode": "dwave"}
      }
    ],
    "qiskit": [                             // List of Qiskit solvers to test
      {
        "solver": "qaoa_sim",
        "solver_opts": {"optimizer": "cobyla"}
      }
    ]
  },
  "benchmarks_dir": "benchmarks",           // Output directory for benchmark results
  "weight_modes": ["cardinality", "cpu"],   // Weight calculation modes to test
  "cardinality_type": "unfiltered",
  "normalize_weights": true,
  "normalize_after_pruning": true,
  "num_reps": 1                             // Number of repetitions per query-solver combo
}

Database Hosts

When connecting from inside Docker (app, webapp containers), use the container hostname with port 5432:

  • pg-sampledb:5432 - Sample database
  • pg-job:5432 - IMDB database
  • pg-tpch-sf1:5432 - TPC-H scale factor 1 (database name: tpch)
  • pg-tpch-sf10:5432 - TPC-H scale factor 10 (database name: tpch)

When connecting from outside Docker (your host machine), use localhost with the mapped host ports:

  • localhost:5433 - Sample database
  • localhost:5434 - IMDB database
  • localhost:5435 - TPC-H scale factor 1
  • localhost:5436 - TPC-H scale factor 10

Running Dedalus

Pipeline

Run the Dedalus pipeline with a configuration file:

make pipeline -- -cfg configs/pipeline/pipeline_config_example.json

  • You can override configuration options from the JSON file by passing one or more CLI flags (for example, --section.option value):

    make pipeline -- -cfg configs/pipeline/pipeline_config_example.json --solver.weight_mode cpu

or manually:

docker compose run --rm app python main.py -cfg configs/pipeline/pipeline_config_example.json

Output: If the user provides the export=True parameter, it creates query_YYYYMMDD_HHMMSS/ directory with pipeline_run.json run results and other intermediate data files.

Benchmark

Run benchmark on any set of database queries you specify.

make benchmark -- -cfg configs/benchmark/benchmark_config_example.json

or manually:

docker compose run --rm app python benchmark.py -cfg configs/benchmark/benchmark_config_example.json

Output: Creates bm_YYYYMMDD_HHMMSS/ directory with benchmarks.json results.

  • You can generate plots from previously produced benchmark JSON files without re-running the full benchmark.

    make plot bm_YYYYMMDD_HHMMSS

    or manually:

    docker compose run --rm app python plot.py bm_YYYYMMDD_HHMMSS

    Output: Creates plots in bm_YYYYMMDD_HHMMSS/plots/

Benchmark Driver & Collector

For paper preparation and cost estimator training, use the benchmark driver to run multiple configurations and automatically collect results showing quantum advantage:

docker compose run --rm app python scripts/benchmark_driver.py run -cfg configs/bench-driver/driver_easy_cpu.json

Output: Creates paper_data/collection_YYYYMMDD_HHMMSS/ with collected plots and JSON results from benchmarks demonstrating quantum advantage.

Available subcommands:

  • run - Run benchmarks and collect results
  • collect - Retroactively collect from existing benchmarks
  • view - View and analyze collected results
  • aggregate - Aggregate data for training

Web App

Launch the Web GUI:

make webapp

or manually:

docker compose up webapp

Open your browser at http://localhost:5001. Backend logs are visible in the webapp service output.

Cost Estimator Training

Dedalus includes a learned Cost Estimator based on Gradient Boosted Decision Trees (LightGBM) to predict solver execution-time ratios and recommend the optimal solver backend (PostgreSQL, Simulated Annealing, D-Wave QPU, or D-Wave Hybrid) for a given query.

Training the LightGBM Solver Classifier

To train the cost estimator from benchmark data and save the portable model artifacts:

python scripts/train_solver_classifier_lightgbm.py \
  --benchmark-dir benchmarks_patched \
  --save-path models/cost_estimator.txt

This performs Stratified K-Fold cross-validation, evaluates macro-F1 and speedup, performs confidence threshold analysis to calibrate safe fallback to PostgreSQL, and exports:

  • models/cost_estimator.txt — The trained LightGBM Booster model
  • models/cost_estimator_meta.json — Solver classes, fitted class IDs, feature schema, and recommended confidence threshold

Using make:

make train-default

Cost Estimator Performance Analysis & Simulation

To evaluate the Cost Estimator against actual execution times across benchmark runs:

python scripts/analyze_ce_performance.py \
  --benchmark-dir benchmarks_patched \
  --output-dir output \
  --ce-model models/cost_estimator.txt \
  --export-analysis

Or run the dispatch simulation:

python scripts/simulate_dispatch.py

Notebook Demos

  1. Launch the jupyter server:

    make jupyter

    or manually

    docker compose up jupyter

  2. Open your browser at http://localhost:8888.

  3. Run the desired demo from the /notebooks folder.

Miscellaneous

Website: dedalus.csd.uoc.gr

About

Dedalus: A quantum-classical hybrid query optimizer for relational databases. Formulates join-order optimization as pruned QUBOs for D-Wave QPUs and hybrid annealers with learned, cost-guided dispatch. [VLDB 2027 Artifact]

dedalus.csd.uoc.gr

Topics

artifact-evaluationd-wavedatabase-systemslearned-systemspostgresqlquantum-annealingquantum-computingquboquery-optimizationvldb

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages