VeWorld Indexer

• Coverage Reports:

  • API
  • Common
  • Indexer

• Tests Results:

  • API
  • Common
  • Indexer
  • E2E

Prerequisites

• Docker
• Java (v21)

GitHub Packages Authentication (optional)

Normal builds resolve dependencies from Maven Central and do not require credentials. GitHub Packages credentials are only needed when resolving a dependency version that has not reached Maven Central yet, such as a SNAPSHOT or pre-release org.vechain:indexer-core.

For local Gradle builds, create a GitHub Personal Access Token (classic) with the read:packages scope and add the following to ~/.gradle/gradle.properties:

gpr.user=your-github-username
gpr.key=your-github-pat

For Docker builds, put those same properties in gradle/docker.gradle.properties or pass another readable file with GRADLE_PROPERTIES_FILE=/path/to/gradle.properties make build-image. The default Docker properties file is git-ignored and excluded from the Docker build context.

Getting Started

• To see a list of all available commands, run make help
• After starting the application, the swagger will be made available at http://localhost:8080

Option 1: Docker only

• Copy env files for the two packages ./package/<package>/.env.example to ./package/<package>/.env and fill in the values for your environment. They should work as-is for docker

• Run:

make start

Option 2: Docker + IntelliJ (Recommended for debugging)

• No need to copy the environment files if you are using IntelliJ. The default variables should connect to the infrastructure using localhost variables
• Run:

make db-all

Connecting to MongoDB

Connect for the various users with the following URIs:

• indexer - mongodb://indexer:password@localhost:27017/vechain?directConnection=true&authMechanism=DEFAULT&authSource=admin
• api - mongodb://api:password@localhost:27017/vechain?directConnection=true&authMechanism=DEFAULT&authSource=admin
• root - mongodb://root:password@localhost:27017/admin?directConnection=true&authMechanism=DEFAULT
• Go to IndexerApplication.kt inside IntelliJ and run/debug:

Restarting

• Clean and restart the DB:

make db-all

Backup MongoDB

You can back up the database by running the following command:

make db-backup

• Will back up the vechain database from localhost:27017
• The backup will be stored in the database/backups/ directory.
• The filename follows this format: database/backups/vechain-YYYYMMDDHHMMSS

You can also specify the host and port of the MongoDB instance you want to backup:

make db-backup MONGO_HOST=my-mongo-host:32423

Restore MongoDB from backup

You can restore the database by running the following command:

make db-restore

• If no backup exists, you will be prompted to specify a backup directory.
• By default, it restores from the latest backup found in the database/backups/ directory.

To restore from a specific backup folder, specify DIR:

make db-restore DIR=backup/mydatabase-20250210

To restore to a different DB:

make db-restore MONGO_HOST=myserver.com

Copy collections between databases

To copy specific collections from one MongoDB cluster to another (e.g. from a local indexer into an Atlas cluster) without doing a full backup/restore round-trip:

make db-copy-collections

This launches the interactive wrapper at database/restore/restore.sh, which prompts for source/destination URIs (with MONGO_PRESET_* env-var presets) and the collection list, then drives the underlying restore_local_dump.sh end-to-end. See database/restore/README.md for full options including non-interactive use.

Features

There are 6 indexers and 6 corresponding APIs. Each indexer can be run in isolation or all together. There is no dependency between indexers for this reason. Each indexer and API pair can be enabled using the corresponding spring profile.

• transactions - enabled with transactions profile
• nfts - enabled with nfts profile
• transfers - enabled with transfers profile
• history - enabled with history profile

As you can see from the list above, the block indexer offers the option to proxy to the Thor node. This is useful if you want the convenience of the Block endpoints without the overhead of indexing the data.

Pruner

Some of the indexers are stateful indexers. This means that records are updated with each block. In order to facilitate rollbacks we must store all previous version of each record. These records are stored in collections with a -archives postfix. As you might imagine these archive collections can get rather large over time. To prevent the collection from blowing up we have implemented an optional pruner service that can be enabled and configured with the following env variablers.

• PRUNER_INTERVAL - How frequently to run the pruner (in blocks)
• PRUNER_REMOVAL_CHUNK_SIZE - Sometimes the number of records to prune can be very large. To prevent mongoDB from blowing up we can set a chunk size for the delete operation
• PRUNER_RECORD_LIMIT - You can set a limit on the number of records to prune in each run.

Testing

• Run all the tests:

make test

Package Testing

• There are 4 packages that can be tested (api, common, e2e, indexer)
• This will run all tests in a package (unit, integration and E2E)
• Run (example for api):

make test-api
make test-common
make test-indexer
make test-e2e

E2E

• Running E2E tests will spin up all the docker infrastructure before the test and tears it down after completion. This is useful for testing the entire system, but it is slow. If you need to debug these tests, it is recommended to spin up the network manually using make clean start and remove the tasks preE2e and postE2e tasks in ./packages/e2e/build.gradle.kts (here)

API Schema Tests

Use the schema-driven harness when you need to validate the public API in a deployed environment. It fetches the OpenAPI document from /api-docs, runs Schemathesis-based checks across every documented operation, and fails if any response exceeds 2 000 ms (override by setting MAX_RESPONSE_MILLISECONDS).

• Local run: packages/api/scripts/run_api_schema_tests.sh [dead|live|<base-url>] [--base-url <base-url>] (defaults to the dead environment). Install schemathesis locally first, for example pip install "schemathesis>=3.19,<4".
• Both colours are reached through CloudFront; the ALB hostnames themselves accept nothing but distribution traffic. Dead is https://mainnet.dead.veworld.vechain.org and https://testnet.dead.veworld.vechain.org, which follow the colour swap automatically.
• GitHub Action: trigger API Tests from the Actions tab and choose the target environment, or provide a full base URL override. The workflow uses the same script and publishes Schemathesis logs and JUnit XML as artifacts.

Disaster Recovery

MongoDB Atlas snapshots are automatically exported daily to S3 (veworld-indexer-atlas-backups). In the event of data loss or cluster corruption where Atlas native snapshot restores are unavailable, the indexer can be restored from these S3 exports.

The restore process involves downloading the gzipped JSON exports to an EC2 instance, streaming them into the target Atlas cluster via parallel mongoimport processes, and rebuilding indexes from the exported metadata files.

For the full step-by-step runbook, see: Disaster Recovery Runbook

Dead Prod Atlas Restore Workflow

For Atlas-native recovery of the current dead prod color, use the Restore Dead Prod From Atlas Snapshots workflow. It restores both dead-prod Atlas clusters from the latest completed snapshots belonging to the current live prod color.

This workflow does not stop or start ECS services. The dead environment must already be quiesced before restore. Use the Stop or Start Dead Prod Environment Services workflow first, then run the restore, then start services again once the restore summary looks correct.

Recommended sequence:

1. Run Stop or Start Dead Prod Environment Services with stop.
2. Run Restore Dead Prod From Atlas Snapshots.
3. Review the restore summary artifact and Atlas restore job IDs.
4. Run Stop or Start Dead Prod Environment Services with start.
5. Run the schema or regression tests against the dead environment before switching traffic.

Content-Addressed Images

An image's identity is the hash of its Docker build context, printed by make content-hash (.github/workflows/scripts/content_hash.sh). Each service is hashed separately over the shared build files plus packages/common and its own package, which is what .dockerignore lets into the context.

The PR build resolves the hash with the default branch's copy of that script, so a branch cannot rename its own content into a tag a later release would reuse. It pushes content.<hash> to GHCR and points pr.<n>.<sha> at it. A release build re-tags the same manifest instead of building whenever that hash is already published — so a terraform-, workflow- or docs-only release does no Docker work at all, and a release that touched only the API leaves the indexer image alone.

A deploy carries this through: it reads the tag each service is actually running on the target colour — resolved through the ECS service, because a task definition family's latest revision may be one no rollout ever completed — hashes that tag's tree, and pins the service to it when the hash matches the release. Only services that changed are promoted to ECR and only their task definitions move, so the deploy has no diff for the rest. Terraform receives the four tags (net x service) as image_version_override rather than an in-place edit of the environment yaml. The step summary lists what was pinned and why.

APP_VERSION is therefore not baked into the image; it comes from the ECS task definition at runtime, so cutting a new version does not change image content. The trade-off is that a release reusing an image does not pick up new Alpine packages — the base image tag is in the hash, so a Renovate base bump is what refreshes them. Dispatch the GHCR publish with force_rebuild to rebuild without one.

Deployment & Testing

To deploy, run the Deploy workflow from the version tag you want to deploy, leaving target at auto. Both networks deploy together. The run works out which colour the release should go to and pauses at a Confirm job whose name states the plan, for example "Deploy v.1.2.3 to prod-green (dead), restoring from prod-blue snapshots first". The Plan Release job summary has the reasoning and the per-service image tags. Approve it to proceed. To do something else, reject it (nothing has been applied) and re-dispatch with target set to live or dead.

The plan follows three rules:

• A running dead colour is the one being staged, so it gets the release.
• A cold dead colour and an indexer unchanged against live: live takes the release directly, with no indexing gap.
• A cold dead colour and a changed indexer: the dead colour gets the release after its Atlas clusters are restored from the latest live snapshots. A changed indexer on live pauses indexing while it restarts, and a cold colour's data is stale.

target: live is refused for a changed indexer. skip_restore only skips the restore: the colour indexes on from whatever its Atlas clusters hold, which for a stopped colour is a stale checkpoint, not genesis. A reindex from genesis is an indexer.version bump (see AGENTS.md "Triggering an Indexer Resync"), not a skipped restore.

One deploy applies every stack — shared infra, observability, the CloudFront distributions, the restore when needed, and the application — in dependency order, so a stack with no change is a no-op rather than a run someone has to remember. AGENTS.md "One Deploy Applies Every Stack" has the order and the reasoning.

After a dead deploy: the DNS switch

A deploy to the dead colour finishes with a Next steps summary. Traffic stays where it was until you run the Switch Live Environment workflow with network all, which also publishes the draft release.

Do that once every indexer on both networks reports fully synced; the summary links the Grafana "blocks behind" panel for that colour and the dead API's indexed head (https://mainnet.dead.veworld.vechain.org/api/v1/blocks?size=1 and the testnet equivalent). A restored colour catches up in hours; a reindex from genesis takes days. If in doubt about which colour is currently live, run the Identify Live/Dead Environments workflow with the default arguments.

Following the DNS switch, please wait at least 48 hours before tearing down the old live (now dead) environment. This is to allow any remote DNS caches to update to the new live environment.

Testing

The dead colour doubles as a testing environment: it is an exact replica of live, Atlas cluster included. To put a release on it regardless of what the plan would pick, dispatch the Deploy workflow with target: dead. A cold colour is restored from the live snapshots first unless skip_restore is ticked. Merging to main deploys nothing. While the dead colour is running, auto sends every release to it, so tear it down or switch to it when testing is done.

Environment Tear-down

When testing is complete, or when a DNS switch has migrated traffic from one environment to the other, the dead environment can be safely torn down until needed again. To do this, run the Cluster Destroy workflow, and select the appropriate environment when prompted.

Leave the dead CloudFront workspace in place. A torn-down colour leaves its origin record pointing at a placeholder, so the distributions return errors until the colour is redeployed — but they cost nothing idle, and keeping them avoids re-issuing the certificate every cycle.

Further details on the CICD process can be found here