Data Trace Agent

An agent that links Git and DVC metadata with direct access to BigQuery and GCS to answer questions about the evolution and content of datasets.

Built with Google ADK and deployed on Vertex AI Agent Engine.

Architecture

flowchart TD
    subgraph local["Local Development"]
        DEV[make dev] --> ADK[adk web :8000]
        ADK --> ROOT[root_agent]
        ROOT --> TOOLS[agent/tools/]
        ROOT --> PROMPTS[prompts/ + prompts.yaml]
    end

    subgraph ci["CI/CD — GitHub Actions"]
        PUSH[git push] --> CI[ci.yml\nlint · format · typecheck · tests]
        PUSH --> SEC[security.yml\nCodeQL · pip-audit · secret scan]
        PR[pull request] --> EVAL[eval.yml\nprompfoo red-team]
        CI & SEC & EVAL -->|all green on main| DEPLOY[deploy.yml]
    end

    subgraph gcp["Google Cloud Platform"]
        DEPLOY --> ENGINE[Vertex AI Agent Engine]
        ENGINE --> MODEL{MODEL_PROVIDER}
        MODEL --> G[Gemini 2.5 Pro]
        MODEL --> CL[Claude via LiteLLM]
        MODEL --> OAI[GPT-4o via LiteLLM]
        ENGINE --> LOG[Cloud Logging]
        ENGINE --> TRACE[Cloud Trace]
    end

    CLIENT[API Consumer] -->|REST| ENGINE

Loading

Agent Architecture

flowchart TD
    Actor([Actor])

    subgraph GCP["Google Cloud Platform — Vertex AI Agent Engine"]
        direction TB
        root_agent((root_agent))

        subgraph SubAgents["Sub-agents"]
            direction LR
            metadata_agent((metadata\nagent))
            data_agent((data analysis\nagent))
            code_agent((code analysis\nagent))
            bq_agent((bigquery\nagent))
        end

        root_agent --> metadata_agent
        root_agent --> data_agent
        root_agent --> code_agent
        root_agent --> bq_agent

        BigQuery[(BigQuery)]
        GCS[(GCS\nDVC remote)]

        bq_agent -->|SQL query| BigQuery
        data_agent -->|dvc pull| GCS
    end

    subgraph GitLab["GitLab"]
        DVC_registry["DVC registry\n(*.meta.yaml + *.dvc)"]
        subgraph ProjectRepos["Project repositories"]
            direction LR
            RepoA[Project A]
            RepoB[Project B]
        end
        DVC_registry -.->|tracks| ProjectRepos
    end

    Actor -->|REST| root_agent
    root_agent & metadata_agent & data_agent -->|clone / read| DVC_registry
    code_agent -->|clone at rev_lock| ProjectRepos

Loading

Quickstart

Prerequisites

• Python 3.11+, uv, Node.js 20+
• gcloud CLI authenticated

Local development

make install              # install dependencies
cp .env.example .env      # configure environment variables
make dev                  # run at http://localhost:8000

Run tests

make test                 # unit tests with coverage
make eval                 # promptfoo red-team evaluation

Deploy to GCP

make setup-gcp            # one-time GCP bootstrap (creates SA, bucket, key)
make deploy-dev           # deploy to dev Agent Engine resource
make deploy-prod          # deploy to prod

Make targets

Target	Description
make dev	Run agent locally at http://localhost:8000
make test	Unit tests with coverage
make eval	Prompt security evaluation (promptfoo)
make lint	Ruff lint check
make format	Ruff formatter
make typecheck	Pyright
make deploy-dev	Deploy to Agent Engine (dev)
make deploy-prod	Deploy to Agent Engine (prod)
make logs	Stream Cloud Logging
make traces	Open Cloud Trace in browser
make setup-gcp	One-time GCP bootstrap
make pre-commit	Run all pre-commit hooks

Environment variables

Variable	Required	Description
GOOGLE_CLOUD_PROJECT	Deploy	GCP project ID
GOOGLE_CLOUD_LOCATION	Deploy	Vertex AI region (default: europe-west1)
GCS_STAGING_BUCKET	Deploy	GCS bucket for Agent Engine artefacts
REPO_URL	No	Remote Git URL of the DVC registry; cloned on first use
GIT_AUTH_TOKEN	No	Git token (deploy token / PAT) for cloning private repos in headless runtimes
GIT_AUTH_HOST	No	Host the token is valid for (defaults to the host of REPO_URL)
GIT_AUTH_USERNAME	No	Username for token-HTTPS clones (default: oauth2)
GIT_AUTH_TOKEN_SECRET	No	Secret Manager secret id for the Git token (preferred at deploy)
AGENT_ENGINE_SERVICE_ACCOUNT	No	Service account the deployed agent runs as; grant it read on DVC remote buckets
DVC_CONFIG_LOCAL	No	Verbatim DVC workspace config for config-based remotes (WebDAV, SSH)
DVC_CONFIG_LOCAL_SECRET	No	Secret Manager secret id for DVC_CONFIG_LOCAL
AGENT_ENGINE_RESOURCE_NAME	No	Existing resource to update (omit = create new)
MODEL_PROVIDER	No	google | anthropic | openai | litellm
GOOGLE_API_KEY	Local dev	Not needed on GCP (uses ADC)
ANTHROPIC_API_KEY	If provider=anthropic
OPENAI_API_KEY	If provider=openai
SERPAPI_API_KEY	No	Enables live web search; omit for stub

Model providers

Set MODEL_PROVIDER in .env:

Value	Model
google (default)	Gemini 2.5 Pro
anthropic	Claude Opus 4.8 via LiteLLM
openai	GPT-4o via LiteLLM
litellm	Any model — set LITELLM_MODEL

Logging and traces

make logs     # stream Cloud Logging (requires GOOGLE_CLOUD_PROJECT in .env)
make traces   # open Cloud Trace console in browser

Agent Engine emits traces and structured logs automatically — no instrumentation needed.

Security

Prompt injection, jailbreak, and PII tests run automatically on every PR via promptfoo. Add test cases in tests/evals/promptfoo.yaml. See SECURITY.md for the vulnerability disclosure policy.

Contributing

See CONTRIBUTING.md. AI assistants: read CLAUDE.md for full project context and working instructions.