AgentBeats: PowerPoint Benchmark

This repo contains a PowerPoint changeset benchmark (lots of heavy .pptx files) and 3 A2A agents:

• Green controller: orchestrates evaluation
• White (GPT-4o): contender 1
• White (GPT-5.2): contender 2

If you encounter any issues running this benchmark, reach out to emil.scheffer@berkeley.edu and I am happy to help.

Git LFS (required)

The benchmark includes many large PowerPoint assets, so *.pptx files are stored in Git Large File Storage (Git LFS). If you clone without LFS, you may only get pointer files and the benchmark won’t run.

Clone with Git LFS

# one-time on your machine
git lfs install

# clone + fetch LFS objects
git clone <YOUR_REPO_URL>
cd <YOUR_REPO_DIR>
git lfs pull

Project Structure

agentbeats/   # green + two white agents (Python)
benchmark/    # benchmark runtime API + scenarios (Node/TS)

Run the benchmark (AgentBeats setup)

0) Start the benchmark server (required)

The agents talk to a local benchmark API server (default: http://localhost:5050).

cd benchmark
npm install
npm run start

1) Launch the 3 agents (Cloudflare tunnel → controller)

For each agent:

• First forward the controller PORT with Cloudflare.
• Then launch the controller with a ROLE and two ports:
  • PORT: the public-facing controller port (this is the one you tunnel)
  • AGENT_PORT: the internal agent port the controller uses
• Set CLOUDRUN_HOST to the Cloudflare tunnel domain only (no https://).

Below is an example port layout:

• green: PORT=8010, AGENT_PORT=8011
• white (GPT-4o): PORT=8020, AGENT_PORT=8021
• white (GPT-5.2): PORT=8030, AGENT_PORT=8031

Green controller

Terminal A (tunnel):

cloudflared tunnel --url http://localhost:8010

Terminal B (controller). Replace my-green.trycloudflare.com with the printed tunnel domain (no https://):

cd agentbeats/green_controller
OPENAI_API_KEY=... \
HTTPS_ENABLED=true \
CLOUDRUN_HOST="my-green.trycloudflare.com" \
ROLE=green \
HOST=0.0.0.0 \
AGENT_PORT=8011 \
PORT=8010 \
agentbeats run_ctrl

White contender 1 (GPT-4o)

Terminal A (tunnel):

cloudflared tunnel --url http://localhost:8020

Terminal B (controller):

cd agentbeats/white_gpt4o
OPENAI_API_KEY=... \
HTTPS_ENABLED=true \
CLOUDRUN_HOST="my-white4o.trycloudflare.com" \
ROLE=white \
HOST=0.0.0.0 \
AGENT_PORT=8021 \
PORT=8020 \
agentbeats run_ctrl

White contender 2 (GPT-5.2)

Terminal A (tunnel):

cloudflared tunnel --url http://localhost:8030

Terminal B (controller):

cd agentbeats/white_gpt52
OPENAI_API_KEY=... \
HTTPS_ENABLED=true \
CLOUDRUN_HOST="my-white52.trycloudflare.com" \
ROLE=white \
HOST=0.0.0.0 \
AGENT_PORT=8031 \
PORT=8030 \
agentbeats run_ctrl

After this step, your 3 agents are live and can now be registered in AgentBeats using the public Cloudflare tunnel domains (the same domains you used for CLOUDRUN_HOST).

(Optional) Run the battle locally (no AgentBeats UI)

This repo also includes a local launcher that runs the green agent and pits both white agents against the benchmark on localhost:

cd agentbeats
uv run python main.py launch