Skip to content

Need help to reproduce the Terminal Bench 2.1 score of GLM 5.2 #100

Description

@mzr1996

Hello, I'm retrying to reproduce the Terminal Bench 2.1 score with Terminus 2.
Here is my command to deploy GLM 5.2 (sglang v0.5.13.post1 on H200)

sglang serve \
    --model-path ~/.cache/huggingface/hub/models--zai-org--GLM-5.2-FP8/snapshots/3722e206c8461c750e5ab73f5528eba42d0c6a37 \
    --served-model-name GLM-5.2-FP8 \
    --tp 8 \
    --mem-fraction-static 0.9 \
    --cuda-graph-max-bs 32 \
    --max-running-requests 32 \
    --reasoning-parser glm45 \
    --tool-call-parser glm47

And here is my Harbor (v0.16.1) config:

job_name: tb21-glm5-2
agents:
  - name: terminus-2
    model_name: hosted_vllm/glm-5.2
    max_timeout_sec: 14400
    kwargs:
      api_base: http://localhost:8000/v1
      parser_name: json
      temperature: 1.0
      max_turns: 500
      interleaved_thinking: true
      llm_call_kwargs:
        extra_body:
          chat_template_kwargs:
            clear_thinking: false
      reasoning_effort: "max"
      model_info:
        max_input_tokens: 256000
        max_output_tokens: 48000
        input_cost_per_token: 0.000003
        output_cost_per_token: 0.000015

timeout_multiplier: 1.0
agent_timeout_multiplier: 16.0
verifier_timeout_multiplier: 8
n_concurrent_trials: 32
verifier:
  max_timeout_sec: 1800.0

The harbor start command:

uv run harbor run -d terminal-bench/terminal-bench-2-1 -c glm5-2.yaml -n 32

And I can only get 62~64 score.
Could you please check my evaluation configuration?

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions