Skip to content

deepseek_harness_recipe.md

Latest commit

 

History

History
161 lines (122 loc) · 5.26 KB

deepseek_harness_recipe.md

File metadata and controls

161 lines (122 loc) · 5.26 KB
Raw

DeepSeek Harness Recipe

This page shows how to use DeepSeek Harness as a reproducibility assistant for ml-quant-trading. The goal is narrow: run the existing benchmark and validation workflows, preserve the evidence bundle, and avoid overstating what the result proves.

DeepSeek Harness is not required by this repository. Treat the checkout as an ordinary workspace that an agent can read, run, and audit.

An optional thin plugin lives at initial-d/dsh-plugin-mlquant-benchmark. The first release is v0.1.0. It only wraps the benchmark, artifact validation, and report-drafting workflow; it does not add a trading agent, model-provider configuration, market-data access, or an agent runtime dependency to this repository.

The plugin is listed in awesome-dsh-plugin, so DSH users can discover the benchmark path from the public plugin index and then land back on this repository with a concrete report template.

Why This Fits

ml-quant-trading already has agent-friendly properties:

  • deterministic synthetic quick start;
  • protocol v1 CPU benchmark with fixed panel, seed, repetitions, and thread counts;
  • machine-readable benchmark JSON;
  • public-data validation docs;
  • explicit Reality Check boundaries;
  • structured issue templates for reproduction reports.

That makes it a useful workspace for testing whether an agent can reproduce a quant research workflow without turning benchmark output into trading claims.

Setup

Open the repository as a DeepSeek Harness workspace, then ask the agent to read:

AGENTS.md
README.md
docs/reality_check.md
docs/benchmarking.md
docs/agent_reproducibility.md

Install the development dependencies:

python -m pip install -e '.[dev]'

If you want DSH-native tool calls instead of asking the agent to run shell commands directly, install the optional plugin from GitHub:

dsh plugin --profile web add github:initial-d/dsh-plugin-mlquant-benchmark

Then add it to a Cordis composition:

- id: mlquant-benchmark
  name: dsh-plugin-mlquant-benchmark

The plugin registers:

  • mlquant_benchmark_v1_cpu;
  • mlquant_read_benchmark_json;
  • mlquant_validate_benchmark_json;
  • mlquant_draft_github_issue.

Run The Benchmark Protocol

Ask the agent to run the fixed CPU benchmark:

python scripts/benchmark_tensor_factors.py \
  --device cpu --n-dates 750 --n-stocks 1000 --window 20 \
  --repeat 10 --warmup 3 --threads 1 --interop-threads 1 --seed 42 \
  --json-out artifacts/benchmark-v1.json

The agent should preserve:

  • exact command;
  • commit SHA;
  • OS, CPU, Python, and PyTorch versions;
  • thread settings;
  • raw Markdown table;
  • artifacts/benchmark-v1.json;
  • warnings, failures, or unusually unstable cases.

The benchmark is an engineering throughput diagnostic. It is not a trading performance result and should not be compared as a controlled hardware ranking unless the environments are controlled.

Suggested DeepSeek Harness Prompt

Read AGENTS.md, docs/benchmarking.md, and docs/reality_check.md.
Run the protocol v1 CPU benchmark exactly as documented.
Return an evidence bundle with commit SHA, command, environment, raw table,
JSON path, and any unstable cases. Do not describe the result as trading alpha
or as a controlled hardware ranking.

Optional Validation Prompt

Read AGENTS.md, docs/reality_check.md, and docs/public_data_validation.md.
Run only the requested public-data validation workflow. If public data fails or
falls back to synthetic data, report that explicitly. Keep costs, turnover,
data source, and caveats next to the headline result.

Good Output

A useful DeepSeek Harness run produces an issue-ready report:

Repository: initial-d/ml-quant-trading
Commit: <sha>
Workflow: protocol v1 CPU benchmark
Command: <exact command>
Environment: <OS / CPU / Python / PyTorch / CUDA availability>
Artifact: artifacts/benchmark-v1.json
Result: <raw table>
Caveats: <warnings, instability, thermal notes, data limits>
Interpretation: engineering benchmark only; no trading claim

Post the result through the benchmark issue or reproduction report template rather than opening a new benchmark format.

If the run was performed through DeepSeek Harness, use the dedicated DeepSeek Harness benchmark report so the prompt, transcript, benchmark artifact, and discovery path stay together.

See the seed DSH benchmark report in #61. The plugin challenge is tracked in initial-d/dsh-plugin-mlquant-benchmark#1.

Plugin Boundary

The optional plugin is intentionally small. It is useful when contributors want repeatable DSH-native tool calls for:

  • protocol v1 CPU benchmark;
  • benchmark JSON validation;
  • issue-body drafting.

The plugin should not become a trading-decision layer. Public-data validation report packaging or benchmark-board update drafts can be added later only if repeated reports show that the extra surface is worth maintaining.