Skip to content

Add DeepSeek benchmark coverage for v0.2.3 - #17

Merged
HuRuilizhen merged 1 commit into
mainfrom
feat/0.2.3-benchmark-deepseek
Apr 18, 2026
Merged

Add DeepSeek benchmark coverage for v0.2.3#17
HuRuilizhen merged 1 commit into
mainfrom
feat/0.2.3-benchmark-deepseek

Conversation

@HuRuilizhen

Copy link
Copy Markdown
Owner

Summary

This PR is the second v0.2.3 slice for evaluating the new DeepSeek provider in the existing local benchmark workflow.

The main goal is to extend the benchmark runner so the same benchmark input can be compared across:

  • heuristic
  • spacy
  • llm:ollama
  • llm:mistral
  • llm:deepseek

Included

  • adds DeepSeek support to the local benchmark runner

    • extends benchmark/run_benchmark.py so --provider deepseek is accepted
    • keeps the benchmark input contract unchanged
    • does not change benchmark result schema or release packaging boundaries
  • validates DeepSeek on the existing 100-sample TechQA benchmark input

    • heuristic: 1497 concepts / 1973 mentions / 0 empty
    • spacy: 649 concepts / 870 mentions / 0 empty
    • llm:ollama:qwen3.5:4b: 374 concepts / 411 mentions / 0 empty
    • llm:mistral:mistral-small: 297 concepts / 319 mentions / 14 empty
    • llm:deepseek:deepseek-chat: 352 concepts / 386 mentions / 5 empty
  • confirms DeepSeek's current position in the provider set

    • noticeably tighter than heuristic and spacy
    • materially fewer empty paragraphs than the current mistral-small run
    • close to ollama in concept density while still remaining selective

Validation

Repository checks:

  • .venv/bin/pytest -q tests/test_benchmark.py
  • .venv/bin/ruff check benchmark/run_benchmark.py tests/test_benchmark.py
  • .venv/bin/pyright benchmark/run_benchmark.py tests/test_benchmark.py

Live benchmark checks:

  • ran deepseek-chat against the existing 100-sample TechQA benchmark input
  • verified the new DeepSeek result file was freshly generated before summarization
  • summarized DeepSeek alongside the existing heuristic / spacy / ollama / mistral runs

Notes

  • benchmark inputs and result files remain local under experiment/ and are intentionally not version-controlled
  • this PR does not update release materials yet; that remains the final release/0.2.3-prep slice
  • the DeepSeek API key used for live validation should be rotated after this verification

@HuRuilizhen
HuRuilizhen merged commit 743d8ec into main Apr 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant