Skip to content

REPRODUCIBILITY.md

Latest commit

 

History

History
61 lines (45 loc) · 1.71 KB

REPRODUCIBILITY.md

File metadata and controls

61 lines (45 loc) · 1.71 KB
Raw

SCM Reproducibility

The July 2026 technical report is backed by local checks, ALB matrices, ablation results, an official Mem0 external baseline, and a gated OpenAI real-LLM brutal artifact.

Local Product Checks

python -m build --wheel --sdist
twine check dist/*

python -m venv /tmp/scm-smoke
/tmp/scm-smoke/bin/pip install dist/scm_memory-*.whl
cd /tmp
/tmp/scm-smoke/bin/python -c "from scm import SCMEngine; print('ok')"
/tmp/scm-smoke/bin/scm --help

Runtime Checks

scm serve --host 127.0.0.1 --port 8000
curl -s http://127.0.0.1:8000/health
curl -s http://127.0.0.1:8000/v1/health
curl -s http://127.0.0.1:8000/v1/tools?format=openai

Publication Gates

Run from the repository root with the project virtual environment:

venv/bin/python research/scripts/review_publication_artifacts.py --strict
venv/bin/python research/scripts/validate_publication_readiness.py --strict
venv/bin/python research/scripts/run_publication_local_checks.py \
  --output-json research/PUBLICATION_LOCAL_CHECKS_2026_07_09.json

Expected state for the July 2026 report:

  • Artifact review: blocker_count=0
  • Readiness: ready_for_top_claims=true
  • Local checks: all commands pass and secret_hits=[]

Paid/External Gates

The external path must read API keys only from environment variables. Do not put secrets in command-line arguments or repository files.

venv/bin/python research/scripts/run_publication_blockers.py \
  --allow-paid \
  --apply-paper-claims

Artifact Bundles

Full raw benchmark outputs are intentionally distributed as GitHub Release assets, not committed directly to Git. See ARTIFACTS.md for the expected bundle names, hashes, row counts, and evidence role.