Skip to content

Commit f3e8548

Browse files
saishshinde15saishshinde15
committed
Prepare public release: remove sensitive files, update README
- Remove internal strategy docs (STATUS, ROADMAP, PITCH, PRODUCT_VISION) - Remove unpublished papers and arxiv submission bundle - Remove research internals (metrics, benchmarks, paper analyses) - Remove deployment configs (fly.toml, railway.json) - Remove reference papers (DeepSeek_V4.pdf) - Update .gitignore with public repo exclusions - Create professional README with quickstart and architecture overview
1 parent 99a0ccc commit f3e8548

195 files changed

Lines changed: 7387 additions & 70365 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,3 +57,52 @@ texput.log
5757
node_modules/
5858
.agents/
5959
.commandcode/
60+
.claude/
61+
62+
# =============================================================================
63+
# PUBLIC REPO EXCLUSIONS - Strategic/Internal documents
64+
# =============================================================================
65+
66+
# Internal strategy & status docs
67+
docs/STATUS.md
68+
docs/ROADMAP.md
69+
docs/PITCH.md
70+
docs/PRODUCT_VISION.md
71+
docs/PROJECT_STATUS.md
72+
docs/SCM_PRODUCTIZATION_STATUS_2026-05-01.md
73+
docs/RESEARCH_DOC_AUDIT_2026-04-29.md
74+
docs/SCM_RESEARCH_LOG.md
75+
docs/HUMAN_MEMORY_REVOLUTION_READINESS.md
76+
docs/HOSTED_DEMO.md
77+
docs/DEMO_VIDEO_SCRIPT.md
78+
docs/HUMAN_LIKE_USER_TRY.md
79+
docs/HUMAN_MORE.md
80+
docs/SCM_PAPER_BASELINE_2026-04-29.md
81+
82+
# Paper strategy & unpublished papers
83+
research/arxiv_submission/
84+
research/SCM_arXiv_Paper_v2.md
85+
research/SleepAI_arXiv_Paper.md
86+
research/SCM_Final_Paper.tex
87+
research/SCM_Final_Paper.pdf
88+
research/SCM_Final_Paper.aux
89+
research/SCM_Final_Paper.out
90+
research/SCM_Final_Paper.toc
91+
research/latex/
92+
research/papers/
93+
research/03_SleepAI_Architecture.md
94+
research/04_Comparative_Gap_Analysis.md
95+
research/10_SCM_Human_Memory_Blueprint.md
96+
research/11_SCM_HME_Phases_Document.md
97+
98+
# Reference papers (copyright)
99+
DeepSeek_V4.pdf
100+
101+
# Deployment configs (internal)
102+
fly.toml
103+
railway.json
104+
105+
# Benchmark artifacts (internal metrics)
106+
research/metrics/
107+
research/reproducibility/
108+
research/benchmarks/

‎DeepSeek_V4.pdf‎

-4.27 MB
Binary file not shown.

‎README.md‎

Lines changed: 93 additions & 158 deletions
Original file line numberDiff line numberDiff line change
@@ -1,205 +1,140 @@
1-
# SCM — Memory that works like yours
1+
# SCM — Sleep-Consolidated Memory for Language Agents
22

3-
> **Other memory layers store facts. SCM learns from them while you're idle.**
3+
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
4+
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
5+
[![Tests](https://img.shields.io/badge/tests-322-green.svg)](tests/)
46

5-
SCM (Sleep-Consolidated Memory) is the first agent memory layer with both a **wake phase** and a **sleep phase** — like the only memory system in nature that actually works.
7+
**Other memory layers store facts. SCM learns from them while you're idle.**
68

7-
[📄 Paper (35 pages)](research/SCM_Final_Paper.pdf) · [🛠 Deployment guide](docs/DEPLOYMENT.md) · [🔌 Integrations](docs/INTEGRATIONS.md) · [🗺 Roadmap](docs/ROADMAP.md) · [📊 Benchmarks](docs/BENCHMARKS.md)
9+
SCM is the first open-source memory architecture for language agents that implements a complete biological memory lifecycle: bounded working memory, selective encoding, sleep-stage consolidation (NREM + REM), adaptive forgetting, contradiction-safe versioning, and autonomous learning during idle time.
810

9-
---
11+
## Key Results
1012

11-
## What's different
13+
| Metric | Value |
14+
|--------|-------|
15+
| Disambiguation recall (with sleep) | **0.9052** |
16+
| Disambiguation recall (awake-only) | **0.0** |
17+
| Noise reduction | **90.9%** |
18+
| One-shot recall accuracy | **1.0** |
19+
| Retrieval latency | **<0.3ms** |
20+
| Regression tests | **322 passing** |
1221

13-
Every other agent memory product does the same thing in different shapes: **store facts, retrieve facts**. They never think between sessions. Your agent forgets to think the moment you stop talking to it.
22+
## Quick Start
1423

15-
That's not how memory works in any system that does it well — including yours. **Sleep is when memory consolidates.** When you sleep, your hippocampus replays the day's experiences and your cortex abstracts patterns from them. You wake up with a *better* version of yesterday's understanding, not the same one.
16-
17-
SCM does both phases:
18-
19-
| Phase | What happens | Bio analog | SCM modules |
20-
|---|---|---|---|
21-
| **Wake** | Selective attention. Encoding-by-importance. Cue-driven retrieval. Contradiction handling. Bounded working memory (~7 items). | Hippocampal encoding, working memory, cue-driven recall | Phases 1-5 |
22-
| **Sleep** | Pattern abstraction. Contradiction resolution. Adaptive forgetting. Knowledge-gap detection and curiosity-driven filling. Wake summary report. | NREM consolidation, REM dreaming, synaptic homeostasis | Phase 7 (M1-M6) |
23-
24-
**The result the user sees:** when they come back from being away, the agent reports what it noticed.
25-
26-
```
27-
> What did you notice while I was away?
28-
29-
While you were away I noticed three things:
30-
• You've changed jobs — I've moved you from Northstar Robotics to Atlas Labs.
31-
• Your Tuesday-morning runs and Friday-night dinners with Mara have become weekly patterns.
32-
• You've mentioned 'OAuth flow' five times without explaining it; I read up on it.
33-
(Authorization protocol, redirect-based, token + scope.)
24+
```bash
25+
pip install scm-memory
3426
```
3527

36-
This moment is the product. **No other open-source memory system has it.**
37-
38-
---
39-
40-
## Five-line quickstart
41-
4228
```python
4329
from scm import SCMEngine
4430

4531
engine = SCMEngine(profile="chatbot")
46-
engine.message("Hi, I'm Saish. I run every Tuesday morning.")
47-
engine.message("Tuesday again — out for a 5K.")
48-
engine.sleep("deep")
49-
print(engine.wake_summary().narrative)
50-
# → "While you were away I noticed Tuesday-morning running has become a pattern..."
51-
```
52-
53-
Or, drop the SCM MCP server into Claude Desktop / Cursor / any MCP client and add five tools (`add_memory`, `search_memory`, `consolidate`, `wake_summary`, `forget`) automatically. See [`docs/INTEGRATIONS.md`](docs/INTEGRATIONS.md).
54-
55-
---
56-
57-
## Install
58-
59-
```bash
60-
pip install scm-memory
61-
```
32+
engine.chat("My name is Saish and I live in Bangalore.")
33+
engine.chat("I love filter coffee.")
6234

63-
For local development from this repository:
35+
# Force a sleep cycle to consolidate
36+
engine.force_sleep("deep")
6437

65-
```bash
66-
git clone https://github.com/Saish15/sleepai.git
67-
cd sleepai
68-
python -m venv venv && source venv/bin/activate
69-
pip install -e ".[dev]"
38+
# Ask about stored memories
39+
response, meta = engine.chat("Where do I live?")
40+
print(response) # "You live in Bangalore."
7041
```
7142

72-
For the recommended privacy-first profile (free, local, no cloud calls):
73-
74-
```bash
75-
ollama pull nomic-embed-text # 274 MB embedding model (recommended)
76-
ollama pull llama3.2:latest # ~4 GB chat / extraction model
77-
```
43+
## Architecture
7844

79-
That's it. SCM auto-detects Ollama and uses it. Four deployment profiles documented in [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md):
45+
SCM implements seven phases of a biological memory lifecycle:
8046

81-
| Profile | LLM | Embedding | Cost / 30 turns | Privacy |
82-
|---|---|---|---|---|
83-
| **A** Offline-only | heuristic regex | sentence-transformers MiniLM | $0 | 100% local |
84-
| **B** Ollama-only (recommended) | Ollama llama3 | Ollama nomic-embed-text | $0 | 100% local |
85-
| **C** Hybrid | DeepSeek-chat | Ollama nomic-embed-text | ~$0.04 | text→cloud, vectors local |
86-
| **D** All-cloud | OpenAI gpt-4o-mini | OpenAI text-embedding-3-large | ~$0.06 | all→cloud |
47+
| Phase | Component | Function |
48+
|-------|-----------|----------|
49+
| 1 | AttentionGate | Selective encoding with 4-tier intensity |
50+
| 2 | EventCompiler | Structured event frames (who/what/when/where/why) |
51+
| 3 | SpreadingActivation | Cue-driven graph propagation retrieval |
52+
| 4 | SleepKernel | Micro-sleep + Deep-sleep (NREM + REM) |
53+
| 5 | ForgettingDynamics | Adaptive value-based forgetting |
54+
| 6 | Guardrails | Paraphrase, evaluation harnesses |
55+
| 7 | IdleLearner | Autonomous learning during user idle time |
8756

88-
---
57+
## Deployment Profiles
8958

90-
## Works seamlessly with any LLM and any harness
59+
| Profile | LLM | Embedding | Cost | Privacy |
60+
|---------|-----|-----------|------|---------|
61+
| A: Offline | heuristic | sentence-transformers | $0 | 100% local |
62+
| B: Ollama | llama3.2 | nomic-embed-text | $0 | 100% local |
63+
| C: Hybrid | DeepSeek | nomic-embed-text | ~$0.04/30 turns | text to cloud |
64+
| D: All-cloud | GPT-4o-mini | text-embedding-3-large | ~$0.06/30 turns | all to cloud |
9165

92-
SCM doesn't care which LLM you use. Concept extraction goes through `LLMExtractor`; switch providers with one env var:
66+
## Examples
9367

9468
```bash
95-
LLM_PROVIDER=ollama # local, free
96-
LLM_PROVIDER=deepseek # cheap cloud
97-
LLM_PROVIDER=openai # premium
98-
# Anthropic / Voyage / Together / any OpenAI-compatible endpoint also works
99-
```
100-
101-
Same for embeddings — sentence-transformers, Ollama, or any OpenAI-compatible provider.
102-
103-
Drop SCM behind any agent framework:
104-
- **MCP server** (Claude Desktop, Cursor, ChatGPT-with-MCP) — `scm mcp` in your config
105-
- **REST API** (`/v1/memories`, `/v1/wake-summary`, etc.) — OpenAPI 3.1 spec at `/v1/openapi.json`
106-
- **Python SDK** — `from scm import SCMEngine`
107-
- **JavaScript SDK** — `import { SCM } from "scm-memory"` (Node 18+, Bun, browsers, Edge runtime)
108-
- **LangChain memory adapter** — drop-in `BaseChatMemory` subclass
109-
- **Plain HTTP** — POST `/v1/memories` from anything
69+
# Run the quickstart
70+
python examples/01_quickstart.py
11071

111-
Tool definitions exported in OpenAI / Anthropic / Gemini / OpenAPI formats from one source. See [`docs/INTEGRATIONS.md`](docs/INTEGRATIONS.md) for 7 integration recipes.
72+
# Run with wake summary
73+
python examples/02_wake_summary.py
11274

113-
---
75+
# Run with Ollama (local LLM)
76+
python examples/03_with_ollama.py
77+
```
11478

115-
## What SCM is NOT
79+
## Testing
11680

117-
- ❌ Not a vector database. It uses one (NetworkX in-memory + SQLite/Postgres backing), but the value is the lifecycle, not the index.
118-
- ❌ Not an LLM. Bring your own.
119-
- ❌ Not a chat UI. It's the memory backend; the UI is your problem (or wire it to the included `/static` demo page).
120-
- ❌ Not just a fact-extraction prompt over a vector DB. SCM is a complete memory pipeline — encoding, binding, retrieval, consolidation, forgetting, schema abstraction.
121-
- ❌ Not 100% production-polished yet. v0.7.x is research-grade with strong tests; the deployment guide closes most rough edges. See [`docs/STATUS.md`](docs/STATUS.md) for an honest current-state read.
81+
```bash
82+
# Run all tests
83+
pytest tests/ -v
12284

123-
---
85+
# Run specific test suite
86+
pytest tests/ -k "sleep" -v
12487

125-
## Honest comparison
88+
# Run with coverage
89+
pytest tests/ --cov=src --cov-report=html
90+
```
12691

127-
We don't claim to dominate every memory benchmark. We dominate a different axis.
92+
## Integrations
12893

129-
| Capability | Stateless vector layer | **SCM** |
130-
|---|---|---|
131-
| Vector retrieval | ✅ | ✅ |
132-
| Working-memory bound | ❌ | ✅ |
133-
| Event-structured encoding | ❌ | ✅ |
134-
| Spreading-activation retrieval | ❌ | ✅ |
135-
| Contradiction-safe versioning | ❌ | ✅ |
136-
| Sleep-stage consolidation | ❌ | ✅ |
137-
| Schema extraction (REM) | ❌ | ✅ |
138-
| Wake-summary endpoint | ❌ | ✅ |
139-
| Curiosity-driven gap-filling | ❌ | ✅ |
140-
| Idle-aware autonomous learning | ❌ | ✅ |
94+
- **LangChain**: `SCMMemory` adapter + tool definitions
95+
- **MCP Server**: stdio + HTTP transports for Claude Desktop, Cursor
96+
- **REST API**: `/v1/memories`, `/v1/wake-summary`, `/v1/health`
97+
- **Python SDK**: `from scm import SCMEngine`
98+
- **JavaScript SDK**: `npm install scm-memory`
14199

142-
SCM does both jobs: vector retrieval (the table-stakes feature) plus continuous learning during idle time (the differentiator).
100+
## Documentation
143101

144-
---
102+
- [API Reference](docs/DEPLOYMENT.md)
103+
- [LangChain Guide](docs/LANGCHAIN_GUIDE.md)
104+
- [Integration Recipes](docs/INTEGRATIONS.md)
105+
- [Benchmark Results](docs/BENCHMARKS.md)
145106

146-
## Status
107+
## How It Works
147108

148-
- **322 regression tests** passing (`pytest tests/ -q`)
149-
- **143 focused regression tests** for Phase 7 + retrieval (`pytest tests/test_*spreading* tests/test_*idle* tests/test_*curiosity* -q`)
150-
- **16/16 brutal LangChain harness scenarios** passing (multi-day persona, contradiction, idle wake-summary, multi-user isolation, failure mode)
151-
- **5,561× p50 latency speedup** on `add_memory` since v0.7.2 (async ingest)
152-
- **5-40× RAM saved at multi-user scale** since v0.7.3 (embedding-model singleton)
109+
### Wake Phase
110+
During conversation, SCM encodes user input into typed semantic concepts, tags each with a 4-dimensional importance vector (novelty, emotion, task relevance, repetition), and stores recent episodes in a bounded 7-item working memory buffer.
153111

154-
See [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) for every measured number with reproduction instructions.
112+
### Sleep Phase
113+
When the user goes idle, SCM enters sleep mode:
114+
- **NREM**: Replays episodes, strengthens co-occurring concepts via Hebbian plasticity, applies synaptic downscaling
115+
- **REM**: Generates novel concept combinations, creates new associative links
116+
- **Forgetting**: Removes low-value memories while preserving important ones
155117

156-
---
118+
### The Result
119+
When you return and ask "What did you notice while I was away?", SCM produces a narrative like:
120+
> "While you were away I noticed three things: You've changed jobs — I've moved you from Northstar Robotics to Atlas Labs. Your Tuesday-morning runs and Friday-night dinners with Mara have become weekly patterns. You've mentioned 'OAuth flow' five times without explaining it; I read up on it."
157121
158-
## Architecture
122+
## Citation
159123

124+
```bibtex
125+
@article{scm2026,
126+
title={SCM: Autonomous Lifelong Learning for Language Agents via Sleep-Stage Memory Consolidation},
127+
author={SCM Research Team},
128+
year={2026}
129+
}
160130
```
161-
┌───────────────────────────────┐
162-
│ YOUR AGENT / HARNESS │
163-
│ (LangChain, Claude, custom) │
164-
└─────────────┬─────────────────┘
165-
│
166-
▼
167-
┌───────────────────────────────────────────┐
168-
│ SCM CORE │
169-
│ Wake-phase: encode → bind → retrieve │
170-
│ Sleep-phase: consolidate → schema → gap │
171-
│ Phase 7: M1 idle daemon │
172-
│ M2 cross-session pool │
173-
│ M3 schema extraction (REM) │
174-
│ M4 wake-summary endpoint │
175-
│ M5 curiosity engine │
176-
│ M6 lifecycle policy │
177-
└────────┬─────────────────────┬─────────────┘
178-
│ │
179-
▼ ▼
180-
┌───────────────────┐ ┌───────────────────┐
181-
│ LLM BACKEND │ │ EMBEDDING │
182-
│ Ollama/DeepSeek/ │ │ Ollama/OpenAI/ │
183-
│ OpenAI/etc. │ │ sentence-trans. │
184-
└───────────────────┘ └───────────────────┘
185-
```
186-
187-
35-page paper at [`research/SCM_Final_Paper.pdf`](research/SCM_Final_Paper.pdf). Documents architecture, formal definitions for all 11 equations, brutal-testing methodology, ALB pilot, LoCoMo + LoCoMo++ honest comparisons, encoder-dependence analysis.
188-
189-
---
190-
191-
## Project status
192-
193-
This project is in **active development**, working toward a public launch. See [`docs/STATUS.md`](docs/STATUS.md) for an unvarnished read of where it stands as a product, and [`docs/ROADMAP.md`](docs/ROADMAP.md) for what's coming next. The paper is ready and staged for arXiv submission but **held until the product-readiness checklist completes** — papers without products fade.
194-
195-
---
196131

197132
## License
198133

199-
MIT. See [`LICENSE`](LICENSE).
200-
201-
---
134+
MIT License — see [LICENSE](LICENSE) for details.
202135

203136
## Contact
204137

205-
`blobopera@proton.me`
138+
- Author: Saish Shinde
139+
- Email: blobopera@proton.me
140+
- GitHub: [github.com/clyrai/SCM_OpenSource](https://github.com/clyrai/SCM_OpenSource)

0 commit comments

Comments
 (0)