Skip to content

Commit 0e089cb

Browse files
committed
docs: Add storage backend performance comparison and migration guides
- Add decision criteria for choosing JSONL vs SQLite backends - Add performance characteristics table (RAM usage, search speed, git-friendliness) - Add comprehensive JSONL → SQLite migration guide with 5-step process - Add SQLite → JSONL reverse migration procedure - Add migration safety tips and best practices - Include Python code examples for both migration directions Addresses documentation gaps from PR #87 (SQLite storage backend).
1 parent be7b766 commit 0e089cb

1 file changed

Lines changed: 180 additions & 0 deletions

File tree

docs/storage.md

Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,29 @@
22

33
CortexGraph supports multiple storage backends for short-term memory (STM).
44

5+
## Choosing a Backend
6+
7+
**Use JSONL when:**
8+
- Dataset size < 10,000 memories
9+
- You want human-readable, git-friendly files
10+
- You need easy inspection and manual editing
11+
- Memory usage is not a concern
12+
13+
**Use SQLite when:**
14+
- Dataset size > 10,000 memories
15+
- Memory efficiency is important (low RAM usage)
16+
- You need fast queries and filtering on large datasets
17+
- You want ACID transaction guarantees
18+
19+
**Performance Characteristics:**
20+
21+
| Backend | Memory Count | RAM Usage | Search Speed | Git-Friendly |
22+
|---------|--------------|-----------|--------------|--------------|
23+
| JSONL | < 10k | High | Fast | ✅ Yes |
24+
| JSONL | 10k - 50k | Very High | Medium | ✅ Yes |
25+
| JSONL | > 50k | Excessive | Slow | ⚠️ Large diffs |
26+
| SQLite | Any size | Low | Fast | ❌ Binary |
27+
528
## JSONL (Default)
629

730
The default backend uses human-readable JSONL (JSON Lines) files.
@@ -92,3 +115,160 @@ strength: 1.5
92115

93116
Memory content goes here...
94117
```
118+
119+
## Migration Guide
120+
121+
### Migrating from JSONL to SQLite
122+
123+
If your dataset has grown large (>10k memories) and you want to switch to SQLite for better performance:
124+
125+
**Step 1: Backup your data**
126+
127+
```bash
128+
# Backup current JSONL files
129+
cp -r ~/.config/cortexgraph/jsonl ~/.config/cortexgraph/jsonl.backup
130+
```
131+
132+
**Step 2: Export to Markdown (optional but recommended)**
133+
134+
```python
135+
from pathlib import Path
136+
from cortexgraph.tools.export import MarkdownExport
137+
from cortexgraph.storage.jsonl_storage import JSONLStorage
138+
139+
# Read from JSONL
140+
storage = JSONLStorage()
141+
storage.connect()
142+
memories = storage.list_memories()
143+
144+
# Export to Markdown as backup
145+
exporter = MarkdownExport(output_dir=Path("./backup_export"))
146+
stats = exporter.export_batch(memories)
147+
print(f"Backed up {stats.success} memories")
148+
```
149+
150+
**Step 3: Copy data from JSONL to SQLite**
151+
152+
```python
153+
from cortexgraph.storage.jsonl_storage import JSONLStorage
154+
from cortexgraph.storage.sqlite_storage import SQLiteStorage
155+
156+
# Read all data from JSONL
157+
jsonl = JSONLStorage()
158+
jsonl.connect()
159+
160+
memories = jsonl.list_memories()
161+
relations = jsonl.list_relations()
162+
163+
# Write to SQLite
164+
sqlite = SQLiteStorage()
165+
sqlite.connect()
166+
167+
# Copy memories
168+
for memory in memories:
169+
sqlite.save_memory(memory)
170+
171+
# Copy relations
172+
for relation in relations:
173+
sqlite.create_relation(relation)
174+
175+
print(f"Migrated {len(memories)} memories and {len(relations)} relations")
176+
177+
jsonl.close()
178+
sqlite.close()
179+
```
180+
181+
**Step 4: Update configuration**
182+
183+
Update `~/.config/cortexgraph/.env`:
184+
185+
```bash
186+
# Change from jsonl to sqlite
187+
CORTEXGRAPH_STORAGE_BACKEND=sqlite
188+
```
189+
190+
**Step 5: Test**
191+
192+
Restart your MCP server or application and verify memories are accessible:
193+
194+
```python
195+
from cortexgraph.storage.sqlite_storage import SQLiteStorage
196+
197+
storage = SQLiteStorage()
198+
storage.connect()
199+
memories = storage.list_memories()
200+
print(f"Found {len(memories)} memories in SQLite")
201+
storage.close()
202+
```
203+
204+
### Migrating from SQLite to JSONL
205+
206+
If you want to switch back to JSONL (e.g., for better git integration):
207+
208+
**Step 1: Backup SQLite database**
209+
210+
```bash
211+
cp ~/.config/cortexgraph/cortexgraph.db ~/.config/cortexgraph/cortexgraph.db.backup
212+
```
213+
214+
**Step 2: Copy data from SQLite to JSONL**
215+
216+
```python
217+
from cortexgraph.storage.jsonl_storage import JSONLStorage
218+
from cortexgraph.storage.sqlite_storage import SQLiteStorage
219+
220+
# Read all data from SQLite
221+
sqlite = SQLiteStorage()
222+
sqlite.connect()
223+
224+
memories = sqlite.list_memories()
225+
relations = sqlite.list_relations()
226+
227+
# Write to JSONL
228+
jsonl = JSONLStorage()
229+
jsonl.connect()
230+
231+
# Copy memories
232+
for memory in memories:
233+
jsonl.save_memory(memory)
234+
235+
# Copy relations
236+
for relation in relations:
237+
jsonl.create_relation(relation)
238+
239+
print(f"Migrated {len(memories)} memories and {len(relations)} relations")
240+
241+
sqlite.close()
242+
jsonl.close()
243+
```
244+
245+
**Step 3: Update configuration**
246+
247+
Update `~/.config/cortexgraph/.env`:
248+
249+
```bash
250+
# Change from sqlite to jsonl
251+
CORTEXGRAPH_STORAGE_BACKEND=jsonl
252+
```
253+
254+
**Step 4: Test**
255+
256+
Verify the migration:
257+
258+
```python
259+
from cortexgraph.storage.jsonl_storage import JSONLStorage
260+
261+
storage = JSONLStorage()
262+
storage.connect()
263+
memories = storage.list_memories()
264+
print(f"Found {len(memories)} memories in JSONL")
265+
storage.close()
266+
```
267+
268+
### Migration Safety Tips
269+
270+
1. **Always backup before migrating** - Keep your original data until you've verified the migration
271+
2. **Test with small subset first** - If you have many memories, test the migration script on a subset
272+
3. **Verify data integrity** - Check memory counts and spot-check a few memories after migration
273+
4. **Keep Markdown exports** - Export to Markdown as a human-readable backup format
274+
5. **No data loss** - Both backends support the same data model, so no information is lost in migration

0 commit comments

Comments
 (0)