|
2 | 2 |
|
3 | 3 | CortexGraph supports multiple storage backends for short-term memory (STM). |
4 | 4 |
|
| 5 | +## Choosing a Backend |
| 6 | + |
| 7 | +**Use JSONL when:** |
| 8 | +- Dataset size < 10,000 memories |
| 9 | +- You want human-readable, git-friendly files |
| 10 | +- You need easy inspection and manual editing |
| 11 | +- Memory usage is not a concern |
| 12 | + |
| 13 | +**Use SQLite when:** |
| 14 | +- Dataset size > 10,000 memories |
| 15 | +- Memory efficiency is important (low RAM usage) |
| 16 | +- You need fast queries and filtering on large datasets |
| 17 | +- You want ACID transaction guarantees |
| 18 | + |
| 19 | +**Performance Characteristics:** |
| 20 | + |
| 21 | +| Backend | Memory Count | RAM Usage | Search Speed | Git-Friendly | |
| 22 | +|---------|--------------|-----------|--------------|--------------| |
| 23 | +| JSONL | < 10k | High | Fast | ✅ Yes | |
| 24 | +| JSONL | 10k - 50k | Very High | Medium | ✅ Yes | |
| 25 | +| JSONL | > 50k | Excessive | Slow | ⚠️ Large diffs | |
| 26 | +| SQLite | Any size | Low | Fast | ❌ Binary | |
| 27 | + |
5 | 28 | ## JSONL (Default) |
6 | 29 |
|
7 | 30 | The default backend uses human-readable JSONL (JSON Lines) files. |
@@ -92,3 +115,160 @@ strength: 1.5 |
92 | 115 |
|
93 | 116 | Memory content goes here... |
94 | 117 | ``` |
| 118 | + |
| 119 | +## Migration Guide |
| 120 | + |
| 121 | +### Migrating from JSONL to SQLite |
| 122 | + |
| 123 | +If your dataset has grown large (>10k memories) and you want to switch to SQLite for better performance: |
| 124 | + |
| 125 | +**Step 1: Backup your data** |
| 126 | + |
| 127 | +```bash |
| 128 | +# Backup current JSONL files |
| 129 | +cp -r ~/.config/cortexgraph/jsonl ~/.config/cortexgraph/jsonl.backup |
| 130 | +``` |
| 131 | + |
| 132 | +**Step 2: Export to Markdown (optional but recommended)** |
| 133 | + |
| 134 | +```python |
| 135 | +from pathlib import Path |
| 136 | +from cortexgraph.tools.export import MarkdownExport |
| 137 | +from cortexgraph.storage.jsonl_storage import JSONLStorage |
| 138 | + |
| 139 | +# Read from JSONL |
| 140 | +storage = JSONLStorage() |
| 141 | +storage.connect() |
| 142 | +memories = storage.list_memories() |
| 143 | + |
| 144 | +# Export to Markdown as backup |
| 145 | +exporter = MarkdownExport(output_dir=Path("./backup_export")) |
| 146 | +stats = exporter.export_batch(memories) |
| 147 | +print(f"Backed up {stats.success} memories") |
| 148 | +``` |
| 149 | + |
| 150 | +**Step 3: Copy data from JSONL to SQLite** |
| 151 | + |
| 152 | +```python |
| 153 | +from cortexgraph.storage.jsonl_storage import JSONLStorage |
| 154 | +from cortexgraph.storage.sqlite_storage import SQLiteStorage |
| 155 | + |
| 156 | +# Read all data from JSONL |
| 157 | +jsonl = JSONLStorage() |
| 158 | +jsonl.connect() |
| 159 | + |
| 160 | +memories = jsonl.list_memories() |
| 161 | +relations = jsonl.list_relations() |
| 162 | + |
| 163 | +# Write to SQLite |
| 164 | +sqlite = SQLiteStorage() |
| 165 | +sqlite.connect() |
| 166 | + |
| 167 | +# Copy memories |
| 168 | +for memory in memories: |
| 169 | + sqlite.save_memory(memory) |
| 170 | + |
| 171 | +# Copy relations |
| 172 | +for relation in relations: |
| 173 | + sqlite.create_relation(relation) |
| 174 | + |
| 175 | +print(f"Migrated {len(memories)} memories and {len(relations)} relations") |
| 176 | + |
| 177 | +jsonl.close() |
| 178 | +sqlite.close() |
| 179 | +``` |
| 180 | + |
| 181 | +**Step 4: Update configuration** |
| 182 | + |
| 183 | +Update `~/.config/cortexgraph/.env`: |
| 184 | + |
| 185 | +```bash |
| 186 | +# Change from jsonl to sqlite |
| 187 | +CORTEXGRAPH_STORAGE_BACKEND=sqlite |
| 188 | +``` |
| 189 | + |
| 190 | +**Step 5: Test** |
| 191 | + |
| 192 | +Restart your MCP server or application and verify memories are accessible: |
| 193 | + |
| 194 | +```python |
| 195 | +from cortexgraph.storage.sqlite_storage import SQLiteStorage |
| 196 | + |
| 197 | +storage = SQLiteStorage() |
| 198 | +storage.connect() |
| 199 | +memories = storage.list_memories() |
| 200 | +print(f"Found {len(memories)} memories in SQLite") |
| 201 | +storage.close() |
| 202 | +``` |
| 203 | + |
| 204 | +### Migrating from SQLite to JSONL |
| 205 | + |
| 206 | +If you want to switch back to JSONL (e.g., for better git integration): |
| 207 | + |
| 208 | +**Step 1: Backup SQLite database** |
| 209 | + |
| 210 | +```bash |
| 211 | +cp ~/.config/cortexgraph/cortexgraph.db ~/.config/cortexgraph/cortexgraph.db.backup |
| 212 | +``` |
| 213 | + |
| 214 | +**Step 2: Copy data from SQLite to JSONL** |
| 215 | + |
| 216 | +```python |
| 217 | +from cortexgraph.storage.jsonl_storage import JSONLStorage |
| 218 | +from cortexgraph.storage.sqlite_storage import SQLiteStorage |
| 219 | + |
| 220 | +# Read all data from SQLite |
| 221 | +sqlite = SQLiteStorage() |
| 222 | +sqlite.connect() |
| 223 | + |
| 224 | +memories = sqlite.list_memories() |
| 225 | +relations = sqlite.list_relations() |
| 226 | + |
| 227 | +# Write to JSONL |
| 228 | +jsonl = JSONLStorage() |
| 229 | +jsonl.connect() |
| 230 | + |
| 231 | +# Copy memories |
| 232 | +for memory in memories: |
| 233 | + jsonl.save_memory(memory) |
| 234 | + |
| 235 | +# Copy relations |
| 236 | +for relation in relations: |
| 237 | + jsonl.create_relation(relation) |
| 238 | + |
| 239 | +print(f"Migrated {len(memories)} memories and {len(relations)} relations") |
| 240 | + |
| 241 | +sqlite.close() |
| 242 | +jsonl.close() |
| 243 | +``` |
| 244 | + |
| 245 | +**Step 3: Update configuration** |
| 246 | + |
| 247 | +Update `~/.config/cortexgraph/.env`: |
| 248 | + |
| 249 | +```bash |
| 250 | +# Change from sqlite to jsonl |
| 251 | +CORTEXGRAPH_STORAGE_BACKEND=jsonl |
| 252 | +``` |
| 253 | + |
| 254 | +**Step 4: Test** |
| 255 | + |
| 256 | +Verify the migration: |
| 257 | + |
| 258 | +```python |
| 259 | +from cortexgraph.storage.jsonl_storage import JSONLStorage |
| 260 | + |
| 261 | +storage = JSONLStorage() |
| 262 | +storage.connect() |
| 263 | +memories = storage.list_memories() |
| 264 | +print(f"Found {len(memories)} memories in JSONL") |
| 265 | +storage.close() |
| 266 | +``` |
| 267 | + |
| 268 | +### Migration Safety Tips |
| 269 | + |
| 270 | +1. **Always backup before migrating** - Keep your original data until you've verified the migration |
| 271 | +2. **Test with small subset first** - If you have many memories, test the migration script on a subset |
| 272 | +3. **Verify data integrity** - Check memory counts and spot-check a few memories after migration |
| 273 | +4. **Keep Markdown exports** - Export to Markdown as a human-readable backup format |
| 274 | +5. **No data loss** - Both backends support the same data model, so no information is lost in migration |
0 commit comments