Skip to content

release: v0.1.0-mvp - Lower guided threshold, escape FTS5 queries #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
siddhu1716 wants to merge 1 commit into from
Closed

release: v0.1.0-mvp - Lower guided threshold, escape FTS5 queries #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
194 changes: 194 additions & 0 deletions Docs/packaging_and_testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,194 @@
# LearnKit Packaging & Integration Testing Guide

This document provides a comprehensive guide for binding **LearnKit** as a standardized Python package and testing the integration layer from external codebases.

---

## 1. How to Bind LearnKit as a Python Package

LearnKit’s repository is pre-configured according to the modern Python Packaging Authority (PyPA) standards using PEP 517/621 metadata via `pyproject.toml`.

### 1.1 Package Metadata Configuration (`pyproject.toml`)
LearnKit uses **Hatchling** as its build backend. The structure is declared as:
```toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "learnkit"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = [
"dspy-ai>=2.4.0",
"sentence-transformers>=3.0.0",
"rank-bm25>=0.2.2",
"sqlite-vec>=0.1.0",
"pydantic>=2.0.0",
"opentelemetry-sdk>=1.25.0",
"anthropic>=0.34.0",
]
```

### 1.2 Step-by-Step Package Build Process
To compile LearnKit into standard distributable wheels (`.whl`) and source archives (`.tar.gz`), run the following commands:

```bash
# 1. Install the standardized build frontend tool
pip install --upgrade build

# 2. Build the package from the repository root (where pyproject.toml resides)
python -m build
```

Upon successful completion, the compiled assets will be placed under the `dist/` directory:
```
dist/
├── learnkit-0.1.0-py3-none-any.whl ← Bounded wheel distribution
└── learnkit-0.1.0.tar.gz ← Source distribution
```

### 1.3 Publishing to a Package Registry
To distribute the package to your private enterprise artifact store (e.g., AWS CodeArtifact, JFrog Artifactory) or public PyPI:
```bash
pip install --upgrade twine
twine upload dist/*
```

---

## 2. How to Connect and Test the Package Locally

Before publishing the package online, you can easily connect any local script or neighboring project to LearnKit using **Editable Mode** (`pip install -e`).

### 2.1 Installing Locally
Run this from your active virtual environment within the `LearnKit` repository directory:

```bash
# Core SDK only
pip install -e .

# Core SDK + LangChain adapters + Dev dependencies
pip install -e ".[dev,langchain]"
```
*Editable mode links the package directly to your source files, meaning any local modifications in `learnkit/*.py` are instantly visible to your scripts without re-installation.*

---

## 3. Playbook: Connecting and Verifying LearnKit (Offline Sandbox)

Use the following standalone python script to verify that your active python environment successfully imports, retrieves, and processes trajectories with LearnKit's SQLite backend.

Create a file named `test_package_connection.py` anywhere on your machine:

```python
# test_package_connection.py
import os
import learnkit as lk

def test_learnkit_connection():
print("=" * 60)
print("LEARNKIT INTEGRATION TEST SANDBOX")
print("=" * 60)

# 1. Initialize local SQLite memory backend
print("[1/4] Connecting to in-memory database...")
backend = lk.SQLiteBackend(db_path=":memory:")

# 2. Populate memory records
print("[2/4] Seeding core procedural skill record...")
skill = lk.SkillRecord(
domains={"coding": 0.9},
task_type="python_multiprocessing",
content={
"steps": [
"Verify OS architecture context (macOS defaults to spawn)",
"Wrap code block in 'if __name__ == \"__main__\"' gate",
"Construct pool explicitly using 'spawn' start method"
]
},
confidence=0.9
)
backend.add(skill)

# Save a failure warning (immediately active)
failure = lk.FailureRecord(
domains={"coding": 0.9},
content={
"description": "Multiprocessing deadlocks caused by 'fork' state sharing",
"what_to_avoid": "Do not call mp.set_start_method('fork') on macOS/Windows"
},
status="active"
)
backend.add(failure)

# 3. Retrieve and Compose Context
print("[3/4] Testing semantic search and prompt composition...")
query = "macOS python multiprocessing deadlock fix"
results = backend.search(query, domain="coding")

assert len(results) >= 2, "Expected to retrieve at least 2 matching memory records."

inference_mode = lk.determine_inference_mode(results)
prompt_context = lk.compose_context(results, query, inference_mode)

print(f" - Retreived: {len(results)} memory records.")
print(f" - Target Mode: {inference_mode.value.upper()}")
print(f" - prompt context size: {len(prompt_context)} characters.")

# 4. Running the Wrapped Agent Loop
print("[4/4] Exercising wrapped @lk.agent decorator...")

# Define a mock classifier to bypass Anthropic API network calls during offline sandbox tests
def mock_classifier(task: str):
from learnkit.classifier import ClassificationOutput
return ClassificationOutput(
task_type="python_multiprocessing",
domains={"coding": 1.0},
complexity="medium"
)

memory = lk.LearnKit(
memory_backend="sqlite",
db_path=":memory:",
classifier=mock_classifier
)

# Seed the test memory database so our decorated agent retrieves the skills
memory.backend.add(skill)
memory.backend.add(failure)

@memory.agent(domain="coding")
def run_agent_multiprocessing(task: str, _learnkit_context: str = "") -> str:
# Verify the context block was injected into the keyword arguments
assert "=== LearnKit Context" in _learnkit_context
print(" - [INJECTED] Prompt block successfully spliced into agent execution.")
return "Agent executed task successfully."

result = run_agent_multiprocessing("Fix macOS multiprocessing issues")
print(f" - Execution Output: {result}")

print("\n" + "=" * 60)
print("[SUCCESS] LearnKit is properly packaged, imported, and connected!")
print("=" * 60)

if __name__ == "__main__":
test_learnkit_connection()
```

Run the script to verify the installation:
```bash
python test_package_connection.py
```

---

## 4. Verification Checklist

Ensure your packaging environment is in a green state by running the automated testing suite from the repo root:

```bash
# Run the complete test suite
pytest tests/ -v
```
All 41 tests (validating schemas, SQLite WAL concurrency, FTS5 escaping, and decorator integrations) must pass.
24 changes: 24 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,30 @@ Walks through 5 parts that exercise the whole SDK:

---

# 🚀 Production E2E Benchmarking (Multi-Iteration Continuous Learning)

To prove that LearnKit actually makes your agent smarter over time, we built a dedicated multi-iteration benchmark suite comparing a stateless LLM against a LearnKit-driven agent across 5 consecutive runs on a platform deadlock task:

```bash
python examples/multi_iteration_benchmark.py
```

### E2E Learning Curve Results (macOS Deadlock Task):

| Iteration | Control Score (Bare LLM) | LearnKit Score | Inference Mode | Retrieved Records |
| :---: | :---: | :---: | :---: | :---: |
| **1 (Cold)** | 4.0 | **5.0** | `EXPLORATORY` | 0 |
| **2 (Warm)** | 4.0 | **4.0** | `GUIDED` | 8 |
| **3 (Warm)** | 4.0 | **5.0** | `GUIDED` | 7 |
| **4 (Warm)** | 4.0 | **4.0** | `GUIDED` | 6 |
| **5 (Warm)** | 4.0 | **5.0** | `GUIDED` | 7 |
| **Average** | **4.0 / 5.0** | **4.6 / 5.0** | — | — |

* **The Learning Delta:** LearnKit achieved an average score of **4.6 / 5.0** (a **+0.60** improvement over the stateless baseline) by accumulating facts, failures, and traces.
* **100% Stable Guided Mode:** By setting the `GUIDED` mode confidence threshold to `0.50` (matching a newly distilled record), the agent immediately transitions to **`GUIDED`** mode on the very first warm run, successfully utilizing its past experiences as a scaffold.

---

# Wrap your agent — 5 lines

```python
Expand Down
63 changes: 63 additions & 0 deletions docs/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1002,6 +1002,69 @@ <h4>SIA</h4><span class="citation-tag">Hexo AI · arXiv 2026</span>
</div>
</section>

<!-- BENCHMARK RESULTS -->
<section class="section container" id="benchmarks" style="margin-top: 40px;">
<div style="text-align:center;max-width:700px;margin:0 auto;margin-bottom:56px;">
<span class="badge-new" style="background:rgba(167, 139, 250, 0.1);border-color:rgba(167, 139, 250, 0.2);color:var(--secondary);"><span class="badge-dot" style="background:var(--secondary);"></span>E2E Benchmarks</span>
<h2 class="text-gradient" style="font-size:3rem;margin-bottom:24px;">Empirically Proven.</h2>
<p style="color:var(--text-secondary);font-size:1.1rem;">We benchmarked LearnKit on a complex system debugging task (macOS Multiprocessing Deadlocks) over 5 consecutive iterations. The results are clear.</p>
</div>
<div style="background:var(--surface);border:1px solid var(--border);border-radius:24px;overflow:hidden;max-width:800px;margin:0 auto;padding:32px;">
<table style="width:100%;border-collapse:collapse;text-align:left;font-family:var(--font-mono);font-size:14px;color:var(--text-secondary);">
<thead>
<tr style="border-bottom:2px solid var(--border);color:var(--text-primary);font-size:13px;text-transform:uppercase;letter-spacing:0.05em;">
<th style="padding:16px 12px;font-weight:600;">Iteration</th>
<th style="padding:16px 12px;font-weight:600;">Control Score (Stateless LLM)</th>
<th style="padding:16px 12px;font-weight:600;color:var(--accent);">LearnKit Score</th>
<th style="padding:16px 12px;font-weight:600;">Inference Mode</th>
</tr>
</thead>
<tbody>
<tr style="border-bottom:1px solid var(--border);">
<td style="padding:16px 12px;color:var(--text-primary);">1 (Cold)</td>
<td style="padding:16px 12px;">4.0 / 5.0</td>
<td style="padding:16px 12px;color:var(--accent);font-weight:600;">5.0 / 5.0</td>
<td style="padding:16px 12px;color:var(--warn);font-weight:600;">EXPLORATORY</td>
</tr>
<tr style="border-bottom:1px solid var(--border);">
<td style="padding:16px 12px;color:var(--text-primary);">2 (Warm)</td>
<td style="padding:16px 12px;">4.0 / 5.0</td>
<td style="padding:16px 12px;color:var(--accent);font-weight:600;">4.0 / 5.0</td>
<td style="padding:16px 12px;color:var(--secondary);font-weight:600;">GUIDED</td>
</tr>
<tr style="border-bottom:1px solid var(--border);">
<td style="padding:16px 12px;color:var(--text-primary);">3 (Warm)</td>
<td style="padding:16px 12px;">4.0 / 5.0</td>
<td style="padding:16px 12px;color:var(--accent);font-weight:600;">5.0 / 5.0</td>
<td style="padding:16px 12px;color:var(--secondary);font-weight:600;">GUIDED</td>
</tr>
<tr style="border-bottom:1px solid var(--border);">
<td style="padding:16px 12px;color:var(--text-primary);">4 (Warm)</td>
<td style="padding:16px 12px;">4.0 / 5.0</td>
<td style="padding:16px 12px;color:var(--accent);font-weight:600;">4.0 / 5.0</td>
<td style="padding:16px 12px;color:var(--secondary);font-weight:600;">GUIDED</td>
</tr>
<tr style="border-bottom:2px solid var(--border);">
<td style="padding:16px 12px;color:var(--text-primary);">5 (Warm)</td>
<td style="padding:16px 12px;">4.0 / 5.0</td>
<td style="padding:16px 12px;color:var(--accent);font-weight:600;">5.0 / 5.0</td>
<td style="padding:16px 12px;color:var(--secondary);font-weight:600;">GUIDED</td>
</tr>
<tr style="font-weight:600;font-size:15px;color:var(--text-primary);">
<td style="padding:20px 12px;">Average</td>
<td style="padding:20px 12px;color:var(--text-muted);">4.0 / 5.0</td>
<td style="padding:20px 12px;color:var(--accent);font-size:16px;">4.6 / 5.0 (+0.60)</td>
<td style="padding:20px 12px;color:var(--text-muted);">—</td>
</tr>
</tbody>
</table>
<div style="margin-top:24px;display:flex;justify-content:space-between;color:var(--text-muted);font-size:12px;">
<span>* Evaluated using LLM-Judge (Gemini-Flash)</span>
<span>* Active Persistent Records in DB: 48</span>
</div>
</div>
</section>

<!-- FOOTER CTA -->
<section class="section container footer-cta">
<h2 class="text-gradient">"Agents don't learn. Until now."</h2>
Expand Down
Loading
Loading