Skip to content

feat: integrate Google NotebookLM as a skill with 12 tools #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cchen362 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

feat: integrate Google NotebookLM as a skill with 12 tools #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e539d06
feat: add cross-platform Windows support
cchen362 Mar 3, 2026
91acdc6
feat: add values-based autonomy framework and channel-agnostic triage
cchen362 Mar 3, 2026
a871142
feat: integrate Google NotebookLM as a skill with 12 tools
cchen362 Mar 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 54 additions & 8 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@ Edward is a full-stack AI assistant with long-term memory, built with Next.js, F

## Running the App

**The backend must run natively on macOS** (not in a container). This is required for the scheduled events scheduler and AppleScript/MCP integrations.
**The backend runs on macOS and Windows.** macOS has full feature support; Windows runs with graceful degradation (Apple-specific features disabled, PWA + push notifications for messaging).

### macOS
```bash
# First-time setup
./setup.sh # Creates venv, installs deps, sets up .env
Expand All @@ -26,7 +27,24 @@ Edward is a full-stack AI assistant with long-term memory, built with Next.js, F
# Individual services
cd backend && ./start.sh # Backend (FastAPI :8000)
cd frontend && npm install && npm run dev # Frontend (Next.js :3000)
```

### Windows (PowerShell)
```powershell
# First-time setup
.\setup.ps1 # Creates venv, installs deps, sets up .env

# Recommended: restart script (handles both services)
.\restart.ps1 # Restart both frontend + backend
.\restart.ps1 frontend # Restart only frontend
.\restart.ps1 backend # Stop and restart only backend

# Individual services
cd backend; .\start.ps1 # Backend (FastAPI :8000)
cd frontend; npm install; npm run dev # Frontend (Next.js :3000)
```

```bash
# Lint frontend
cd frontend && npm run lint
```
Expand Down Expand Up @@ -72,12 +90,13 @@ Init order matters — tool registry must come after skills/MCP:
2. `init_skills()` — Load skill enabled states
3. MCP clients — WhatsApp, Apple Services subprocesses
4. Custom MCP servers — User-added servers from DB
5. Tool registry — Must be after all tool sources are initialized
6. Scheduler — Polls every 30s for due scheduled events
7. Heartbeat — iMessage listener + triage loop
8. Consolidation — Hourly memory clustering
9. Evolution — Check for pending deploys after restart
10. Orchestrator — Recover crashed worker tasks
5. NotebookLM client — Lazy init if credentials exist
6. Tool registry — Must be after all tool sources are initialized
7. Scheduler — Polls every 30s for due scheduled events
8. Heartbeat — iMessage listener + triage loop
9. Consolidation — Hourly memory clustering
10. Evolution — Check for pending deploys after restart
11. Orchestrator — Recover crashed worker tasks

All have matching shutdown hooks in reverse order.

Expand Down Expand Up @@ -148,7 +167,7 @@ Tool loop (in respond node):
- GitHub search uses `GITHUB_TOKEN` env var for API access

**Skills System** (`backend/services/skills_service.py`)
- Manages integrations (iMessage AppleScript, Twilio SMS, Twilio WhatsApp, WhatsApp MCP, Brave Search, Code Interpreter, JavaScript Interpreter, SQL Database, Shell/Bash, Apple Services, HTML Hosting)
- Manages integrations (iMessage AppleScript, Twilio SMS, Twilio WhatsApp, WhatsApp MCP, Brave Search, Code Interpreter, JavaScript Interpreter, SQL Database, Shell/Bash, Apple Services, HTML Hosting, Google NotebookLM)
- Tracks enabled/disabled state in database
- Reports connection status for each skill
- Initializes MCP client on skill enable (not just DB toggle)
Expand All @@ -164,6 +183,14 @@ Tool loop (in respond node):
- Uses v2 JSON API with `X-API-Key` authentication
- Tools: `create_hosted_page`, `update_hosted_page`, `delete_hosted_page`, `check_hosted_slug`

**Google NotebookLM** (`backend/services/notebooklm_service.py`)
- Curated, source-grounded knowledge bases via `notebooklm-py` library (undocumented Google APIs)
- Lazy singleton client — created on first use, persists until shutdown
- Name-based notebook references (case-insensitive, no raw IDs exposed to LLM)
- Credential check at startup, graceful skip if missing
- Requires one-time browser login (`notebooklm login`), credentials expire ~1-2 weeks
- Tools: `nlm_list_notebooks`, `nlm_create_notebook`, `nlm_delete_notebook`, `nlm_add_source`, `nlm_list_sources`, `nlm_get_source_text`, `nlm_ask`, `nlm_research`, `nlm_generate_artifact`, `nlm_wait_artifact`, `nlm_push_document`, `nlm_push_file`

**Execution System** (`backend/services/execution/`)
- Shared base: `base.py` with `ExecutionResult`, sandbox management, `run_subprocess()` helper
- **Python** (`python_execution.py`): Sandboxed via subprocess, blocked dangerous modules
Expand Down Expand Up @@ -273,6 +300,20 @@ HTML hosting tools available to LLM (when skill enabled):
- `delete_hosted_page` - Delete a hosted page
- `check_hosted_slug` - Check if a URL slug is available

NotebookLM tools available to LLM (when notebooklm skill enabled):
- `nlm_list_notebooks` - List all notebooks
- `nlm_create_notebook` - Create a new notebook
- `nlm_delete_notebook` - Delete a notebook
- `nlm_add_source` - Add source (url/youtube/text/file) to notebook
- `nlm_list_sources` - List sources in a notebook
- `nlm_get_source_text` - Extract indexed fulltext from a source
- `nlm_ask` - Ask a question with source citations
- `nlm_research` - Run web research, auto-import sources
- `nlm_generate_artifact` - Generate audio/video/quiz/flashcards/slides/infographic/mind_map/data_table/report
- `nlm_wait_artifact` - Check artifact generation status
- `nlm_push_document` - Push Edward document to notebook as text source
- `nlm_push_file` - Push Edward PDF file to notebook as file source

Apple Services tools available to LLM (when apple_services skill enabled):
- Calendar tools - Read/manage calendar events
- Reminders tools - Manage user's Apple Reminders (NOT for Edward's internal scheduling)
Expand Down Expand Up @@ -539,6 +580,10 @@ FILE_STORAGE_ROOT=./storage # Optional: defaults to ./storage
# Claude Code (for evolution service + orchestrator CC tasks)
# Requires `claude-agent-sdk` in requirements.txt
# Claude Code CLI must be installed and authenticated on the host

# Google NotebookLM (requires one-time browser login: notebooklm login)
# NOTEBOOKLM_STORAGE_PATH=~/.notebooklm/storage_state.json # Optional: override credential path
# NOTEBOOKLM_AUTH_JSON= # Optional: inline auth for headless environments
```

Database defaults to `edward`/`edward`/`edward` (user/password/database).
Expand All @@ -564,6 +609,7 @@ Database defaults to `edward`/`edward`/`edward` (user/password/database).
- Code execution tools filtered by skill state: code_interpreter, javascript_interpreter, sql_interpreter, shell_interpreter
- Apple Services tools filtered by skill state: apple_services
- HTML hosting tools filtered by skill state: html_hosting
- NotebookLM tools filtered by skill state: notebooklm
- Twilio inbound webhooks process SMS and WhatsApp asynchronously to avoid timeouts, responds via API not TwiML
- WhatsApp and SMS from same phone number share the same external contact and conversation; `last_channel` tracks reply channel
- MCP client manages multiple subprocess servers: WhatsApp, Apple Services (includes Messages)
Expand Down
195 changes: 195 additions & 0 deletions IMPLEMENTATION_PLANS/000_MASTER_PLAN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,195 @@
# Plan 000: Master Plan — Edward Cross-Platform + Autonomous Knowledge

## STOP: Read This Entire Document Before Making Any Changes

This is the master architecture document for the Edward project fork. Every subsequent plan (001-004) is derived from the decisions documented here.

**Revised**: Based on brainstorming workshop. Original scope (Windows + Telegram) expanded to cross-platform foundation + autonomy framework + NotebookLM knowledge system.

---

## Why We're Forking

The original Edward is a powerful full-stack AI assistant, but it has gaps:

| Problem | Root Cause | Impact |
|---------|-----------|--------|
| **Can't run on Windows** | Shell scripts use bash/brew, startup assumes macOS | Backend won't start at all |
| **os.uname() crashes** | `os.uname()` doesn't exist on Windows | Import errors on 2 services |
| **Shell exec hardcoded** | PATH hardcoded to `/opt/homebrew/bin:...` | Shell execution skill broken |
| **Thin system prompt** | Paper-thin persona, no self-awareness or judgment framework | Autonomous behavior is unpredictable |
| **No deep knowledge system** | Documents embed only first 500 chars, no chunking or ingestion | Can't act as a knowledge base |
| **Triage hardcodes iMessage** | Heartbeat triggers say "send via iMessage" | Broken on Windows, inflexible |
| **No prompt caching** | All 9 LLM call sites pay full token price | Higher cost than necessary |

**This fork addresses ALL of these while preserving 100% of existing functionality on macOS.**

---

## Architecture: What Changes vs. What Stays

### Stays Exactly The Same (No Touch)
- Frontend (Next.js) — fully cross-platform already
- Core backend (FastAPI, LangGraph, memory, documents, scheduling)
- Database (PostgreSQL + pgvector + asyncpg)
- Twilio SMS/WhatsApp integration
- Code execution (Python, JS, SQL)
- Web search (Brave), HTML hosting, file storage
- Orchestrator, evolution service
- Memory system (extraction, reflection, deep retrieval, consolidation)

### Changes

| Component | Before | After | Why |
|-----------|--------|-------|-----|
| **Startup scripts** | Bash only | + PowerShell equivalents (.ps1) | Windows can't run bash |
| **Platform checks** | `os.uname()` | `sys.platform == "darwin"` | os.uname() crashes on Windows |
| **Shell execution** | Hardcoded bash + macOS PATH | Platform-aware (cmd.exe on Windows) | Shell skill works on Windows |
| **System prompt** | 1-sentence persona | + Values, capabilities map, platform context, autonomy calibration | Autonomous agent needs self-awareness |
| **Triage prompts** | Hardcoded "iMessage" | Channel-agnostic, dynamic | Works on any platform |
| **Knowledge** | Documents only | + NotebookLM skill (13 tools) | Deep, source-grounded knowledge bases |
| **LLM calls** | No prompt caching | Ephemeral cache on all call sites | ~30-50% token savings |

### New Architecture Diagram
```
┌─────────────────────────────┐
│ Frontend │
│ (Next.js :3000 / PWA) │
└──────────┬──────────────────┘
│ SSE + HTTP
┌──────────┴──────────────────┐
│ Backend (FastAPI :8000) │
│ │
│ ┌────────────────────────┐ │
│ │ LangGraph Agent │ │
│ │ (memory, tools, LLM) │ │
│ └────────┬───────────────┘ │
│ │ │
│ ┌────────┴───────────────┐ │
│ │ Tool Registry │ │
│ │ (skill-gated) │ │
│ └────────────────────────┘ │
│ │ │
│ ┌────────┴───────────────┐ │
│ │ Services │ │
│ │ ├─ Messaging (Twilio) │ │
│ │ ├─ iMessage (macOS) │ │
│ │ ├─ NotebookLM (NEW) │ │
│ │ └─ Push (VAPID) │ │
│ └────────────────────────┘ │
└──────────┬──────────────────┘
│
┌──────────┴──────────────────┐
│ PostgreSQL + pgvector │
│ (memories, conversations, │
│ contacts, checkpoints) │
└──────────────────────────────┘
│
┌──────────┴──────────────────┐
│ Google NotebookLM (NEW) │
│ (via notebooklm-py library) │
│ Notebooks, sources, Q&A, │
│ research, artifacts │
└──────────────────────────────┘
```

---

## Key Design Decisions

### 1. Cross-Platform (Not Windows-Only)
- **Decision**: Support both macOS and Windows. Don't migrate to one OS.
- **Rationale**: User is on Windows now but may switch to macOS later. Build once, run anywhere.

### 2. PWA as Primary Interface
- **Decision**: Push notifications + PWA chat as the primary user interaction channel.
- **Rationale**: Already fully implemented (VAPID Web Push, installable app, mobile-responsive). No need for Telegram or additional messaging channels.

### 3. NotebookLM for Deep Knowledge
- **Decision**: Integrate Google NotebookLM as a skill via `notebooklm-py` library.
- **Rationale**: Provides source-grounded Q&A, cross-source reasoning, and artifact generation (audio, quizzes, mind maps) that would take months to build in-house. Acceptable trade-off: uses undocumented APIs, suitable for personal projects.

### 4. Values-Based Autonomy (Not Rules)
- **Decision**: Add lightweight system prompt sections for identity, capabilities, platform awareness, and autonomy calibration. No rigid behavior rules.
- **Rationale**: Preserves the original creator's "non-deterministic programming" philosophy while giving Edward self-awareness and judgment principles.

### 5. Prompt Caching: Ephemeral on All Static Content
- **Decision**: Add `cache_control: {"type": "ephemeral"}` to all static prompt prefixes.
- **Rationale**: 9+ LLM call sites, all have static instruction text. ~30-50% savings on main chat.

### 6. Telegram: Deferred
- **Decision**: Deprioritize Telegram integration. PWA covers the use case.
- **Rationale**: Telegram would just be another Edward↔user channel, not outbound messaging to others. Can be revisited if push notifications prove unreliable.

---

## Implementation Order & Dependencies

```
Plan 001: Cross-Platform Foundation
│ (no dependencies — pure infrastructure)
│
Plan 002: Autonomy Framework
│ (depends on 001 — prompt references platform context)
│
Plan 003: NotebookLM Integration
│ (depends on 002 — Edward needs autonomy framework to use NLM with judgment)
│
Plan 004: Prompt Caching
(depends on 003 — apply caching after all LLM call sites are finalized)
```

| # | Plan | Status | Effort |
|---|------|--------|--------|
| 001 | [Cross-Platform Foundation](001_CROSS_PLATFORM_FOUNDATION.md) | **Complete** | 0.5-1 day |
| 002 | [Autonomy Framework](002_AUTONOMY_FRAMEWORK.md) | **Complete** | 0.5-1 day |
| 003 | [NotebookLM Integration](003_NOTEBOOKLM_INTEGRATION.md) | **Complete** | 2-3 days |
| 004 | [Prompt Caching](004_PROMPT_CACHING.md) | Active | 0.5-1 day |
| — | [Telegram Integration](DEFERRED_TELEGRAM_INTEGRATION.md) | Deferred | — |

**Total estimated effort**: ~4-6 days

---

## Cost Estimates

### Per-Message Cost (with caching, Plan 004)

| Component | Model | Without Cache | With Cache | Frequency |
|-----------|-------|-------------|-----------|-----------|
| Main response | Sonnet 4.5/4.6 | $0.02-0.04 | $0.01-0.025 | Every message |
| Memory extraction | Haiku 4.5 | $0.001 | $0.0002 | Every message |
| Search tags | Haiku 4.5 | $0.0005 | $0.0001 | Every message |
| Reflection | Haiku 4.5 | $0.001 | $0.0002 | Every message |
| Deep retrieval | Haiku 4.5 | $0.001 | $0.0002 | ~30% of messages |

**Monthly estimate (50 messages/day with caching): ~$10-20/month**

### NotebookLM (Plan 003)
- **Google NotebookLM**: Free for personal use (as of 2026)
- **notebooklm-py**: MIT license, no API costs
- **Only cost**: Anthropic tokens for Edward's tool calls that trigger NLM operations

---

## Success Criteria

- [ ] Backend starts on both Windows and macOS without errors
- [ ] macOS-only skills show "unavailable" (not crash) on Windows
- [ ] System prompt includes identity, capabilities, platform context, and autonomy sections
- [ ] Triage prompts are channel-agnostic (no hardcoded "iMessage")
- [ ] NotebookLM skill creates notebooks, adds sources, queries, and generates artifacts
- [ ] Prompt caching reduces token usage by 30%+
- [ ] All existing macOS functionality preserved (no regressions)
- [ ] Edward demonstrates autonomous knowledge-building behavior

---

## Non-Goals (Explicitly Out of Scope)

- Docker support (use native installs for now)
- WSL2 setup guide (native Windows is simpler)
- Discord/Telegram integration (deferred, PWA is sufficient)
- Rich UI for NotebookLM (skill toggle + chat tools is sufficient)
- Windows Contacts integration (no equivalent to AppleScript)
- Renaming Edward to "Edweird" (deferred, can be done anytime)
Loading