Continuous Self-Learning from Dialog (Automatic Skill Evolution)

[Edouard-Legoupil]

Problem

CodeWhale has a powerful Skills System where users can manually create SKILL.md files that the agent can reference. However, the knowledge remains static:

1. No automatic pattern extraction - The agent doesn't recognize when it's solving the same type of problem repeatedly
2. No skill evolution - When a user provides corrections or better approaches, the agent doesn't update existing skills
3. No cross-session learning - Lessons learned in one session don't automatically improve future sessions
4. Manual overhead - Users must proactively identify patterns and write skills themselves, which breaks flow

Example: A developer builds 3 FastAPI endpoints with MongoDB over a week. Each time, they go through similar steps (configure connection, create models, implement CRUD, add error handling, write tests). The agent executes these steps each time but never "learns" that this is a repeatable pattern worth codifying as a skill.

Proposed Solution

Implement a self-learning loop that automatically observes, extracts, and codifies patterns from successful interactions:

Core Components:

1. Pattern Detection Engine

• Monitor dialog history for repeated solution patterns
• Identify multi-step workflows that occur 3+ times
• Detect corrections and incorporate them into skills
• Track successful outcomes vs. failures

2. Automated Skill Generation

• Create SKILL.md files based on detected patterns
• Generate structured YAML frontmatter with:
  • Name, description, version
  • Step-by-step instructions
  • Common pitfalls (learned from corrections)
  • Usage examples from real dialog
  • Confidence scores based on success rate

3. Skill Evolution System

• Update existing skills when better approaches are discovered
• Merge related skills into more comprehensive ones
• Archive unused or low-confidence skills
• Version tracking with change history

4. Learning Triggers

# Example configuration
[learning]
auto_create_skills = true          # Default: false (requires confirmation)
confidence_threshold = 0.8          # Only auto-create if pattern confidence > 80%
pattern_occurrence_threshold = 3    # Pattern must appear 3+ times
enable_cross_session_learning = true
skill_review_frequency = "weekly"   # Suggest reviewing skills periodically

5. User Control & Transparency

• Preview suggested skills before creation (/suggest-skills)
• Approve/deny skill creation (/approve-skill fastapi-crud)
• Review learning history (/learning-history)
• Manual override and pruning (/delete-skill old-pattern, /merge-skills)

Use Case

Scenario: A development team building a FastAPI + React + MCP application over 6 months:

Month 1: Manual Mode

• Developers work with CodeWhale, solving problems

• Agent silently observes patterns but doesn't act

• Team reviews /suggest-skills weekly and creates the most useful ones manually

Month 2-3: Semi-Automated Mode

• Agent automatically creates skills for patterns with 90%+ confidence

• Team reviews and approves skills weekly

• Skills begin covering 40% of repetitive tasks

Month 4-6: Fully Automated Mode

• Agent creates, updates, and merges skills autonomously

• Skills cover 80% of repetitive patterns

• Development speed increases 2-3x for common tasks

Specific Example:

Session 1: "Build FastAPI endpoint with JWT auth"
Session 2: "Build FastAPI endpoint with JWT auth and role-based permissions"
Session 3: "Build FastAPI endpoint with JWT auth, roles, and rate limiting"

→ Agent detects the pattern and creates skill "fastapi-jwt-crud"
→ Future request: "Build user management API" 
→ Agent uses the skill, saving 30 minutes of work
→ User adds "also needs email verification" 
→ Agent updates the skill with email verification step
→ Next team member gets the improved skill automatically

Alternatives considered

Alternative	Pros	Cons
Manual Skills	Full control, simple to implement	High overhead, knowledge loss between sessions
Third-party Learning Tools	Could use external ML	Integration complexity, additional dependencies
Prompt-based Learning	Simple to implement	Temporary, lost between sessions
MCP-based Learning	Extensible, modular	Requires separate MCP server, not built-in

This proposal leverages CodeWhale's existing infrastructure (skills, hooks, MCP) while adding intelligence on top, keeping everything integrated and user-friendly.

Impact

This feature would transform CodeWhale from a static assistant into an adaptive pair programmer that grows with its users. It's technically feasible using CodeWhale's existing infrastructure (skills, hooks, MCP) and would provide immense value to both individual developers and teams.

For Individual Developers:

• ⚡ 2-3x faster for repetitive coding tasks

• 🧠 Preserves knowledge - don't need to remember every best practice

• 🔄 Continuous improvement - your assistant gets smarter every day

For Teams:

• 📚 Shared knowledge base - skills become team documentation

• 🎯 Consistent code quality - best practices automatically enforced

• 🚀 Faster onboarding - new members learn patterns through skills

For the CodeWhale Project:

• 🏆 Unique competitive advantage - no other open-source agent has this

• 🌟 Stronger community - users will share their learned skills

• 📈 Increased adoption - teams will choose CodeWhale for the learning capability

Metrics to Track:

• Number of skills created automatically vs. manually

• Time saved per task (before/after learning)

• Skill usage rates and success rates

• User satisfaction scores

Additional context

Why Now?

• The skills system is already built - this adds the "learning" layer

• MCP provides the perfect hook for pattern detection

• Users are already asking for smarter assistants

Potential Challenges & Mitigations:

• False positives → Confidence thresholds and human approval (especially initially)

• Skill bloat → Automatic archiving and merging of similar skills

• Security → Version control and rollback for skills, opt-in feature with clear controls

Implementation Phases (Suggestion)

Phase 1: Pattern Detection (MVP)

• Add /suggest-skills command that analyzes current session

• Show proposed skill template for user approval

• Implement via MCP server (external)

Phase 2: Automated Creation

• Detect patterns without prompting

• Auto-create skills with configurable confidence threshold

• Add /approve-skill and /reject-skill commands

Phase 3: Evolution

• Update skills when corrections are detected

• Merge related skills automatically

• Track skill usage and confidence scores

Phase 4: Cross-Session Learning

• Share learned skills across all projects

• Aggregate best practices from multiple sessions

• Privacy controls for sensitive code

Example

$ codewhale

> Build a FastAPI endpoint for user registration with email validation

[Agent builds the feature through 8 steps]

> Now build a product listing endpoint

[Agent notices pattern...]

🧠 I've noticed you're building FastAPI CRUD endpoints with validation.
   I've seen this pattern 3 times this week.

   Would you like me to create a skill called "fastapi-crud-with-validation"? [y/n] y

✅ Skill created! It includes:
   - MongoDB connection setup
   - Pydantic model generation
   - CRUD endpoint implementation
   - Email validation pattern
   - Error handling best practices

   Type `/skill fastapi-crud-with-validation` to use it next time.

[Next week...]

> Build a blog post API

💡 I see you're building FastAPI CRUD again. Using skill: fastapi-crud-with-validation
   [Executes in 3 steps instead of 8, with best practices applied]

> Also add category filtering

🔄 I've added category filtering to the skill. This will be available for future tasks.

[github-actions]

Thanks @Edouard-Legoupil for the report.

This issue is staying open for maintainer triage. CodeWhale gets better because people bring us real edge cases from real machines, providers, regions, and workflows.

If you can add a reproduction, logs, version output, screenshots, or the provider/model involved, that makes it much easier for us to verify and harvest the fix. Maintainers may comment /lgtmi to mark recurring issue reporters as approved so this intake note is skipped next time.