Skip to content

SELF_AWARENESS.md

Latest commit

 

History

History
306 lines (216 loc) · 13.1 KB

SELF_AWARENESS.md

File metadata and controls

306 lines (216 loc) · 13.1 KB
Raw

Self-Awareness: Local Model Skill Generation

AlexClaw can generate valid dynamic skills using a local 14B parameter model, with zero cloud API cost. This document describes the full process — from embedding documentation to a working skill generated by a local LLM.

Overview

AlexClaw runs inside a Docker container with no access to its own source code at runtime. To enable a local LLM to understand and extend the system, we built a knowledge pipeline that embeds AlexClaw's own documentation into the same vector database used for hexdocs and other knowledge. The local model can then query this knowledge via RAG to generate valid Elixir code that compiles, loads into the running BEAM VM, and executes as a workflow step.

This is the foundation for evolving AlexClaw from a reactive agent (cron + commands) to an autonomous agent that reasons about goals and writes its own skills.

Step-by-Step: What We Did

1. Embedded the Elixir Ecosystem Documentation (hexdocs.pm)

AlexClaw has a dynamic skill called hexdocs_scraper that crawls hexdocs.pm, parses API documentation for Elixir packages, chunks it by module/function, and stores each chunk as a 768-dimension vector embedding in the knowledge_entries table.

We ran this as a workflow (Admin > Workflows > HexDocs Scraper > Run Now). The scraper covered 22 packages:

  • Elixir stdlib (full standard library)
  • Phoenix, Phoenix LiveView, Phoenix HTML, Phoenix PubSub
  • Ecto, Ecto SQL
  • Plug, Plug Crypto
  • Jason, Req, Finch, Mint
  • Floki, SweetXml
  • Telemetry, Telemetry Metrics, Telemetry Poller
  • Bandit, NimbleOptions, NimblePool, GenStage

Result: ~4400 documentation chunks with embeddings, giving the local model comprehensive Elixir API knowledge.

2. Built the Self-Awareness Knowledge Loader

We created AlexClaw.Knowledge.SelfAwareness — a module that runs on every container boot as a background task. It reads four files that define what AlexClaw is and how to extend it:

File Source Key What It Teaches the Model
ALEXCLAW_ARCHITECTURE.md self:architecture Supervision tree, components, data flows, how skills fit in
SECURITY.md self:security Security policy, what's allowed, deployment constraints
lib/alex_claw/skill.ex self:skill_behaviour The exact @callback definitions — run/1, description/0, permissions/0, version/0, routes/0 and their return types
test/fixtures/skills/skill_template.ex self:skill_template A complete working skill with the full SkillAPI reference as comments — every available function call

These files are bundled into priv/self_awareness/ during the Docker build. On each boot, old entries are deleted and re-embedded to ensure fresh vectors — this guarantees the model always has up-to-date knowledge even after architecture changes.

Result: 30 chunks (16 architecture + 12 security + 1 behaviour + 1 template), all with embeddings.

3. Set Up Local Models in LM Studio

Two models loaded simultaneously on a single RTX 4080 (16GB VRAM):

Chat model — Qwen 2.5 14B Instruct:

  • Handles reasoning and code generation
  • Connected via OpenAI-compatible API at http://host.docker.internal:1234
  • Configured as a local tier provider in AlexClaw's LLM router

Embedding model — nomic-embed-text-v1.5 (GGUF):

  • Generates 768-dimension vectors for all knowledge entries
  • ~140MB VRAM, runs instantly alongside the chat model
  • Connected via LM Studio's /v1/embeddings endpoint

Configuration in Admin > Config:

Key Value
llm.lmstudio_enabled true
llm.lmstudio_host http://host.docker.internal:1234
llm.lmstudio_model (model name as shown in LM Studio)
embedding.provider (dropdown of enabled provider names in Admin UI)
embedding.model text-embedding-nomic-embed-text-v1.5

4. First Test — Without Proper Embeddings (Failed)

We tested before the self-awareness embeddings were properly generated (the embedding model wasn't loaded when the container booted, so the 30 self-awareness chunks had no vectors).

Prompt used in Chat (Docs context, Qwen model):

Based on your knowledge of AlexClaw architecture, write a complete dynamic skill module called AlexClaw.Skills.Dynamic.SystemInfo that returns the current date, time, hostname, and Elixir version. It must follow the AlexClaw.Skill behaviour, use the AlexClaw.Skills.Dynamic namespace, and return a triple tuple. Output only the Elixir code, no explanation.

Result — broken code:

defmodule AlexClaw.Skills.Dynamic.SystemInfo do
  @behaviour AlexClaw.Skill

  def call(%{} = _input) do
    date_time = DateTime.utc_now()
    hostname = :os.cmd('hostname'). |> to_string() |> String.trim()
    elixir_version = System.version()
    {:ok, %{date: ..., hostname: hostname, elixir_version: elixir_version}}
  end
end

Issues:

  • call/1 instead of run/1 — wrong callback name
  • Missing @impl true annotations
  • Missing description/0 callback
  • Syntax error: stray . after :os.cmd('hostname')
  • Returned {:ok, result} (legacy two-tuple) instead of {:ok, result, :on_success}
  • No permissions/0, version/0, or routes/0

The model got the namespace right and the general idea, but without the skill template and behaviour definition in its context, it guessed at the API and got it wrong.

5. Fixed Embeddings and Retested (Success)

We ensured LM Studio's embedding model was loaded, then restarted the container. The self-awareness loader ran, deleted old entries, and re-embedded all 30 chunks with proper vectors. Verified with:

SELECT source, COUNT(*), COUNT(embedding)
FROM knowledge_entries WHERE kind = 'self_awareness'
GROUP BY source;

All 30/30 entries had embeddings.

Prompt used in Chat (Docs context, Qwen model):

You are AlexClaw. Using your knowledge of your own architecture, skill behaviour, and the dynamic skill template, write a complete dynamic skill module called AlexClaw.Skills.Dynamic.SystemInfo that returns the current UTC date/time, hostname, and Elixir version as a formatted string. Output only the Elixir code.

Result — valid code, first attempt, unedited:

defmodule AlexClaw.Skills.Dynamic.SystemInfo do
  @behaviour AlexClaw.Skill

  @impl true
  def run(%{} = _args) do
    date_time = DateTime.utc_now() |> DateTime.to_string()
    hostname = :os.cmd('hostname') |> to_string() |> String.trim()
    elixir_version = System.version()

    result_string = "UTC Date/Time: #{date_time}, Hostname: #{hostname}, Elixir Version: #{elixir_version}"
    {:ok, result_string, :on_success}
  end

  @impl true
  def description() do
    "Returns current UTC date/time, hostname, and Elixir version."
  end

  @impl true
  def permissions() do
    []
  end

  @impl true
  def version() do
    "1.0.0"
  end

  @impl true
  def routes() do
    [:on_success]
  end
end

6. Loaded and Executed

  1. Saved as system_info.ex in the skills volume
  2. Loaded via /skill load system_info.ex in Telegram
  3. Created a workflow in Admin > Workflows with two steps:
    • Step 1: system_info skill (no config needed)
    • Step 2: telegram_notify skill (delivers output)
  4. Clicked "Run Now"

Workflow output (delivered via Telegram):

UTC Date/Time: 2026-03-21 07:12:14.205190Z, Hostname: cad43eb9b8d9, Elixir Version: 1.19.5

Workflow run status: completed. Both steps succeeded — system_info returned :on_success, telegram_notify returned :on_delivered.

What Made It Work

The model got everything right:

  • Correct namespace: AlexClaw.Skills.Dynamic.*
  • Correct behaviour: @behaviour AlexClaw.Skill
  • All callbacks marked with @impl true
  • Correct return format: {:ok, result_string, :on_success} (triple tuple)
  • All optional callbacks implemented: description/0, permissions/0, version/0, routes/0
  • Valid Erlang interop: :os.cmd/1 for hostname
  • No external dependencies
  • Clean, idiomatic Elixir

Why it worked with embeddings but not without:

The RAG retrieval pulled the skill template (with the full SkillAPI reference and correct callback signatures) and the behaviour definition (with exact @callback specs) into the model's context. Without those chunks, the model was guessing — with them, it was following a specification.

How the Chat UI Works for This

AlexClaw's Chat page (Admin > Chat) has two key selectors:

  1. Model picker — choose any configured LLM provider (cloud or local). For this test, we selected the local Qwen 2.5 14B via LM Studio.

  2. Context source — determines what knowledge is injected into the system prompt:

    • Docs only — searches knowledge_entries (hexdocs + self-awareness docs)
    • Memory only — searches memories (news, conversations, facts)
    • Both — searches both tables
    • None — no RAG context

For skill generation, use Docs only — this retrieves the architecture docs, skill template, and Elixir API documentation.

The system prompt instructs the LLM to cite provided documentation over general knowledge, ensuring it follows AlexClaw's actual contracts rather than hallucinating an API.

Knowledge Base Summary

Source Chunks Description
hexdocs.pm (22 packages) ~4400 Elixir ecosystem API documentation
self:architecture 16 AlexClaw system design and component docs
self:security 12 Security policy and hardening guidance
self:skill_behaviour 1 AlexClaw.Skill callback definitions
self:skill_template 1 Complete dynamic skill template with SkillAPI reference
Total ~4430 Full context for local model code generation

All embeddings are 768-dimension vectors stored in PostgreSQL via pgvector, indexed with HNSW for fast cosine similarity search.

Verification Commands

Check self-awareness loading (after container start)

Look for log lines:

SelfAwareness: loaded ALEXCLAW_ARCHITECTURE.md (16 chunks)
SelfAwareness: loaded SECURITY.md (12 chunks)
SelfAwareness: loaded lib/alex_claw/skill.ex (1 chunks)
SelfAwareness: loaded test/fixtures/skills/skill_template.ex (1 chunks)

Verify embeddings in database

-- Self-awareness entries
SELECT source, COUNT(*) as total, COUNT(embedding) as embedded
FROM knowledge_entries WHERE kind = 'self_awareness'
GROUP BY source;

-- All knowledge entries
SELECT kind, COUNT(*) as total, COUNT(embedding) as embedded
FROM knowledge_entries GROUP BY kind;

All entries should have non-null embeddings.

Next Steps — Progress

Step Status Implementation
1. Guided skill generation Done Manual testing via Chat UI proved the approach works reliably with embeddings
2. Feedback loop Done Coder.generation_loop/5 — compile error appended to prompt, retry (configurable, default 3)
3. Autonomous workflow creation Done Coder creates workflow + telegram_notify step when create_workflow: true (disabled by default)
4. Goal-driven autonomy Done /coder <goal> — model receives goal, searches knowledge base, generates skill, loads it, optionally wires workflow

What's New: The Coder Skill

AlexClaw.Skills.Coder (lib/alex_claw/skills/coder.ex) is a core skill that orchestrates the full generate, write, load, wire cycle:

  1. Derives a snake_case module name from the goal
  2. Searches knowledge base for architecture docs + skill template (RAG context)
  3. Sends goal + context to local LLM with a system prompt specifying the skill contract
  4. Extracts code block from response
  5. Writes to skills directory via SkillAPI.write_skill/3 (filename validated)
  6. Loads via SkillAPI.load_skill/2 (namespace, behaviour, permissions validated by SkillRegistry)
  7. On compile error: appends error to prompt, retries (up to max_retries)
  8. On success: optionally creates a disabled workflow with the skill + telegram_notify

SkillAPI extensions enabling this:

  • :skill_write — write/read .ex files (path traversal prevention)
  • :skill_manage — load/unload/reload dynamic skills
  • :workflow_manage — create workflows, add steps, run, get results

Remaining Goals

  • Self-improvement loop — model evaluates its own generated skills, identifies issues, and iterates
  • Multi-skill composition — model generates multiple skills that work together in a pipeline
  • Proactive goal generation — model identifies gaps in its capabilities and proposes new skills

Key Insight

The bottleneck is not model size — it's retrieval quality. A 14B model with the right documentation chunks in context produces valid, compilable Elixir that follows the exact behaviour contract. The same model without those chunks produces broken code that guesses at the API.

What matters:

  1. The behaviour contract is small (5 callbacks) — fits easily in context
  2. The template is self-documenting — includes the full API reference as comments
  3. The architecture doc provides system-level understanding
  4. 768-dim nomic embeddings retrieve the right chunks
  5. 14B parameters are sufficient for template-guided code generation

The self-awareness embeddings turned a failing code generation attempt into a first-try success.