[OMEGA-76] Add workflow plugin to allow adding new skills via MarkDown files

.github/workflows/common.yml

@@ -103,6 +103,10 @@ jobs:
           TEST_SERVER_IP: 'host.docker.internal'
           IMPORT_KB_ON_START: '0' # TODO: this is turned off because Python dependencies are not installed to the workflow environment
 
+      - name: Unit tests
+        if: inputs.run_tests
+        run: docker exec -e PETTA_PATH=/PeTTa omegaclaw sh /PeTTa/repos/OmegaClaw-Core/tests/mettatest.sh
+
       - name: Autotests - Phase 1 (Blocking Checks)
         if: inputs.run_tests
         working-directory: Autotests

config/plugins.yaml

@@ -55,3 +55,7 @@
 - name: mockprovider
   loader: python
   location: "{REPO}/providers"
+
+- name: workflow
+  loader: metta
+  location: "{REPO}/plugins/workflow"

lib_omegaclaw.metta

@@ -2,14 +2,16 @@
 !(import! &self (library lib_llm))
 !(import! &self (library lib_vector))
 !(import! &self (library lib_combinatorics))
+!(import! &self (library lib_he))
 !(import! &self (library OmegaClaw-Core lib_nal))
 !(import! &self (library OmegaClaw-Core lib_pln))
 !(import! &self (library OmegaClaw-Core ./providers/lib_llm_ext.py))
+!(import! &self (library OmegaClaw-Core ./src/helper.py))
+!(import! &self (library OmegaClaw-Core ./src/utils))
 !(import! &self (library OmegaClaw-Core ./src/logger.py))
 !(import! &self (library OmegaClaw-Core ./src/log))
-!(import! &self (library OmegaClaw-Core ./src/utils))
 !(import! &self (library OmegaClaw-Core ./src/plugin))
-!(import! &self (library OmegaClaw-Core ./src/helper.py))
+!(import! &self (library OmegaClaw-Core ./src/pluginapi))
 !(import! &self (library OmegaClaw-Core ./src/agentverse.py))
 !(import! &self (library OmegaClaw-Core ./src/channels))
 !(import! &self (library OmegaClaw-Core ./src/skills))

plugins/workflow/README.md

@@ -0,0 +1,77 @@
+# Workflow loading plugin
+
+Workflow loading plugin allows loading agent skills written according to the
+[common specification](https://agentskills.io/specification). Each workflow
+consists of short description, detailed description in `SKILL.md` file and
+optionally `skill.metta` file which contains related skills implementation in
+MeTTa language.
+
+[Research workflow](./instructions/research-workflow) is a ready-to-use example
+of the workflow. One can try it starting the agent and asking it doing a
+research on some topic:
+```
+Start research. Build a classifier for iris dataset using sklearn, compare
+logistic regression and random forest.
+```
+
+## Workflow files
+
+`description.txt` (required) contains short usually one-line workflow
+description for the agent. For example:
+```
+When user asks to demonstrate workflow plugin load test-workflow instructions: (workflow-load-instructions \"test-workflow\")
+```
+
+`SKILL.md` (required) is an agent skills file in the [common
+format](https://agentskills.io/specification). For example:
+```md
+---
+name: test-workflow
+description: Created to check how SKILL.md is loaded to OmegaClaw.
+---
+Next are instructions and MeTTa  functions  that should be performed step by step
+# Test Workflow (OmegaClaw)
+## Step 1 - demonstrate usage of skills
+- Call test-skill with "This is a test workflow demonstration" message
+## Step 2 - complete workflow
+- Call `(workflow-unload-instructions)`
+```
+
+`skill.metta` (optional) contains the list of the OmegaClaw skills to load
+when workflow is active and additional MeTTa functions which are mentioned in
+`SKILL.md` file.
+
+Skill description are added as high-level expressions. Each such expression
+adds one skill to the OmegaClaw. First atom of the expression is `skill` symbol
+and other atoms are parameters of the `add-skill` function. See
+[pluginapi.metta](/src/pluginapi.metta) for details. Skill implementations are
+added as MeTTa functions.
+
+For example:
+```metta
+(skill test-skill "Test skill to demonstrate workflow by sending message to the user" (message_in_quotes))
+
+(= (test-skill $message)
+   (send $message))
+```
+
+## Using workflow
+
+Start an agent and ask it to use the workflow. For the test workflow above
+send: `Demonstrate workflow plugin`
+
+## Workflow parameters
+
+Wofkflow plugin OmegaClaw configuration parameters:
+- `pluginWorkflowInstructionsDir` - path to the directory which contains
+  available workflows. Default value is `<project
+  root>/plugins/workflow/instructions`
+- `pluginWorkflowMemoryDir` - path to the directory to keep workflow working
+  files when workflow is active. Default value is `<project
+  root>/memory/workflow_space`
+
+One can set this parameters passing them as a command line arguments. For
+example:
+```sh
+sh run.sh run.metta pluginWorkflowInstructionsDir="<path>"
+```

plugins/workflow/instructions/research-workflow/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: research-workflow
+description: End-to-end research workflow for OmegaClaw. Iterative planning,
+  data acquisition, experiments, and write-up.
+---
+# Research Workflow (OmegaClaw)
+This workflow guides you through problem definition, online research,
+and creating a detailed execution plan. Once the user approves the plan,
+it replaces these instructions and you follow it step by step.
+## Project Structure
+All projects live under `(let $d (researchDir) (strings-concat ($d "/<research-name>/")))`.
+(researchDir) - is a skill (function) which returns directory the research folders and files should be located
+The `research-start` skill creates the base folders.
+(researchDir)/<research-name>/
+  topic.txt            # created by research-start
+  00_problem.md        # research question, scope, metrics
+  01_theory.md         # data sources, methods, assumptions
+  02_plan.md           # full execution plan with concrete commands
+  src/                 # all scripts
+  data/                # raw and processed data
+  runs/                # configs + metrics (JSON)
+  figures/             # plots and charts
+## Step-by-Step
+Next are instructions and MeTTa  functions  that should be performed step by step
+### Step 1 — Define the problem
+- Call `(research-start <research-name>_in_quotes "topic")` to create project folders
+- Write research question, scope, success metrics, constraints:
+  `(let $d (researchDir) (write-file (strings-concat ($d "/<research-name>/00_problem.md")) "content"))`
+- `(research-step <research-name>_in_quotes "problem-defined" "question: X metric: Y"
+                  "search online for related methods and data sources")`
+### Step 2 — Research and create plan
+- Search online for related methods and data sources:
+  `(tavily-search "query")`
+- Save findings:
+  `(let $d (researchDir) (write-file (strings-concat ($d "/<research-name>/01_theory.md")) "content"))`
+- `(research-step <research-name>_in_quotes "theory-saved" "sources: X methods: Y"
+                  "create plan and present to user")`
+- Create the full execution plan using findings and the Plan Template below
+- Send plan to user for review:
+  `(send "PROPOSED PLAN:\n<full plan text>")`
+- `(research-checkpoint <research-name>_in_quotes "Plan ready. Approve or suggest changes?")`
+- **WAIT. Do NOT advance until user responds.**
+### Step 3 — Plan approval
+- If user approves:
+save plan:
+  `(let $d (researchDir) (write-file (strings-concat ($d "/<research-name>/02_plan.md")) "approved plan text"))`
+  `(research-step <research-name>_in_quotes "plan-approved" "milestones A B C"
+                  "follow plan from Milestone A")`
+IMPORTANT!!!: load plan to active &active_instructions variable:
+  `(load-research-dynamic-instructions <research-name>_in_quotes "02_plan.md")` 
+  take into account that this function receives 2 arguments <research-name> and "02_plan.md" both in quotes.
+- If user requests changes:
+  Revise plan, send again, wait again
+## Plan Template
+When writing `02_plan.md`, include ALL sections below.
+The plan must be self-contained — after loading it replaces this file.
+# Research Plan: <topic>
+## Operating Rules
+- Work autonomously. Make your own decisions on implementation details.
+- Use `pin` to track current step between iterations.
+- All file paths must be built from `(researchDir)` via `let`: e.g. `(let $d (researchDir) (strings-concat ($d "/<research-name>/...")))`
+- Write code via `write-file`, run via `shell`. Never output code in `send`.
+- Save seeds and versions in every script.
+- Store metrics as JSON in `runs/`.
+- Use `(research-step <research-name>_in_quotes "step" "result" "next")` after each milestone.
+- Do NOT advance past a checkpoint until user responds.
+- When to ask the user (batch questions into one message):
+  - Missing credentials, access, or files
+  - Ambiguous goal or success metric
+  - Methodological choice that materially changes results
+## Milestone A — Data Ready + Baseline
+### A1 — Prepare data and baseline
+- `<concrete data source and acquisition method>`
+- Write: `(let $d (researchDir) (write-file (strings-concat ($d "/<research-name>/src/baseline.py")) "code"))`
+  Script: load data, preprocess, simple model, save runs/baseline.json
+- Run: `(let $d (researchDir) (shell (strings-concat ("cd " $d "/<research-name> && python src/baseline.py"))))`
+- `(research-step <research-name>_in_quotes "data-ready" "N rows, baseline=X"
+                  "call research-checkpoint")`
+- `(research-checkpoint <research-name>_in_quotes "Data ready. Baseline: X. Proceed?")`
+- **WAIT for user.**
+## Milestone B — Experiments
+### B1 — <experiment name>
+- Hypothesis: <what you expect>
+- Write: `(let $d (researchDir) (write-file (strings-concat ($d "/<research-name>/src/experiment_1.py")) "code"))`
+- Run: `(let $d (researchDir) (shell (strings-concat ("cd " $d "/<research-name> && python src/experiment_1.py"))))`
+- Save: `(let $d (researchDir) (write-file (strings-concat ($d "/<research-name>/runs/exp1.json")) "metrics"))`
+- `(research-step <research-name>_in_quotes "experiment-1" "metric=Y" "next experiment")`
+### B2 — <next experiment>
+...
+## Milestone C — Results + Conclusions
+### C1 — Write results
+- `(let $d (researchDir) (write-file (strings-concat ($d "/<research-name>/05_results.md")) "content"))`
+- `(research-step <research-name>_in_quotes "results-written" "best: X" "write conclusions")`
+### C2 — Conclusions
+- `(let $d (researchDir) (write-file (strings-concat ($d "/<research-name>/06_conclusions.md")) "content"))`
+- `(research-step <research-name>_in_quotes "conclusions-done" "summary"
+                  "call research-checkpoint")`
+- `(research-checkpoint <research-name>_in_quotes "Results ready. Next iteration? Proceed?")`
+- **WAIT for user.**
+## Milestone D — Complete
+- `(research-complete <research-name>_in_quotes)`
+## Stop/Pivot Conditions
+- `<when to stop or change approach>`

plugins/workflow/instructions/research-workflow/description.txt

@@ -0,0 +1 @@
+Every time user asks to start research task, investigate, study, analyze, or build ML model load research-workflow instructions :(workflow-load-instructions \"research-workflow\")

plugins/workflow/instructions/research-workflow/skill.metta

@@ -0,0 +1,122 @@
+; Declare skills to be added to the agent skills when workflow is loaded and
+; removed on unloading workflow
+
+(skill research-start "Create research project folders" (research_name_in_quotes topic_in_quotes))
+(skill research-step "Mark research step done" (research_name_in_quotes step_in_quotes result_in_quotes next_action_in_quotes))
+(skill research-checkpoint "Pause for user approval" (researchname_in_quotes message_in_quotes))
+(skill research-complete "Finish research and unload workflow" (researchname_in_quotes))
+(skill load-research-dynamic-instructions " Load a generated file into active context" (researchname_in_quotes filename_in_quotes))
+(skill researchDir "Get research directory" ())
+
+;; Actual skills implementation
+
+;; Build a safe path or empty string if validation fails
+(= (safe-instruction-path $project_dir $research-name $filename)
+   ; TODO: calling Python validation code is replaced by stub.
+   ; Can be returned back when https://github.com/trueagi-io/PeTTa/issues/196 is
+   ; fixed.
+   ;(py-call (utils.safe_path $project_dir $research-name $filename)))
+   (joinPath ($project_dir $research-name $filename)))
+
+(= (load-research-dynamic-instructions  $research-name $filename)
+    (let $d (researchDir) (load-dynamic-instructions $d $research-name $filename)))
+
+; loads instructions created during workflow
+(= (load-dynamic-instructions $project_dir $research-name $filename)
+   (if (not (valid-file-or-folder-name $research-name))
+       (send (strings-concat ("ERROR: invalid name: " $research-name)))
+       (if (not (valid-file-or-folder-name $filename))
+           (send (strings-concat ("ERROR: invalid filename: " $filename)))
+           (let $path (safe-instruction-path $project_dir $research-name $filename)
+                (if (== $path "")
+                    (send (strings-concat ("ERROR: path traversal rejected for " $research-name "/" $filename)))
+                    (load-dynamic-instructions-safe $path $research-name $filename))))))
+
+; actually loads the file (internal). Only called after validation.
+(= (load-dynamic-instructions-safe $path $research-name $filename)
+   (if (not (exists-file $path))
+       (send (strings-concat ("ERROR: file not found: " $path)))
+       (let $content (read-file $path)
+            (progn
+              (workflow-change-active-instructions $content)
+              (appendToHistory ((get_time_as_string)
+                                (newline)
+                                (strings-concat ("LOADED [" $research-name "/" $filename "]"))
+                                (newline)))
+              (pin (strings-concat ("ACTIVE: " $research-name
+                            " FILE: " $filename
+                            " STATUS: executing"
+                            " NEXT: follow loaded instructions")))
+              (send (strings-concat ("Loaded " $filename " for " $research-name ". Following instructions.")))
+              $content))))
+
+
+(= (researchDir)
+   (let $dir (get-existing-dir (pluginWorkflowMemoryDir))
+     (joinPath ($dir "research"))))
+
+
+;; LLM decides to start research
+;; creates folders for research and asks to create 00_problem.md then search online
+(= (research-start $research-name $topic)
+    (if (valid-file-or-folder-name $research-name)
+       (progn (shell (strings-concat ("mkdir -p " (researchDir) "/" $research-name "/src")))
+              (shell (strings-concat ("mkdir -p " (researchDir) "/" $research-name "/data")))
+              (shell (strings-concat ("mkdir -p " (researchDir) "/" $research-name "/runs")))
+              (shell (strings-concat ("mkdir -p " (researchDir) "/" $research-name "/figures")))
+              (write-file (strings-concat ((researchDir) "/" $research-name "/topic.txt")) $topic)
+              (send (strings-concat ("Created project: " (researchDir) "/" $research-name "\n"
+                             "Folders: src/ data/ runs/ figures/")))
+              (pin (strings-concat ("RESEARCH_ACTIVE: " $research-name
+                            " DIR: " (researchDir) "/" $research-name
+                            " TOPIC: " $topic
+                            " STEP: 1-problem"
+                            " ACTION_REQUIRED: Write 00_problem.md then search online"
+                            " for related methods and data sources.")))
+              $research-name)
+        (send (strings-concat ("Invalid name: " $research-name ". Only a-z, 0-9, - and _ allowed.")))
+       )
+)
+
+;; update current step
+(= (research-step $research-name $step $result $next)
+    (if (valid-file-or-folder-name $research-name)
+       (progn (appendToHistory  ((get_time_as_string) (newline) (strings-concat  ("RESEARCH [" $research-name "] done: " $step " — " $result)) (newline)) )
+              (send (strings-concat ("[" $research-name "] done: " $step)))
+              (pin (strings-concat ("RESEARCH_ACTIVE: " $research-name
+                            " DONE: " $step
+                            " RESULT: " $result
+                            " NEXT: " $next)))
+              (strings-concat ("Done: " $step)))
+        (send (strings-concat ("Invalid name: " $research-name ". Only a-z, 0-9, - and _ allowed.")))
+     )
+)
+
+
+;; Checkpoint: wait user's response
+(= (research-checkpoint $research-name $message)
+    (if (valid-file-or-folder-name $research-name)
+        (progn (pin (strings-concat ("RESEARCH_ACTIVE: " $research-name
+                            " STATUS: WAITING_FOR_USER"
+                            " DO_NOT_PROCEED"
+                            )))
+              (send (strings-concat ("CHECKPOINT [" $research-name "]\n" $message "\n"
+                             "Reply to continue.")))
+              "Waiting for user")
+        (send (strings-concat ("Invalid name: " $research-name ". Only a-z, 0-9, - and _ allowed.")))
+     )
+)
+
+;; Unload research skill
+(= (research-complete $research-name)
+    (if (valid-file-or-folder-name $research-name)
+        (progn (workflow-unload-instructions)
+              (send (strings-concat ("Research complete: " $research-name
+                             "\nResults in " (researchDir) "/" $research-name "/"
+                             "\nWorkflow unloaded from context.")))
+              (appendToHistory  ((get_time_as_string) (newline) (strings-concat  ("RESEARCH [" $research-name "] COMPLETED")) (newline)) )
+              (pin "No active research. No active workflow.")
+              "Research complete, workflow unloaded")
+         (send (strings-concat ("Invalid name: " $research-name ". Only a-z, 0-9, - and _ allowed.")))
+      )
+)

plugins/workflow/instructions/test-workflow/SKILL.md

@@ -0,0 +1,10 @@
+---
+name: test-workflow
+description: Created to check how SKILL.md is loaded to OmegaClaw.
+---
+Next are instructions and MeTTa  functions  that should be performed step by step
+# Test Workflow (OmegaClaw)
+## Step 1 - demonstrate usage of skills
+- Call test-skill with "This is a test workflow demonstration" message
+## Step 2 - complete workflow
+- Call `(workflow-unload-instructions)`

plugins/workflow/instructions/test-workflow/description.txt

@@ -0,0 +1 @@
+This is a test workflow description, to load the workflow use the command: (workflow-load-instructions \"test-workflow\")

plugins/workflow/instructions/test-workflow/skill.metta

@@ -0,0 +1,4 @@
+(skill test-skill "Test skill to demonstrate workflow by sending message to the user" (message_in_quotes))
+
+(= (test-skill $message)
+   (send $message))

plugins/workflow/utils.py

@@ -0,0 +1,35 @@
+import os
+import re
+
+def validate_file_or_folder_name(filename: str) -> str:
+    """
+    Validate filename: letters, digits, hyphen, underscore, dot.
+    Returns "valid" or "invalid".
+    """
+    if not filename:
+        return "invalid"
+    if not re.match(r'^[a-zA-Z0-9_.-]+$', filename):
+        return "invalid"
+    if filename.startswith('.') or '..' in filename:
+        return "invalid"
+    return "valid"
+
+
+def safe_path(base_dir: str, folder: str, filename: str) -> str:
+    """
+    Safely build a file path inside base_dir.
+    Returns the resolved path or empty string if unsafe.
+    """
+    if validate_file_or_folder_name(folder) != "valid":
+        return ""
+    if validate_file_or_folder_name(filename) != "valid":
+        return ""
+
+    base = os.path.realpath(base_dir)
+    target = os.path.realpath(os.path.join(base, folder, filename))
+
+    # Ensure target is inside base directory (prevents path traversal via symlinks)
+    if not (target == base or target.startswith(base + os.sep)):
+        return ""
+
+    return target