Aod activity on demand - #1107
Open
Ashutoshx7 wants to merge 28 commits into
Open
Conversation
Activity on Demand (AOD) lets any learner describe an activity in plain language and receive a working, installable XO bundle. A model-agnostic LLM router, RAG corpus built from real Sugar Activity source code, and AST-based security validator work together to produce genuine, safe Python with proper GTK3 widgets, toolbar scaffolding, and Journal hooks. Backend (src/jarabe/model/aod*.py): - aodspec: ActivitySpec dataclass, structured prompt parsing, validation - aodgenerator: template selection, plan building, project assembly - aodpipeline: full generation pipeline (spec -> RAG -> plan -> codegen -> validate -> assemble), 3 retries with validation feedback to LLM - aodllm: multi-provider support (Gemini, OpenAI, OpenRouter, DeepSeek, Qwen, Moonshot, OpenCode, FreeModel, Claude, Ollama, local-template) - aodrag: keyword-based RAG corpus from real installed Sugar activities - aodvalidator: AST-based security (forbidden imports/calls, class conformance, required methods, prompt-specific checks) - aodcodegen: codegen prompt builder, fenced code extraction, truncated response detection - aodjobs: file-backed job store with state machine - aodservice: local backend service with job queue and session tracking - aodsessions: session store for revision history and chat messages - aodcredentials: API key storage with keyring/profile-file fallback - aodqueue: threaded job queue worker - aodprompts: LLM planner system/user prompt builders - aodtemplates: Sugar activity template archetypes - aodlicenses: SPDX license ID registry Frontend (src/jarabe/desktop/): - homebox.py: AOD panel with prompt screen, generation animation with step cards, studio review with code viewer, live edit, version history - homewindow.py: lazy import hook for AOD panel - viewtoolbar.py: Create with AI toolbar button entry point Tooling: - aod_test_cli.py: CLI harness for testing generation with any provider - aod_benchmark.py: multi-provider benchmark runner - start_sugar_xephyr.sh: nested Xephyr test environment Tests: 80 tests covering spec validation, generator, pipeline, LLM providers, RAG search, validator, prompts, credentials, and service.
Replace 'keep compact' with detailed quality requirements: real UI with titled sections, multiple interaction modes via Gtk.Stack, proper domain logic, polished GTK3 with Pango markup, rich toolbar with custom actions, keyboard shortcuts, and 400-900 line target for most requests. Add per-request-type richness examples (drawing, quiz, board game, writing, two-learner).
New aodrefine.py module implements Aider-style SEARCH/REPLACE blocks for targeted edits to existing activity.py: - build_refine_system_prompt(): tells model to return only changed regions as <<<<<<< SEARCH / ======= / >>>>>>> REPLACE blocks - build_refine_user_prompt(): sends current source + refinement request - parse_search_replace(): parses blocks, handles FULLREGEN signal - apply_patches(): whitespace-tolerant matching, counts applied/failed Pipeline refine_activity() tries SEARCH/REPLACE first (~1k tokens, ~5s), falls back to full regen if: - Model outputs FULLREGEN - Any SEARCH block doesn't match current source - Patched code fails AST validation Service _run_refinement_job() routes refinement jobs through refine_activity() for LLM providers, skips planner call by reusing existing plan. Local-template refinements still use generate_activity(). Cost per refinement on Kimi K2.6 via OpenRouter: - SEARCH/REPLACE: ~/usr/bin/bash.0006, ~5-10s - Full regen fallback: ~/usr/bin/bash.007, ~25-40s - Previous (plan + full regen): ~/usr/bin/bash.012, ~40-60s 19 new tests: parsing, patching, whitespace tolerance, multi-block, FULLREGEN, error cases, prompt builders.
Bug 1 (critical): SEARCH/REPLACE path was completely dead because all providers ran the response through extract_activity_source() which expects a complete activity.py, not SEARCH/REPLACE blocks. Added generate_text() to every provider (Gemini, OpenAI, FreeModel, Claude, Ollama) that returns raw model text without extraction. Pipeline now calls generate_text() for refinement instead of generate_activity_source(). Bug 2: refine_method metadata was dropped by normalize_plan() because it was not in the passthrough whitelist. Added 'refine_method' to the whitelist so callers can tell whether SEARCH/REPLACE or full regen was used. Bug 3: bundle_id changed on every refinement because normalize_plan() recomputed it from spec.prompt (which differs for each refinement). Now preserves the parent bundle_id from the incoming plan. Bug 5: Model received the source twice (truncated in spec.prompt + full in refine prompt). Removed the source excerpt from _build_refinement_spec() since refine_activity() sends the full source via build_refine_user_prompt(). Bug 7: Removed dead extract_activity_source_from_response import from pipeline.
- Fix sidebar preview vanish: drop show_all() (was revealing hidden toolbar buttons), store live canvas ref, call queue_draw() directly on canvas + deferred 200ms second pass - Fix SEARCH/REPLACE marker collision: anchor ======= and >>>>>>> REPLACE searches to line boundaries so identical strings in source code are not mistaken for block dividers - Fix Gemini empty response: both streaming and non-streaming paths now raise ProviderError instead of returning empty string - Fix aodsessions double disk-load: add append_messages() and append_revision_and_message() for atomic multi-mutation saves - Fix aodqueue unbounded growth: cap at 64 jobs with put_nowait() - Add code_size field to ActivitySpec (compact/standard/full) - Thread max_output_tokens through all five LLM providers - Add code size dropdown to prompt UI (~500/~1000 lines/Full output) - Adjust codegen system prompt length instruction per code_size selection
- aodrag: cache build_corpus() result at process level, invalidated by bundle directory mtime so new installs are picked up automatically - aodpipeline: add 1s/2s/4s exponential backoff between codegen retry attempts instead of retrying immediately after validation failure - aodpreview: add cleanup() method to PreviewActivity that removes the tempfile.mkdtemp() activity root; previously accumulated on every generation with no cleanup - homebox: store live preview instance in _live_preview_activity and call cleanup() when the preview is replaced or cleared
When Live Edit Mode is ON, clicking any widget in the live-generated activity preview now identifies exactly what was clicked (e.g. 'button: Reset', 'drawing canvas', 'text area', 'slider') instead of always reporting 'activity canvas'. Implementation: - Walk the live-generated GTK widget tree recursively on preview render (_walk_and_attach_live_edit), attaching button-press-event handlers to every identifiable leaf widget (Button, ToolButton, Entry, DrawingArea, Label, Scale, TextView, Grid, Toolbar) - _describe_widget_for_live_edit derives a human-readable target string from widget type + label/tooltip/placeholder - __live_edit_widget_press_cb updates the target label, applies a yellow CSS highlight ring (.live-edit-selected), and returns True to stop propagation (prevents outer shell from overriding with 'activity canvas') - Clicking empty space still falls through to the shell handler and reports 'activity canvas' as before - _detach_live_edit_handlers disconnects all handlers and removes the CSS highlight when the preview is cleared or replaced
…r challenges - Fix blank preview: move live-edit handler attachment after show_all() and wrap in try/except so GLib cannot swallow exceptions before pack_start - Fix walker: return after attaching to leaf widget to avoid double-handler on button's internal label child (which broke button interaction) - After generation, emit a conversational AI chat bubble instead of a plain status log line, describing what was built (kind, summary, interaction model) - Learning sidebar challenges are now derived from the AOD plan (template, activity_kind, features, learner_steps) and replaced after each generation instead of showing 8 hardcoded generic prompts
- Remove 1900px min-width from studio workspace so the three-panel layout fits within a standard 1366px screen without clipping the sidebar - Shrink learning sidebar from 380px to 260px to leave more room for preview - Replace single long AI chat blob with 2-4 short separate bubbles: name+kind, one-line summary, first learner step, action prompt
Wrap the learning sidebar in a Gtk.Revealer with SLIDE_LEFT transition (200ms). Toggle reveal_child() instead of show()/hide() so the panel slides in/out smoothly. Fullscreen mode also animates through the revealer.
Gtk.Revealer keeps its full width allocated during the slide, so the preview panel would jump to fill the freed space only when the animation ended. Replace with a 16ms GLib timer that directly changes the sidebar set_size_request() each frame using a smoothstep ease (250ms). The HBox reallocates on every tick, so the preview expands in perfect sync with the sidebar closing. - _animate_sidebar(opening): starts the animation, shows panel before fade-in - _sidebar_anim_tick(): smoothstep interpolation, hides panel at end of close - _sidebar_snap(): instant snap for fullscreen toggle (no animation)
Learning area: - Expanded from 4 to 6 areas: Science (explore & measure) and Language (stories & words) added alongside Logic, Tools, Games, Creation Constructionist additions: - New 'Who learns' option group (age_band: 6-9 / 10-13 / 14+ / any) feeds ActivitySpec.age_band so the planner adjusts vocabulary/complexity - New 'How they work' option group (collab: solo / pair / class) prepends collaboration context to the generation prompt Generation options: - Planner now has two real choices: RAG (Sugar examples) and Direct (no examples, faster); Direct sets use_rag=False in the pipeline - Policy adds Creative option (fewer structural constraints) Layout: - Replaced inline 'Start with' / 'Learning area' headings with subtle uppercase section labels via _create_section_label() - LLM provider selector collapsed into a Gtk.Expander (hidden by default) to reduce visual clutter for users who don't need it - Cards use border-radius 8px with 120ms CSS ease transition on hover - Hint texts updated for all 6 learning areas with richer descriptions
Before: one dense row with 6+ widgets — combo, key, Paste, Model, Endpoint, Save & test, Remove key all visible simultaneously. After: - Primary row: provider picker + API key + Paste + Save + Remove (clean, one clear action; labels shortened to 'Save' / 'Remove') - Advanced row (Model override + Endpoint URL): shown only when the selected provider actually supports those fields (cloud or Ollama); hidden for 'Automatic' and local providers - 'Save & test' → 'Save' (the test happens implicitly on next generate)
- Drop 'Who learns' and 'How they work' option groups — Sugar's existing activity templates already encode these implicitly - Remove Model override and Endpoint URL input boxes from the provider row entirely; keep the Gtk.Entry stubs as non-rendered widgets so the save callback continues to work without layout noise
…quest The manual smoothstep loop called set_size_request(w, -1) each tick, but GTK allocates non-expand widgets at their natural content size regardless of the size request minimum — so the widget never actually shrank and the sidebar just snapped on hide(). Gtk.Revealer overrides get_preferred_width during animation so the HBox correctly shrinks/grows the revealer on every frame, the preview expands in sync, and the content is clipped to the animated width. - SLIDE_LEFT transition (260ms): sidebar slides from/to the right edge - notify::child-revealed signal fires _refresh_preview_layout() once done - Fullscreen toggle uses duration=0 snap (no animation, just instant hide)
Preview: replace the static generating screen with a bigger XO logo in an animated orbit canvas (breathing halo, comet that closes into the progress ring, staggered entrance, completion ripple). Stopping the animation early (instant failure or cancel) now restores every faded widget and settles the ring, so fast failures no longer leave a blank preview panel. Pipeline: add science and language as first-class learning categories end to end (spec, planner steering, template ranking); coerce unknown soft spec fields instead of failing the job; retry transient HTTP failures (429/5xx/network) with backoff across all LLM providers; retry planning once on a malformed plan response; lower the validator size floor for compact activities. Also add the Flatpak export module with tests.
Ashutoshx7
force-pushed
the
aod-activity-on-demand
branch
from
July 3, 2026 11:08
4652135 to
199a222
Compare
pygame/sugargame are now validated against the actual runtime: a missing one fails validation with a rewrite-with-cairo instruction fed back through the codegen retry loop, and the codegen system prompt stops offering pygame on machines without it.
A new aodenhance module turns a learner's few words into a detailed brief. The pipeline auto-enhances short prompts before RAG/planning (fail-soft, per-job toggle), the create view gains a ✨ Enhance button and an Enhance Auto/Off chip, and the chat shows the brief the AI understood so learners see what a strong prompt looks like. Original learner words are kept for sessions, versions, and refinements.
Generated code now clears three gates before a learner sees it: static validation, an actual run in a sandboxed GTK subprocess (aodruntime + aodruntimeharness — start, pump events, Journal write/read round-trip; crashes and degraded startups become retry feedback, blocking __init__ hits a timeout), and one self-review round (aodcritic — checklist review answered with OK or minimal SEARCH/REPLACE fixes, kept only if the patched code re-passes the first two gates). Validator warnings ride along on retries as "Also consider" hints. Every activity also gets its own icon (aodicons): deterministic glyph per template/category, name-hash accent, Sugar entity header so it recolors like any shell icon. Outcomes land in the saved plan under runtime_check / critic; env knobs AOD_RUNTIME_CHECK, AOD_RUNTIME_CHECK_TIMEOUT, AOD_CRITIC. .flake8: keep W503/W504 in the ignore list (flake8 ignores both by default; a bare ignore= re-enables two mutually exclusive checks).
Port of the studio change: after code is accepted, one small generate_text call asks the model for a 55x55 SVG on Sugar's &stroke_color;/&fill_color; entities, so the icon shows the activity's own metaphor and still recolors to the learner's XO colors. Replies are strictly sanitized (no scripts, images, text, gradients, event attributes, or external references; entity header re-applied; XML parse check); anything doubtful falls back to the deterministic glyph. Outcome recorded under plan['icon_source']; disable with AOD_AI_ICON=off.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.