Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,28 @@ Die alten Claude-Commands unter `C:\Users\admin\.claude\commands\` sollen in Cod

Wenn ein Skill nicht verfuegbar ist, direkt ueber lokale Skripte, MCP-Tools, HTTP-Bridge oder API arbeiten. Nicht an Claude-Command-Dateien haengen bleiben.

## 0.2 MasterSelects Debug Bridge
## 0.2 gstack Integration

`gstack` ist fuer strukturierte Planung, Reviews, Browser-QA und Security-Checks verfuegbar. Global installieren, nicht ins Repo vendorisieren.

Installationspfade:

- Codex: `git clone --single-branch --depth 1 https://github.com/garrytan/gstack.git ~/gstack && cd ~/gstack && ./setup --host codex`
- Claude Code: `git clone --single-branch --depth 1 https://github.com/garrytan/gstack.git ~/.claude/skills/gstack && cd ~/.claude/skills/gstack && ./setup --team`
- Windows: Git Bash oder WSL verwenden; `bun` und `node` muessen installiert sein
- Nach der Codex-Installation Codex neu starten, damit neue Skills geladen werden

Einsatzregeln:

- `masterselects` bleibt erste Wahl fuer Timeline-, Preview-, Clip- und Debug-Bridge-Automation in der lokalen App
- `gstack-office-hours`, `gstack-autoplan`, `gstack-plan-eng-review`, `gstack-plan-design-review` und `gstack-plan-devex-review` fuer Discovery, Scope und Plan-Qualitaet
- `gstack-review` fuer unabhaengige Code-Reviews
- `gstack-investigate` fuer Root-Cause-Debugging statt Trial-and-Error-Fixes
- `gstack-cso` fuer Security-Reviews
- `gstack-qa`, `gstack-qa-only`, `gstack-browse`, `gstack-open-gstack-browser` und `gstack-setup-browser-cookies` fuer Browser-QA, Repros und auth-geschuetzte Flows
- `gstack-upgrade` verwenden, statt eine Repo-lokale gstack-Kopie zu pflegen

## 0.3 MasterSelects Debug Bridge

Fuer App-Debugging existieren lokale AI-Tools hinter `POST http://localhost:5173/api/ai-tools`. Voraussetzung: Dev-Server laeuft und die App ist im Browser geoeffnet.

Expand Down
24 changes: 24 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,30 @@ Die `/masterselects` Skill stellt 4 Debug-Tools bereit, die über den HTTP Bridg

---

## 0.2 gstack Integration

`gstack` ist fuer strukturierte Planung, Reviews, Browser-QA und Security-Checks verfuegbar. Global installieren, nicht ins Repo vendorisieren.

Installationspfade:

- Codex: `git clone --single-branch --depth 1 https://github.com/garrytan/gstack.git ~/gstack && cd ~/gstack && ./setup --host codex`
- Claude Code: `git clone --single-branch --depth 1 https://github.com/garrytan/gstack.git ~/.claude/skills/gstack && cd ~/.claude/skills/gstack && ./setup --team`
- Windows: Git Bash oder WSL verwenden; `bun` und `node` muessen installiert sein
- Nach der Codex-Installation Codex neu starten, damit neue Skills geladen werden

Einsatzregeln:

- `/masterselects` bleibt erste Wahl fuer Timeline-, Preview-, Clip- und Debug-Bridge-Automation in der lokalen App
- `gstack-office-hours`, `gstack-autoplan`, `gstack-plan-eng-review`, `gstack-plan-design-review` und `gstack-plan-devex-review` fuer Discovery, Scope und Plan-Qualitaet in Codex
- `/office-hours`, `/autoplan`, `/plan-eng-review`, `/plan-design-review` und `/plan-devex-review` fuer denselben Workflow in Claude Code
- `gstack-review` oder `/review` fuer unabhaengige Code-Reviews
- `gstack-investigate` oder `/investigate` fuer Root-Cause-Debugging statt Trial-and-Error-Fixes
- `gstack-cso` oder `/cso` fuer Security-Reviews
- `gstack-qa`, `gstack-qa-only`, `gstack-browse`, `gstack-open-gstack-browser`, `gstack-setup-browser-cookies` oder die entsprechenden Slash-Commands fuer Browser-QA, Repros und auth-geschuetzte Flows
- `gstack-upgrade` oder `/gstack-upgrade` verwenden, statt eine Repo-lokale gstack-Kopie zu pflegen

---

## 1. Workflow (WICHTIG!)

### Branch-Regeln
Expand Down
37 changes: 24 additions & 13 deletions docs/Features/3D-Layers.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,12 @@ Legacy gaussian-avatar support still exists in code for migration and old projec
|---|---|---|
| Per-layer 3D toggle | Stable | Any normal video/image layer can be switched between 2D and 3D. |
| OBJ / glTF / GLB / FBX model import | Stable | Model clips are always 3D and render through Three.js. |
| Numbered GLB sequences | Stable | Numbered `.glb` frames import as a single 30fps model-sequence clip. |
| Primitive mesh clips | Stable | Cube, sphere, plane, cylinder, torus, cone, and 3D text are created from the Media Panel. |
| Scene camera clips | Stable | Timeline camera clips control the shared Three.js scene. |
| Gaussian splat clips | Stable but specialized | Default route is shared-scene Three.js; native WebGPU rendering is optional. |
| Splat effector clips | Stable but specialized | They deform Three.js splats live at playback time. |
| Numbered PLY / SPLAT sequences | Stable | Numbered `.ply` / `.splat` frames import as one shared-scene gaussian-splat sequence clip. |
| 3D effector clips | Stable but specialized | They influence shared-scene 3D layers live at playback time. |
| Gaussian avatar import | Legacy only | Import is blocked; existing projects may still expose blendshape editing. |
| Temporal / particle splat settings | Experimental | Wired in the engine/export path, but not yet exposed as a dedicated properties tab. |

Expand All @@ -34,7 +36,7 @@ Legacy gaussian-avatar support still exists in code for migration and old projec

Three.js is used as the shared 3D scene for classic 3D layers and for the default gaussian-splat route. The native gaussian-splat renderer is a separate WebGPU path, enabled by the clip-level `useNativeRenderer` setting.

Camera clips and splat effectors only affect the shared Three.js scene. They do not drive the native gaussian-splat renderer.
Camera clips and 3D effectors only affect the shared Three.js scene. They do not drive the native gaussian-splat renderer.

## Stable 3D Features

Expand All @@ -51,6 +53,8 @@ Camera clips and splat effectors only affect the shared Three.js scene. They do
- Models are auto-centered and normalized to fit the viewport.
- Default lighting is Ambient plus Directional lighting.
- The Transform panel exposes a wireframe debug toggle for model clips.
- Numbered `.glb` files like `frame000000.glb`, `frame000001.glb`, `frame000002.glb` are grouped into one model-sequence asset during import.
- GLB sequences currently default to 30fps and use frame-based playback through the existing model clip path.

### Primitive Meshes and 3D Text

Expand Down Expand Up @@ -90,11 +94,14 @@ The Transform tab also turns into camera-orbit controls for the active scene cam
Gaussian splat clips are imported from `.ply` and `.splat` files.

- Clips are created as `is3D: true`.
- The clip-level render tab exposes `useNativeRenderer`, `maxSplats`, `sortFrequency`, `splatScale`, `nearPlane`, and `farPlane`.
- The clip-level render tab exposes `useNativeRenderer`, `maxSplats`, `sortFrequency`, `splatScale`, `orientationPreset`, `nearPlane`, and `farPlane`.
- The default renderer is the shared Three.js scene path.
- Native WebGPU rendering is optional and off by default.
- The shared-scene route participates in scene cameras and splat effectors.
- Numbered `.ply` or `.splat` files like `scan000000.ply`, `scan000001.ply`, `scan000002.ply` are grouped into one gaussian-splat sequence asset during batch import.
- Gaussian-splat sequences currently stay on the shared Three.js scene path even if a clip was previously set to native render.
- The shared-scene route participates in scene cameras and 3D effectors.
- The native route uses its own camera-style navigation controls in the Transform tab.
- A per-clip `3D Effector` toggle in the Transform tab lets you opt shared-scene splat/model layers in or out.

Some gaussian-splat settings exist in the data model and export pipeline but are not yet surfaced as a full dedicated UI:

Expand All @@ -104,16 +111,20 @@ Some gaussian-splat settings exist in the data model and export pipeline but are

Those are wired through the renderer and export code, but they should still be treated as in-progress surface area.

## Splat Effectors
## 3D Effectors

Splat effector clips are timeline clips that only affect Three.js splats.
3D effector clips are non-rendering timeline clips that influence shared-scene 3D layers.

- Modes: `repel`, `attract`, `swirl`, and `noise`
- Controls: strength, falloff, speed, and seed
- Transform scale acts as the effector radius
- They do not render visible content on their own
- Shared-scene gaussian splats use the direct splat deformation path
- Shared-scene gaussian-splat sequences also use that direct splat deformation path frame by frame
- Models, primitive meshes, and 3D text receive object-level motion
- Native gaussian splats ignore 3D effectors because they do not run in the shared Three.js scene

This is a good example of a specialized 3D feature that is stable in the UI, but limited to the shared-scene splat path.
This is a shared-scene 3D feature that is stable in the UI, but it does not extend into the native gaussian-splat renderer.

## Legacy Gaussian Avatars

Expand All @@ -133,7 +144,7 @@ If you see avatar-specific code paths in the renderer or AI tooling, treat them
| Regular 2D clip | Transform, Effects, Masks, Transcript, Analysis |
| Camera clip | Transform, Camera |
| Gaussian splat clip | Transform, Gaussian, Effects, Masks, Transcript, Analysis |
| Splat effector clip | Transform, Effector, Effects, Masks, Transcript, Analysis |
| 3D effector clip | Transform, Effector, Effects, Masks, Transcript, Analysis |
| 3D text clip | 3D Text, Transform, Effects, Masks |
| Legacy gaussian avatar clip | Transform, Blendshapes |

Expand Down Expand Up @@ -164,11 +175,11 @@ The Transform tab is context-sensitive:
| `src/stores/timeline/meshClipSlice.ts` | Primitive mesh and 3D text clip creation |
| `src/stores/timeline/cameraClipSlice.ts` | Timeline camera clip creation |
| `src/stores/timeline/clip/addGaussianSplatClip.ts` | Gaussian splat clip creation |
| `src/stores/timeline/splatEffectorClipSlice.ts` | Splat effector clip creation |
| `src/stores/timeline/splatEffectorClipSlice.ts` | 3D effector clip creation |
| `src/components/panels/properties/TransformTab.tsx` | Context-sensitive 3D transform and camera controls |
| `src/components/panels/properties/GaussianSplatTab.tsx` | Gaussian splat render settings tab |
| `src/components/panels/properties/CameraTab.tsx` | Scene camera settings tab |
| `src/components/panels/properties/SplatEffectorTab.tsx` | Splat effector settings tab |
| `src/components/panels/properties/SplatEffectorTab.tsx` | 3D effector settings tab |
| `src/components/panels/properties/BlendshapesTab.tsx` | Legacy gaussian-avatar blendshapes tab |
| `src/engine/featureFlags.ts` | 3D feature flags |

Expand All @@ -178,10 +189,10 @@ The Transform tab is context-sensitive:
|---|---|---|
| `.obj` | Supported | Imported as a Three.js model clip. |
| `.gltf` | Supported | Imported as a Three.js model clip. |
| `.glb` | Supported | Imported as a Three.js model clip. |
| `.glb` | Supported | Imported as a Three.js model clip. Numbered `.glb` frames are grouped into a model-sequence clip. |
| `.fbx` | Supported | Imported as a Three.js model clip. |
| `.ply` | Supported | Gaussian splat import. |
| `.splat` | Supported | Gaussian splat import. |
| `.ply` | Supported | Gaussian splat import. Numbered `.ply` frames are grouped into a shared-scene sequence clip. |
| `.splat` | Supported | Gaussian splat import. Numbered `.splat` frames are grouped into a shared-scene sequence clip. |
| `.ksplat` | Not yet supported | Parser stubs exist, but the file is rejected today. |
| `.gsplat-zip` | Not yet supported | Parser stubs exist, but the file is rejected today. |
| Gaussian avatar `.zip` | Legacy only | Import is blocked in the current product surface. |
Expand Down
24 changes: 18 additions & 6 deletions docs/Features/AI-Integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,29 +29,40 @@ GPT-powered editing with 79 exported tools across 15 exported definition groups,

### Features
- Interactive chat interface
- Model selection dropdown
- Model selection badge with popover
- Shared hosted-credit pricing per model
- Conversation history
- Clear chat button
- Auto-scrolling
- Tool execution indicators
- Styled approval cards for mutating/sensitive tool calls
- Compact control pills below the prompt box
- Optional per-panel approval bypass toggle below the send button
- Thinking-effort selector for supported OpenAI chat models

### Available Models
```
GPT-5.2, GPT-5.2 Pro
GPT-5.1, GPT-5.1 Codex, GPT-5.1 Codex Mini
GPT-5.4, GPT-5.4 Mini, GPT-5.4 Nano
GPT-5.3 Chat, GPT-5.3 Codex
GPT-5.2, GPT-5.2 Codex
GPT-5.1, GPT-5.1 Codex Mini
GPT-5, GPT-5 Mini, GPT-5 Nano
GPT-4.1, GPT-4.1 Mini, GPT-4.1 Nano
GPT-4o, GPT-4o Mini
o3, o4-mini, o3-pro (reasoning)
o3, o4-mini, o3-mini
```

Default model: `gpt-5.1`

The dropdown is sourced from `src/shared/openAiModelCatalog.ts`. That same catalog also drives hosted chat billing in `functions/lib/modelPricing.ts`, so the UI label and the charged credits stay in sync.
The same catalog also marks which models support OpenAI reasoning effort controls, so the chat panel only shows the Thinking control when the selected model can use it.

### Editor Mode
When enabled:
The AI chat panel always runs with editor tools enabled:
- Includes timeline context in prompts
- Uses the exported AI tool catalog from `src/services/aiTools/definitions`
- The chat UI applies its own approval gate before calling mutating or sensitive tools
- Users can temporarily bypass those confirmations from the panel
- AI can manipulate timeline directly

The current tool surface is 79 exported tool definitions across 15 exported definition groups. Two dispatch gaps remain in the shared registry:
Expand Down Expand Up @@ -92,6 +103,7 @@ That console surface is dev-only. The Vite dev bridge and the Native Helper HTTP
- Active IN / OUT / REF assignments appear as removable color badges around the prompt box
- Hovering a prompt-box badge strongly emphasizes the linked board node for as long as the badge is hovered
- Nano Banana 2 accepts up to 14 ordered reference images; the composer labels them as `REF 1`, `REF 2`, ... so prompts can refer to them explicitly
- FlashBoard's Kie.ai catalog includes Kling 3.0, Seedance 2.0, and Nano Banana 2; Seedance 2.0 currently ships as a Board-only Kie option with text-to-video and image-to-video support
- IN / OUT / REF outlines scale with zoom and use a stronger glow so references stay readable while navigating the board

### Current Backends
Expand All @@ -100,7 +112,7 @@ The current AI Video stack is no longer best described as "PiAPI as one unified

| Backend | Where it is used | Notes |
|---------|------------------|-------|
| `Kie.ai` | Classic generator and FlashBoard | Current provider list comes from `getKieAiProviders()`; user-supplied key in Settings |
| `Kie.ai` | Classic generator and FlashBoard | Classic provider list comes from `getKieAiProviders()`; FlashBoard extends the Kie catalog with Seedance 2.0; user-supplied key in Settings |
| `MasterSelects Cloud` | Classic generator and FlashBoard when hosted access is available | Hosted credits/account flow; board mode resolves to hosted Kling when no local Kie key is present |
| `PiAPI` | Legacy compatibility and some catalog/pricing metadata | Still present in older history/key migration paths and FlashBoard pricing/catalog helpers, but not the primary runtime path the current panel describes |

Expand Down
5 changes: 4 additions & 1 deletion docs/Features/FlashBoard.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ FlashBoard is the AI canvas workspace behind the AI Video panel's Board mode. It
FlashBoard is not a separate model backend. It is a workspace layer on top of the existing AI services:

- `piapi` for the PiAPI catalog
- `kieai` for Kie.ai Kling 3.0 and Nano Banana 2
- `kieai` for Kie.ai Kling 3.0, Seedance 2.0, and Nano Banana 2
- `cloud` for hosted Kling 3.0 and hosted Nano Banana 2

The AI Video panel switches into FlashBoard when the user selects Board mode. If the user has no Kie.ai key and is signed in to MasterSelects Cloud, the board uses the hosted cloud service scope. Otherwise it stays on Kie.ai.
Expand Down Expand Up @@ -68,10 +68,13 @@ The board uses the shared catalog from `FlashBoardModelCatalog`:

- PiAPI video providers from the shared PiAPI catalog
- Kie.ai Kling 3.0 video
- Kie.ai Seedance 2.0 video
- Kie.ai Nano Banana 2 image generation
- Cloud Kling 3.0 video
- Cloud Nano Banana 2 image generation

Seedance 2.0 is currently exposed only in FlashBoard on the Kie.ai path. The existing Classic AI Video flow remains narrower and continues to focus on the classic Kie provider list.

The classic AI Video flow is narrower: it currently exposes only the Kie.ai Kling 3.0 provider list, while FlashBoard exposes the richer catalog.

---
Expand Down
18 changes: 12 additions & 6 deletions docs/Features/Media-Panel.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ The panel also accepts a few specialized asset types that flow into the timeline
- `model` files: OBJ, glTF/GLB, FBX
- `gaussian-splat` files: PLY, SPLAT

Numbered `.glb` files with a shared prefix are detected as a sequence during batch import. A folder containing `frame000000.glb`, `frame000001.glb`, `frame000002.glb` and so on will appear as one model asset instead of many separate files.
Numbered `.ply` or `.splat` files with a shared prefix are also detected as a sequence during batch import. A folder containing `scan000000.ply`, `scan000001.ply`, `scan000002.ply` and so on will appear as one gaussian-splat asset instead of many separate files.

Lottie imports are treated as first-class media items. `.json` files are only accepted when their contents actually match Lottie structure, so arbitrary JSON data is not misclassified as animation.

### Import Methods
Expand All @@ -54,7 +57,7 @@ Click the **+ Add** button for creating new items:
- **3D Text** - New 3D text mesh item
- **Solid** - New solid color item (placed in auto-created "Solids" folder)
- **Camera** - New camera item
- **Splat Effector** - New splat-effector item
- **3D Effector** - New shared-scene 3D effector item
- **Mesh** ▶ - Submenu with 3D primitive meshes (placed in auto-created "Meshes" folder):
- Cube, Sphere, Plane, Cylinder, Torus, Cone
- Creates a `MeshItem` which can be dragged to the timeline as a 3D clip
Expand All @@ -75,11 +78,13 @@ Imports use a two-phase approach:

1. **Phase 1 (instant):** A placeholder entry appears immediately in the panel with `isImporting: true`, showing file name and size
2. **Phase 2 (background):** Full processing runs in the background:
- Media info extraction (dimensions, duration, FPS, codec, bitrate, audio detection)
- Thumbnail generation (for video and image files)
- File hash calculation (for deduplication and proxy matching)
- Copy to project RAW folder when `copyMediaToProject` is enabled, or when the import is forced
- Existing proxy detection (by file hash)
- Media info extraction (dimensions, duration, FPS, codec, bitrate, audio detection)
- Thumbnail generation (for video and image files)
- File hash calculation (for deduplication and proxy matching)
- Copy to project RAW folder when `copyMediaToProject` is enabled, or when the import is forced
- Numbered GLB sequence grouping into a single 30fps model-sequence asset
- Numbered PLY/SPLAT sequence grouping into a single 30fps gaussian-splat sequence asset
- Existing proxy detection (by file hash)

**Deduplication:** Files with matching name + size are automatically skipped.

Expand Down Expand Up @@ -448,6 +453,7 @@ interface MediaFile {
- Uses actual media duration
- Audio-only files restricted to audio tracks
- Files still importing or missing cannot be dragged to timeline
- Numbered gaussian-splat sequences always stay on the shared Three.js 3D renderer path after drop
- Compositions cannot be dragged into themselves (active comp check)
- Mesh items create 3D clips with `is3D: true` and `meshType` (rendered via Three.js)

Expand Down
Loading
Loading