Feature: Multiview / Picture-in-Picture Support

## Problem

The integration currently does not support **SDVoE Multiview mode**, which enables advanced picture-in-picture (PiP), picture-and-picture (PaP), quad-split, and custom multi-source layouts on a single display. This is a key SDVoE feature for professional AV installations.

## Current State

**Current Functionality:**
- ✅ Single source routing to single display
- ✅ Stream index 0 only (hardcoded)
- ❌ No multiview layout support
- ❌ No picture-in-picture configuration
- ❌ No multi-source single-display capability

## SDVoE Multiview Capabilities

According to the SDVoE API documentation, multiview mode provides:

### Supported Multiview Layouts
- **Picture-in-Picture (PiP)** - Small window over full-screen background
- **Picture-and-Picture (PaP)** - Multiple equal-sized windows side-by-side
- **Quad Split** - 2x2 grid of four sources
- **Matrix Grid** - NxM grid of multiple sources
- **L-Shape** - Asymmetric layouts
- **Custom Layouts** - Fully customizable window positions and sizes

## How Multiview Works

Multiview mode uses a different approach than standard routing:

### 1. Layout Creation
Define a multiview layout with:
- **Name:** Unique identifier for the layout
- **Output Resolution:** Overall canvas size (e.g., 3840x2160)
- **Surfaces:** Virtual frame buffers for each source
- **Windows:** Visual tiles showing each surface

### 2. Window Configuration
Each window has:
- **Position:** X,Y coordinates on canvas
- **Size:** Width and height in pixels
- **Index/Z-order:** Layering (which appears on top)
- **Target Surface:** Which source stream to display
- **Offsets:** Crop/pan within source

### 3. Stream Configuration
Transmitters (sources) need special setup:
- **Scaled Streams:** Use stream index 1 (not 0!)
- **Scaler Resolution:** Set output resolution for the stream
- **Multiple Indexes:** Each source uses different stream index

### 4. Receiver Configuration
Apply the multiview layout:
- **Set Multiview Mode:** Load the named layout
- **Subscribe to Streams:** Each surface subscribes to a source
- **Video Format:** Set overall output format

## Example: Picture-in-Picture Configuration

```python
# From SDVoE API documentation

# Step 1: Stop existing streams and set scaler for large window
# Transmitter 1 (background) - stream index 1 at full resolution
{
  "op": "stop",
  "device": "d88039e83991",
  "stream_type": "HDMI",
  "stream_index": 0,
  "free": True
}
{
  "op": "set:scaler",
  "device": "d88039e83991",
  "width": 3840,
  "height": 2160
}
{
  "op": "start",
  "device": "d88039e83991",
  "stream_type": "HDMI",
  "stream_index": 1  # Note: index 1, not 0!
}

# Step 2: Configure transmitter 2 (PiP window) - stream index 1 at smaller resolution
{
  "op": "set:scaler",
  "device": "d88039e55d9a",
  "width": 960,
  "height": 540
}
{
  "op": "start",
  "device": "d88039e55d9a",
  "stream_type": "HDMI",
  "stream_index": 1
}

# Step 3: Create layout
{
  "op": "create",
  "layout": "PiP_Sample",
  "width": 3840,
  "height": 2160
}

# Step 4: Create windows
# Background window (index 1 = behind)
{
  "op": "create",
  "layout": "PiP_Sample",
  "window": 1,
  "horizontal_position": 0,
  "vertical_position": 0,
  "width": 3840,
  "height": 2160,
  "target_surface": 0
}

# PiP window (index 0 = on top)
{
  "op": "create",
  "layout": "PiP_Sample",
  "window": 0,
  "horizontal_position": 400,
  "vertical_position": 400,
  "width": 960,
  "height": 540,
  "target_surface": 1
}

# Step 5: Set receiver to multiview mode
{
  "op": "set:multiview",
  "device": "0016c04c46d7",
  "layout": "PiP_Sample",
  "video": {
    "width": 3840,
    "height": 2160,
    "fps": 30
  },
  "subscriptions": [
    {
      "source_device": "d88039e83991",
      "stream_index": 1,
      "subscription_index": 0
    },
    {
      "source_device": "d88039e55d9a",
      "stream_index": 1,
      "subscription_index": 1
    }
  ]
}
```

## Key Technical Details

### Stream Index Requirements
- **Standard Mode:** Uses stream index 0
- **Multiview Mode:** Uses stream index 1+
- **Current Integration:** Hardcoded to index 0 (blocks multiview!)
- **Related Issue:** This is why issue #TBD (hardcoded stream index) must be fixed first

### Scaler Configuration
- Transmitters must output scaled streams
- Each transmitter can have different resolution
- Scaler settings stored in device configuration
- Must match window sizes for best quality

### Layout Management
- Layouts stored on API server
- Can be created, modified, deleted
- Reusable across different receivers
- Persistent across reboots

## Proposed Implementation

### Phase 1: Foundation (Requires fixing hardcoded stream index issue)
- [ ] Remove hardcoded stream index 0 limitation
- [ ] Support multiple stream indexes per device
- [ ] Add stream index selection in UI
- [ ] Update coordinator to parse stream indexes
- [ ] Update API client for multi-index operations

### Phase 2: Basic Multiview Support
- [ ] Add multiview display mode constant
- [ ] Create layout management API methods
  - `async_create_layout(name, width, height)`
  - `async_create_window(layout, index, position, size, surface)`
  - `async_delete_layout(name)`
  - `async_list_layouts()`
- [ ] Add scaler configuration API methods
  - `async_set_scaler(device_id, width, height)`
- [ ] Add multiview mode setting
  - `async_set_multiview(device_id, layout, video, subscriptions)`

### Phase 3: UI Implementation
- [ ] Multiview layout builder UI
- [ ] Visual layout designer (drag-and-drop)
- [ ] Preset layouts (PiP, PaP, Quad, etc.)
- [ ] Window size/position configuration
- [ ] Source assignment per window
- [ ] Preview/test functionality

### Phase 4: Advanced Features
- [ ] Layout templates library
- [ ] Dynamic layout switching
- [ ] Layout presets per room/use case
- [ ] Save/load custom layouts
- [ ] Layout sharing between receivers

## Use Cases

### Conference Room with Confidence Monitor
```yaml
# Main display shows presentation (large window)
# Small PiP window shows presenter camera
# Switched via automation or manual control
Layout: PiP_Presenter
- Background: 1920x1080 (presentation)
- PiP: 480x270 at bottom-right (camera)
```

### Video Production Monitoring
```yaml
# Quad-split showing 4 camera feeds
# Equal-sized windows in 2x2 grid
# Director can monitor all sources simultaneously
Layout: Quad_Cameras
- Window 1: Camera 1 (top-left)
- Window 2: Camera 2 (top-right)
- Window 3: Camera 3 (bottom-left)
- Window 4: Camera 4 (bottom-right)
```

### Training Room with Instructor Feed
```yaml
# Picture-and-picture with instructor and slides
# Side-by-side equal display
Layout: PaP_Training
- Left: Instructor camera (960x1080)
- Right: Presentation slides (960x1080)
```

### Control Room Matrix View
```yaml
# 3x3 grid showing 9 surveillance feeds
# Compact overview of all sources
Layout: Matrix_3x3
- 9 windows, each 640x360
- Grid arrangement
```

## API Commands Required

### Layout Management
```
POST /api/multiview/layout/{name}
GET /api/multiview/layout
GET /api/multiview/layout/{name}
DELETE /api/multiview/layout/{name}
```

### Window Management
```
POST /api/multiview/layout/{name}/window/{index}
DELETE /api/multiview/layout/{name}/window/{index}
```

### Device Configuration
```
POST /api/device/{id} {"op": "set:scaler", "width": 1920, "height": 1080}
POST /api/device/{id} {"op": "set:multiview", "layout": "name", ...}
```

## Technical Considerations

### Device Requirements
- **Requires AVP:** Advanced Video Processing Engine
- **Not All Devices:** Only receivers with multiview capability
- **Capability Detection:** Must check device features
- **Version Requirements:** API 3.0+ for multiview

### Performance Implications
- **Bandwidth:** Multiple streams to single receiver
- **Processing:** Real-time scaling and composition
- **Latency:** Slight increase vs single-source mode
- **Stream Management:** Coordinate multiple transmitters

### Coordinator Updates
- Parse FRAME_BUFFER node for multiview config
- Track layout names and configurations
- Expose window positions and assignments
- Monitor multiview-specific status

## Benefits

- **Professional Features:** Match commercial control systems
- **Monitoring:** Multiple sources on single display
- **Presentation:** Dynamic layouts for meetings
- **Production:** Video monitoring and switching
- **Flexibility:** Custom layouts for any use case
- **Space Efficient:** Less displays needed

## Priority

🟡 **Medium**

## Effort & Complexity

- **Effort:** Very High (complex, multi-phase)
- **Complexity:** Very High (requires stream index fix + new APIs + UI)
- **Dependencies:** 
  - **Must fix hardcoded stream index first** (Issue #TBD)
  - Requires hardware with multiview support
  - Complex UI development

## References

- SDVoE API Documentation: Section 5 Multiview Command Reference
- SDVoE API Documentation: Section 5.4 Multiview Examples
- SDVoE API Documentation: Section 5.4.1 Picture-in-Picture Example
- SDVoE API Documentation: Section 6.8.8 Frame Buffer Node (multiview_layout)
- Related Issue: #TBD (Hardcoded stream index) - **MUST BE FIXED FIRST**
- Related Issue: #10 (Video Wall Support)

Feature: Multiview / Picture-in-Picture Support #11

Description

Problem

Current State

SDVoE Multiview Capabilities

Supported Multiview Layouts

How Multiview Works

1. Layout Creation

2. Window Configuration

3. Stream Configuration

4. Receiver Configuration

Example: Picture-in-Picture Configuration

Key Technical Details

Stream Index Requirements

Scaler Configuration

Layout Management

Proposed Implementation

Phase 1: Foundation (Requires fixing hardcoded stream index issue)

Phase 2: Basic Multiview Support

Phase 3: UI Implementation

Phase 4: Advanced Features

Use Cases

Conference Room with Confidence Monitor

Video Production Monitoring

Training Room with Instructor Feed

Control Room Matrix View

API Commands Required

Layout Management

Window Management

Device Configuration

Technical Considerations

Device Requirements

Performance Implications

Coordinator Updates

Benefits

Priority

Effort & Complexity

References

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions