Controller audio: mic + speaker over the Satellite protocol

.github/actions/setup-build-deps/action.yml

@@ -34,7 +34,7 @@ runs:
         # shellcheck disable=SC2086
         sudo apt-get install -y --no-install-recommends \
           build-essential cmake ninja-build pkg-config ccache \
-          libsodium-dev libsdl2-dev libssl-dev \
+          libsodium-dev libsdl2-dev libssl-dev libopus-dev \
           catch2 clang-tidy ${EXTRA_PACKAGES}
 
     - name: Restore compiler cache

.github/workflows/codeql.yml

@@ -52,7 +52,7 @@ jobs:
           sudo apt-get update
           sudo apt-get install -y --no-install-recommends \
             build-essential cmake ninja-build pkg-config \
-            libsodium-dev libsdl2-dev libssl-dev
+            libsodium-dev libsdl2-dev libssl-dev libopus-dev
 
       # Not apt: noble ships Qt 6.4.2 and this project's floor is 6.7. The
       # action pulls the official binaries and exports CMAKE_PREFIX_PATH.

.github/workflows/linux-ci.yml

@@ -260,7 +260,7 @@ jobs:
             build-essential cmake ninja-build pkg-config \
             qt6-base-dev qt6-base-dev-tools qt6-declarative-dev qt6-svg-dev \
             qt6-tools-dev qt6-tools-dev-tools qt6-l10n-tools \
-            libsodium-dev libsdl2-dev libssl-dev libdbus-1-dev \
+            libsodium-dev libsdl2-dev libssl-dev libopus-dev libdbus-1-dev \
             dpkg-dev fakeroot file gzip \
             librsvg2-bin desktop-file-utils appstream lintian
 

.github/workflows/release.yml

@@ -164,7 +164,7 @@ jobs:
             build-essential cmake ninja-build pkg-config \
             qt6-base-dev qt6-base-dev-tools qt6-declarative-dev qt6-svg-dev \
             qt6-tools-dev qt6-tools-dev-tools qt6-l10n-tools \
-            libsodium-dev libsdl2-dev libssl-dev libdbus-1-dev \
+            libsodium-dev libsdl2-dev libssl-dev libopus-dev libdbus-1-dev \
             dpkg-dev fakeroot file gzip \
             librsvg2-bin desktop-file-utils appstream lintian
 
@@ -323,7 +323,7 @@ jobs:
             git-core ca-certificates \
             gcc-c++ cmake ninja-build pkgconf-pkg-config rpm-build \
             qt6-qtbase-devel qt6-qtdeclarative-devel qt6-qtsvg-devel qt6-qttools-devel \
-            libsodium-devel SDL2-devel openssl-devel dbus-devel \
+            libsodium-devel SDL2-devel openssl-devel opus-devel dbus-devel \
             librsvg2-tools desktop-file-utils libappstream-glib gzip
 
       - name: Checkout
@@ -466,7 +466,7 @@ jobs:
           # resolve on the build host.
           sudo apt-get install -y --no-install-recommends \
             build-essential cmake ninja-build pkg-config \
-            libsodium-dev libssl-dev libudev-dev libusb-1.0-0-dev \
+            libsodium-dev libssl-dev libopus-dev libpulse-dev libudev-dev libusb-1.0-0-dev \
             libxkbcommon-x11-0 libxcb-cursor0 libxcb-icccm4 libxcb-image0 \
             libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-shape0 \
             libxcb-xinerama0 libxcb-xkb1 \
@@ -480,8 +480,12 @@ jobs:
 
       # Not apt for the same reason as Qt: jammy ships SDL 2.0.20 and the code
       # uses SDL_GameControllerPath, which needs 2.24. Same version, checksum
-      # and gamepad-only feature set as the Flatpak manifest's sdl2 module;
-      # linuxdeploy bundles the resulting library into the AppImage. Runs
+      # and feature set as the Flatpak manifest's sdl2 module: gamepads plus
+      # the audio subsystem controller audio opens the pads' own endpoints
+      # through (libpulse-dev above is what lets its pulse backend configure;
+      # SDL dlopen's the host's libpulse at run time, so nothing extra is
+      # bundled). linuxdeploy bundles the resulting library into the AppImage.
+      # Runs
       # after setup-qt because that action overwrites CMAKE_PREFIX_PATH and
       # this step prepends to it.
       - name: Build SDL2 (jammy's predates SDL_GameControllerPath)
@@ -498,7 +502,7 @@ jobs:
             -DCMAKE_BUILD_TYPE=Release \
             -DCMAKE_INSTALL_PREFIX="${RUNNER_TEMP}/sdl-prefix" \
             -DSDL_STATIC=OFF -DSDL_SHARED=ON -DSDL_TEST=OFF \
-            -DSDL_RENDER=OFF -DSDL_AUDIO=OFF \
+            -DSDL_RENDER=OFF -DSDL_AUDIO=ON \
             -DSDL_HIDAPI=ON -DSDL_JOYSTICK=ON -DSDL_HAPTIC=ON
           cmake --build "${RUNNER_TEMP}/sdl-build" --parallel
           cmake --install "${RUNNER_TEMP}/sdl-build"

CHANGELOG.md

@@ -20,6 +20,64 @@ the repos share a version number.
 
 ### Added
 
+- **Controller audio, wave 1: wire + capability model** `[wire-coordinated]`
+  (satellite's `MSG_MIC_AUDIO`/`MSG_SPEAKER_AUDIO`/`MSG_MIC_LED`; dish-android
+  shipped the client reference; dish-windows companion). This lands the
+  protocol-2 audio extension's plumbing without yet turning any audio on:
+  - the wire: `MSG_MIC_AUDIO` (0x0012) send path, `MSG_SPEAKER_AUDIO` (0x0013)
+    and `MSG_MIC_LED` (0x0014) dispatch, the `mic`/`speaker` descriptor caps,
+    and the datagram ceilings (1500-byte receive buffer, 1472-byte inner
+    payload guard) a full-size audio frame needs;
+  - the pure cores: the 2-frame reorder window (`core/audio/AudioJitter.h`,
+    the third mirror of satellite's — edit together) and the pinned Opus
+    formats (mic mono VOIP 32 kbps DTX, speaker stereo AUDIO 96 kbps, both
+    VBR + in-band FEC) behind `core/audio/AudioCodec.h`, libopus-backed in
+    `source/audio/OpusAudioCodec.*` (new system dependency: `libopus`, found
+    through `pkg-config` like SDL2 and libsodium);
+  - the host verdict: `GET /api/server/capabilities` is now probed after every
+    session PUT for the `controllerAudio` block (per-backend `audio` fallback),
+    so the capability table's mic/speaker host layer reflects what the host
+    will actually carry — conservative "no audio" until a probe says yes;
+  - the model and UI: Microphone and Controller sound rows in the capability
+    matrix and per-binding toggles (mic defaults OFF for privacy, speaker ON),
+    persisted like the motion toggle; `wButtons` bit 0x0800 reserved as the
+    DualSense mic-mute state.
+- **Controller audio, wave 2: the audio itself** `[wire-coordinated]`. A
+  Direct-claimed DualSense (or DS4 v2) now carries real audio end to end:
+  - pad-to-endpoint routing: the claimed pad's USB product string (iProduct,
+    read from sysfs — the same string pipewire and pulse embed in the pad's
+    audio device names) is matched against the endpoints SDL enumerates, with
+    every ambiguity resolving to "no route" (two pads sharing a name,
+    duplicate endpoint names, a name containing two pads' strings) — routes
+    re-resolve on claim changes and audio hotplug, and a change re-declares
+    the slot's descriptor;
+  - the capture engine: the pad's own headset mic, windowed to exact 20 ms
+    frames, Opus-encoded and sent as `MSG_MIC_AUDIO`, one seq per window
+    including failed encodes. THE PRIVACY INVARIANT: muted, toggled off,
+    unrouted, unstreaming or unwelcome at the host means the capture device is
+    CLOSED and zero packets leave — never silence in their place;
+  - the playout engine: `MSG_SPEAKER_AUDIO` through the reorder window and
+    Opus FEC/PLC to the pad's own speaker endpoint, with a two-frame start
+    cushion rebuilt as silence after the satellite's suppressed-silence
+    stretches;
+  - the DualSense mute button: decoded as an edge onto a latch that folds the
+    mute STATE into `wButtons` (0x0800) on the read thread, mirrored to the
+    app's mute state, stripped from Moonlight's button words; the slot card
+    and Configure binding show the local truth with a click-to-toggle control,
+    and the pad's mute lamp answers locally at once (a later host `MSG_MIC_LED`
+    repaints it — last writer wins on the pad);
+  - `MSG_MIC_LED` actuation: the DS5 lamp + mic-amp power-save bit, shadowed in
+    the per-claim feedback state so rumble/lightbar/player-LED/trigger writes
+    re-assert it instead of stomping it;
+  - `SDL_INIT_AUDIO` is owned by the audio gateway, not the gamepad bridge, so
+    gamepad re-inits never take a live stream down; the bridge's event loop
+    forwards audio hotplug. The Flatpak gains the `pulseaudio` socket (served
+    by `pipewire-pulse` on PipeWire desktops), and the AppImage's bundled SDL
+    is now built with its audio subsystem on.
+  Mute is deliberately session-scoped (not persisted): it clears when the pad
+  leaves, the way the hardware's own mute does; the durable off-switch is the
+  per-binding Microphone toggle, which still defaults OFF.
+
 - **Protocol 2** `[wire-coordinated]` (satellite #86, #87; dish-android #174,
   #175; dish-windows companion). The version is now negotiated rather than
   assumed: the client offers 2, the satellite settles the session on that

CMakeLists.txt

@@ -137,6 +137,9 @@ find_package(Qt6 6.7 QUIET OPTIONAL_COMPONENTS LinguistTools)
 find_package(PkgConfig REQUIRED)
 pkg_check_modules(SODIUM REQUIRED IMPORTED_TARGET libsodium)
 pkg_check_modules(SDL2   REQUIRED IMPORTED_TARGET sdl2)
+# Controller audio (source/audio/OpusAudioCodec): the reference Opus codec,
+# found the same pkg-config way as the two above — every distro ships opus.pc.
+pkg_check_modules(OPUS   REQUIRED IMPORTED_TARGET opus)
 # OpenSSL's libcrypto backs the Moonlight-host pairing (AES-128, RSA, self-
 # signed X.509) and the control-stream AES-GCM, none of which libsodium offers.
 # Only libcrypto is used; the TLS client itself is Qt Network's.
@@ -222,6 +225,24 @@ set(DISH_CORE_SOURCES
     src/core/reducer/LatencyWindow.h
     src/core/reducer/CapabilitySolver.h
     src/core/reducer/CapabilitySolver.cpp
+    src/core/reducer/HostAudioVerdict.h
+    # Controller audio: the reorder window, the pure codec seam, the
+    # pad-to-endpoint matcher and the engine rules in core; the libopus link,
+    # the SDL device gateway and the engines at the source edge with the other
+    # third-party-facing code.
+    src/core/audio/AudioJitter.h
+    src/core/audio/AudioCodec.h
+    src/core/audio/PadAudioMatcher.h
+    src/core/audio/AudioEnginePolicy.h
+    src/source/audio/OpusAudioCodec.h
+    src/source/audio/OpusAudioCodec.cpp
+    src/source/audio/AudioDeviceGateway.h
+    src/source/audio/SdlAudioGateway.h
+    src/source/audio/SdlAudioGateway.cpp
+    src/source/audio/MicCaptureEngine.h
+    src/source/audio/MicCaptureEngine.cpp
+    src/source/audio/SpeakerPlayoutEngine.h
+    src/source/audio/SpeakerPlayoutEngine.cpp
     src/core/reducer/ApplyBindingMachine.h
     src/core/reducer/CatalogFeatureGate.h
     src/core/reducer/CatalogPrewarm.h
@@ -308,6 +329,10 @@ set(DISH_CORE_SOURCES
     src/source/store/ControllerTypeStore.cpp
     src/source/store/MotionEnabledStore.h
     src/source/store/MotionEnabledStore.cpp
+    src/source/store/AudioEnabledStore.h
+    src/source/store/AudioEnabledStore.cpp
+    src/source/store/MicMuteStore.h
+    src/source/store/MicMuteStore.cpp
     src/source/store/SatelliteMotionBackendStatusStore.h
     src/source/store/SatelliteMotionBackendStatusStore.cpp
     src/source/store/UsbPathPreferenceStore.h
@@ -388,6 +413,8 @@ set(DISH_CORE_SOURCES
     src/repository/DeadzoneRepository.cpp
     src/repository/MotionPreferenceRepository.h
     src/repository/MotionPreferenceRepository.cpp
+    src/repository/AudioPreferenceRepository.h
+    src/repository/AudioPreferenceRepository.cpp
     src/repository/TouchpadModeRepository.h
     src/composer/ConnectionsComposer.h
     src/composer/ConnectionsComposer.cpp
@@ -465,6 +492,8 @@ target_link_libraries(dish_core
         Qt6::Concurrent
         PkgConfig::SODIUM
         PkgConfig::SDL2
+        # Controller audio (see the pkg_check_modules comment above).
+        PkgConfig::OPUS
         Threads::Threads
     PRIVATE
         dish_warnings

README.md

@@ -103,7 +103,7 @@ it in place. What the check sends is spelled out in [`PRIVACY.md`](PRIVACY.md).
 - GCC 12+ or Clang 15+, CMake 3.21+, Ninja
 - Qt 6.7+ (Core, Gui, Network, DBus, Svg, Quick, Qml, QuickControls2; Linguist
   tools for the translation catalogues)
-- libsodium, SDL2, OpenSSL (libcrypto), Catch2 v3
+- libsodium, SDL2, Opus, OpenSSL (libcrypto), Catch2 v3
 - Optional: `rsvg-convert`, which renders the rest of the launcher-icon ladder
   from the SVG. Without it the build says so and installs only the scalable and
   512x512 icons, which is enough for a working menu entry.
@@ -114,7 +114,7 @@ On Debian and Ubuntu:
 sudo apt install build-essential cmake ninja-build pkg-config \
   qt6-base-dev qt6-base-dev-tools qt6-declarative-dev qt6-svg-dev \
   qt6-tools-dev qt6-l10n-tools \
-  libsodium-dev libsdl2-dev libssl-dev libdbus-1-dev catch2 \
+  libsodium-dev libsdl2-dev libssl-dev libopus-dev libdbus-1-dev catch2 \
   librsvg2-bin
 ```
 

THIRD_PARTY.md

@@ -5,10 +5,10 @@ third-party component that is linked into `dish`, embedded in it as a resource,
 or used only to build and test it. It also states what someone redistributing
 the binary has to do.
 
-Nothing is bundled. Qt, SDL2 and libsodium are dynamically linked against the
-system copies your distribution provides, so a redistributor of a package built
-from this tree is redistributing those libraries under their own distribution's
-terms, not through this repository.
+Nothing is bundled. Qt, SDL2, libsodium and Opus are dynamically linked against
+the system copies your distribution provides, so a redistributor of a package
+built from this tree is redistributing those libraries under their own
+distribution's terms, not through this repository.
 
 The app has an in-app version of this list at Settings, Licenses, rendered from
 [`assets/licenses/licenses.json`](assets/licenses/licenses.json). That manifest
@@ -25,6 +25,7 @@ today.
 | [Qt 6](#2-qt-6) | CMake requires >= 6.7 | `LGPL-3.0-only` | Dynamically linked against the system Qt. Nothing is bundled. | Notice, license text, relink freedom. See section 2. |
 | [SDL2](#sdl2) | >= 2.0.18 | `Zlib` | Dynamically linked against the system SDL2. | Keep the notice, do not claim authorship |
 | [libsodium](#libsodium) | >= 1.0.18 | `ISC` | Dynamically linked against the system libsodium. | Keep the copyright and permission notice |
+| [Opus](#opus) | >= 1.3 | `BSD-3-Clause` | Dynamically linked against the system libopus. | Keep the copyright notice and disclaimer |
 | [Inter](#4-inter) | 4.001 | `OFL-1.1` | Four `.ttf` faces embedded in `dish` as Qt resources under `:/fonts/`. | Ship the license text with every copy. See section 4. |
 | [Catch2](#5-catch2) | 3.x | `BSL-1.0` | Test binary only. Not linked into `dish`. | None for redistributors of the app |
 | [ENet (cgutman fork)](#9-enet) | commit `4cde9cc` | `MIT` | Vendored C sources under `third_party/enet/`, compiled into `dish`. | Ship the copyright + permission notice |
@@ -130,9 +131,9 @@ obligations on redistribution.
 
 ## 3. Linked system libraries
 
-SDL2 and libsodium are found through `pkg-config` and dynamically linked against
-whatever the build host provides. Neither is vendored into this tree and neither
-is bundled with the binary.
+SDL2, libsodium and Opus are found through `pkg-config` and dynamically linked
+against whatever the build host provides. None of them is vendored into this
+tree and none is bundled with the binary.
 
 ### SDL2
 
@@ -190,6 +191,50 @@ ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 ```
 
+### Opus
+
+Opus audio codec (libopus). SPDX `BSD-3-Clause`. Upstream:
+<https://opus-codec.org/>.
+
+Used for the controller-audio streams only: encoding a claimed pad's
+microphone capture and decoding host audio for the pad's own speaker
+(`src/source/audio/OpusAudioCodec.*`).
+
+```
+Copyright 2001-2023 Xiph.Org, Skype Limited, Octasic,
+                    Jean-Marc Valin, Timothy B. Terriberry,
+                    CSIRO, Gregory Maxwell, Mark Borgerding,
+                    Erik de Castro Lopo, Mozilla, Amazon
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+- Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+
+- Redistributions in binary form must reproduce the above copyright
+notice, this list of conditions and the following disclaimer in the
+documentation and/or other materials provided with the distribution.
+
+- Neither the name of Internet Society, IETF or IETF Trust, nor the
+names of specific contributors, may be used to endorse or promote
+products derived from this software without specific prior written
+permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+```
+
 
 ## 4. Inter
 
@@ -354,8 +399,8 @@ Apollo. Wolf's documentation and MIT source were sufficient.
 [`assets/licenses/licenses.json`](assets/licenses/licenses.json) is the manifest
 the in-app Licenses screen renders, parsed by `src/UI/licenses/LicenseManifest.*`.
 It is hand-authored, not generated, so it can drift. It currently lists Qt 6,
-SDL2, libsodium, Catch2 and Inter, which is the same set as this file, with the
-same licenses. One difference is worth knowing about:
+SDL2, libsodium, Opus, Catch2 and Inter, which is the same set as this file,
+with the same licenses. One difference is worth knowing about:
 
 - The manifest lists Catch2, which is test-only and is not in the shipped
   binary. Showing it to a user is harmless but inaccurate.

assets/licenses/licenses.json

@@ -40,6 +40,19 @@
         }
       ]
     },
+    {
+      "group": "org.opus-codec",
+      "artifact": "opus",
+      "version": "1.3+",
+      "name": "Opus",
+      "url": "https://opus-codec.org/",
+      "licenses": [
+        {
+          "name": "BSD 3-Clause License",
+          "url": "https://opensource.org/license/bsd-3-clause"
+        }
+      ]
+    },
     {
       "group": "org.catchorg",
       "artifact": "Catch2",