diff --git a/apps/light/README.en.md b/apps/light/README.en.md index d9307ceb..089e3488 100644 --- a/apps/light/README.en.md +++ b/apps/light/README.en.md @@ -82,6 +82,8 @@ Enter one JSON object per line in the terminal, using the received channel ID: {"command":"devices"} {"command":"set-input","deviceId":"DEVICE-ID"} {"command":"set-output","deviceId":null} +{"command":"set-input-mode","mode":"push_to_talk"} +{"command":"ptt","active":true} {"command":"stats","id":"my-query"} {"command":"leave"} {"command":"reconnect"} @@ -129,6 +131,33 @@ quitting remain available. Denial keeps the microphone muted and emits `microphone-error`; it does not try to bypass system settings. The macOS permission flow still requires execution on a Mac. +### Speaking indicator and push-to-talk + +During a call, the core emits `speaking` with `sessionId` and +`speaking:true|false` whenever someone, including the local user, starts or +stops speaking. As in the full client, this is **not** published to the server: +each client derives who is speaking from the audio. + +- **Remote participants:** the level comes from the RTP header (RFC 6464) the + sender already includes, read every 100 ms. Nothing is decoded or measured + separately. While the user is deafened, no one is shown speaking. +- **Local user:** the level is measured on raw capture, before echo + cancellation and noise suppression. Loud, steady background noise can count + as speech. +- The threshold is -40 dBov (the 0.01 `audioLevel` the full client uses), and + speech ends only after 400 ms of silence so it does not flicker between words. +- Sampling exists only during a call; outside one, no thread wakes up. + +`set-input-mode` selects `voice_activity` (default: always transmitting) or +`push_to_talk` and saves the choice in the profile. In push-to-talk, `ptt` with +`active:true`/`false` opens and closes transmission. The core does not capture +keys: the global hotkey belongs to whatever controls the process; the tray has +no global shortcuts yet. Capture stays open while the key is released, so a press takes +effect immediately. The track sends silence, and the local user is not shown +speaking. Mute, deafen, and server restrictions still apply while the key is +held. Switching modes or leaving the call releases the key. `ptt` outside +push-to-talk mode is rejected. + ## Tray interface `--tray` adds the icon to the Windows notification area or the macOS status bar. @@ -226,7 +255,7 @@ require explicit recovery, never silent replacement. Directories containing another application's files are rejected. `monky-light-settings.json` stores only user preferences (currently audio -devices) and never participates in identity validation. It is replaced +devices and the input mode) and never participates in identity validation. It is replaced atomically under the profile lock. Invalid or unreadable settings emit a `warning`, fall back to defaults, and are replaced by the next save; an interrupted settings write does not require recovery. diff --git a/apps/light/README.md b/apps/light/README.md index 14f83343..26f544fa 100644 --- a/apps/light/README.md +++ b/apps/light/README.md @@ -82,6 +82,8 @@ Digite um objeto JSON por linha no terminal, usando o ID recebido: {"command":"devices"} {"command":"set-input","deviceId":"ID-DO-DISPOSITIVO"} {"command":"set-output","deviceId":null} +{"command":"set-input-mode","mode":"push_to_talk"} +{"command":"ptt","active":true} {"command":"stats","id":"minha-consulta"} {"command":"leave"} {"command":"reconnect"} @@ -130,6 +132,33 @@ precisa transmitir. Enquanto a permissão está pendente, é possível receber mutado e gera `microphone-error`; não tenta contornar as configurações do sistema. O fluxo de permissão do macOS ainda requer execução em um Mac. +### Indicador de fala e push-to-talk + +Durante uma chamada, o núcleo emite `speaking` com `sessionId` e +`speaking:true|false` sempre que alguém, inclusive o próprio usuário, começa ou +para de falar. Como no cliente completo, isso **não** é publicado no servidor: +cada cliente deduz quem fala a partir do áudio. + +- **Participantes remotos:** o nível vem do cabeçalho RTP (RFC 6464) que o + remetente já envia, consultado a cada 100 ms. Nada é decodificado ou medido + à parte. Enquanto o usuário está em deafen, ninguém aparece falando. +- **Próprio usuário:** o nível é medido na captura bruta, antes do cancelamento + de eco e da supressão de ruído. Ruído de fundo alto e constante pode contar + como fala. +- O limiar é -40 dBov (o `audioLevel` 0,01 usado pelo cliente completo), e a + fala só termina após 400 ms de silêncio, para não piscar entre palavras. +- A amostragem só existe durante a chamada; fora dela nenhuma thread acorda. + +`set-input-mode` escolhe `voice_activity` (padrão: transmite sempre) ou +`push_to_talk` e salva a escolha no perfil. Em push-to-talk, `ptt` com +`active:true`/`false` abre e fecha a transmissão. O núcleo não captura teclas: +o atalho global é responsabilidade de quem controla o processo; a bandeja +ainda não tem atalhos globais. A captura continua aberta enquanto a tecla está solta, então +pressionar tem efeito imediato. A faixa envia silêncio, e o próprio usuário não +aparece falando. Mute, deafen e restrições do servidor continuam valendo mesmo +com a tecla pressionada. Trocar de modo ou sair da chamada solta a tecla. +`ptt` fora do modo push-to-talk é recusado. + ## Interface de bandeja `--tray` adiciona o ícone na área de notificação do Windows ou na barra de status @@ -226,7 +255,7 @@ componentes da identidade exigem recuperação explícita, nunca recriação sil Diretórios com arquivos de outros aplicativos são recusados. `monky-light-settings.json` guarda apenas preferências do usuário (hoje, os -dispositivos de áudio) e nunca participa da validação da identidade. É +dispositivos de áudio e o modo de entrada) e nunca participa da validação da identidade. É substituído de forma atômica sob o lock do perfil. Configurações inválidas ou ilegíveis geram `warning`, usam os padrões e são substituídas no próximo salvamento; uma gravação interrompida das configurações não exige recuperação. diff --git a/apps/light/src/headless_application.cpp b/apps/light/src/headless_application.cpp index c2bad990..2d18dfdf 100644 --- a/apps/light/src/headless_application.cpp +++ b/apps/light/src/headless_application.cpp @@ -112,6 +112,7 @@ const char* mediaEventName(media::EventKind kind) { case media::EventKind::stats: return "media-stats"; case media::EventKind::warning: return "media-warning"; case media::EventKind::devices: return "audio-device-selected"; + case media::EventKind::speaking: return "speaking"; } throw std::logic_error("Unknown media event"); } @@ -347,6 +348,7 @@ class Application final { result.muted = session_.policy().suppressMicrophone() || (audio_.requestMicrophoneAccess && !microphoneAuthorized_); result.deafened = session_.policy().suppressPlayback(); + result.transmit = !settings_.pushToTalk || pushToTalkActive_; return result; } @@ -429,6 +431,11 @@ class Application final { const auto bytes = event.stats.dump().size(); loop_.post([this, generation, event = std::move(event)] { if (generation != session_.generation() || generation != activeGeneration_) return; + if (event.kind == media::EventKind::speaking) { + output({{"event", "speaking"}, {"generation", generation}, {"sessionId", event.session_id}, + {"speaking", event.stats.at("speaking")}}); + return; + } output({{"event", mediaEventName(event.kind)}, {"generation", generation}, {"channelId", event.channel_id}, {"sessionId", event.session_id}, {"detail", event.detail}, {"stats", event.stats}}); @@ -512,6 +519,39 @@ class Application final { if (active_) active_->select_devices(devicePreference()); } + void setInputMode(const Json& value) { + const auto mode = value.at("mode").get(); + if (mode != "voice_activity" && mode != "push_to_talk") { + throw std::invalid_argument("mode must be voice_activity or push_to_talk"); + } + auto settings = settings_; + settings.pushToTalk = mode == "push_to_talk"; + try { + identity_.saveSettings(settings); + } catch (const std::invalid_argument&) { + throw; + } catch (const std::exception& error) { + throw std::invalid_argument(std::string("Could not save input mode: ") + error.what()); + } + settings_ = std::move(settings); + // A key held while switching modes must not keep transmitting later. + pushToTalkActive_ = false; + applyAudioPolicy(); + } + + // The key itself is the controller's business (a global hotkey in the tray + // app); the core only gates transmission. + void pushToTalk(bool active) { + if (!settings_.pushToTalk) throw std::invalid_argument("Push-to-talk is not the input mode"); + if (pushToTalkActive_ == active) return; + pushToTalkActive_ = active; + applyAudioPolicy(); + } + + void applyAudioPolicy() { + if (active_ && activeGeneration_ == session_.generation()) active_->update_policy(audioPolicy()); + } + void stopWatchingDevices() { if (!stopWatchingDevices_) return; auto stop = std::move(stopWatchingDevices_); @@ -567,10 +607,12 @@ class Application final { break; case SessionEventKind::teardown: if (active_ && activeGeneration_ < event.generation) retireVoice(); + // Like the full client, a key held across calls does not carry over. + pushToTalkActive_ = false; output({{"event", "voice-left"}, {"generation", event.generation}}); break; case SessionEventKind::policyChanged: - if (active_ && activeGeneration_ == session_.generation()) active_->update_policy(audioPolicy()); + applyAudioPolicy(); authorizeMicrophone(); if (!closing_) output({{"event", "voice-policy"}, {"policy", event.payload}}); break; @@ -716,6 +758,8 @@ class Application final { {"voiceMode", session_.voiceMode() == VoiceMode::sfu ? "sfu" : "p2p"}, {"mediaActive", active_ != nullptr}, {"retiringMedia", retired_.size()}, {"policy", session_.policy().describe()}, + {"inputMode", settings_.pushToTalk ? "push_to_talk" : "voice_activity"}, + {"pushToTalkActive", pushToTalkActive_}, {"audioDevice", audio_.diagnostics ? audio_.diagnostics() : Json(nullptr)}}; result["microphoneAccessPending"] = microphoneRequestPending_; result["channelId"] = session_.admission() ? Json(session_.admission()->channelId) : Json(nullptr); @@ -752,6 +796,8 @@ class Application final { if (name.starts_with("fixture-") && audio_.fixtureCommand) audio_.fixtureCommand(value); else if (name == "set-input") selectDevice(true, value); else if (name == "set-output") selectDevice(false, value); + else if (name == "set-input-mode") setInputMode(value); + else if (name == "ptt") pushToTalk(value.at("active").get()); else if (name == "join") session_.join(value.at("channelId").get(), Clock::now()); else if (name == "leave") session_.leave(); else if (name == "mute") session_.setMuted(value.at("enabled").get()); @@ -759,7 +805,7 @@ class Application final { else if (name == "reconnect") restartConnection(); else if (name != "quit") { throw std::invalid_argument("Unknown command; use join, leave, mute, deafen, devices, set-input, " - "set-output, stats, channels, reconnect or quit"); + "set-output, set-input-mode, ptt, stats, channels, reconnect or quit"); } output({{"event", "command-accepted"}, {"id", id}, {"command", name}}); if (name == "quit") quit(); @@ -799,6 +845,7 @@ class Application final { const bool outputIsConsole_ = standardOutputIsConsole(); bool microphoneAuthorized_ = false; bool microphoneRequestPending_ = false; + bool pushToTalkActive_ = false; CancelMicrophoneAccess cancelMicrophoneAccess_; std::function observer_; int exitCode_ = 0; @@ -826,6 +873,8 @@ int runHeadless(std::vector arguments, HeadlessConfiguration audio) "{\"command\":\"deafen\",\"enabled\":false}\n" "{\"command\":\"devices\"} / {\"command\":\"set-input\",\"deviceId\":\"...\"|null}\n" "{\"command\":\"set-output\",\"deviceId\":\"...\"|null}\n" + "{\"command\":\"set-input-mode\",\"mode\":\"voice_activity\"|\"push_to_talk\"}\n" + "{\"command\":\"ptt\",\"active\":true|false}\n" "{\"command\":\"stats\"} / {\"command\":\"channels\"}\n" "{\"command\":\"leave\"} / {\"command\":\"reconnect\"} / {\"command\":\"quit\"}\n" "EOF and Ctrl+C release the connection and media resources.\n" diff --git a/apps/light/src/media/policy_audio_device.hpp b/apps/light/src/media/policy_audio_device.hpp index 8b5de5ca..eb416e16 100644 --- a/apps/light/src/media/policy_audio_device.hpp +++ b/apps/light/src/media/policy_audio_device.hpp @@ -1,6 +1,7 @@ #pragma once #include "audio_devices.hpp" +#include "speech_activity.hpp" #include #include @@ -54,6 +55,18 @@ class PolicyAudioDevice : public webrtc::AudioDeviceModule { } const AudioDeviceSelection &input_selection() const { return input_; } const AudioDeviceSelection &output_selection() const { return output_; } + // Safe from any thread: loudest raw capture level since the last read. + std::uint8_t TakeInputLevel() { return meter_.Take(); } + + int32_t RegisterAudioCallback(webrtc::AudioTransport *callback) override { + if (!callback) { + const auto result = native_->RegisterAudioCallback(nullptr); + meter_.SetTarget(nullptr); + return result; + } + meter_.SetTarget(callback); + return native_->RegisterAudioCallback(&meter_); + } int32_t InitRecording() override { return capture_allowed_ ? native_->InitRecording() : 0; @@ -91,8 +104,6 @@ class PolicyAudioDevice : public webrtc::AudioDeviceModule { #define MONKY_ADM_CONST(result, name, parameters, arguments) \ result name parameters const override { return native_->name arguments; } MONKY_ADM_CONST(int32_t, ActiveAudioLayer, (AudioLayer * layer), (layer)) - MONKY_ADM_FORWARD(int32_t, RegisterAudioCallback, - (webrtc::AudioTransport * callback), (callback)) MONKY_ADM_FORWARD(int32_t, Init, (), ()) MONKY_ADM_FORWARD(int32_t, Terminate, (), ()) MONKY_ADM_CONST(bool, Initialized, (), ()) @@ -232,6 +243,7 @@ class PolicyAudioDevice : public webrtc::AudioDeviceModule { } webrtc::scoped_refptr native_; + InputLevelMeter meter_; AudioDeviceSelection input_; AudioDeviceSelection output_; // The first selection always applies, even for the default device. diff --git a/apps/light/src/media/speech_activity.hpp b/apps/light/src/media/speech_activity.hpp new file mode 100644 index 00000000..769166c1 --- /dev/null +++ b/apps/light/src/media/speech_activity.hpp @@ -0,0 +1,137 @@ +#pragma once + +#include + +#include +#include +#include +#include +#include +#include + +namespace monky::light::media { + +// Levels use the RFC 6464 scale carried by RTP: 0 is full scale, 127 silence, +// in -dBov. Remote levels come from that header; local ones are measured here. +constexpr std::uint8_t kSilentLevel = 127; +// -40 dBov is the 0.01 linear audioLevel the Chromium client treats as voice. +constexpr std::uint8_t kSpeechLevel = 40; +// Consecutive quiet samples before speech ends, so word gaps do not flicker. +constexpr unsigned kSpeechHoldSamples = 4; + +inline std::uint8_t LevelOf(const std::int16_t *samples, std::size_t count) { + if (count == 0) + return kSilentLevel; + double energy = 0; + for (std::size_t index = 0; index < count; ++index) + energy += static_cast(samples[index]) * samples[index]; + const double rms = std::sqrt(energy / static_cast(count)) / 32768.0; + if (rms <= 0) + return kSilentLevel; + return static_cast( + std::clamp(-20.0 * std::log10(rms), 0.0, double{kSilentLevel})); +} + +// Hysteresis over periodic samples. Missing samples (no packets, stopped +// capture) count as silence. Owned by a single thread. +class SpeechDetector { +public: + // Returns the new state only when it changes. + std::optional Sample(std::uint8_t level) { + if (level <= kSpeechLevel) { + quiet_ = 0; + return Set(true); + } + if (++quiet_ < kSpeechHoldSamples) + return std::nullopt; + return Set(false); + } + // Ends speech immediately, e.g. when transmission is gated. + std::optional Silence() { + quiet_ = kSpeechHoldSamples; + return Set(false); + } + bool speaking() const { return speaking_; } + +private: + std::optional Set(bool value) { + if (speaking_ == value) + return std::nullopt; + speaking_ = value; + return value; + } + bool speaking_ = false; + unsigned quiet_ = 0; +}; + +// Sits between the native ADM and WebRTC's audio transport and keeps the +// loudest capture level since the last read. Measuring costs one pass over +// each 10 ms block on the capture thread; nothing is copied or allocated. +// The level is raw device input, before echo cancellation and noise +// suppression, so steady loud background noise can count as speech. +class InputLevelMeter final : public webrtc::AudioTransport { +public: + void SetTarget(webrtc::AudioTransport *target) { target_.store(target); } + // Loudest level since the previous call; silence when nothing was captured. + std::uint8_t Take() { return loudest_.exchange(kSilentLevel); } + + int32_t RecordedDataIsAvailable(const void *samples, size_t count, + size_t bytes_per_sample, size_t channels, + uint32_t rate, uint32_t delay, + int32_t drift, uint32_t mic_level, + bool key_pressed, + uint32_t &new_mic_level) override { + return RecordedDataIsAvailable(samples, count, bytes_per_sample, channels, + rate, delay, drift, mic_level, key_pressed, + new_mic_level, std::nullopt); + } + int32_t RecordedDataIsAvailable( + const void *samples, size_t count, size_t bytes_per_sample, + size_t channels, uint32_t rate, uint32_t delay, int32_t drift, + uint32_t mic_level, bool key_pressed, uint32_t &new_mic_level, + std::optional capture_time_ns) override { + if (bytes_per_sample == channels * sizeof(std::int16_t)) + Record(LevelOf(static_cast(samples), + count * channels)); + auto *target = target_.load(); + if (!target) { + new_mic_level = mic_level; + return 0; + } + return target->RecordedDataIsAvailable( + samples, count, bytes_per_sample, channels, rate, delay, drift, + mic_level, key_pressed, new_mic_level, capture_time_ns); + } + int32_t NeedMorePlayData(size_t count, size_t bytes_per_sample, + size_t channels, uint32_t rate, void *samples, + size_t &samples_out, int64_t *elapsed_ms, + int64_t *ntp_ms) override { + auto *target = target_.load(); + if (!target) { + samples_out = 0; + return 0; + } + return target->NeedMorePlayData(count, bytes_per_sample, channels, rate, + samples, samples_out, elapsed_ms, ntp_ms); + } + void PullRenderData(int bits_per_sample, int rate, size_t channels, + size_t frames, void *data, int64_t *elapsed_ms, + int64_t *ntp_ms) override { + if (auto *target = target_.load()) + target->PullRenderData(bits_per_sample, rate, channels, frames, data, + elapsed_ms, ntp_ms); + } + +private: + void Record(std::uint8_t level) { + auto current = loudest_.load(); + while (level < current && !loudest_.compare_exchange_weak(current, level)) { + } + } + // WebRTC registers with both directions stopped, but test devices do not + // enforce that; an atomic keeps a late callback from reading a torn value. + std::atomic target_{nullptr}; + std::atomic loudest_{kSilentLevel}; +}; + +} // namespace monky::light::media diff --git a/apps/light/src/media/voice_engine.cpp b/apps/light/src/media/voice_engine.cpp index 019ee356..b510a497 100644 --- a/apps/light/src/media/voice_engine.cpp +++ b/apps/light/src/media/voice_engine.cpp @@ -20,6 +20,7 @@ #include #include #include +#include #if defined(WEBRTC_WIN) #include #endif @@ -355,6 +356,7 @@ struct VoiceEngine::Impl : std::enable_shared_from_this { std::set failed_producers; std::map sfu_transport_states; Json applied_devices; + std::map speech; bool initialized = false; // The server refused the microphone: the call only receives, with capture // stopped, until the next policy change tries again. @@ -540,6 +542,9 @@ struct VoiceEngine::Impl : std::enable_shared_from_this { void Consume(const std::string &id); void CloseConsumer(const std::string &id); void Stats(); + void ScheduleSpeech(); + void SampleSpeech(); + void ReportSpeech(const std::string &session, std::optional change); }; struct VoiceEngine::Impl::Peer : webrtc::PeerConnectionObserver, @@ -875,6 +880,7 @@ void VoiceEngine::Impl::EnsureMicrophone() { microphone = factory->CreateAudioTrack("monky-microphone", source.get()); if (!microphone) throw std::runtime_error("Could not create microphone track"); + microphone->set_enabled(config.policy.transmit); for (auto &[id, peer] : peers) if (!peer->mic_transceiver->sender()->SetTrack(microphone.get())) throw std::runtime_error("Could not attach microphone to peer " + id); @@ -938,6 +944,12 @@ void VoiceEngine::Impl::ApplyPolicy() { StopMicrophone(true); else EnsureMicrophone(); + if (microphone) + microphone->set_enabled(config.policy.transmit); + // Report gated speech now instead of after the quiet hold. + if (muted || !config.policy.transmit) + ReportSpeech(config.self_session_id, + speech[config.self_session_id].Silence()); for (auto &[id, peer] : peers) ConfigureTracks(*peer); for (auto &[id, consumer] : consumers) { @@ -1042,6 +1054,7 @@ std::string VoiceEngine::Impl::Cleanup() { } environment.reset(); applied_devices = nullptr; + speech.clear(); initialized = false; publish_refused = false; sfu_recovery_pending = false; @@ -1094,6 +1107,7 @@ void VoiceEngine::Impl::Join(JoinConfig value, } token->CheckActive(); initialized = true; + ScheduleSpeech(); Emit(EventKind::initialized, peers.empty() && consumers.empty() ? "Voice initialized; no remote microphone connected" @@ -1784,6 +1798,77 @@ void VoiceEngine::Impl::Stats() { } } +// Polling at a human-perceptible rate is cheaper than per-frame sinks and keeps +// all speech state on media control. Remote levels are the RFC 6464 values +// the sender put in each RTP header, so nothing is decoded or measured here. +constexpr auto kSpeechSampleInterval = std::chrono::milliseconds(100); +// Opus DTX still sends a packet every 400 ms; older sources mean no stream. +constexpr std::int64_t kSpeechSourceAgeMs = 500; + +void VoiceEngine::Impl::ScheduleSpeech() { + auto weak = weak_from_this(); + PostAfter(kSpeechSampleInterval, [weak, call = token] { + auto self = weak.lock(); + if (!self || !self->Current(call)) + return; + self->SampleSpeech(); + self->ScheduleSpeech(); + }); +} + +void VoiceEngine::Impl::SampleSpeech() { + std::map levels; + const bool deafened = token->deafened.load(); + // Always drain the meter so audio captured while gated is never reported. + const auto local = adm->TakeInputLevel(); + const bool transmitting = microphone && config.policy.transmit && + !token->muted.load() && !deafened; + levels[config.self_session_id] = transmitting ? local : kSilentLevel; + const auto now = webrtc::TimeMillis(); + const auto remote = [&](const std::string &session, + webrtc::RtpReceiverInterface *receiver) { + auto &level = levels.try_emplace(session, kSilentLevel).first->second; + // Deafened users hear no one; SFU consumers are paused anyway. + if (!receiver || deafened) + return; + for (const auto &source : receiver->GetSources()) + if (source.audio_level() && + now - source.timestamp().ms() <= kSpeechSourceAgeMs) + level = std::min(level, *source.audio_level()); + }; + for (const auto &[id, peer] : peers) + if (peer->mic_transceiver && + peer->participant.user_id != config.self_user_id) + remote(id, peer->mic_transceiver->receiver().get()); + for (const auto &[id, consumer] : consumers) + if (const auto found = known_producers.find(id); + found != known_producers.end()) + remote(Text(found->second, "producerSessionId"), + consumer->GetRtpReceiver()); + // Whoever left the call stops speaking at once. + for (auto entry = speech.begin(); entry != speech.end();) { + if (roster.contains(entry->first)) { + ++entry; + continue; + } + ReportSpeech(entry->first, entry->second.Silence()); + entry = speech.erase(entry); + } + for (const auto &[session, level] : levels) + if (roster.contains(session)) + ReportSpeech(session, speech[session].Sample(level)); + for (auto &[session, detector] : speech) + if (!levels.contains(session)) + ReportSpeech(session, detector.Sample(kSilentLevel)); +} + +void VoiceEngine::Impl::ReportSpeech(const std::string &session, + std::optional change) { + if (change) + Emit(EventKind::speaking, *change ? "Speaking" : "Silent", session, + {{"speaking", *change}}); +} + struct VoiceEngine::Owner { std::shared_ptr impl; std::thread thread; diff --git a/apps/light/src/media/voice_engine.hpp b/apps/light/src/media/voice_engine.hpp index 7874955b..2d41fb1b 100644 --- a/apps/light/src/media/voice_engine.hpp +++ b/apps/light/src/media/voice_engine.hpp @@ -25,6 +25,9 @@ struct AudioPolicy { // These are EFFECTIVE values, including server restrictions. bool muted = false; bool deafened = false; + // Push-to-talk gate. The device keeps capturing so a press takes effect + // at once; the track sends silence and local speech is not reported. + bool transmit = true; bool echo_cancellation = true; bool automatic_gain = true; NoiseSuppression noise_suppression = NoiseSuppression::moderate; @@ -80,7 +83,11 @@ enum class EventKind { stats, warning, // `stats` carries the effective input/output device selection. - devices + devices, + // A participant (self included) started or stopped speaking; `stats` holds + // {"speaking": bool}. Sampled only during a call, never published to the + // server: receivers derive it from audio, as the Chromium client does. + speaking }; struct Event { EventKind kind; diff --git a/apps/light/src/profile_identity.cpp b/apps/light/src/profile_identity.cpp index 95f6df91..ee0f284a 100644 --- a/apps/light/src/profile_identity.cpp +++ b/apps/light/src/profile_identity.cpp @@ -47,6 +47,9 @@ constexpr char kSettingsFile[] = "monky-light-settings.json"; constexpr char kSettingsPendingFile[] = "monky-light-settings.json.pending"; constexpr char kSettingsFormat[] = "monky-light-settings"; constexpr std::size_t kMaxSettingsBytes = 16 * 1024; +// Same names as the full client's settings. +constexpr char kVoiceActivity[] = "voice_activity"; +constexpr char kPushToTalk[] = "push_to_talk"; constexpr std::size_t kMaxDeviceIdBytes = 1024; // server-key-auth (#821): the identity key pinned for each server this profile signed in to. constexpr char kServerKeysFile[] = "monky-light-servers.json"; @@ -550,7 +553,12 @@ ProfileSettings parseSettings(const std::string& contents) { !data.contains("version") || !data["version"].is_number_integer() || data["version"] != 1) { throw std::runtime_error("Profile settings have an unsupported format or version"); } - return {settingsDevice(data, "inputDeviceId"), settingsDevice(data, "outputDeviceId")}; + // Absent in profiles saved before push-to-talk existed. + const auto mode = data.value("inputMode", Json(kVoiceActivity)); + if (mode != kVoiceActivity && mode != kPushToTalk) { + throw std::runtime_error("Profile settings contain an invalid input mode"); + } + return {settingsDevice(data, "inputDeviceId"), settingsDevice(data, "outputDeviceId"), mode == kPushToTalk}; } std::string encodeSettings(const ProfileSettings& settings) { @@ -566,7 +574,8 @@ std::string encodeSettings(const ProfileSettings& settings) { try { result = Json{{"format", kSettingsFormat}, {"version", 1}, {"inputDeviceId", optional(settings.inputDeviceId)}, - {"outputDeviceId", optional(settings.outputDeviceId)}}.dump(2) + "\n"; + {"outputDeviceId", optional(settings.outputDeviceId)}, + {"inputMode", settings.pushToTalk ? kPushToTalk : kVoiceActivity}}.dump(2) + "\n"; } catch (const Json::exception&) { throw std::invalid_argument("Audio device ID must be valid UTF-8"); } diff --git a/apps/light/src/profile_identity.hpp b/apps/light/src/profile_identity.hpp index 43c5ef33..05ba6cd5 100644 --- a/apps/light/src/profile_identity.hpp +++ b/apps/light/src/profile_identity.hpp @@ -16,6 +16,7 @@ namespace monky::light { struct ProfileSettings { std::optional inputDeviceId; // nullopt: system default std::optional outputDeviceId; // nullopt: system default + bool pushToTalk = false; // false: voice activity }; // Owns the signing key and exclusive profile lock for its entire lifetime. diff --git a/apps/light/test/fixture_audio_device_test.cpp b/apps/light/test/fixture_audio_device_test.cpp index 536e586c..844b7e7c 100644 --- a/apps/light/test/fixture_audio_device_test.cpp +++ b/apps/light/test/fixture_audio_device_test.cpp @@ -442,6 +442,9 @@ void CheckDeviceSelection() { Require(policy->SetPolicy(true, true) == 0 && policy->StartRecording() == 0 && policy->StartPlayout() == 0 && transport.WaitFor(3, 3), "Selected devices start"); + // The synthetic tone is a 2048-amplitude sine: RMS 20*log10(1448/32768) = -27 dBov. + const auto level = policy->TakeInputLevel(); + Require(level >= 26 && level <= 28, "The policy device measures capture before forwarding it"); Require(policy->SelectDevices({std::string("mic-a"), std::string("out-b")}) == 0 && device->Recording() && device->Playing() && @@ -462,6 +465,8 @@ void CheckDeviceSelection() { "A reconnected preferred device is selected again"); Require(policy->SetPolicy(false, true) == 0 && !device->Recording(), "Mute stops capture"); + static_cast(policy->TakeInputLevel()); + Require(policy->TakeInputLevel() == media::kSilentLevel, "Stopped capture reports silence"); Require(policy->SelectDevices({std::string("mic-b"), std::string("out-b")}) == 0 && !device->Recording() && device->Snapshot().selected_input == "mic-b", "Switching the input while muted must not reopen capture"); @@ -470,6 +475,28 @@ void CheckDeviceSelection() { Require(policy->RegisterAudioCallback(nullptr) == 0, "Detach the policy transport"); } +void CheckSpeechActivity() { + const std::array silent{}; + const std::array full{32767, -32768, 32767, -32768}; + Require(media::LevelOf(silent.data(), silent.size()) == media::kSilentLevel && + media::LevelOf(full.data(), full.size()) == 0 && + media::LevelOf(full.data(), 0) == media::kSilentLevel, + "Levels use the RFC 6464 -dBov scale"); + media::SpeechDetector detector; + Require(!detector.Sample(media::kSilentLevel) && detector.Sample(media::kSpeechLevel) == true && + !detector.Sample(media::kSpeechLevel + 1) && !detector.Sample(media::kSilentLevel) && + !detector.Sample(media::kSilentLevel) && !detector.Sample(10), + "Speech starts at the threshold and survives short gaps"); + for (unsigned index = 1; index < media::kSpeechHoldSamples; ++index) { + Require(!detector.Sample(media::kSilentLevel), "Speech holds through the quiet window"); + } + Require(detector.Sample(media::kSilentLevel) == false && !detector.speaking(), + "Speech ends after the quiet window"); + Require(detector.Sample(0) == true && detector.Silence() == false && !detector.Silence() && + !detector.Sample(media::kSilentLevel), + "Silence ends speech at once and reports only one change"); +} + void CheckFailures() { Transport transport; auto device = FixtureAudioDevice::Create(); @@ -545,6 +572,7 @@ int main() { CheckInflightStopAndTerminate(false); CheckInflightStopAndTerminate(true); CheckDeviceSelection(); + CheckSpeechActivity(); CheckFailures(); CheckDestruction(); CheckTransportDestructionAfterRemoval(); diff --git a/apps/light/test/headless_interop.test.cjs b/apps/light/test/headless_interop.test.cjs index dbf1b6f5..6a6751b6 100644 --- a/apps/light/test/headless_interop.test.cjs +++ b/apps/light/test/headless_interop.test.cjs @@ -203,6 +203,49 @@ for (const voiceMode of ['p2p', 'sfu']) { }); } +for (const voiceMode of ['p2p', 'sfu']) { + test(`headless clients report ${voiceMode} speech and gate it with push-to-talk`, { timeout: 60_000 }, async t => { + const fixture = await createServerFixture(t, { voiceMode }); + const alice = new NativeClient(fixture, { nickname: 'Light Alice' }); + const bob = new NativeClient(fixture, { nickname: 'Light Bob' }); + const auth = await alice.wait('authenticated'); + await bob.wait('authenticated'); + const channelId = auth.channels.find(channel => channel.type === 'VOICE').id; + await alice.join(channelId); + await bob.join(channelId); + // The synthetic microphone plays a constant tone, so both sides see Alice speak. + // Without an action the change may already have happened: Alice speaks as + // soon as her own call starts, possibly before Bob's join returns. + const speech = async (speaking, action) => { + const since = action ? [alice.events.length, bob.events.length] : [0, 0]; + if (action) await action(); + await Promise.all([alice, bob].map((client, index) => client.wait(event => event.event === 'speaking' && + event.sessionId === auth.sessionId && event.speaking === speaking, since[index]))); + }; + await speech(true); + + assert.equal((await alice.command('ptt', { active: true })).event, 'command-error', + 'Push-to-talk must not be accepted in voice-activity mode'); + await speech(false, () => alice.command('set-input-mode', { mode: 'push_to_talk' })); + const gated = await alice.state(); + assert.equal(gated.inputMode, 'push_to_talk'); + assert.equal(gated.pushToTalkActive, false); + assert.equal(gated.audioDevice.recording, true, 'Push-to-talk keeps capture open so a press is immediate'); + + await speech(true, () => alice.command('ptt', { active: true })); + await speech(false, () => alice.command('ptt', { active: false })); + await speech(true, () => alice.command('ptt', { active: true })); + await speech(false, () => alice.command('mute', { enabled: true })); + await speech(true, () => alice.command('mute', { enabled: false })); + await alice.command('set-input-mode', { mode: 'voice_activity' }); + const restored = await alice.state(); + assert.equal(restored.inputMode, 'voice_activity'); + assert.equal(restored.pushToTalkActive, false, 'Switching modes releases a held key'); + await alice.close(); + await bob.close(); + }); +} + test('native calls survive both live P2P/SFU topology transitions and preserve moderation', { timeout: 60_000 }, async t => { const fixture = await createServerFixture(t); const admin = await fixture.connectHuman('Fixture admin'); diff --git a/apps/light/test/profile_identity_test.cpp b/apps/light/test/profile_identity_test.cpp index b5af5bcd..b04dac71 100644 --- a/apps/light/test/profile_identity_test.cpp +++ b/apps/light/test/profile_identity_test.cpp @@ -399,9 +399,10 @@ void settingsBesideIdentity(const fs::path& root) { ProfileIdentity identity(profile.path); publicKey = identity.publicKeyHex(); const auto defaults = identity.loadSettings(); - require(!defaults.inputDeviceId && !defaults.outputDeviceId, "Missing settings did not yield defaults"); + require(!defaults.inputDeviceId && !defaults.outputDeviceId && !defaults.pushToTalk, + "Missing settings did not yield defaults"); identity.saveSettings({std::string("input-\xc3\xa1"), std::nullopt}); - identity.saveSettings({std::string("input-\xc3\xa1"), std::string("output")}); + identity.saveSettings({std::string("input-\xc3\xa1"), std::string("output"), true}); require(!fs::exists(profile.path / "monky-light-settings.json.pending"), "Settings write left its marker"); requireFailure([&] { identity.saveSettings({std::string(), std::nullopt}); }, "An empty device ID was saved"); @@ -418,12 +419,23 @@ void settingsBesideIdentity(const fs::path& root) { require(identity.publicKeyHex() == publicKey, "Settings changed the profile identity"); const auto loaded = identity.loadSettings(); require(loaded.inputDeviceId == std::optional("input-\xc3\xa1") && - loaded.outputDeviceId == std::optional("output"), "Settings did not round-trip"); + loaded.outputDeviceId == std::optional("output") && loaded.pushToTalk, + "Settings did not round-trip"); identity.saveSettings({std::nullopt, std::string("output")}); require(!fs::exists(profile.path / "monky-light-settings.json.pending"), "Settings write did not replace an interrupted marker"); } + // Profiles saved before push-to-talk existed keep voice activity. + writeFile(settingsPath, "{\"format\":\"monky-light-settings\",\"version\":1,\"inputDeviceId\":\"mic\"}"); + { + ProfileIdentity identity(profile.path); + const auto legacy = identity.loadSettings(); + require(legacy.inputDeviceId == std::optional("mic") && !legacy.pushToTalk, + "Settings without an input mode did not default to voice activity"); + } for (const auto* invalid : {"", "{", "[]", "{\"format\":\"monky-light-settings\",\"version\":2}", + "{\"format\":\"monky-light-settings\",\"version\":1,\"inputMode\":\"always\"}", + "{\"format\":\"monky-light-settings\",\"version\":1,\"inputMode\":true}", "{\"format\":\"other\",\"version\":1}", "{\"format\":\"monky-light-settings\",\"version\":1,\"inputDeviceId\":7}", "{\"format\":\"monky-light-settings\",\"version\":1,\"inputDeviceId\":{}}"}) { diff --git a/release-notes/658-light-speaking-ptt.json b/release-notes/658-light-speaking-ptt.json new file mode 100644 index 00000000..b4813341 --- /dev/null +++ b/release-notes/658-light-speaking-ptt.json @@ -0,0 +1,5 @@ +{ + "group": "novidades", + "pt-BR": "O núcleo do Monky Light agora indica quem está falando na chamada, inclusive você, e ganhou o modo push-to-talk: a escolha fica salva no perfil, e o microfone só transmite enquanto a tecla está pressionada.", + "en": "The Monky Light core now shows who is speaking in the call, including you, and gained a push-to-talk mode: the choice is saved in the profile, and the microphone only transmits while the key is held." +}