Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 30 additions & 1 deletion apps/light/README.en.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,8 @@ Enter one JSON object per line in the terminal, using the received channel ID:
{"command":"devices"}
{"command":"set-input","deviceId":"DEVICE-ID"}
{"command":"set-output","deviceId":null}
{"command":"set-input-mode","mode":"push_to_talk"}
{"command":"ptt","active":true}
{"command":"stats","id":"my-query"}
{"command":"leave"}
{"command":"reconnect"}
Expand Down Expand Up @@ -129,6 +131,33 @@ quitting remain available. Denial keeps the microphone muted and emits
`microphone-error`; it does not try to bypass system settings. The macOS
permission flow still requires execution on a Mac.

### Speaking indicator and push-to-talk

During a call, the core emits `speaking` with `sessionId` and
`speaking:true|false` whenever someone, including the local user, starts or
stops speaking. As in the full client, this is **not** published to the server:
each client derives who is speaking from the audio.

- **Remote participants:** the level comes from the RTP header (RFC 6464) the
sender already includes, read every 100 ms. Nothing is decoded or measured
separately. While the user is deafened, no one is shown speaking.
- **Local user:** the level is measured on raw capture, before echo
cancellation and noise suppression. Loud, steady background noise can count
as speech.
- The threshold is -40 dBov (the 0.01 `audioLevel` the full client uses), and
speech ends only after 400 ms of silence so it does not flicker between words.
- Sampling exists only during a call; outside one, no thread wakes up.

`set-input-mode` selects `voice_activity` (default: always transmitting) or
`push_to_talk` and saves the choice in the profile. In push-to-talk, `ptt` with
`active:true`/`false` opens and closes transmission. The core does not capture
keys: the global hotkey belongs to whatever controls the process; the tray has
no global shortcuts yet. Capture stays open while the key is released, so a press takes
effect immediately. The track sends silence, and the local user is not shown
speaking. Mute, deafen, and server restrictions still apply while the key is
held. Switching modes or leaving the call releases the key. `ptt` outside
push-to-talk mode is rejected.

## Tray interface

`--tray` adds the icon to the Windows notification area or the macOS status bar.
Expand Down Expand Up @@ -226,7 +255,7 @@ require explicit recovery, never silent replacement. Directories containing
another application's files are rejected.

`monky-light-settings.json` stores only user preferences (currently audio
devices) and never participates in identity validation. It is replaced
devices and the input mode) and never participates in identity validation. It is replaced
atomically under the profile lock. Invalid or unreadable settings emit a
`warning`, fall back to defaults, and are replaced by the next save; an
interrupted settings write does not require recovery.
Expand Down
31 changes: 30 additions & 1 deletion apps/light/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,8 @@ Digite um objeto JSON por linha no terminal, usando o ID recebido:
{"command":"devices"}
{"command":"set-input","deviceId":"ID-DO-DISPOSITIVO"}
{"command":"set-output","deviceId":null}
{"command":"set-input-mode","mode":"push_to_talk"}
{"command":"ptt","active":true}
{"command":"stats","id":"minha-consulta"}
{"command":"leave"}
{"command":"reconnect"}
Expand Down Expand Up @@ -130,6 +132,33 @@ precisa transmitir. Enquanto a permissão está pendente, é possível receber
mutado e gera `microphone-error`; não tenta contornar as configurações do sistema.
O fluxo de permissão do macOS ainda requer execução em um Mac.

### Indicador de fala e push-to-talk

Durante uma chamada, o núcleo emite `speaking` com `sessionId` e
`speaking:true|false` sempre que alguém, inclusive o próprio usuário, começa ou
para de falar. Como no cliente completo, isso **não** é publicado no servidor:
cada cliente deduz quem fala a partir do áudio.

- **Participantes remotos:** o nível vem do cabeçalho RTP (RFC 6464) que o
remetente já envia, consultado a cada 100 ms. Nada é decodificado ou medido
à parte. Enquanto o usuário está em deafen, ninguém aparece falando.
- **Próprio usuário:** o nível é medido na captura bruta, antes do cancelamento
de eco e da supressão de ruído. Ruído de fundo alto e constante pode contar
como fala.
- O limiar é -40 dBov (o `audioLevel` 0,01 usado pelo cliente completo), e a
fala só termina após 400 ms de silêncio, para não piscar entre palavras.
- A amostragem só existe durante a chamada; fora dela nenhuma thread acorda.

`set-input-mode` escolhe `voice_activity` (padrão: transmite sempre) ou
`push_to_talk` e salva a escolha no perfil. Em push-to-talk, `ptt` com
`active:true`/`false` abre e fecha a transmissão. O núcleo não captura teclas:
o atalho global é responsabilidade de quem controla o processo; a bandeja
ainda não tem atalhos globais. A captura continua aberta enquanto a tecla está solta, então
pressionar tem efeito imediato. A faixa envia silêncio, e o próprio usuário não
aparece falando. Mute, deafen e restrições do servidor continuam valendo mesmo
com a tecla pressionada. Trocar de modo ou sair da chamada solta a tecla.
`ptt` fora do modo push-to-talk é recusado.

## Interface de bandeja

`--tray` adiciona o ícone na área de notificação do Windows ou na barra de status
Expand Down Expand Up @@ -226,7 +255,7 @@ componentes da identidade exigem recuperação explícita, nunca recriação sil
Diretórios com arquivos de outros aplicativos são recusados.

`monky-light-settings.json` guarda apenas preferências do usuário (hoje, os
dispositivos de áudio) e nunca participa da validação da identidade. É
dispositivos de áudio e o modo de entrada) e nunca participa da validação da identidade. É
substituído de forma atômica sob o lock do perfil. Configurações inválidas ou
ilegíveis geram `warning`, usam os padrões e são substituídas no próximo
salvamento; uma gravação interrompida das configurações não exige recuperação.
Expand Down
53 changes: 51 additions & 2 deletions apps/light/src/headless_application.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ const char* mediaEventName(media::EventKind kind) {
case media::EventKind::stats: return "media-stats";
case media::EventKind::warning: return "media-warning";
case media::EventKind::devices: return "audio-device-selected";
case media::EventKind::speaking: return "speaking";
}
throw std::logic_error("Unknown media event");
}
Expand Down Expand Up @@ -347,6 +348,7 @@ class Application final {
result.muted = session_.policy().suppressMicrophone() ||
(audio_.requestMicrophoneAccess && !microphoneAuthorized_);
result.deafened = session_.policy().suppressPlayback();
result.transmit = !settings_.pushToTalk || pushToTalkActive_;
return result;
}

Expand Down Expand Up @@ -429,6 +431,11 @@ class Application final {
const auto bytes = event.stats.dump().size();
loop_.post([this, generation, event = std::move(event)] {
if (generation != session_.generation() || generation != activeGeneration_) return;
if (event.kind == media::EventKind::speaking) {
output({{"event", "speaking"}, {"generation", generation}, {"sessionId", event.session_id},
{"speaking", event.stats.at("speaking")}});
return;
}
output({{"event", mediaEventName(event.kind)}, {"generation", generation},
{"channelId", event.channel_id}, {"sessionId", event.session_id},
{"detail", event.detail}, {"stats", event.stats}});
Expand Down Expand Up @@ -512,6 +519,39 @@ class Application final {
if (active_) active_->select_devices(devicePreference());
}

void setInputMode(const Json& value) {
const auto mode = value.at("mode").get<std::string>();
if (mode != "voice_activity" && mode != "push_to_talk") {
throw std::invalid_argument("mode must be voice_activity or push_to_talk");
}
auto settings = settings_;
settings.pushToTalk = mode == "push_to_talk";
try {
identity_.saveSettings(settings);
} catch (const std::invalid_argument&) {
throw;
} catch (const std::exception& error) {
throw std::invalid_argument(std::string("Could not save input mode: ") + error.what());
}
settings_ = std::move(settings);
// A key held while switching modes must not keep transmitting later.
pushToTalkActive_ = false;
applyAudioPolicy();
}

// The key itself is the controller's business (a global hotkey in the tray
// app); the core only gates transmission.
void pushToTalk(bool active) {
if (!settings_.pushToTalk) throw std::invalid_argument("Push-to-talk is not the input mode");
if (pushToTalkActive_ == active) return;
pushToTalkActive_ = active;
applyAudioPolicy();
}

void applyAudioPolicy() {
if (active_ && activeGeneration_ == session_.generation()) active_->update_policy(audioPolicy());
}

void stopWatchingDevices() {
if (!stopWatchingDevices_) return;
auto stop = std::move(stopWatchingDevices_);
Expand Down Expand Up @@ -567,10 +607,12 @@ class Application final {
break;
case SessionEventKind::teardown:
if (active_ && activeGeneration_ < event.generation) retireVoice();
// Like the full client, a key held across calls does not carry over.
pushToTalkActive_ = false;
output({{"event", "voice-left"}, {"generation", event.generation}});
break;
case SessionEventKind::policyChanged:
if (active_ && activeGeneration_ == session_.generation()) active_->update_policy(audioPolicy());
applyAudioPolicy();
authorizeMicrophone();
if (!closing_) output({{"event", "voice-policy"}, {"policy", event.payload}});
break;
Expand Down Expand Up @@ -716,6 +758,8 @@ class Application final {
{"voiceMode", session_.voiceMode() == VoiceMode::sfu ? "sfu" : "p2p"},
{"mediaActive", active_ != nullptr}, {"retiringMedia", retired_.size()},
{"policy", session_.policy().describe()},
{"inputMode", settings_.pushToTalk ? "push_to_talk" : "voice_activity"},
{"pushToTalkActive", pushToTalkActive_},
{"audioDevice", audio_.diagnostics ? audio_.diagnostics() : Json(nullptr)}};
result["microphoneAccessPending"] = microphoneRequestPending_;
result["channelId"] = session_.admission() ? Json(session_.admission()->channelId) : Json(nullptr);
Expand Down Expand Up @@ -752,14 +796,16 @@ class Application final {
if (name.starts_with("fixture-") && audio_.fixtureCommand) audio_.fixtureCommand(value);
else if (name == "set-input") selectDevice(true, value);
else if (name == "set-output") selectDevice(false, value);
else if (name == "set-input-mode") setInputMode(value);
else if (name == "ptt") pushToTalk(value.at("active").get<bool>());
else if (name == "join") session_.join(value.at("channelId").get<std::string>(), Clock::now());
else if (name == "leave") session_.leave();
else if (name == "mute") session_.setMuted(value.at("enabled").get<bool>());
else if (name == "deafen") session_.setDeafened(value.at("enabled").get<bool>());
else if (name == "reconnect") restartConnection();
else if (name != "quit") {
throw std::invalid_argument("Unknown command; use join, leave, mute, deafen, devices, set-input, "
"set-output, stats, channels, reconnect or quit");
"set-output, set-input-mode, ptt, stats, channels, reconnect or quit");
}
output({{"event", "command-accepted"}, {"id", id}, {"command", name}});
if (name == "quit") quit();
Expand Down Expand Up @@ -799,6 +845,7 @@ class Application final {
const bool outputIsConsole_ = standardOutputIsConsole();
bool microphoneAuthorized_ = false;
bool microphoneRequestPending_ = false;
bool pushToTalkActive_ = false;
CancelMicrophoneAccess cancelMicrophoneAccess_;
std::function<void(const Json&)> observer_;
int exitCode_ = 0;
Expand Down Expand Up @@ -826,6 +873,8 @@ int runHeadless(std::vector<std::string> arguments, HeadlessConfiguration audio)
"{\"command\":\"deafen\",\"enabled\":false}\n"
"{\"command\":\"devices\"} / {\"command\":\"set-input\",\"deviceId\":\"...\"|null}\n"
"{\"command\":\"set-output\",\"deviceId\":\"...\"|null}\n"
"{\"command\":\"set-input-mode\",\"mode\":\"voice_activity\"|\"push_to_talk\"}\n"
"{\"command\":\"ptt\",\"active\":true|false}\n"
"{\"command\":\"stats\"} / {\"command\":\"channels\"}\n"
"{\"command\":\"leave\"} / {\"command\":\"reconnect\"} / {\"command\":\"quit\"}\n"
"EOF and Ctrl+C release the connection and media resources.\n"
Expand Down
16 changes: 14 additions & 2 deletions apps/light/src/media/policy_audio_device.hpp
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#pragma once

#include "audio_devices.hpp"
#include "speech_activity.hpp"

#include <api/audio/audio_device.h>
#include <api/scoped_refptr.h>
Expand Down Expand Up @@ -54,6 +55,18 @@ class PolicyAudioDevice : public webrtc::AudioDeviceModule {
}
const AudioDeviceSelection &input_selection() const { return input_; }
const AudioDeviceSelection &output_selection() const { return output_; }
// Safe from any thread: loudest raw capture level since the last read.
std::uint8_t TakeInputLevel() { return meter_.Take(); }

int32_t RegisterAudioCallback(webrtc::AudioTransport *callback) override {
if (!callback) {
const auto result = native_->RegisterAudioCallback(nullptr);
meter_.SetTarget(nullptr);
return result;
}
meter_.SetTarget(callback);
return native_->RegisterAudioCallback(&meter_);
}

int32_t InitRecording() override {
return capture_allowed_ ? native_->InitRecording() : 0;
Expand Down Expand Up @@ -91,8 +104,6 @@ class PolicyAudioDevice : public webrtc::AudioDeviceModule {
#define MONKY_ADM_CONST(result, name, parameters, arguments) \
result name parameters const override { return native_->name arguments; }
MONKY_ADM_CONST(int32_t, ActiveAudioLayer, (AudioLayer * layer), (layer))
MONKY_ADM_FORWARD(int32_t, RegisterAudioCallback,
(webrtc::AudioTransport * callback), (callback))
MONKY_ADM_FORWARD(int32_t, Init, (), ())
MONKY_ADM_FORWARD(int32_t, Terminate, (), ())
MONKY_ADM_CONST(bool, Initialized, (), ())
Expand Down Expand Up @@ -232,6 +243,7 @@ class PolicyAudioDevice : public webrtc::AudioDeviceModule {
}

webrtc::scoped_refptr<webrtc::AudioDeviceModule> native_;
InputLevelMeter meter_;
AudioDeviceSelection input_;
AudioDeviceSelection output_;
// The first selection always applies, even for the default device.
Expand Down
Loading
Loading