Voice OS

Executive summary

One microphone and one voice for the whole application: streaming recognition for notes and charting, spoken answers, and a speech-to-speech assistant that can also see through the intraoral camera. A single ownership registry decides who holds the microphone, and spoken output always yields to clinical capture. Recognition and synthesis are production; the live speech-to-speech assistant and its camera lane are in pilot behind per-user grants.

Overview

ProductionCapture · Voice OS
Give every module the same speech input and output, so the clinician never manages microphones per screen and nothing the app says can talk over a recording.

Pipeline

A module asks the microphone ownership registry for the mic. Clinical capture (Scribe, Perio, dictation) outranks the command listener; spoken output never acquires the mic at all (VOUT-01).
Audio streams as PCM over a ticket-authenticated WebSocket to the server, which forwards it to an EU speech-recognition endpoint and streams partial and final transcripts back.
Short utterances outside a capture pass a semantic intent classifier that maps them to app actions or to Copilot.
Answers are spoken through the voice output bus, which arbitrates against active capture and stops mid-sentence when a capture claims the microphone (VOUT-02/03).
Live mode opens a speech-to-speech session bridged to Vertex AI. Tools let the model read the chart, open views and ask the clinical lane for evidence-backed answers; camera frames stream alongside the audio at 1 fps idle and 2 fps while the clinician speaks.

Mic ownership registry
one owner at a time
Clinical capture
Scribe · Perio · dictation
Command listener
intents · wake
Live session
speech-to-speech · camera

Streaming STT
EU endpoints · fallback
Semantic intent
lite-tier classifier
Live bridge
Vertex AI EU

Voice output bus
yields to capture
Speech synthesis
streaming TTS
Copilot clinical lane
evidence-backed answers

Connections (6)

Mic ownership registry → Clinical capture
Clinical capture → Streaming STT
Command listener → Semantic intent
Live session → Live bridge
Live bridge → Copilot clinical lane · tool call
Speech synthesis → Voice output bus

Voice OS. The ownership registry is the single arbiter; spoken output observes it and never claims the microphone.

Data owned

Listener leases and capture leases
Holds
Which clinician and device hold listening or capture, with hand-offs between devices
Live voice proposals
Holds
Actions the live assistant proposed and whether they were confirmed
Voice dispatch assertions
Holds
What a spoken command was resolved to, for audit
Rollout grants
Holds
Per-user and per-clinic enablement for semantic intent, synthesis and live mode

Store	Holds
Listener leases and capture leases	Which clinician and device hold listening or capture, with hand-offs between devices
Live voice proposals	Actions the live assistant proposed and whether they were confirmed
Voice dispatch assertions	What a spoken command was resolved to, for audit
Rollout grants	Per-user and per-clinic enablement for semantic intent, synthesis and live mode

Models and engines

Streaming recognition
Model or engine
Soniox real-time (EU), Google Cloud Speech as circuit-breaker fallback
Notes
Clinician-only diarized mode refuses the fallback, which cannot diarize
Batch recognition
Model or engine
Soniox async (EU)
Notes
Unbound dictation recorded before a patient is identified
Speech synthesis
Model or engine
Soniox TTS; ElevenLabs as alternative
Notes
Browser speech may speak only fixed phrase keys, never free text (VOUT-09)
Semantic intent
Model or engine
Gemini 3 Flash-Lite (lite tier)
Notes
Short classification calls, no clinical content
Live speech-to-speech and camera
Model or engine
Gemini 3 Live via Vertex AI EU
Notes
No AI Studio path; the bridge refuses non-Vertex upstreams

Step	Model or engine	Notes
Streaming recognition	Soniox real-time (EU), Google Cloud Speech as circuit-breaker fallback	Clinician-only diarized mode refuses the fallback, which cannot diarize
Batch recognition	Soniox async (EU)	Unbound dictation recorded before a patient is identified
Speech synthesis	Soniox TTS; ElevenLabs as alternative	Browser speech may speak only fixed phrase keys, never free text (VOUT-09)
Semantic intent	Gemini 3 Flash-Lite (lite tier)	Short classification calls, no clinical content
Live speech-to-speech and camera	Gemini 3 Live via Vertex AI EU	No AI Studio path; the bridge refuses non-Vertex upstreams

Guardrails

One microphone owner at a time, enforced by the registry; spoken output observes and never participates.
Spoken output stops the instant clinical capture claims the microphone, and reports that stop as its own operational signal.
Actions that change clinical state require the confirmation step that action defines; held commands make no write until confirmed.
WebSocket sessions authenticate with 60-second tickets in the subprotocol header and are re-checked against revocation every 15 seconds.
Speech providers are reached only on EU endpoints; failover between engines is supervised and visible in telemetry.

Rollout and status

ProductionStreaming recognition and synthesis
Used by Scribe, Perio and Copilot voice
PilotLive speech-to-speech
Enabled in production but admits only allow-listed users or roles
PilotLive camera (Vision) lane
Frames at 1–2 fps with ROI-weighted sharpness selection, inside a live session
PlannedSpoken confirmation of live actions
ADR 0030 is proposed, not accepted
PlannedSpoken-output review
Implemented with a shadow mode; off in production

Interfaces

Consumes
What
Copilot clinical lane (answers to spoken clinical questions)
Consumes
What
Chart and workspace state (live tools that read the app)
Emits
What
Transcripts to Scribe, Perio and dictation
Emits
What
Resolved intents to workspace navigation and Copilot
Emits
What
Proposals and dispatch assertions to audit

Direction	What
Consumes	Copilot clinical lane (answers to spoken clinical questions)
Consumes	Chart and workspace state (live tools that read the app)
Emits	Transcripts to Scribe, Perio and dictation
Emits	Resolved intents to workspace navigation and Copilot
Emits	Proposals and dispatch assertions to audit

Invariants

VOUT-01…10 — spoken output vs. clinical capture
ADR 0026 one voice; ADR 0027 one microphone; ADR 0031 voice speaks by default

Known limits

Wake-word and several live action lanes are implemented behind flags that default off.

SOURCE OF TRUTH

server/services/voiceS2s/
server/services/speechToText/
server/services/textToSpeech/
src/services/audio/
docs/clinical-invariants/voice-output.md
docs/adr/0027-faurenis-voice-os-one-microphone.md