TTS / Audio System

🚨 Start here — the four things that are easy to get wrong

speak() is the ONE entry point (features/audio/service/speak.ts). Never hand-roll a TTS call, never construct a player, never read a voice preference yourself. React surfaces use useSpeech() on top of it. To let someone HEAR a catalog model's voice (e.g. a speech-script speaker), call speak({ text, sample: { model, voice } }) — the catalog lane plays the vendor's own sample (ElevenLabs) or a line the server renders once per (model, voice) via POST /audio/voice-preview.
Voice / speed / language come from the tiered listening config (features/audio/service/listeningConfig.ts) — system → org → user, user wins. Never read userPreferences.voice.* for playback; those fields survive only as the pre-fetch boot fallback. See the tiered-config section below.
Audio output must be unlocked inside a user gesture or iOS plays silence. Call primeAudioOutput() (features/audio/unlock.ts) synchronously in any handler that will later start audio. See the iOS section below.
One voice at a time, app-wide. playbackLock arbitrates; the playbackQueue + audioSessionRegistry own what is playing and its history. Producing audio without a session trips the runtime bypass guard.

Architecture Overview

Two independent TTS engines, one STT engine, and a voice assistant pipeline.

TTS Engines

Engine	Provider	Transport	Primary Hook	API Route
Cartesia	Cartesia API	WebSocket (PCM F32LE, 44.1kHz)	`useCartesiaSpeaker`	`GET /api/cartesia`
Groq/PlayAI	Groq SDK	REST → WAV blob	`useTextToSpeech`	`POST /api/audio/text-to-speech`

STT Engine

Route	Provider	Auth	Max Size
`POST /api/audio/transcribe`	Groq Whisper	Supabase cookie/Bearer	4.5 MB
`POST /api/audio/transcribe-url`	Groq Whisper	Supabase cookie/Bearer	100 MB (URL)
`POST /api/audio/log-error`	Supabase	Supabase cookie/Bearer	N/A

Auth

All audio API routes use resolveUser from utils/supabase/resolveUser.ts — dual-mode: Supabase session cookie OR Bearer token. Both TTS routes (/api/cartesia, /api/audio/text-to-speech) require authentication.

Directory Map

features/tts/              ← Production TTS feature (Cartesia + Groq)
  hooks/
    useCartesiaSpeaker.ts   ← PRIMARY Cartesia hook (lazy, self-contained)
    useTextToSpeech.ts      ← Groq/PlayAI hook (REST, WAV blob)
  components/
    SpeakerButton.tsx       ← Single play/pause toggle (lazy-loads core)
    (SpeakerButton itself lives in @ai-matrx/media/react; this file re-exports it)
    SpeakerGroup.tsx        ← 3-button group shell (Play, Pause, Stop)
    SpeakerGroupCore.tsx    ← Dynamically imported 3-button core
    SpeakerCompactGroup.tsx ← 2-button group shell (Play/Pause, Stop)
    SpeakerCompactGroupCore.tsx
    AudioPlayerButton.tsx   ← Groq/PlayAI button (uses useTextToSpeech)
  types.ts                  ← Shared types (EnglishVoice, SpeakerVariant, etc.)

hooks/tts/                  ← Legacy/specialized hooks
  simple/
    useCartesiaControls.ts  ← Full controls (connect-on-mount, script state)
    useCartesiaWithPreferences.ts ← Redux voice prefs + connect-on-mount
    useSimpleCartesia.ts    ← Minimal Cartesia (connect-on-mount)
  useCartesia.ts            ← Legacy full-featured (direct API key, eager)
  usePlayer.ts              ← Raw PCM stream player (Web Audio API, 24kHz)
  usePlayerSafe.ts          ← Enhanced usePlayer with explicit init
  useVoiceChat.ts           ← Voice assistant pipeline (VAD + STT + AI + TTS)
  useVoiceChatCdn.ts        ← CDN variant of voice chat
  useVoiceChatWithAutoSleep.ts ← Auto-sleep extension

features/audio/             ← THE modern audio system (start here, not features/tts/)
  service/
    speak.ts                ← 🚨 THE app-wide "turn text into audio" entry point
    useSpeech.ts            ← React face of speak() (per-surface status)
    listeningConfig.ts      ← 🚨 tiered voice/speed/language (system→org→user)
    useListeningSettings.ts ← settings-pane face; update() writes MY tier
    engines.ts              ← AV engine registry (voice lists live here ONCE)
  unlock.ts                 ← 🚨 iOS/WebKit gesture unlock (see iOS section)
  activation.ts             ← one-way latch that mounts the lazy audio system
  playback/
    playbackQueue.ts        ← the single app-wide queue (one at a time)
    playbackLock.ts         ← app-wide single-output arbiter (start-always-wins)
    AudioPlaybackHost.tsx   ← queue → Redux mirror; installs unlock listeners
    adapters/cartesiaAdapter.ts  ← streaming PCM lane (SinkAwarePlayer)
    adapters/catalogAdapter.ts   ← server-catalog lane (HTMLAudioElement)
  session/
    audioSessionRegistry.ts ← ALL audio activity (in + out, live + history)
    usePlaybackSessionController.ts / useMediaElementPlaybackSession.ts
    bypassGuard.ts          ← screams when audio is produced with no session
  sinkAwarePlayer.ts        ← our WebPlayer fork; output-device routing
  hooks/ components/ services/ voice/   ← recording, mic UI, voice selection UI

features/window-panels/windows/listen/
  ListenSummaryWindow.tsx   ← the Listen panel (summarize-for-listening player)

providers/AudioSystemHost.tsx        ← the ONE ssr:false boundary (lazy mount)
providers/AudioOutputHostImpl.tsx    ← app-root streaming speaker owner

lib/cartesia/               ← Low-level Cartesia client & types
  client.ts                 ← LEGACY: direct NEXT_PUBLIC_CARTESIA_API_KEY
  cartesia.types.ts         ← Full Cartesia type definitions
  voices.ts                 ← re-export of the one voice catalog (@ai-matrx/media/voices)

app/api/cartesia/route.ts   ← Token endpoint (authenticated)
app/api/audio/              ← TTS + STT API routes

utils/supabase/resolveUser.ts  ← Shared auth resolution
@ai-matrx/media/voices parseMarkdownToText ← MD → plain text

Hook Selection Guide

Adding TTS to a new component? Use useCartesiaSpeaker or a Speaker* component.

Need	Solution
Single play/pause button	`<SpeakerButton text={content} />`
3-button group (Play/Pause/Stop)	`<SpeakerGroup text={content} />`
2-button group (toggle + Stop)	`<SpeakerCompactGroup text={content} />`
Programmatic TTS with Redux prefs	`useCartesiaSpeaker({ processMarkdown: true })`
Groq/PlayAI TTS (non-Cartesia)	`useTextToSpeech()` or `<AudioPlayerButton text={content} />`
Full voice config UI (demo/playground)	`useCartesiaControls()` or `useSimpleCartesia()`
Voice assistant with VAD	`useVoiceChat()` or `useVoiceChatWithAutoSleep()`

Why multiple hooks exist

The legacy hooks (useCartesiaControls, useSimpleCartesia, useCartesia) connect eagerly on mount and manage their own voice/emotion/speed state. They exist for playground and demo pages where the user needs full control over Cartesia parameters.

useCartesiaSpeaker is the production hook: lazy (does nothing until speak() is called), reads voice preferences from Redux, and has proper error handling. Always prefer it for new features.

Speaker Component Contracts

All Speaker* components follow the same pattern:

Thin shell renders static disabled buttons (zero JS loaded)
First click triggers React.lazy() → dynamically imports Core
Core initializes useCartesiaSpeaker and begins playback
Shape never changes — buttons are always rendered, unavailable actions are disabled

Props (all variants)

interface SpeakerProps {
  text: string;           // Content to speak
  processMarkdown?: boolean; // Strip markdown (default: true)
  className?: string;     // Applied to outer container
  disabled?: boolean;     // Disable all buttons
}

// SpeakerButton also accepts:
variant?: 'glass' | 'transparent' | 'solid' | 'group';

Icon System

Speaker buttons use raw SVG icons from @ai-matrx/tap-target/buttons, NOT Lucide. Available: PlayTapButton, PauseTapButton, StopTapButton, Volume2TapButton. (Formerly components/icons/tap-buttons.tsx, deleted in the 2026-08-30 C9 package adoption.)

Styling

Uses TapTargetButton / TapTargetButtonGroup from the @ai-matrx/tap-target package (C9 adoption, 2026-08-30 — the old in-repo copy is gone). Glass styling via matrx-glass CSS classes (globally available in app/globals.css).

API Contracts

GET /api/cartesia

Auth: Required (cookie or Bearer) Response: { token: string } — short-lived Cartesia access token Errors: 401 (no auth), 500 (token generation failed)

POST /api/audio/text-to-speech

Auth: Required (cookie or Bearer) Body: { text: string; voice?: EnglishVoice; model?: string } Response: audio/wav binary (Cache-Control: 1 year) Limits: text max 10,000 chars, voice must be valid PlayAI voice Errors: 400 (validation), 401 (no auth), 429 (rate limit), 500

POST /api/audio/transcribe

Auth: Required Body: multipart/form-data with file (audio), optional language, prompt Response: { success, text, language, duration, segments, _meta: { attempts } } Limits: 4.5 MB file size, allowed types: flac/mp3/mp4/mpeg/mpga/m4a/ogg/wav/webm Retries: 3 with exponential backoff, logged to audio_transcription_errors

POST /api/audio/transcribe-url

Auth: Required Body: { url: string; language?; prompt? } — URL must be on the route's allowlist (Python backend tiers + AWS S3 hostnames; signed cld_files URLs qualify) Response: Same as /transcribe Limits: 100 MB via Groq URL parameter

Voice / speed / language — the tiered listening config (2026-08-28)

🚨 Cartesia voice, speed, and language NO LONGER live in userPreferences.voice for playback. They resolve through the tiered surface-config listening namespace (system → org → user, user wins) via features/audio/service/listeningConfig.ts:

React reads: selectListeningVoice/Speed/Language (+ selectListeningVoiceId(state, purpose)).
Imperative reads (speak(), playback adapters): getListeningSettings() — resolves at START time so replays honor current settings.
Writes: useListeningSettings().update(patch) — ONE row at the user's tier (never a whole-preferences write).
System default: platform-global row, admin-edited at /administration/ui/surfaces/matrx-user/assistant-message (Config namespaces). Org rows override; user rows win.

userPreferences.voice.{voice,speed,language} remain ONLY as the pre-fetch boot fallback; emotion/wakeWord/microphone/speaker still live there.

Redux Voice Preferences (legacy fields — see above for voice/speed/language)

// state.userPreferences.voice (Cartesia)
interface VoicePreferences {
  voice: string;    // LEGACY for playback — boot fallback only
  language: string; // LEGACY for playback — boot fallback only
  speed: number;    // LEGACY for playback — boot fallback only
  emotion: string;
  microphone: boolean;
  speaker: boolean;
  wakeWord: string;
}

// state.userPreferences.textToSpeech (Groq/PlayAI)
interface TextToSpeechPreferences {
  preferredVoice: GroqTtsVoice; // e.g. "Cheyenne-PlayAI"
  autoPlay: boolean;
  processMarkdown: boolean;
}

🚨 iOS / WebKit — the output-unlock law (2026-08-30)

Every browser on iPhone is WebKit — Chrome and Firefox included — so these rules are not "a Safari edge case", they are the whole mobile platform.

Two WebKit behaviors made all mobile audio silent, with no error:

An AudioContext created (or resumed) outside a user gesture starts suspended and stays silent. Our TTS starts audio from websocket callbacks seconds after the tap, so per-utterance contexts were always suspended.
Web Audio is muted by the ringer/silent switch unless the page declares navigator.audioSession.type = "playback" (iOS 16.4+).

features/audio/unlock.ts is the fix and the only place this logic lives:

installAudioUnlockListeners() — capture-phase pointerdown/keydown/ touchend listeners; mounted ONCE by AudioPlaybackHost. Unlocks on the user's first interaction anywhere, so the page is usually already unlocked before any audio is requested.
primeAudioOutput() — call this synchronously in every handler that will later start audio. Idempotent and cheap. Already called by enqueuePlayback / resumePlayback / playPlaybackItem, both Listen actions, and the Listen panel transport.
getUnlockedAudioContext() / getPrimedMediaElement() — the shared, gesture-unlocked handles. SinkAwarePlayer schedules into the shared context when it exists (per-utterance GainNode; never closes it); catalogAdapter plays through the shared element.

Rules when you touch audio:

Adding a new "start audio" button? Call primeAudioOutput() in the handler.
Never mint a bare new AudioContext() for playback — go through the queue.
Never close the shared context (stop() in shared mode tears down the utterance only).
A context that will not reach running throws a user-facing error ("The browser blocked audio output — tap the play button to start sound"). Keep it that way: silence with no error is the bug this replaced.

The Listen panel — summarize-for-listening

features/window-panels/windows/listen/ListenSummaryWindow.tsx (overlay listenSummaryWindow). Two entry points, both in one "Listen" context-menu submenu and in the assistant action bar's ⋯ menu:

Summarize without playing — summary streams in as text; user presses Play.
Summarize & listen — stream-to-stream: the summary is spoken as it is written, via the app-root speaker (voicePlaybackBus request with includeActive: true).

The summarizing agent is resolved from the spoken_summary surface role, falling back to the platform home surface (LISTENING_HOME_SURFACE) whose ambient.spoken_summary mandate holds the default agent — so it works on every surface for every user with no personal binding. Never put an agent UUID in code here. Menu wiring: @ai-matrx/context-menu (listen submenu role) + messageActionRegistry.listeningItems.

Known Deferred Issues

These are documented design decisions, not bugs:

Five Cartesia hook variants — Consolidation into one hook requires updating ~15 consumer files. Deferred as separate effort. Use useCartesiaSpeaker for all new code.
lib/cartesia/client.ts leaks API key — Uses NEXT_PUBLIC_CARTESIA_API_KEY. Legacy hook useCartesia depends on it. Will be removed when legacy hook is consolidated.
No connection health monitoring — No heartbeat/ping. Silent WebSocket drops discovered only on next speak().
No playback progress for Cartesia — the catalog (element) lane tracks duration/currentTime; the streaming PCM lane does not.
Token expiry — Cartesia token fetched once, no refresh. If it expires mid-session, connectCartesiaTts refreshes and retries once automatically.

Two entries were REMOVED on 2026-09-08 because they became false — do not reinstate them from an old copy of this file: "no global TTS instance management" (the playbackQueue + playbackLock + audioSessionRegistry trio has been the single owner since the unified-audio work) and "no abort for in-flight TTS" (skipPlayback / clearPlayback / a session's stop control all abort; playbackLock preempts across paths).

Adding a New Speaker Variant

Create SpeakerNewVariant.tsx (thin shell) in features/tts/components/
Create SpeakerNewVariantCore.tsx (default export, uses useCartesiaSpeaker)
Use React.lazy() in shell to import core
Use TapTargetButton* from @ai-matrx/tap-target (icons: @ai-matrx/tap-target/buttons)
Export from features/tts/components/index.ts
Never hide buttons — disable unavailable actions
Never change component shape during state transitions

Modifying API Routes

All audio API routes use resolveUser from utils/supabase/resolveUser.ts. When adding new routes:

Import and call resolveUser(request) first
Return 401 if !user
Use structured error responses with code field
For Groq routes: implement retry with exponential backoff
Log errors to audio_transcription_errors via logTranscriptionError()

tts-audio-system

Files2

SKILL.md.claude/skills/tts-audio-system/SKILL.md