Voice Chat

Voice chat provides push-to-talk speech input and audio playback for AI agent conversations. The frontend captures microphone audio, streams it over WebSocket to the background-tasks voice service, receives transcripts, and sends them as chat messages through the existing agentChat pipeline.

Architecture

useVoiceChat Hook

File: lib/hooks/useVoiceChat.ts

Usage

const {
  isConnected,
  isRecording,
  isSpeaking,
  sessionId,
  error,
  connect,
  disconnect,
  startRecording,
  stopRecording,
  synthesize,
  stopSpeaking,
} = useVoiceChat({
  conversationId,
  language: "en-US",
  voiceId: "tiffany",
  onTranscript: (text, isPartial) => { /* handle transcript */ },
  onAudioReceived: (audioData, durationMs) => { /* handle TTS audio */ },
  onError: (error) => { /* handle error */ },
  onSessionStarted: (sessionId) => { /* session ready */ },
  onSessionEnded: () => { /* session closed */ },
})

Options

Option	Type	Required	Description
conversationId	string | null	Yes	Agent conversation ID for the WebSocket URL
language	VoiceLanguage	No	Language code (default: "en-US")
voiceId	VoiceId	No	Voice for TTS (default: "tiffany")
onTranscript	(text, isPartial) => void	No	Called when transcript received
onAudioReceived	(audioData, durationMs) => void	No	Called when TTS audio received
onError	(error: VoiceError) => void	No	Called on any error
onSessionStarted	(sessionId) => void	No	Called when voice session starts
onSessionEnded	() => void	No	Called when voice session ends

State

State	Type	Description
isConnected	boolean	WebSocket connection is open
isRecording	boolean	Microphone is actively capturing audio
isSpeaking	boolean	TTS audio is playing back
sessionId	string | null	Active voice session ID from server
error	VoiceError | null	Last error ({ code, message })

Methods

Method	Returns	Description
connect()	Promise<void>	Open WebSocket and send start session message
disconnect()	void	Stop recording (sends end), close WebSocket
startRecording()	Promise<void>	Request microphone, begin streaming PCM audio
stopRecording()	void	Stop mic capture, send end to trigger final transcript
synthesize(text)	void	Send TTS request over WebSocket
stopSpeaking()	void	Clear audio playback queue

Callbacks

Callback	Triggered When
onTranscript(text, isPartial)	Speech-to-text result received (partial or final)
onAudioReceived(audioData, durationMs)	TTS audio response ready for playback
onSessionStarted(sessionId)	Voice session created on server
onSessionEnded()	WebSocket connection closed
onError(error)	Any error occurs (connection, microphone, server)

Error Codes

Code	Source	Description
NO_CONVERSATION	Client	No conversationId provided
CONNECTION_ERROR	Client	WebSocket connection failed
CONNECTION_FAILED	Client	Exception during connect attempt
MICROPHONE_ERROR	Client	getUserMedia failed (permissions denied, no mic)
TRANSCRIPTION_FAILED	Server	STT transcription failed
SYNTHESIS_FAILED	Server	TTS synthesis failed
BILLING_ERROR	Server	Insufficient credits or inactive subscription
VOICE_DISABLED	Server	Voice service is disabled
SERVICE_UNAVAILABLE	Server	Voice service health check failed

Supported Languages

Code	Language	Code	Language
en-US	English (US)	es-ES	Spanish
en-GB	English (UK)	de-DE	German
en-IN	English (India)	fr-FR	French
en-AU	English (Australia)	it-IT	Italian
pt-BR	Portuguese (Brazil)	hi-IN	Hindi

Voices

Two voice options: tiffany (female, default) and matthew (male). These map to language-specific Amazon Polly neural voices on the backend — see Voice Service for the full mapping.

ChatModal Integration

File: components/reusable/chatbot/ChatModal.tsx

Voice Mode Toggle

The ChatModal has a voice mode toggle button that:

1. Requires conversationId to be set (disabled until conversation initializes)
2. Connects to the voice WebSocket when enabled
3. Disconnects when disabled or when the modal closes

Push-to-Talk Flow

Voice mode uses a push-to-talk pattern controlled by the mic button:

Transcript Handling

The onTranscript callback in ChatModal handles two cases:

• Partial transcripts (isPartial: true): Displayed in the textarea as visual feedback while the user speaks. NOT sent as messages.
• Final transcripts (isPartial: false): Automatically sent as a chat message via handleSendMessage(). This triggers the agentChat subscription, which handles LLM processing, RAG, and tool calling.

Voice Error Handling

When the backend sends TRANSCRIPTION_FAILED, the ChatModal:

1. Shows a warning toast: "Voice not available — Speech recognition failed"
2. Automatically disables voice mode (setIsVoiceMode(false))
3. Falls back to text input

Other errors show a generic toast with the error message.

Audio Processing

Microphone Capture

The hook uses the Web Audio API to capture and process microphone audio:

1. getUserMedia — Requests mono audio with echo cancellation, noise suppression, and auto gain control
2. AudioContext — Created at the browser's native sample rate (typically 44.1kHz or 48kHz)
3. ScriptProcessorNode — Processes audio in 4096-sample buffers (~93ms at 44.1kHz)
4. Downsampling — Converts from browser sample rate to 16kHz using linear interpolation
5. PCM conversion — Converts Float32 samples to 16-bit signed integer PCM
6. WebSocket send — Binary PCM data sent directly (no JSON wrapping, no base64)

note

ScriptProcessorNode is deprecated in favor of AudioWorkletNode. A future update should migrate to AudioWorklet for better performance and to avoid running audio processing on the main thread.

Audio Playback (TTS)

When TTS audio arrives from the server:

1. Base64-encoded PCM is decoded to an ArrayBuffer
2. Audio is queued in audioQueueRef for sequential playback
3. Each chunk creates a new AudioContext at 16kHz
4. Int16 PCM is converted to Float32 for the Web Audio API
5. An AudioBufferSourceNode plays the audio
6. The isSpeaking state is true while the queue is being processed

Cleanup

The hook cleans up resources on:

• stopRecording(): Disconnects ScriptProcessor, closes AudioContext, stops MediaStream tracks, sends "end" to WebSocket
• disconnect(): Calls stopRecording(), then closes WebSocket
• Component unmount: Stops all audio resources and sends "end" before closing WebSocket

Environment Variables

Variable	Purpose
NEXT_PUBLIC_API_URL	Backend HTTP/WSS base URL — voice WS path is derived as ${apiUrl.replace(/^http/, "ws")}/voice/ws/<conversationId> and proxied through the NestJS AI gateway. The legacy NEXT_PUBLIC_AI_VOICE_WS_URL env var was removed in the Phase 3.7 cutover.

Related

• Voice Service (Backend) — WebSocket protocol, billing, AWS services
• AI Agents — Agent system and agentChat subscription
• AI Service API — AI service GraphQL operations