Building a Voice Agent with LangChain

Who this is for and what you will be able to do

This guide is for developers who already build LangChain agents and want to give them a voice. By the end you should be able to tell the sandwich architecture apart from speech-to-speech models, follow the data flow of a three-stage voice agent, stream each stage, and chain the stages into a pipeline that serves a WebSocket.

What a voice agent is

Voice agents are agents that hold natural spoken conversations with users. They combine speech recognition, natural language processing, generative AI, and text-to-speech^[1]. Customer support, personal assistants, hands-free interfaces, and coaching and training are typical use cases^[1].

Every voice agent has to listen (capture audio and transcribe it), think (interpret intent, reason, and plan), and speak (generate audio and stream it back to the user)^[1].

Choosing an architecture

The sandwich architecture puts speech-to-text (STT), a text-based LangChain agent, and text-to-speech (TTS) in sequence^[1]. Each component can be controlled and swapped, and you get access to the latest capabilities of text-modality models with clear boundaries between components^[1]. The costs are that you must orchestrate several services, and converting speech to text loses information such as tone and emotion^[1].

Speech-to-speech uses one multimodal model that takes audio in and produces audio out^[1]. The architecture is simpler and typically has lower latency for simple interactions^[1]. The trade-offs are fewer model options, a greater risk of provider lock-in, less transparency about how audio is processed, and fewer options for control and customization^[1].

This guide uses the sandwich architecture. With some STT and TTS providers, the sandwich architecture can reach sub-700ms latency while keeping the components modular^[1].

The example: a sandwich-shop order taker

The example is a voice agent that manages orders for a sandwich shop. It uses AssemblyAI for speech-to-text and Cartesia for text-to-speech^[1]. The reference application is the voice-sandwich-demo repository, and its README covers installation and setup^[1].

The browser captures microphone audio, encodes it as PCM, streams the chunks to the server over a WebSocket, and plays back the synthesized speech it receives^[1]. WebSockets provide real-time, bidirectional communication. The same architecture can be adapted to telephony systems such as Twilio and Vonage, or to WebRTC^[1].

Each stage uses async generators or async iterators, so a downstream stage can start before the upstream stage finishes, which reduces end-to-end latency^[1].

Stage 1: speech-to-text

The STT stage uses a producer-consumer pattern. Audio chunks are sent to the STT service while transcript events are received, so transcription can begin before all the audio has arrived^[1].

There are two event types. stt_chunk carries partial transcripts while the service is still processing audio. stt_output carries final, formatted transcripts, and those trigger agent processing^[1]. The example keeps a persistent WebSocket open to AssemblyAI's real-time STT API, configured for 16kHz PCM audio with automatic turn formatting^[1].

Stage 2: the LangChain agent

The agent stage takes transcripts, runs them through a LangChain agent, and streams all the text content blocks the agent generates^[1]. The example defines two tools, one that adds an item to the order and one that confirms it:

# Define agent tools
def add_to_order(item: str, quantity: int) -> str:
    """Add an item to the customer's sandwich order."""
    return f"Added {quantity} x {item} to the order."
def confirm_order(order_summary: str) -> str:
    """Confirm the final order with the customer."""
    return f"Order confirmed: {order_summary}. Sending to kitchen."

The sample system prompt asks for responses that are concise and friendly, with no emojis, special characters, or markdown, because a text-to-speech engine reads the responses aloud^[1].

The agent streams with stream_events(version="v3") and stream.messages, so tokens are emitted as they are generated rather than after the complete response^[1]. A checkpointer keeps the conversation state across turns, keyed by a unique thread ID, so the agent can refer to earlier exchanges^[1].

Stage 3: text-to-speech

The TTS stage turns the agent's text into audio and streams it back to the client. It also uses a producer-consumer pattern, merging two concurrent streams. One stream passes upstream events through and sends agent text to the TTS provider. The other receives the synthesized audio chunks^[1].

Some TTS providers, such as Cartesia, start synthesizing as soon as they receive text. Playback can therefore begin before the agent has finished its whole response^[1]. All upstream events pass through this stage unchanged, so the client or other observers can track the state of the whole pipeline^[1].

Chaining the stages

The three stages are connected with RunnableGenerator:

pipeline = (
    RunnableGenerator(stt_stream)      # Audio → STT events

On the WebSocket side, incoming audio bytes feed the pipeline, and the TTS audio that comes out is sent back to the client:

    # Transform audio through pipeline
    output_stream = pipeline.atransform(websocket_audio_stream())
    # Send TTS audio back to client
    async for event in output_stream:
        if event.type == "tts_chunk":
            await websocket.send_bytes(event.audio)

Each stage processes events independently and concurrently. Transcription starts as soon as audio arrives, the agent starts as soon as a transcript is available, and speech synthesis starts as soon as agent text is generated^[1].

Tracing with LangSmith

Applications with several steps make many LLM calls, which makes problems hard to find. Set LANGSMITH_TRACING to "true" and provide LANGSMITH_API_KEY, and traces are logged to LangSmith, so each call can be inspected^[1].

Limits and common questions

• The sandwich architecture involves several services, and speech-to-text drops information such as tone and emotion^[1].
• Speech-to-speech offers fewer model choices, a higher lock-in risk, and less visibility into audio processing^[1].
• The sample system prompt forbids emojis and markdown because the output is spoken; a text-screen interface would need a different prompt^[1].

Next steps

For installation and setup, follow the README of the reference repository^[1].

Adapted from the voice agent guide in LangChain's official documentation^[1], released under the MIT License.