Hugging Face’s speech-to-speech pipeline makes local voice agents a CLI install away

Hugging Face released speech-to-speech, a modular pipeline for building voice agents entirely with open-source models. The library strings together voice activity detection, speech-to-text, a language model, and text-to-speech into a low-latency cascade. Every component is swappable. The LLM slot speaks OpenAI-compatible protocols, so developers can point it at a hosted provider, at Hugging Face Inference Providers, or at a vLLM or llama.cpp server on their own hardware for a fully local, fully open stack.

The default configuration uses Silero VAD v5 for voice activity detection, Parakeet TDT for local STT, an OpenAI-compatible LLM, and Qwen3-TTS for speech output. On a Mac, one command with the --local_mac_optimal_settings flag sets MPS acceleration, Parakeet TDT, MLX LM for the LLM, and Qwen3-TTS with mlx-audio. The pipeline runs in production as the conversation backend for thousands of Reachy Mini robots, according to the repository.

This is not an academic demonstration. The library ships as a pip install speech-to-speech package. It exposes an OpenAI Realtime-compatible WebSocket API at /v1/realtime. Any client built against the OpenAI Realtime protocol can connect. The repository includes a full event reference: input_audio_buffer.append, session.update, conversation.item.create, response.create, and response.cancel inbound, with speech start/stop, streaming transcription, audio deltas, and tool calls outbound.

The significance is not the architecture. Cascade pipelines for voice agents are well understood. The significance is the packaging, the modularity, and the fact that Hugging Face has made it trivially easy to run a fully open-source voice agent on consumer hardware. The LLM backend can be a local llama.cpp server running Gemma 4, or a hosted provider like OpenRouter or Hugging Face Inference Providers. The TTS backend supports Qwen3-TTS, Kokoro-82M, Pocket TTS, ChatTTS, and MMS TTS. The STT backend supports Parakeet TDT, Whisper through Transformers, Faster Whisper, Lightning Whisper MLX, MLX Audio Whisper, and Paraformer.

The library is opinionated in its defaults. The default LLM is gpt-5.4-mini through the OpenAI Responses API. But the design makes it easy to swap in a local model. The repository shows how to serve Gemma 4 with llama.cpp, then point the pipeline at it with --responses_api_base_url "http://127.0.0.1:8080/v1". The LLM is the most compute-intensive and highest-latency component in the pipeline, as the documentation notes. “A single forward pass through a large model can dominate end-to-end response time,” the README states. Choosing the right backend for hardware and latency budget matters.

Hugging Face is making a bet that developers want to build voice agents without vendor lock-in. The OpenAI Realtime API is a proprietary protocol. Hugging Face has built an open implementation of that protocol, then layered on the ability to swap every component. A developer can prototype against GPT-5.4-mini, then switch to a local Gemma 4 or Qwen3.5-9B server for production, without changing a line of client code. The protocol compatibility means existing OpenAI Realtime clients work out of the box.

The timing matters. Voice is becoming a primary interface for AI agents. OpenAI, Google, and Anthropic all ship voice modes for their frontier models. But those are closed, server-side implementations. A developer building a voice agent for a medical device, a factory floor, or a privacy-sensitive application cannot send audio to a third-party API. Hugging Face’s speech-to-speech library is a direct answer to that constraint. It runs on a laptop, on a robot, or on a server with no internet connection.

The library also exposes multiple run modes. The default realtime mode uses the WebSocket-based OpenAI Realtime protocol. The local mode uses the machine’s microphone and speakers directly. The websocket mode sends raw PCM audio over WebSocket for minimal custom clients. The socket mode streams raw PCM over TCP for remote server deployments. Each mode serves a different deployment scenario: prototyping, local testing, embedded devices, and distributed inference.

The choice of Qwen3-TTS as the default TTS engine is notable. Qwen3-TTS is a 1.7B-parameter model from Alibaba’s Qwen team. It supports custom voices and multiple languages. The default speaker is named “Aiden.” The GGML backend runs on CUDA 12.x on Linux and on Apple Silicon through mlx-audio. The repository includes explicit instructions for installing the correct CUDA wheel, a sign that the team expects developers to actually run this on their own machines.

The VAD component uses Silero VAD v5, which is a well-known, lightweight model for detecting speech boundaries and turn-taking. The pipeline runs each component in its own thread, connected by queues. This is a pragmatic design choice for low-latency streaming. The documentation includes a component table with 14 supported backends across VAD, STT, LLM, and TTS stages. The install commands for each optional backend are listed with their pip extras: kokoro, pocket, chattts, facebook-mms, faster-whisper, whisper-mlx, paraformer, mlx-lm.

One limitation is the dependency management. DeepFilterNet, used for optional audio enhancement in VAD, requires numpy<2 and conflicts with Pocket TTS, which requires numpy>=2. The documentation notes this conflict and advises manual installation only in environments without Pocket TTS. This is the kind of friction that emerges when a project tries to support many optional backends. It is not a dealbreaker, but it is a reminder that local AI stacks still require some system administration.

The library is open source under a permissive license. The repository includes a CONTRIBUTING.md file and a star history chart. The project has been in development long enough to accumulate citations and a Docker compose file that starts a llama.cpp server with Gemma 4 alongside the speech-to-speech TCP socket server. The Docker setup exposes ports 8080, 12345, and 12346.

What this means for AI builders is straightforward. The cost of building a voice agent has dropped to a single pip install command and a few configuration flags. The quality of the open-source components is high enough for production use, as demonstrated by the Reachy Mini robot deployment. The modular design means builders can optimize each stage independently, swapping in a faster STT model or a cheaper TTS model as needed.

The open question is whether the local voice-agent stack can match the latency and quality of the proprietary APIs. The repository’s documentation is honest about the LLM being the bottleneck. A local llama.cpp server running Gemma 4 will not match the inference speed of OpenAI’s infrastructure. But for applications where latency tolerance is measured in seconds rather than milliseconds, or where data sovereignty is a hard requirement, Hugging Face’s speech-to-speech library is now the default starting point.