How to Set Up Local Voice Control for Home Assistant Using Wyoming and Ollama

In the modern smart home ecosystem, the convenience of voice control often comes at a steep price: your privacy. For years, users have had to choose between the seamless functionality of Amazon Alexa or Google Assistant and the unsettling reality that their private conversations are being processed on distant corporate servers. Home Assistant has long been the gold standard for local automation, but until recently, achieving a “Star Trek-like” voice interface that doesn’t rely on the cloud was a complex, fragmented task. Many enthusiasts found themselves trapped between clunky, pre-programmed voice commands and the high latency of early DIY solutions.

The landscape changed dramatically with the introduction of the Year of the Voice by the Home Assistant team and the explosion of Large Language Models (LLMs) like those powered by Ollama. By leveraging the Wyoming protocol, we can now bridge the gap between high-performance local hardware and the Home Assistant interface. This setup allows for a truly private, locally hosted voice assistant that understands natural language, processes commands locally, and doesn’t share your data with big tech. In this guide, we will dive deep into how you can orchestrate Home Assistant, Wyoming, and Ollama to build a voice control system that is as smart as it is secure.

Understanding the Architecture: Home Assistant, Wyoming, and Ollama

To build a robust local voice system, it is crucial to understand how these three components interact. Home Assistant acts as the central nervous system, managing your devices and providing the user interface. However, Home Assistant itself isn’t a voice engine. This is where the Wyoming protocol comes in. Wyoming is a lightweight, standardized communication protocol designed specifically to connect Home Assistant with external voice services like Text-to-Speech (TTS), Speech-to-Text (STT), and wake word detection engines. It allows these services to run in separate containers or even on separate machines, which is vital for performance optimization.

The “brain” of our operation is Ollama. While traditional voice assistants use simple pattern matching (e.g., “turn on [device]”), Ollama allows us to run powerful Large Language Models (like Llama 3 or Mistral) locally. When you speak to your smart home, the STT engine converts your voice to text, and instead of a basic script, that text is sent to Ollama. The LLM interprets your intent, even if you don’t use exact phrasing. For example, instead of saying “Turn on the heater,” you could say “It’s a bit chilly in here,” and the LLM, integrated via the Assist pipeline, can infer that you want the climate control adjusted.

This decentralized approach provides incredible flexibility. You can run Home Assistant on a Raspberry Pi while hosting the resource-heavy Ollama instance on a more powerful desktop PC or a dedicated server with a GPU. The Wyoming protocol ensures that the audio data and text intent flow seamlessly between these devices with minimal latency, creating a unified experience that feels like a single, cohesive product.

The Prerequisites: Hardware and Software Requirements

Before diving into the configuration, you must ensure your hardware is up to the task. Local AI is computationally expensive. While you can run basic Speech-to-Text and Text-to-Speech on a Raspberry Pi 4 or 5, running a Large Language Model via Ollama requires more “oomph.” For a smooth experience, a machine with at least 16GB of RAM and a dedicated NVIDIA GPU (for CUDA acceleration) is highly recommended. If you are running on a CPU-only system, ensure it is a modern multi-core processor (like an Intel i5/i7 or AMD Ryzen 5/7) to keep response times under a few seconds.

• Home Assistant Instance: Running Home Assistant OS, Supervised, or Container (version 2023.12 or newer).
• Ollama Server: Installed on your network (can be the same machine as HA or a separate server).
• Wyoming Add-ons: Specifically Faster Whisper (STT) and Piper (TTS).
• Audio Hardware: A USB microphone, an ESP32-S3 powered voice satellite, or even your smartphone running the Home Assistant app.

On the software side, you will need the Ollama Home Assistant Integration (available via HACS or as a native integration in newer versions). This integration allows Home Assistant to use Ollama as a “Conversation Agent.” You should also have the Wyoming Integration installed within Home Assistant to manage the communication with your STT and TTS services. This modular setup ensures that if you want to upgrade your STT engine later, you don’t have to rebuild your entire voice pipeline.

Setting Up Ollama for Natural Language Processing

The first practical step is getting Ollama ready to act as your home’s interpreter. Once Ollama is installed on your server, you need to pull a model that balances speed and intelligence. For most home automation tasks, llama3:8b or mistral are excellent choices. They are small enough to run quickly but smart enough to handle complex logic. You can download these by running ollama pull llama3 in your terminal.

Once the model is downloaded, you need to expose Ollama’s API to Home Assistant. By default, Ollama listens on localhost:11434. If Ollama is on a different machine than Home Assistant, you must set the environment variable OLLAMA_HOST=0.0.0.0 to allow external connections. In Home Assistant, go to Settings > Devices & Services > Add Integration and search for “Ollama.” Enter the IP address and port of your Ollama server. During the setup, you will be asked to provide a “System Prompt.” This is vital—it tells the AI how to behave. A good prompt is: “You are a helpful smart home assistant. You have access to the following devices. Keep your answers concise and focus on controlling the home.”

After the integration is configured, you’ll see Ollama listed as a Conversation Agent. This is a transformative moment for your smart home. Unlike the default “Assist” agent, which relies on strict sentence matching, the Ollama agent can handle nuance. It can resolve ambiguities, understand context from previous commands, and even handle requests that involve multiple steps, provided you’ve configured your “exposed entities” correctly in Home Assistant’s voice settings.

Configuring the Wyoming Protocol Services

With the brain (Ollama) ready, we need the “ears” and “mouth.” This is achieved through the Wyoming services: Faster Whisper for Speech-to-Text and Piper for Text-to-Speech. If you are using Home Assistant OS, these are available as Add-ons. If you are using Docker, you will need to run the respective Wyoming containers. Faster Whisper is particularly impressive because it uses OpenAI’s Whisper model but is optimized to run much faster on local hardware.

In the Faster Whisper configuration, you can choose model sizes: tiny, base, small, medium. For a local setup, “tiny” or “base” is usually sufficient and offers the lowest latency. Piper, on the other hand, provides incredibly natural-sounding voices that are generated locally. Unlike old-school robotic TTS, Piper uses neural models to produce human-like intonation. Once these add-ons are running, you must add the Wyoming Integration in Home Assistant. It will auto-discover the Whisper and Piper services on your network, effectively linking your audio processing pipeline.

The beauty of the Wyoming protocol is its efficiency. It handles the streaming of audio data in chunks, meaning the STT engine can start transcribing before you’ve even finished speaking. This reduces the perceived “lag” that often plagues local voice control systems. By offloading these tasks to Wyoming-compatible containers, you ensure that Home Assistant’s core process remains snappy and responsive, even when the AI is working hard to transcribe a long sentence.

Building the Assist Pipeline

The “Assist Pipeline” is where all these components merge into a single workflow. In Home Assistant, navigate to Settings > Voice Assistants. Here, you will create a new pipeline. You will be prompted to select your STT, TTS, and Conversation Agent. For STT, select Faster Whisper; for TTS, select Piper; and for the Conversation Agent, select Ollama.

This configuration defines the sequence of events when you trigger a voice command:

1. Wake Word: (Optional) A local engine like openwakeword detects your trigger phrase (e.g., “Hey Jarvis”).
2. STT: Faster Whisper listens to your speech and converts it to a text string.
3. Intent Processing: The text is sent to Ollama. Ollama looks at the devices you’ve “exposed” to it and decides what action to take.
4. Action: Home Assistant executes the command (e.g., toggling a light).
5. TTS: Piper converts the AI’s text response into audio and plays it back to you.

One critical tip for the Assist Pipeline: Entity Exposure. By default, Home Assistant might not share all your devices with the Conversation Agent. You must go to the Voice Assistant settings and explicitly “Expose” the lights, switches, and thermostats you want Ollama to control. Be selective; exposing 500 entities to an LLM can sometimes lead to confusion or slower processing times. Start with the most used devices and expand as you refine your system prompts.

Deploying Voice Satellites for Whole-Home Coverage

A voice assistant is only useful if it can hear you. While you can use the Home Assistant mobile app, the ultimate goal is dedicated hardware in every room. This is where Voice Satellites come in. These are small, inexpensive devices—often based on the ESP32-S3 chip—that run a minimal firmware designed to stream audio to your Wyoming server. The Atom Echo or the ESP32-S3-BOX-3 are popular choices for this purpose.

Using ESPHome, you can flash these devices with a voice-assistant configuration. Once powered on, they connect to Home Assistant via the API. Because the heavy lifting (STT/TTS/LLM) is done on your server, these tiny satellites don’t need much power. They simply listen for a wake word, stream the audio to the Wyoming integration, and play back the response through a small speaker. This mimics the functionality of an Echo Dot but keeps every bit of audio data within your local network.

If you prefer a more “pro” DIY approach, you can use an old Raspberry Pi with a high-quality USB microphone array (like the ReSpeaker). By installing the wyoming-satellite software on the Pi, you can turn any Linux-based device into a high-end microphone for your Home Assistant setup. This is particularly useful for large living rooms where a tiny ESP32 might struggle to pick up voices from across the room.

Common Mistakes and Performance Considerations

Setting up local voice control is rewarding, but there are several pitfalls that can lead to a frustrating experience. The most common issue is Latency. If your AI takes 10 seconds to respond, you’ll likely stop using it. Latency usually stems from underpowered hardware running Ollama or Faster Whisper. If you’re seeing slow response times, try using a smaller model (like tiny for STT) or moving Ollama to a machine with a GPU. Network congestion can also play a role, especially if you are streaming high-quality audio over a weak 2.4GHz Wi-Fi connection.

• Conflicting Wake Words: Having multiple satellites too close together can lead to “double triggers.” Use the Assist settings to manage which satellite takes priority.
• Over-Exposing Entities: Giving the LLM access to every single sensor and automation can “confuse” it. Only expose entities that actually need voice control.
• Vague System Prompts: If Ollama starts hallucinating or giving long-winded answers, refine your system prompt to demand brevity and strict adherence to home control.
• Microphone Quality: Many users blame the software when the culprit is a poor-quality microphone that captures too much background noise. Use microphone arrays with built-in noise cancellation (AEC) whenever possible.

Another consideration is Privacy vs. Utility. While this setup is local, you must ensure your Home Assistant instance itself is secure. If you use remote access (like Nabu Casa or a VPN), ensure you have strong 2FA enabled. The power of a local LLM is that it knows a lot about your home’s state; you don’t want that information accessible to unauthorized users. Also, remember that local LLMs are “stateless” by default in some configurations—they might not remember what you said two minutes ago unless you configure a memory buffer within the integration.

Conclusion: The Future of the Private Smart Home

Setting up local voice control using Wyoming and Ollama represents a significant milestone for the smart home enthusiast. We have moved past the era of simple voice commands and into a time where our homes can truly understand us without compromising our personal data. By combining the flexibility of Home Assistant, the efficiency of the Wyoming protocol, and the raw intelligence of Ollama’s Large Language Models, you can create a bespoke AI assistant tailored specifically to your family’s needs.

The transition from cloud-dependent systems to a fully local stack is not just about privacy; it’s about reliability. Your voice control will no longer break just because your internet connection is down or a corporate server is undergoing maintenance. As you continue to refine your setup, consider experimenting with different LLM models or custom ESPHome satellite designs. The next logical step is to explore “Local Vision” (using LLMs to describe security camera feeds) or integrating more complex logic through Home Assistant Blueprints. The foundation you’ve built today with Wyoming and Ollama is the gateway to a truly intelligent, autonomous, and private home of the future.