Over the last few years, we've seen huge advances in voice technology as it's now significantly more reliable and able to understand a wide variety of accents.

We've moved on from the clunky voice assistants of the last decade, where you had to speak slowly and just hope that it understood what you said. Today's voice agents are much more capable, as they're able to hold proper conversations and complete tasks for us - which is incredibly exciting to see.

In one weekend, I built an AI coach that helps me prepare for upcoming interviews, think through the big decisions out loud, and explore ideas in greater depth. In this guide, I'll cover exactly how I did it, what voice agents actually are under the hood, and how you can build your own.

Special thanks to AssemblyAI, who are sponsoring this post and have just launched their new realtime speech-to-text model. I've put the model through its paces and included my honest experience of voice agents - what worked, where it struggled, and where the real opportunities are for businesses.

This guide will walk you through the entire process, so don't worry if you're not technical. AssemblyAI are also offering Loop readers $50 in free credits, which is more than enough to build your first voice agent.

Since the technology has improved so much in recent years, there's a growing list of use cases that simply weren't possible before. Whether it's for your own personal work or business, here are some use cases that work well:

Personal work:

Business workflows:

In the rest of this guide, I'll walk you through how I built my own personal coach, and how you can do the same. I've written it for anyone, regardless of how technical you are - so don't worry if voice agents are new for you.

I always like to get a second person's opinion and have my ideas challenged, but people are busy and it's not something that can always be done. So I built an AI coach that helps me to explore different ideas, prepare for upcoming presentations, and rehearse the tricky questions I might get afterwards.

I simply start talking and the voice agent listens to my presentation, then it gives feedback at the end - with a list of different things that I could improve. This also works well for brainstorming ideas and talking through my decisions, as I can get the same pushback from the AI agent.

While LLMs are incredibly powerful, I'm starting to become overloaded with walls of generated text. There's only so much that you can read and I've found that I'm skimming more than I'm actually reading. Voice models are different as they allow us to have a proper back-and-forth conversation, which makes the AI a better thinking partner compared to the narrow experience you get in a chat window.

The best thing is that I can use this anywhere. I can go for a walk and talk about a problem with the voice agent, then head back home with detailed notes waiting for me. I was honestly surprised by how cheap and fast it is to run. The speech layer only costs $0.45 per hour, and responses come back almost instantly - which is low enough for my use case and allows me to run it everyday.

I only needed three things to get this working:

That's it. You can get setup in just a few minutes and Claude Code can walk you through the entire process.

To get started, simply download my template and you'll have your own voice agent running in a few minutes.

Now that you've got setup, you can change some of the agent's settings and tailor it for your specific use case. While most people will simply adjust the system prompt, there's a lot more you can do to improve the agent's performance.

Give the agent more context about your conversation

When your agent asks the user a question, you should send that question into agent_context. This is really helpful for the AI model, as it can use this context to figure out what the user is likely saying - especially when the audio is unclear or muffled.

For example, if the agent has just asked "what's your email?", there's the risk that the model could write it as "user at example dot com". But if you include the agent_context, the agent will write [email protected] instead. The same approach works for names, dates, account numbers, and confirmations - anything where you know the rough shape of what's coming.

If you've got specific names, products, or terms that come up often and the model is unlikely to know them, you can use keyterms_prompt to improve their recognition. The model will use this list of words when it hears something ambiguous and ensures that it makes fewer mistakes. For example, "Vercel" might come out as "Ver sell" without the keyterms_prompt.

Focus the agent on the right speaker

You can use voice_focus to filter out any background noise and lock onto the primary speaker.

You can also select two modes for the voice focus, which allow you to handle different audiences and environments:

Stop the agent cutting you off mid-sentence

You can use turn_detection to control when the agent thinks you've finished speaking. By default, it uses a combination of silence, tonality, and pacing to work this out - which works well for most cases. But if the agent feels a bit too jumpy and is cutting you off, or is too slow and waiting awkwardly, you can adjust the settings.

To do this, you can change:

For example, if you're building a coaching agent or anything where the user pauses to think, you should set min_turn_silence a bit higher (around 800 to 1000ms) so that the agent waits before responding. But if you're developing a voice agent for drive-throughs or somewhere people speak quickly, you can keep this low.

Shape the agent's personality and give clear instructions

You can use system_prompt to give the agent a clear idea of what its role is, how it should respond to users, and what it should do. It's another important aspect, as it shapes the personality and behaviour of your agent.

For my coach build, the system prompt does most of the work for me. It tells the agent to act as an interview coach, keep responses to two sentences, push back on weak answers, and give structured feedback at the end. Without those instructions, the LLM defaults to giving long-winded essays that aren't well suited for voice.

When you're writing your own prompt, here are a few things you should consider adding:

If you want to step this up a notch, you can also change the system prompt mid-session - which is really useful for agents that have to handle a wide range of conversations.

For example, a customer support agent could start with a general triage prompt that handles common questions, then swap to a more specialised prompt when the conversation moves into billing or technical topics. The conversation history stays intact, so that the user doesn't have to constantly repeat themselves.

Give the agent memory

For the agent to hold a proper conversation, it needs to remember what's already been said. The way this works is that you keep a running list of the conversation and send it to Claude on every turn, so each reply has the full context behind it.

This works really well for short sessions, like preparing for interview questions or brainstorming a single decision. For longer sessions that span several days, you'll want to summarise the older parts of the conversation yourself and feed that summary back in via the system prompt at the start of each session, so the history doesn't grow forever.

Include the expected language

While this isn't necessary and the agent can automatically detect the language for you, it is good practice and helps to eliminate any unnecessary mistakes.

To pin one, set language_code to the two-letter code for your target language:

It's also worth mentioning SpeakerRevision, which allows you to label different speakers in the same conversation, with the agent correcting any earlier mistakes as it learns more about each voice. This is really useful if you're building something like a meeting notetaker, where you need to know who said what and manage several different speakers.

We went into a lot of the technical details, but it's worth stepping back and thinking about where this technology is actually heading - and how businesses can use it.

In the last few years, we have seen huge advances in Large Language Models - which are generating more text than ever and changing the way that we work. But more text isn't always better, as I've found myself skimming more than I'm actually reading - and I'm not the only one doing this.

As models become more capable, I've noticed that my colleagues are increasingly using voice models to talk with their AI - as it's much faster than typing and they can give it more context about their problem.

This trend is only going to accelerate in the coming years, as more industries start to use the technology and bring it into their workflows. Now that we have voice models that are incredibly quick, perform well, and can store memory - the sky's the limit with the potential use cases and the problems we can solve.

Now that we've gone through the process of building an AI agent, here's a summary of what we covered and some tips that you can apply when you're trying this yourself.

Don't overlook the personal use cases

While a lot of people focus on business use cases - of which there are plenty - don't forget to explore how this could help you personally and change the way you work. We did this by building an AI voice coach that helps us prepare for upcoming interviews, make important decisions and explore them in greater detail, or brainstorm different ideas.

I've become quite tired of the chatbot experience in recent months, which is why I was keen to build my own thinking partner and talk through the ideas instead.

It'll be interesting to see how this evolves in the coming years, but I do think this is an emerging area that will only become bigger as LLMs do more tasks for us.

You don't need to be technical to build your own voice agent

Technology has changed a lot recently and you no longer need to write the code yourself, especially if it's a pretty straightforward idea. Claude Code is a great tool, as you can simply paste AssemblyAI's documentation and ask it to build a voice agent for you.

It's incredibly easy to get started and the technical aspect is not a huge barrier anymore for testing ideas. Instead, it's deciding what you want it to do and how you can scale that efficiently.

The system prompt does most of the work

I showed you several important settings and how you can tweak them to get better results from your voice agent. They're worth bearing in mind, as they give you extra control on top of the prompt.

But the core prompt is important for shaping your agent and how it interacts with people. As you start to use the agent more often and see how it performs, you can iterate on this prompt over time and constantly improve it.

Don't over-engineer it

Voice agents have a reputation for being complicated, but most of that comes from teams that got stuck in framework conversations before they'd even built anything.

Instead, you should start small and test your ideas. As I've shown already, you can do that in just 30 minutes and build on the idea from there.

Once you've got the prototype working well and consistently, you can start thinking about adding more complexity.

Summary

Overall, I've been really impressed by how far voice agents have come. They're now reliable enough to build something genuinely useful in just a weekend, and by combining AssemblyAI's Universal-3.5 Pro Realtime model with Claude and Cartesia, you can go from an idea in your head to a working voice agent - without having to write any code yourself.

Build your own voice agent

Try AssemblyAI's new Universal-3.5 Pro Realtime model for yourself. They give you $50 in free credits to start, which is more than enough to build and test your first agent - as I only used a few dollars to create my own voice agent.

Let me know what you think - I'd love to hear how you’re using the technology and your thoughts on how it will impact the wider industry.

Or if you have any questions, simply reply to this email and I’ll get back to you.

Liam

Share with Others

If you found something interesting in this week’s edition, feel free to share this newsletter with your colleagues.

Liam McCormick is a Senior AI Engineer and works within the AI team at Bright. He identifies business value in emerging technologies, implements them, and then shares these insights with others.