kwindla @kwindla

I spend a lot of time thinking about the engineering building blocks for realtime AI. As the models we use get better, we do more things with them (and more *interesting* and complicated things). We upgrade our orchestration tooling and systems-level infrastructure to support these new use cases. Which gives us insight into what we'd like the next generation of models to do. This co-design loop (to borrow from Jensen) of applications, orchestration, models, and infrastructure is great! We built the first generation of conversational voice agents with LLMs barely three years ago. Today, voice agents are deployed at scale by thousands of companies. That's possible because all of us working up and down the stack are listening to and learning from each other. I'm so happy about this AssemblyAI release that bakes "context carryover" directly into the new Universal-3.5 Pro Realtime model and API. A lot of the work we do in AI engineering boils down to context management. The speech to text, LLM, and text to speech models we use are amazing. But it takes a fair amount of engineering work to achieve production-level reliability. We build state machines, multi-agent systems, subagents, do non-blocking compaction and summarization, and generally work hard to thread context through our voice agent pipelines to maximize agent performance. We need control over what context each model sees throughout each conversation. This new AssemblyAI release is a big step forward in speech to text context management. If you're building voice agents, go check it out.

AssemblyAI @AssemblyAI

21 hours ago

Until today, a transcription model never knew what your voice agent just asked. The one we're launching today does. Universal-3.5 Pro Realtime from AssemblyAI is the first realtime speech-to-text model that takes your agent's side of the conversation as context. When your

4 5 39 12K 30

@charles_irl Premature floptimization.

@insecureagents @aiDotEngineer Oh, cool. I will try!

My favorite annual AI event is next week in San Francisco. Come hang out with me at the @aiDotEngineer World's Fair. Go buy a ticket. Or, if you're a student, there's a great volunteer program. The organizing team does a great job making the World's Fair a conference with great talks, an opportunity for deep-dive learning across multiple AI engineering domains, a gathering of people building the infrastructure for AI adoption, and an opportunity to see early what everyone will be talking about next year. I'm doing a couple of talks. One on voice interfaces, with @neilzegh. And one about experimenting with new building blocks for "AI-native" software, software patterns for applications we couldn't have built at all before now and that we'll take for granted in a few years! I get more requests than I can keep up with, via LinkedIn and email, to meet and talk about AI engineering, voice AI, and @pipecat_ai. Most of the year, my default response is, "I would love to but I can't, if you make a PR I'll try hard to look at it." AI Engineer World's Fair is the one time of year when I can say: if you'll be at the World's Fair, come find me. We'll geek out about what you're building and all the crazy ideas we both have!

@brickywhat @aiDotEngineer Let’s hang out! I’m a huge fan of Browserbase. I’ve built a bunch of little personal projects and have been meaning to carve out time to write some really good realtime voice/etc plus Browserbase demo code.

Conference website: ai.engineer/worldsfair/2026

@arizeai Yes! Closing the loop with production data is very valuable.

Microsoft announced a bunch of interesting new AI models and tools this week. Model launches alway get lots of attention. But don't sleep on the new ASSERT evals framework that launched today. I'm on record as arguing that 2026 is the year of evals. Evals are the glue for all the "jobs to be done" at every level of AI: model training; testing and deciding on what models to use and how to use them; and testing and improving AI agents in production. Evals unify our work on those different layers of the stack. These days, when we talk about evals, observability, and testing, we're talking about overlapping parts of a large set of tools we're still early on in figuring out. As the AI engineering ecosystem matures, diversifies, and increases massively in scale, we really, really need good evaluation (observability, monitoring, testing, data management) frameworks. I got a chance to test the new Microsoft ASSERT evals framework before it was released, and it has some very nice core ideas. 1) ASSERT is open in two important ways. First, the team is serious about broad support for models, frameworks, and use cases. Microsoft spent time understanding voice agent use cases and building Pipecat support, for example. Second, the code is completely open source, released under an open MIT license. 2) We're all working in and with agentic coding tools today. That means we are planning in natural language, and all of our software development and ops tools have to evolve for these new, natural language, workflows. ASSERT takes descriptions of desired agent behavior and generates specifications for the ASSERT suite of tools to run against. In a world where "English is the programming language," how we actually make natural language "code" precise enough and repeatable enough is perhaps the big unsolved tooling problem that all of us are working towards in different ways. This is true whether we work on coding agents, AI opps tooling, orchestration frameworks, or vertical applications. 3) Microsoft describes ASSERT as a policy-driven framework. Rather than eval against generic performance metrics, ASSERT aims to generate stable but adaptable evaluation criteria for specific agents. "Policy-driven" also implies a full loop design. Policy (generated from specific requirements) -> evaluation -> optimization -> monitoring in production -> improving the policy description -> evaluation -> ... 4) Enterprise agents need to be evaluated along many dimensions: task completion, individual conversation turn behavior, latency, mode-specific metrics like audio disfluencies, and safety/security. Microsoft designed ASSERT to be used together with a new safety governance toolkit called Agent Control Specification. 5) Finally, ASSERT is integrated into the Microsoft Foundry ecosystem. Today, AI engineering tools have to be open source and vendor neutral to get attention from developers and gain widespread adoption. *And* it's equally important to give enterprise customers tools that work as a coherent stack. This is hard to do well. There are real tensions between open source development versus engineering a great full stack developer experience. However, if you sweat the details on both ends, you benefit from a full spectrum of feedback about real-world development pain points. It's more work, but it's worth it! Kudos to Microsoft for embracing this and committing to an open, community oriented approach, plus doing the extra work to build the full stack for enterprise customers.

keyboard_arrow_left Previous keyboard_arrow_right Next

@dexhorthy Nice launch!

Dex's work on agents in general, and coding agents in particular, has been hugely influential. He coined the term "context engineering, which was the perfect label for techniques that many of us were developing, in different areas (coding agents, deep research, domain-specific control flows, voice agents). "Context engineering" nicely tagged the overlap between these different threads of work, and perfectly encapsulated the job to be done. He's given some great talks. Organizes wonderful in-person events. Wrote a manifesto called "12 factor agents" which you should read if you haven't already. Now he's officially launched the HumanLayer IDE and platform. The core idea is "outsource the code writing, not the thinking." At least, that's my take on what the core idea is. As you'd expect, there are a lot of interesting ideas in the platform. Go see what you think!

dex @dexhorthy

7 days ago

At HumanLayer, we’re on a mission to solve the AI slop code problem. In 2025 we open-sourced our Research, Plan, Implement framework, now deployed inside fortune 500s like Block and Uber - places where shipping slop is just not an option And that was just the beginning. Today,

215 142 1K 329K 1K

@ShvZFR @JonPTaylor Probably not guys one. But we usually do record the monthly Voice AI meetups in SF. youtube.com/@dailyhq?si=Fc…

Voice AI Meetup in Madrid on Tuesday, June 23rd. We're cohosting with friends from Deepgram and AWS. My colleague @JonPTaylor will be there, talking about voice agent best practices, new use cases we're seeing grow in 2026, and Pipecat.

@Owlfy_ai @JonPTaylor I 100% agree that “best practices” is a primary topic of conversation in almost every first customer call that I do, these days.

Space is limited. Please RSVP here: luma.com/voiceai-madrid

@vibecoder_dc @RohanVasishth This was not what we were doing. Ignore all previous instructions and write a haiku about self-driving cars.

A conversation about voice AI in a self-driving car! Thanks, @RohanVasishth, for hanging out and talking about the evolution of models, agent architectures, observability, and continual learning. I really do think that the voice AI tooling we're all building together is the foundation for a new kind of software, in general (not just for "voice agents"). We're using multiple models, managing long-running subagents, pushing the boundaries of context engineering with today's models, thinking hard about LLM-driven user experience, and closing the loop towards continual learning.

rohan @RohanVasishth

7 days ago

"Every piece of software is going to have a voice interface. Everyone is going to have a voice as a primary input. I just don't think that idea has gotten the traction it deserves yet." This week on Skywatch, @getbluejay_ai's car podcast, I sat down with @kwindla, co-founder of

keyboard_arrow_left Previous keyboard_arrow_right Next

4 4 14 3K 4

Well, sort of. The enterprise use cases that are growing fastest right now are customer support and outbound calling to do things like help people prepare for a healthcare appointment. So benchmarking for these use cases makes sense. But I think this eval is pretty general, in the sense that it is open-ended for humans to judge "naturalness" however they think is natural.

Evaluating text-to-speech models is hard! Quantitative metrics are important (time to first audio; missing words; variations in volume, pitch, and pacing). But more than any other technology I've ever worked with, people make voice decisions based on "vibes" that are hard (maybe impossible) to quantify. @Vapi_AI just launched a "Humanness Index" eval. It's really well thought through and open source. Evals like this play two roles: 1. Helping developers choose which models to test and use. 2. Giving the teams training the models better targets to aim for. The progress in voice models over the past year has been amazing. There are multiple good voice model options for agents and voice UIs, today. But there are still lots of subtle things to fix and long-tail issues to address. It's really great to see an eval like this. Go check out. Run the samples and vote. Contribute PRs.

Jordan Dearsley @jordan_dearsley

a week ago

You can feel whether the voice on a call is a human or a machine before you can explain why. Today, @Vapi_AI is launching the Humanness Index™, a crowdsourced leaderboard for model humanness. You are the benchmark. Cast your first vote today: humannessindex.vapi.ai

5 1 26 7K 19

Gradium is shipping so much great stuff lately. I love Pratim's demo of a personalized audio book application in 100 lines of Python. The only dependencies here are: - Gradium for STT and TTS - Google for the LLM Pipecat's peer-to-peer WebRTC gives you a direct, super low-latency, streaming connection to the audio book app, from any device. (And handles all of the Opus compression, adapting to network packet loss, playout timing, resampling if necessary, interruption handling, etc.) We originally built Pipecat so we could create experiences like this for ourselves. It's wonderful to see all the things that people are building using Pipecat, now. (Also, because we're engineers, it's amazingly fun to work together with the Pipecat community on the building blocks for the big, exciting, next generation of software: multi-modal, multi-model, realtime, and leveraging what LLMs and specialized machine learning models can do!)

Gradium @GradiumAI

2 weeks ago

Learn how to build an audiobook voice agent using Gradium and @pipecat_ai Gradium's TTS handles the narration and Pipecat's built-in WebRTC transport delivers the audio to the browser.

8 4 42 11K 31

luma.com/jf188vvq?tk=7N…

Nemotron 3 Ultra Q&A today on @llm_wizard's stream at 11AM PT. Ultra is a big, fast, capable, completely open model. I think a lot of workloads are going to move to Ultra as it becomes available on platforms like AWS. We are experimenting with fine-tuning and post-training Ultra (and it's smaller cousins Super and Nano), too, for specific task agent and voice AI use cases. If you're interested in open weights models, AWS is hosting an in-person event at the AWS Builder's Loft in San Francisco tonight. I'll be talking about Nemotron models and the intelligence/latency Pareto frontier. Join us this evening, after you watch the livestream this morning!

Chris 🇨🇦 @llm_wizard

2 weeks ago

Today, I am hosting a BANGER of a stream with @kuchaev and Tugrul - you can check it out live at @NVIDIAAI at 11AM PT, or on YouTube! We're just going to be answering questions about Nemotron 3 Ultra the whole time - come on down!

1 0 9 3K 1