We Kept Rebuilding Our AI Agent. Here's What Survived - Fuad Ali, Arize AI

[music] >> Uh what's up guys? Thank you guys for coming to this talk. I know we're getting to the end of the day. You had lunch, you're getting tired. So I really appreciate the patience. You've been hearing a lot from all of us at Our Eyes. You just heard Sophia's talk maybe if you were in the room earlier. So thank you again. My name's Fuad. I did a talk yesterday if you weren't here. I work on product at Our Eyes. So I'm what they call an AI PM which I'll get into a little bit about you kind of how that relates to shipping agents. Um today I want to talk about uh two years of shipping our in-product production agent and some of the lessons we've learned. Hopefully there are good takeaways for you guys as you are kind of building your own agents in the wild. Cool. So before I start, we're going to get a little bit meta here uh and hopefully you guys are all kind of following along. But you know this slide was made with an agent. Yes, it was. Um we have coding agents um that kind of operate with Alex, with Signal, with other parts of our product that are themselves agents.

Uh we have Signal which is kind of the trace reading agent that reads our agent's traces, right? So we have an agent reading the agent traces. We have Alex which is the in-product agent. You can think of it like kind of a cursor for AI engineering. That's the agent I'm talking about today. Uh we actually use Alex to debug Alex. Um so it gets a little bit meta. Um and then hopefully you get you know kind of an idea of how Alex could help you evaluate your agent and some of the design patterns to be taken from Alex and how that could be translated to your agent. So anyways, turtles all the way down, agents all the way down. Uh and if you are confused about which agent we're talking about, I will make sure to make that clear. So um what we're going to cover is kind of like how Alex got here. I've got some nice throwback screenshots from 2023 which feels like ages ago, about uh of how Alex used to look. Uh where it used to be just a router over skills tool calling loop. Um and kind of how we've changed that. Uh some of the hardest decisions we made along the way. So, specifically who owns the plan, how to end a turn, how we dealt with memory across the agent, should we use one agent, should we use many sub agents? Uh what default model to choose and sort of how we test it as well. Um then, like I mentioned, it gets a bit meta, but we debug Alex with Alex. So, and now we have signals as well. So, how signals is finding Alex's retry loops and some of the real issues we found

with our agent using signal. Uh and then, importantly, what what still needs a person as well. And so, where the human value kind of adds judgment and the AIPM world kind of comes in. Uh and some of the decisions we kept that have lasted throughout the architecture of Alex. Um Alex is the agent inside Arize X. Alex can read your traces, turn findings into tests, so building your evals for you, data sets, experiments. It can plan work. It can run multi-step tasks. It can create synthetic data for you based on your traces. Um and it can do a lot of different things. I would say, yeah, this is a quote from one of our staff engineers, on the first prototype back in late 2023 with like 3.5 error models, it was about 50% hallucination, 50% And you saw that say we're getting a bit better as models have gotten better as well. Uh this is just a little before and after of the visuals just to give you an idea.

So, you can give it, you know, prompts that you stored in Arize, you can ask it to optimize those prompts. And obviously, it had, you know, kind of tools to read prompts from our database and things like that. Now, we have, you know, much more in-depth flow with planning, with much more beautiful UI. Alex kind of explains what it's doing. Uh it'll reason through tasks much more strongly. So, here I have like a request to optimize Alex's own system prompt and actually makes a plan to check, you know, kind of resources around the application as well. So, yeah, this window shows up in a rise. You can use it to debug your traces. Um How do we get here? So, quick kind of like timeline. We started the prototype in late 2023. This is like with 3.5 error models. Um We then, you know, very quickly decided to move a lot of math out of the model and into the harness code because if you remember asking agents to do math back then, I mean nowadays they're solving Navier-Stokes level problems, but back then they were not very good at math. Um Then we kind of had the official launch. We called it the copilot back then when things were still called copilots.

A router of a set of skills, a decision tree. We had some rails. We rewrote a lot of it in August 2025. Um Iterative tool calling and a to-do tool come in. So, the agent can actually kind of plan what it's doing. Alex 2.0 was our big relaunch in February 2026. Sonar 4.5 became the default model and we'll talk through some of how we test that model as well. One Alex was just released in June. So, instead of specialist agents, we merged it into kind of like one agent that just loads skills on demand. And finally, memory. We now have long-term memory, so as the user kind of interacts with Alex throughout the course of their session, they are kind of continuously the agent continuously knows sort of the history of that user and that's written in the background. >> [snorts] >> Um and obviously, we didn't train these models. Everything we rebuilt was harness code, which I think um actually gives me a bit of faith in the world of engineering that there's still very interesting problems to solve here and it's not all up to the model and you can't just run a frontier model and expect it to do everything for you.

>> [snorts] >> Uh models got better on their own schedule for sure and we made some decisions around that, but pretty much everything we built was that harness code. We don't use an agent framework. Um The loop, the plan, the stopping rule, and the memory are all ours. What that means is we have a lot of control over it and you know we get to learn a lot along the way as well. >> [snorts] >> So first one planning and to-do list. The analogy I have a bunch of stuff on the slides but the analogy I kind of think about around you know planning is like Let's say you I don't I don't have kids maybe some of you guys have kids and you can kind of relate to this analogy but let's say you want you know your kid to do some chores. You can give it a list of chores and ask it to or ask them to check off the list of chores as they go. Um You can check in with them continuously and hold the list yourself and not tell them about the list except for the next task, right? So that's one paradigm.

Um And there are trade-offs to kind of like each scenario. We initially just gave the model to do list and what we saw was it spent most of its time checking off the to do list. So a lot of LM calls for um pretty much nothing like no useful unit of work other than updating a to do list. It also we also realized it tended to lose focus after multiple tool calls and yeah 20 20 plus iterations on the to do tools themselves. So we added this as an extra tool in its arsenal. And we eventually moved the plan into the harness one piece at a time. So what I mean by that is first thing we did was we added an in progress so the the model wouldn't lose track of where it was. Um Then we decided to put the plan as a message so the harness renders the plan to the model call every time the model gets a new call and the rules are next to the data so it's a little bit easier for the model to interpret where it's at. We actually ended up dropping the in progress identifier because we realized we're just making two calls for effectively you know anything that isn't finished is in in a stack ranked order is the next thing to do, so we remove that.

Um and we're actually testing now without a to-do tool at all, uh just keeping everything in line in the messages uh completely. So, the harness is keeping the state and the completion check, and the model just consumes that and goes and does the next thing. Um so, maybe some lessons for you guys as you're building to-dos, uh models can be very eager with uh lists. Um then, kind of like what ends the turn uh finishing became a gate the harness enforces. Um so, what we at first tried was, you know, kind of text without tool calls, uh meant and done. That broke on models that announce a step in one turn and make the call in the next, which [snorts] um you know, as you change models, you'll you'll quickly find some of these breaking things. And one of the things we do in Aux is we actually allow users to choose their own provider. Uh so, we can't necessarily guarantee that we've tested every single model someone will try with Aux. So, a lot of um you know, some of the decisions we made here are to guard against those. And yes, we run regression tests and we check against a bunch of different models and we benchmark different models on Aux. We have an opinion on a default, but you know, obviously some of our customers are exclusive Anthropic customers or exclusive OpenAI customers.

So, to actually empower them to use their own models, we kind of had to design against everything that was available right? Um so, now we have an explicit finish call that bounces back uh when plan items are open up to three times, uh a warning set one step before the iteration limit, and then a force finish as well. Um memory, this is a good one. Um there are actually a number of problems here, and I'm I I like analogies, so I'm going to use another analogy here. It's like, you know, you you're working at your desk, um and a big book lands on your desk. And maybe one big book is fine. You can open the book, kind of search through the book. Uh if the book gets too long, then it's kind of useless to now cuz you're going to spend your entire time reading that book. That's one issue. Uh and that's sort of like, you know, one big tool result comes back and it's too big for you to read, right? Um but you can imagine there's actually multiple books on your desk, right? So, eventually not just are the books too long for you to read on your desk, but there's no space left on your desk, right? You just have too many books, right? So, there's a yeah, you know, in this analogy, um there's a lot of ways you can approach this, right? You can decide to store that book and just have a pointer to the book, right? So, that's kind of what we did with the tool results, where Okay, I now have a sticky note on my desk that says where the book is and what, you know, kind of high level the book contains. And so, if I need to go read that book, I'll just reference my sticky note and then go find the book on the shelf.

My desk clears up a little bit, right? So, that's kind of what we do with tool calls uh and tool results. For example, like Alex's uh you know, can help you with trace analysis. Some of these traces are other agents' traces, and they can have a ton of stuff in the trace, right? So, um if it's a big JSON blob, for example, that just blows up our context window and then you can pretty much do no useful work. So, instead, what we do is we truncate within uh the JSON and we just let the agent code to go find it if it needs the full context with JQ. Um one thing that agents are very, very good at is writing code and using code-like tools to uh kind of infer things. And so, that's kind of what we try to leverage with that. Um the agent's own trajectory also going past uh a certain uh window. Um so, what we now have is the harness makes the model write a checkpoint uh and then drops what it replaced. Uh we have fixed token budget for history um and across sessions, so I kind of got to like the long-term memory piece. Uh basically, we just ask the a background job to edit one small file per user. I'll dive a little bit more into like why we decided to do this with with long-term memory uh later in the presentation, but essentially, that's how it works.

Um kind of talk through this. So, treat a huge result like a file and just scrap it. Uh don't dump it all into context. Don't worry about doing anything fancy. Uh for long-term memory, a small file actually beat an all graph and we we tried things like vector retrieval, episodic and semantic stores, uh temporal knowledge graphs, etc. etc. What we eventually landed on was we just have one file per user. It's a markdown file. It's capped at 8,000 characters. Um and you know, I I think there there is a lesson here that, you know, as engineers um we like to build things and it's really fun to build things and sometimes we get in our own way and we over-engineer. Um and this file is just loaded in on every full request. Uh it's written into in a background job after the session finishes. It's small. It's exact. It's easy to load in. It doesn't blow up context and it's very easy for a human to audit as well. Uh so, you can actually kind of have a human labeler go look at these memory files and and determine whether the right things are being stored and kind of edit instructions for the background job. So, um and then you also only have one job at the very end, which uh you can make very safe as well. Uh so, you're you're making sure you're not writing memory from other users into the file. Um so, yeah, very good lesson on not over-engineering uh your problems.

Um briefly on specialists, um you know, we had one trace agent that initially just did trace analysis back in 2025. Then we had 10 specialists, so every page kind of had its own specialist. If you're on the data set page, there's, you know, special sub agent that can generate synthetic data for you and look look and introspect on all uh all your data sets. Uh we ended up kind of rolling that back as well and just using one agent where the agent can load skill instructions on demand and tools on demand and we just let the agent decide what tools to do, right? So, it is a bit of an evolution also in in model capability, but I think simplification as well. >> [snorts] >> Um we on model capability, we also kind of stand by that the newest model isn't always the best by default. Um there are times where we upgrade the model by default, right? And so, we have a perspective on that. Um I think we have um you know, what is the best model is a very subjective question. Uh what is the best model on a generalized frontier is very different from what may be best for you guys in what you're building or what we're building. Uh we generally just default to fast and good enough and spend a lot of time on the harness instead. Um we are now starting to explore auto routing, uh so choosing models per task and training our own models, so small post training SLMs, things like that on very narrow surfaces. Um and some of these surfaces actually don't even need the loop at all. You can just make a single model

call. So, an example of that is I wonder if I can pull it up. We have this like, you know, filter with AI. Um we've come a long way from SQL generation in the in the past and it's actually very, very easy to generate filters with very, very small models now. So, um that's an example where maybe we don't need the entire loop and we're just over-engineering again. Um Cool. So, that's a good takeaway. Um production sessions becoming our test. So, this is where I'll talk a little bit more about arise as a platform. Uh tracing everything is really important, right? So, we trace Actually, if you look at this, these are the traces from this agent. Uh so, it's very meta here. Um but every time you ask Alex a ques- question here, the the traces these are the traces from Alex. We we still call it Copilot prod. Um so, those are our live traces coming in from the tool. Um this makes it very easy to add, you know, regression data sets and and keep track of different user queries that you want to go test against and iterate against.

Um and so, those production sessions become our test cases and become our living fixtures over time. They grow, they change um as we see new error cases. Um and we also now have the ability to use session data sets as well. So, you can store an entire session as a data set and kind of play a specific turn through the agent if you need. Um finally, we use arise experiments throughout all this as well to make sure that we're you know, kind of AB testing uh harness changes as we go through this. Uh one takeaway I want you guys to to kind of keep in mind also is that in agents, bugs rarely show up as explicit errors. Um in the old world software, this was a lot I don't know if it was easier, but it was different. Um now it's more of like a qualitative okay, the status says okay, but if there are 43 identical tool calls in the same turn, that's probably something we want to fix. Um software metrics like latency, right? So, I think we can all agree 227 seconds is way too long to wait for a response, but what actually is too raw too long it really depends on your use case.

Um so, we use signal a lot now to kind of automate this reveal. We we use less and less of the human reading the trace. Um our staff engineers are very grateful for that. Uh but we still do use kind of that human expertise, which I'll get to in a bit. >> [snorts] >> Um the fix can also be the next box. So, these regression data sets are are really important and I would encourage you guys to keep them in mind. Judges need that human baseline, so that's where we kind of impart the human task. We align our evaluators and we make sure that our judges actually accurately reflect what you know, our engineers think as well. Uh so, that's very important. Um I think we we still keep a lot of things with the human, those gold labels, those golden data sets uh for judges come from people. Um the speed versus quality is a human judgment call. I don't think there's any single answer across the stack on when you should be a little bit faster, but it's okay to degrade quality. It is still you know, kind of like a human judgment call even if you have the metrics around it like okay, changing models will increase latency by X and decrease uh or increase quality response by Y. If we just use Astra everywhere, would that be the right decision? Still a human judgment call, I think. We have the metrics to support that, but we still have to make those decisions.

Um And these are kind of the the six things that survived all these rebuilds. Tracing to our own products that we of course still trace everything that Alex does. Math and data in code. So, code computes the aggregations, the model reason and explains. And then exposing those as tools to them to the model. So, we're not giving the model plain text math questions. We're actually asking it to use code to go do that. Golden data sets in CI, errors with an obvious fix, streaming and persistent, and skills. Um We've changed what we call skills. I think, you know, routers, tools, etc. But really it's give the model a capability around an external system. So, a lot of decisions you'll face. And here's some of what we chose. So, if you'll go ahead and take a picture of the slide. Who owns the plan? What ends the turn? If your tool output is too big for your context, what do you do? Who should write the long-term memory? Should you have one? Should you have multiple sub agents? What model is good enough and fast enough?

And who checks the agent's own fixes? So, yeah, that's me. On the bottom right of the screen there my LinkedIn and X. The tagline is everyday we use Alex to build Alex. We have a bunch of blog posts on sort of like technical deep dives into how we built this. And yeah, I encourage you guys to all reach reach out. And big shout out to the Alex team and our engineers for building this. Thank you. >> [applause]