How to Test Your Agents on Real Tasks With Harbor - Xander Song, Arize AI

[music] I'll go ahead and get started. Uh, my name is Xander Song. I'm an engineer at AriseAI where I work on Phoenix, our open-source observability experimentation and evaluation platform. And today I'm going to introduce you to Harbor, which is an open- source framework for evaluating agents on real tasks with real data in isolated sandboxed environments. The theme of today's talk is shipping based on evidence rather than vibes. And my goal is to give you a conceptual overview of how Harbor works as a framework and to share some of the ways in which my team is using Harbor to iterate more quickly and with more confidence on our agents, harnesses, skills, prompts, and tools. This talk is going to be a discussion rather than a interactive workshop. But my hope is that you walk away with a concrete sense of how Harbor works as a framework and also a sense of how you might use it in your day-to-day work to make your life and your agents a little bit better. A disclaimer before I begin. I am not a maintainer of Harbor. Um I do however consider myself to be an avid power user at this point.

So, so what is harbor? At a high level, Harbor is a framework that allows you to define realistic tasks in the form of natural language instructions to an agent. These are tasks that you would like your agent to be able to accomplish. Harbor feeds those tasks to your agent running in a fresh sandboxed environment with real data relevant to the given task. And then it enables you to define what are called verifiers. These verifiers are specific to each task and they just check that your agent produced the intended side effects inside of the environment. Verifiers produce rewards. Examples of rewards are binary scores such as zero or one or pass fail. So this picture is like really really simple, right? But I want to just take a moment to appreciate why it's powerful. Because if my task and my data are realistic and if I've done a good job of defining verifiers that accurately describe what constitutes success or failure on the given task, what that means is that I can hold my task and my verifier constant while I change the implementation of my agent. And by simply executing this pipeline from start to finish, I can quickly and efficiently gain a signal telling me whether or not the change I made to my

agent produced an improvement or a regression for this particular task. Um, and not only that, I can take it a step further. I can compile not just one task, but many tasks that cover the broad surface area of the capabilities of my agent. And this lets me close the loop here. So that now I'm actually able to start iterating with a degree of confidence that I might not be able to um if I was shipping based on vibes alone. So that's really the idea of harbor in a nutshell. Let's break it down a little bit further. I'm going to talk about like the architectural details of how Harbor works. So on this slide here, I have the task that I was talking about on the previous slide. Each task specifies a set of instructions in the form of a markdown file. And once again, that is just a piece of natural language that you're going to feed to your agent.

The task also describes an environment in which the agent will run. And that typically takes the form of a Docker file. And that Docker file typically contains data that is relevant to your task at hand. And we'll give examples of this in just a minute. But the thing to realize here is that um Harbor's actually agnostic with respect to what kind of data you give it. So some examples could be like a SQL light file. A very common example for coding agents would be like a git repository at a particular commit. And Harbor also lets you define a set of verifiers inside of a test directory. And as I said, verifiers are just scripts. They're just scripts that run on your environment after your agent is done running that verify that the intended side effect took place. And like I said, typically we don't have just one task, but we have many collected together in what is called a data set. And for each task in the data set, Harbor allows you to run a configurable number of trials. These trials are just independent executions of the task to account for the fact that LLMs are stochastic. Agents are stochastic. They don't behave the same every time. So each trial is going to build a fresh environment in the form of a brand new Docker container seated with that data we were just talking about. Harbor is going to drop your agent of choice into this environment. It's going to feed it the instruction markdown file and it's

going to let it do its thing. And of course, the agent under test is going to produce side effects in the environment usually. So if it's a coding agent, it's going to make changes to source files. Um, it might make changes to a database if you put it in there. It could do any number of things, right? And after the agent finishes, the verifiers are going to run on the environment. And again, a verifier is just a script that checks the environment. Those verifiers produce rewards which get written to a reward.ext file and that that's the most important thing if you take away nothing else. This is like the architecture of how harbor works. There's another thing which is kind of an additional optional thing you can do which is you can configure your agents to emit so-called trajectories in the form of ATIF. ATIF stands for agent trajectory interchange format. ATIF is Harbor's specification for a JSON based tracing format which basically lets you peer into the internal details and the internal steps that your agent took to produce the side effect on the system.

So it's basically there for debugging purposes and for logs and and recording and all that. this whole pipeline executing end to end where we take all the tasks in a data set run a certain number of trials for those tasks run our verifiers to produce rewards and collect trajectories this whole thing all taking together is what Harper calls a job and there's one more thing I want to point out while I'm on this slide which is you'll notice right here um I said I said agent/oracle um so Harbor also has this notion of an oracle you can drop not only an agent agent into the environment, but also an oracle. And an oracle is just a solution to your task. It's just a script that solves your task. And we'll talk about oracles more in a minute, but oracles turn out to be like this really useful thing because oracles basically let you work out all the the the kinks in your pipeline before you ever throw an agent into the mix. It lets you make sure that your data is good, that your environment is configured properly, that your verifiers work as expected, and you get to do all that before you spend any money on tokens for an agent.

All right, so that is like the high level. Um, before I pause for questions, I will I will just continue on for just a second to kind of illustrate it with a concrete example. So, I'm going to break down a concrete task. Let's suppose that our agent that we care about is a coding agent um or a coding agent with a particular set of skills or a coding agent configured with [clears throat] like a custom MCP server that we're running. And let's suppose that the task is to fix a failing migration. So if that's the case, the instruction markdown file would you know it would it would be some set of instructions that you would pass to the coding agent as a user. So it might say, "Hey, I'm noticing that this migration is failing. Could you please try to fix it?" Right? And in the environment, the environment would contain perhaps an actual um SQL file. Maybe it would contain some DDL. Um and maybe it would contain some source code containing an up migration. And we've, you know, we've seated the data so that the up migration fails. Um in addition, we'll have our tests. And again, the tests, they could be just scripts that run the up migration that ensures that the up migration exit with status code zero. And then it maybe it checks like some tables. Maybe it just checks the tables to make sure that the columns are there that we expect, the indexes are there that we expect to make sure that the um migration has actually had the intended side effects on the system.

And you'll also notice that here optionally we have the solution, right? So the solution is just going to be the literal solution, the literal uh script that you could run um that would fix the the migration. And again, why would why would you go through the trouble of doing this? Um, the reason again is it kind of lets you quickly and confidently build a little bit of assurance that you've got your system set up, that your verifiers are correct, and you can it kind of lets you um vibe a little harder, I think, with the system because you can gain a certain amount of confidence in your verifiers without necessarily like reviewing them line by line. At least that's my experience with it. And so after the the agent runs the task in the environment, um it's going to edit that broken migration file, it's going to leave a couple modifications on the file system and then the verifiers are going to run.

And typically if you're running um Harbor in Docker, which is like the default local setting, those verifiers are going to mount in the Docker container after the agent finishes. And that's to prevent the agent from cheating. So the the agent doesn't have access to the verifiers and the solutions um during the actual execution. Those verifiers are going to run the sort of checks that we talked about. Maybe it literally runs the up migration. Maybe it um you know checks the tables, checks the indexes. And I think the thing to really take away from this slide is um the verifiers are agnostic of the implementation of your agent. They actually don't know anything about the agent at all. All they care about is what the agent leaves behind. So it is possible to like write some verifiers that are not like that that peak into the ATIF trajectory that kind of look at the implementation or the execution of the agent and evaluate it that way. But that's not really the canonical kind of verifier.

The verifiers, you know, canonically treat the agent as a little bit of a black box and they're grading the agent solely based on its side effects and its output in the environment rather than the exact manner in which it produced those side effects and outcome. And there's kind of a natural parallel here with integration tests because you can imagine that if you have some integration tests for an interface or an API, those integration tests um if they're really well written are hopefully agnostic of the implementation that is behind that interface or API. And in the same way, if we have really good verifiers, our verifiers are kind of describing what we expect the agent to do, the side effects we expected to produce in the environment. And they're doing so in a way that is decoupled from the implementation. And that's a really nice feature to have for a set of tests because it allows you to refactor under green, for example. It allows you to hold your tasks constant while you like completely change the implementation of your agent and then it allows you to like fire this pipeline really quickly to figure out if everything still works.

That's kind of what we're talking about when we're talking about iterating quickly um and shipping confidently. So before I go any further, let me just pause right here for questions or comments or um Yeah, let me just pause before I go any further. Yeah. Yeah. Yeah. You just Yeah. Again, it's just a Docker container. So, you just pop whatever you want in there, right? [laughter] >> It's a primitive that you're you're proposing. >> Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. So, you know, it could be it could be any framework, right? Uh it could be any language. It could be any piece of code, any executable. Concretely, like, you know, I'll give you a little sneak sneak peek or sneak preview, right? Like, we've had to like change tools like um you know, adjust adjust prompts, right? Um, I'm going through a process right now where I'm trying to consolidate our skills, for example. So, we kind of have like uh several different agent experiences that we try to support. Um, there's a lot of overlap and then there's like certain differences and we're trying to like basically get our skills all kind of like reusable, right?

Um, and agnostic of the particular like direction that the agent is coming from, whether it's MCP or a CLI or via like our our own like server side agent. we're trying to get it agnostic and and so those are the kinds of things where it kind of helps to be able to specify this set of tests that is like completely agnostic of the underlying implementation, you know, and we we kind of we came to this like maybe um through the pain of doing it the other way where we were like kind of like had a set of tests that were um that were coupled and it was becoming difficult to maintain because you know I would change a tool and the tests would break right even though the not even though the true functionality of the agent was still intact, right? Things like that. Um, we'll get to more of these, right? We'll get to more of these at the end. Um, any other questions before I go on? Okay. Okay. So, uh, this this slide is going to talk about scale. Um, so as we as we already talked about, right, each task and each trial, uh, is independent, runs in its own environment, its own docker container. Um, and that makes this whole pipeline completely horizontally scalable.

And that means that it's very compatible with a sandbox provider such as Daytona or Modal or E2B or any of those. Um, and it means you can kind of rip through these tests like really fast if you want to, which is kind of fun. So in this diagram arrayed vertically, we have a set of four tasks in a data set, including the fixed migration task that we were just imagining. Um, we've got a couple of different coding agents that we might want to test out. claude code with opus 5 codeex with gbt5 and then we've got um a couple of different trials. Let's suppose that we're doing three trials for k equals 1 2 3. Again, the point of that is just to kind of handle the handle the fact that these things are stocastic and they they're not perfect. They don't behave the same every time. Um so because all these are independent um you can run them all in parallel on these sandbox providers. So we use Daytona um and it runs quite fast with like 200 concurrent workers. So even if you have a large test suite you can rip through it quickly. Um and then from these individual trials you are able to compute aggregate metrics for the kinds of things that you care about. So for example you could compute a pass at Kmetric that just tells you in this case did I pass after three attempts at the at the task. And so I can see that oh for like this particular task the fixed migration task both of my agents like passed at K right passed at three uh for this one only one

of my two agents passed at three. You can actually also compute those kind of aggregate metrics over the entire data set right and so this kind of like gives you this very quick picture of how your change is performing because again if you have like a a good data set that covers the set of functionality that you care about you can get these aggregate metrics pretty pretty rapidly. Yeah. Go ahead. Each Yeah, that's a great question. So, this is a really important point, right? Is each of these boxes is a totally separate Docker container, identical in the beginning, but then you you put an agent, a separate agent into each one, and you let it go and do its thing, and they do different things, right? Um, so each one is totally independent. They're completely isolated, right? They're independent trials of the same task with the same agent. Um you're basically run yeah you're basically running claude code three times and saying hey try to fix this migration and then you're saying oh after three tries did claude code fix the migration. Okay so now now I've now we've talked about harbor at a at a high level. Um but I want to switch gears and I want to talk about how Harbor plugs into Phoenix. So again I'm not a maintainer of Harbor. Um I'm just a fan.

Uh but I am a maintainer of of Phoenix uh which is our which is ARIS's open source AI observability tool. And the point of this slide is just to basically convey that we have analogous concepts for the primitives in Harbor. So we're able to map your data sets to our notion of a data set. We're able to track your experiments. We're able to um record your trials. Uh we're able to convert your rewards into our notion of reward which we call an evaluation. Um, and I guess the other thing I I want to take away from this slide is also we we do something where we convert those trajectories that I talked about into so-called open inference format. Um so open inference is a open standard of semantic conventions so-called semantic conventions which is built on top of open telemetry which is an open source framework for telemetry data such as logs metrics and traces and these conventions basically define data structures that uh your AI I application can emit which then downstream consumers such as Phoenix can ingest and we can use those to build certain kinds of UI experiences and workflows that wouldn't otherwise be possible. So full disclosure, I am a maintainer of both

imprints. Um we recently donated this the spec to cloud native, right? Um and to the hotel working group. So it has some traction outside of arise. And so that's maybe a reason you might want to um use open inference. Um so this diagram on the left, you can see the ATIF trajectory for that fixed migration task. you can see the individual steps that the agent might take um including an LLM call and including a tool call. And over on the right, uh this is kind of what it winds up looking like in open inference. And in the product, like I said, open inference allows us to build out the kinds of opinionated UIs and workflows that you see here. So we can do things like we can render your trajectories with some opinionated UI, some specialized UI for LLMs and tools. We can version your tasks and your data sets so that at any given time you know exactly what the task look like for a particular job in Harbor.

We track your experiments and we help you visualize metrics and rewards as you iterate. So in this case the reward you could imagine it being pass fail um zero or one and you can run a sequence of experiments and you can kind of see over time is my reward moving in the right direction. We also have skills CLIs and an MCP server so that claude and codecs and cursor can easily get access to this data. [clears throat] And at this point I do want to note that Harbor does have its own open- source UI um that you can run locally on your machine. It also has a hosted web service. Um, so you you definitely don't have to use Phoenix to use Harbor. Um, but if you're interested in some of the features I just mentioned, that would be a reason to give Phoenix a try. Okay, so hopefully at this point I've given you kind of a bird's eye view of how everything works. Now I want to talk about some of the ways in which our team has been using Harbor. And hopefully this answers um answers your point over here just around getting some concrete examples. Um so in order to do that I just want to give kind of this high level of like what our architecture works with. So again this is the Phoenix architecture. This is particular to our team right this is the this is what our team is kind of like dealing with on a day-to-day basis. Um on the left here you have an enduser's AI application or agent. And again this application is going to have been instrumented with open telemetry and

open inference. And it's going to be exporting and emitting traces to the Phoenix server over OTLP, which is the open telemetry protocol. So, it's basically sending data to Phoenix, right? Saying, "Hey, I'm doing this and this is like what my these are the sequence of steps that I did to accomplish this task. This could be like a production application. It could be an application that you're testing out. Those traces are ingested by Phoenix and they are written to one of our SQL backends, either SQLite or Postgres. And once the data is in the system, we have a set of core services for things like traces, spans, sessions, data sets, experiments, and evaluations. And these core services are exposed via various kinds of APIs, REST or GraphQL. But our goal is not only to like make those programmatic interfaces, but it's also to make it easy for our end users and for their coding agents and other kinds of agents to get access to the data and use the data. And so we support several different kinds of agent experiences.

Um the first of which are coding agents. So for coding agents, we have a CLI that the coding agent can use which will invoke REST or it will invoke GraphQL. We also have an MCP server that you can configure your coding agent to use um which is going to do the same things. And we have a third agent interface that we maintain for the team which is a server side agent that we call Pixie which also has access to the MCP server and therefore has access to REST and GraphQL. Um it also has like a bash emulator. So it's it's kind of like a coding agent but running server side. Um so these are the three surfaces that we that we want to support and that we have to test and ensure are working. the coding agent with the CLI, the coding agent with an MCP server, and a server side agent, which is our own harness. And when it comes time to actually like test all of this out, architecturally, we have a pretty big advantage. We have this really nice feature, which is that this server runs on a single node. It's kind of like a graphana- like architecture. Um, and that means it's very easy to dockerize. So, you just pop the whole thing in a Docker container.

What we do is we just take a SQLite file which contains some realistic data that we've tested on. Um we replace the browser with a test client. Um and we just put it in Docker. And once it's in Docker, we can start using it in Harbor. So we can feed the system the kinds of tasks that our particular users care about. So, our particular users might ask a coding agent or they might ask our Pixie server agent, find some hallucinated LLM calls from the last week of my traces and add them to a data set. That's a realistic query that one of our users might ask. And we can go ahead and let any of these three kinds of agents go wild inside of this environment. And at the end, we can run our verifiers. And the kinds of verifiers that we can run for us in this particular setup are really quite simple because we have a server running inside of here with the SQL life file. So we could either hit a rest endpoint to verify that the intended side effect has changed or we could literally like query the SQL life file if we wanted to because it's just in the container. Um and just verify for example that the particular tupil for this data sets table has been written in the way that we expect.

>> Yeah. Yeah. No, thanks for stopping me because I think there's a lot going on on this slide. So maybe I should pause and and kind of like make sure it's clear before I go on. Right. Um, so I think there were a couple of questions there and I'll I'll try and address address them. Um, so in this slide there are three different agents in in any given trial. I would only be testing one of them if that makes sense. So you could imagine maybe I'm only testing Claude code with the CLI, maybe I'm only testing Claude code with MCP or maybe I'm only testing you know our Pixie server server agent. So just may let's imagine like it's just one at a time in terms of how does harbor feed at the prompt. Harbor and this is maybe a nuance that I haven't really talked about, right? Harbor itself um does not Harbor is a program. It's a command line tool. Um and it doesn't actually run in the environment per se in the Docker environment per se. It runs on a host machine which could be your laptop if you're running it locally or it runs in like a GitHub action runner, you know, if you're running it in CI. Um, and you can run the environment in a Docker container locally. That's like the default behavior. But if you're trying to do something more scalable, you would be running it on a sandbox provider in all likelihood. Um, so you'd be running it on Daytona. And so Harbor is this orchestrator um which is running on your laptop or running in GitHub actions which is um kicking off the run. It's basically saying, you know, hey, start

start the agent, right? Um, and the way it does that is, and I can show you what this looks like on the file system, but it basically, you know, um, expects a particular script. Harbor expects a particular script to be written for the task, right? It's all driven via scripts. So, you like when you're when you're setting up Harbor, you have to kind of make sure that all these scripts are in the right place in the Docker image. Um, so there's going to be a script to start the task. There's going to be a script to run the verifiers. There's going to be a script to run the oracles. It's all driven by scripts. So the scripts are in the Docker container. Um, but doc but Harborself is a program that lives on a on a host machine and just kind of orchestrates things and kicks things off. That's right. And the interf Yeah. So maybe I'll just repeat what he what you just said because I think it's well put is like Harbor is a CLI um and it has certain conventions for if you like make your docker container or your docker image look this way it will be able to run it um and those conventions are almost all like is there a script in the right place so that's kind of like the contract right if there was a a contract that you had to honor with harbor so that your your image would run it would be like did I put my scripts in the right place right That's kind of the contract.

>> Got it. >> Yeah. Does that Does that clarify? >> Yes. >> Okay. Yeah. So again on this one on this slide, right? Basically I think the takeaway is um the takeaway on this slide is like you have a great deal of freedom um and flexibility when it comes to customizing your harbor environment, right? Um the the simplest like the simplest example that people always get started with is the example of a coding agent operating on a git repo. But this example is different, right? This example is like I literally put some data from a SQL life file in there. I threw my whole application in there with it along with its APIs along with its own serverside agent. It's all it's a running server in the environment in the Docker container. Um something you know there's a CLI in the environment. So what's not even included here is like there's actually a set of skills literally in the environment on the file system that the agent has access to. Um, so all that is to say is like you have a lot of flexibility in setting up your environment. Um, you know, it has to fit in a Docker image, but if you can fit it in a Docker image, um, you know, you can you can make Harbor run it basically.

All right. And now any other any other questions before I move on? Maybe because this is a complex slide. I think this is and and you know this is like again this slide is very much like particular to to our project right this is like what our project looks like your project may not look like this right if you if you're not running like a single node server or something but yeah I mean that's such a good question and I haven't had to deal with that problem so it would require a lot of thought I think right but basically the question is like my system is not a single node it's not a single you know I can't just like throw in a SQL single SQLite file I have like you know, some distributed system with like S3 and all this other stuff that I'm trying to maintain. How would I test harbor out with that? The answer is like that's probably harder. Yeah, that's probably harder. So, in that case, in that case, my initial thought is like, you know, you have to perhaps carve out your agent a little more, right? So, here we've had this advantage where like it's simple for us to throw the whole damn thing in there in the Docker container. And so, we did.

If that wasn't simple to do, if you can modularize your agent somehow into some kind of like interface, hide hide the implementation behind an interface um and if you can like if there's some like stable API boundary or interface that is relatively stable, right? Where you know um I'm feeding like this agent data in this format, right? Whatever that format may be, like eventually this agent has access to data in this format, right? Um, that would be one approach, right? Um, and I realize that that is maybe not a satisfactory answer depending on what your exact setup looks like because maybe your agent like literally queries S3 and then like what are you to do, right? Are you going to mock out all those S3 calls? And then once you start mocking out S3 calls, you're kind of in a world of pain because now you're in the world of mocking stuff again, right? And and that's really like what we're trying to get away from with Harbor. We were trying to get away from that world. Um, so I don't know that there's like I actually don't know because I haven't had to explore this problem yet. Um, I do know that you know I've been kind of like eliding this particular feature of Harbor, but Harbor does have the ability to um like run multiple Docker images, right? Spin up multiple things. So like you could you could wonder I would wonder about that is like oh is there a way where I could somehow encapsulate my application or something? Yeah, go ahead. The question was like what scale do you use? Right? if you have some massive SQL life file or some you know

really large set of traces do you like actually evaluate on the whole thing we don't currently do that and I think it would probably be okay for us to do that um right now we you know right now the kind of format is like oh like we have some data here I'm going to have Claude kind of help me clean it up a little bit for this particular set of tasks and again the kind of the nice thing about Harbor is you know you you can um use the exact same data for many different tasks but you don't have to. Um, and so if there's a particular question that you care about and it only requires a small subset of the data to answer, you don't have to like upload all of the data to answer that particular question. Um, but yeah, I mean it it just depends too like on what you're trying to test, I think, right? If part of what you were trying to test was the performance of your agent when the system is full, when the database is like quite full, then I think it would totally make sense to um put in a lot of data. We we haven't done that yet. Well, I tried to Yeah, that's such a good question. I think the question is like what is what what do you get by using Phoenix with Harbor?

Why not just use Harbor maybe? Um and the answer is like you totally can use just Harbor, right? I'll I'll give you a little demo um in just a second where hopefully some of some of what we offer is is on display, right? Because I I do think that there are some genuinely unique things around like data set versioning, around the experiments, around the visualizations, around the the agent experiences that you get. um the ability to like actually inspect and drill down quickly, right? So, we've been we've been solving those kinds of problems um for several years and and our UI is like really dialed in for that sort of thing. And I'll give you a demo in just a sec. Yeah. And then you can ask me again, see if I answered the question. All right. Uh All right. I'm going to go to the next slide. Right. So this particular example that we're going to think about is um this problem that we encountered recently. Uh the problem was you know in this picture that I've shown you I talked about rest and I talked about GraphQL. So our MCP server our CLIs they had access to these APIs but we found that people were asking questions that are not easily answerable via REST or GraphQL. So they were asking aggregate kind of questions, right? They were asking, you know, what was the busiest day that I had in the last few months or something, right? Um how many traces did I get on that day? Or they were asking questions like how much am I spending on uh each of these models, right? Like which model is actually costing me the most out of all my LLM calls, right?

Where am I spending the most? Um and so what the agents would try to do is they would uh you know when possible they would uh look for the the rest route or the uh the GraphQL resolver. Um but these were not aggregate resolvers. So they would try to like basically pageionate through stuff and and do the aggregation themselves. And as you can imagine that results in a problem because if there's enough data you wind up uh you wind up doing a lot of a lot of pageionations right and a lot of tool calls. Um, and so part of what we did was we actually added this this line right here. So this line uh is meant to represent a tool that we added to our MCP server um, which basically allows the agent to write a readonly SQL query and execute it against the database. Um, and a lot went into making that tool. I won't really go into the implementation details but the thing that we noticed even after we made that tool was that oh sometimes sometimes the agent isn't calling the right tool because now the agent has this other problem right now the agent is here and it's like well I got rest I got graphql and I got SQL which one which one should I be using the agent doesn't really know and so we realized oh yeah well naturally we have to like somehow update a prompt we got to tell it what to do right um but even after you tell it what to do how do you verify that it's actually doing it Okay. Um, you know, it's one thing to change the prompt, but again, we're trying to like

ship based on some evidence, right? And not just on vibes. Um, so this is this is the change that we we wound up testing out. It's like a small little tweak to a to a prompt, right? So before it it only mentioned uh GraphQL, but after it says, "Hey, if you're dealing with like a some kind of aggregate like a count or group, um why don't you use our SQL tool rather than reaching for GraphQL first?" Right? Um, and of course the most important the most important thing that we need in order to answer this question is first of all some like realistic data. So we throw in a you know semi-realistic um SQL life file that we've been kind of just like collecting as we develop ourselves as we work ourselves and like dog food the product ourselves. Um, but then here's the really the part that I found quite cool or novel about Harbor, which is like um I literally just asked Claude like can you look at like the domains of my application, you know, traces, spans, um, data sets, experiments. Can you look at all these domains? Can you brainstorm a list of user queries, some of which would be aggregate, some of which would not be aggregate, right? And then can you write an Oracle solution to those queries, right? Um the Oracle solution literally just being a REST call or perhaps a database query and the coding agent will just do that. So we generated just this data set of synthetic questions. Um I spot check the

questions, right? Just to make sure they sort of make sense. Um and I have a certain degree of confidence in the actual verifiers which again are just scripts that are that are running in the environment. But because those verifiers um are passing against an oracle again those verifiers are just like you know kind of passing against this oracle I have a certain amount of confidence in them right some of them were LLM as a judge some of them were SQL queries or or REST calls themselves but you know just the ability to kind of work out those kinks before the agent ever gets to the system it gives you a certain level of confidence um and then you know I was able to test out this particular change and I was able to see like, oh, you know, as you might expect, making a change like this, it causes the system to use SQL more frequently and more aggressively, which is what I wanted it to do, right? I wanted to overuse SQL rather than underuse it because I don't want it to like start trying to pageionate through trying to do like aggregate queries. That would be bad. And then the other interesting thing is just by changing this one prompt I can also see the number of tool calls in each in each uh trajectory the average number of tool calls dropping which again is like also you know the reason that we did this in the first place is I wanted to reduce the number of tool calls and this data set it wasn't even like a big data it wasn't a big data set so it didn't have to do a lot of pageionations I think it these numbers like these modest drops they kind of understate the issue because the data was not large and So let me let me show you now what this looks like inside of

Phoenix. So the first thing that you can see here um are the examples in our data set, the tasks in our data set. And again, these are versioned. So we actually have a whole version history here every time there's a change. Um and we can inspect what some of these look like. So again, some of these are aggregate queries, some of them are not. So here's an example of an aggregate query that I might care about. What are the three most expensive models by total cost? How much does each cost? So this question is more easily answered with a SQL than than by pageionating through like a REST API. And down here you can see the answer and again this answer um it comes from an oracle. Um and so that's part of why I have a certain level of faith in it is because it's it's coming from a SQL query run against a concrete set of data. It's not something that I feel um a strong need to check actually for this particular example because the answer is deterministic and I'm I'm sort of allow I'm I'm placing a certain amount of trust in Claude I guess to like write a good SQL query. Um so we got all our examples here and then we got our um we got our experiments and our notion of an experiment that's that's what I've been calling a job in Harvard. So just imagine this to be all the jobs that we've run. I've run a lot of jobs in preparation for this talk. Um, these charts show you the kinds of metrics that you might care about, whether those are cost or run latency. Uh, the reward,

as you can see, for this particular task is quite high. Um, and that's because I haven't really done anything that's going to like hopefully break the system. It's not like something was truly genuinely broken. The system was getting the right answer in the beginning. So, the reward was always high. Um, it's just that it was just taking too many tool calls, right? So this was more of an optimization rather than a correctness fix. You can see tool call count. You can see um whether or not we use SQL. >> How do you calculate the reward? >> So the reward if we go back in the slide, the reward is just coming from the verifier. And the verifier is something that you the user have in your control. Right? So when you're defining your task, for each task, you get to write a script that says what constitutes success or failure. And that script is going to emit a zero or a one, a pass or a fail. You get to name the reward, right? So the reward could have a different name other than pass fail. Um that's all a reward is though. you as the developer, you as the maintainer, you think about for this particular task, what what does it mean to succeed and what does it mean to fail? How could I figure out just by looking at the side effects in the environment, looking at the file system or database file for example, how could I figure out whether the agent succeeded and that's what the reward is and so that's what we're tracking over time. There are the aggregate rewards for the entire job.

That's what we're tracking in this view. Yeah, go ahead. >> I mean this is such a good question, right? I kind of get it kind of like gets to one of these big big qu big big picture questions about like what do you use Harbor for? Actually the question was around like well hey you said that you know typically your verifiers are agnostic with respect to the the trajectory or the execution path of the agent but hey like look some of your verifiers actually like kind of it looks like they're peeking in to see whether the agent use SQL. Isn't that cheating? Um the answer is yes. So it's not I think the times when you would do what I said in the purest form of like your verifiers are truly agnostic and they really don't look at the trajectory and they really just look at the file system you know or the the side effects the situation for that is really like um when you care about correctness right so may you know one of the other things that I'm not showing here which is a different example that we've done is like we've worked on some regression tests and those tests are really like does it get to the right answer in the end and for Those tests I think our verifiers are quite pure. They don't really look at the trajectory. They just look at like oh did it get the right answer right? For this one for this particular problem I was concerned with something maybe subtly different. Like I wasn't really just trying to get the right answer. I was trying to optimize a metric. Right? I was trying to optimize tool calling in particular tool calls.

Harbor is one of these things where you can use it for many things. You could use it for uh you know I I talked about integration testing. You can use it for like kind of endto-end testing um for assuring correctness um for running in CI right or you could also use it because you're like oh right now I'm trying to solve a really particular problem which is like oh I kind of noticed that it sucks on these aggregate queries like how can I make it better how could I in a quick and dirty way kind of convince myself that it's actually doing what I hope it's doing right um so like at least for for us like what I found so far is it just depends on what you're trying to do um sometimes sometimes it is worth peing inside the trajectory I think you know [laughter] yeah even though even though it makes the test less robust to change perhaps that that's a really good question you know my team is so small I don't know that we can I can really speak from personal experience um that being said like one of the reasons that you you might use a tool like Phoenix is because it allows you know it has like the ability to share data right to curate data from different sources to add it you know to to like add traces to a data set um it has the ability to like store data as a you know in an external file and like upsert it to the database and then run it has those kinds of abilities. So like that might help you in terms of the the uh logistics of like you know actually like dealing with this in a larger team. I I don't I don't know because really it is just me and a handful of people working on this right now. All right, I'm going to show you one last little demo and then I'll just take questions. I think we got only a

handful of minutes here left. So you know back to back to your question around like what are some of the things that you might get out of Phoenix. I think this is like kind of the the one of the bigger ones, right? Is like the ability to compare individual jobs at the task level, right? So right here I have two two particular jobs selected. I have opus 5 before and after making that single prompt change. This column is opus 5 before. This column is opus 5 after these these metrics at the top. They're telling me in aggregate how did these like metrics that I care about, how did these rewards change after I made this tweak to the prompt, right? And I can see that like GraphQL tool calls went down. So it's calling GraphQL in aggre in it's calling GraphQL less than it was. Um the reward is staying the same which means everything is still passing. Right? SQL calls are going up. Tool counts are going down. The number of turns the agent is taking is going down because it's encapsulating a lot of that a lot of those turns that would have been multi-turn steps into a single turn because it's using SQL. uh it's using more SQL, it's using less GraphQL, right? So you get those kind of like top level metrics which are the ones that in my mind verify, okay, this is a decent change. Um and then I'm able to see in this table like four particular tasks, right? Here's the here's that one we looked at like what are the three most expensive models by by total cost. I'm able to see what I expected the answer to be, right? I'm able to see the actual answer in each job. I'm able to see the

metrics and how they changed. So for this one um yeah for this one the number of tools went down the number of turns went down interestingly enough the number of SQL calls actually went down too which is kind of unexpected and I can do things like I can I can filter by these right so I can here's a fun little feature that Miko built recently um which is I can say show me the runs where the second experiment had um where the second experiment ment had lower tool call count and I'll use the exact the exact metric to help it out. Lower tool count than the first experiment. We'll see if this works. So, what this is doing is it's actually running a a local model in the browser to convert that piece of natural language into a query DSL.

I think it messed that up though. We gota we got to use a more powerful one. >> The prompt API is a little bit >> I'll just I'll just copy in what I know to be the correct uh correct expression here. So I'm able to filter down. So now I'm just looking at the um now I'm looking at the rows where it improved basically like I'm looking at the ones where it didn't use SQL before where now it's using SQL. Um and you can see here this one used SQL it went up by one and if we actually like look at the uh look at the trace we can actually see that right so those ATIF trajectories that get converted into open inference traces here you can see it you can see it was using bash before um and when it's using bash under the hood it is actually using graphql it's calling has a bash emulator which is calling graphql underneath the hood right so I'm able to see that right here I'm able to see in the second job the same trace or the analogous trace and I'm able to see oh instead of instead of calling bash and using graphql here it actually issued a SQL query directly to the database so I'm able to like visually inspect and say oh okay in this particular example it did the right thing um and I can even I can even ask you know the agent itself so this is kind of like what we do um you know I can literally ask the agent

itself like hey look at a particular um set of experiments, you know, look at this particular task and it will tell me exactly what happened. And so this is kind of like how we get some of our data in the first place is we're kind of like using using the agent um recording its traces uh getting its data and using the agent to like iterate on itself. >> Yeah, I'd say with respect to the regression stuff, we're still really early days. I mean, we're trying to get like happy paths going, I would say. Um, and I think like the way that the way that we're structuring it is is around skills because I think in my mind when you're doing those regression tests, they are sort of analogous to like playright in a certain way where it's like an endto-end thing and it just includes like an agent in the loop. It includes like LLM calls in the loop. Um, and so I think skills are a really good like thing to write those kinds of tests for because at least in our case like the skills are describing workflows. They're saying, "Oh, if this happens to this, if that happens to this, right?

It's describing a workflow that you want to have happen and and so right now we're just kind of in the process of trying to nail down happy paths on our most important skills. um and ensure that the happy path passes across like the the broad array of like models and providers that we're trying to support. Right. Right. Right. Yeah. So this this particular comment is around like the ability of uh you could make the the greatest skill in the world, but like if somebody else makes another skill that kind of like where the front matter of the skill kind of treads on your feet a little bit, it's going to it may not fire anymore, right? And then it's no good if it never fires. Um, and for that sort of thing, it's an interesting question, right? It's like there's multiple ways of approaching it, right? Like one thing that you could do would you you could just say, well, I don't care which one fired as long as I have a good test and I know that like, oh well, like if this skill didn't fire, it's probably not going to pass this test, right? Um, that would be one approach. Another thing you can do and maybe this is the appropriate thing right is you can always as we said reach into the trajectory with your verifiers you can always look in you can always peek you know you can cheat a little bit you can say oh well like given this question did it call this particular skill and we have had tests like that um I think the challenge is maintainability right it's always around maintainability um if you change the skill name or something the test is going to flake um

but we've had tests where the thing we were trying to test was skillfiring, right? And those tests are not they're not the hardest to write, right? I mean, don't even because if if you're only testing one turn and all you want to know is like, did the right skill fire? Um, you don't even really need great data for that. You just ask it a question and and see did the skillfire. >> Um, and like you can totally, you know, maintain a suite of tests that that do that, right? And and you know, then it becomes less of a end to end test. it becomes more of like a almost like it's like almost like you know on the spectrum between you know fully decoupled from the implementation and coupled to the implementation is a little bit more coupled um and yet it could still be useful to you right um so there's no reason to not write a test like that um there's no reason to not write a test like that I think so it really just depends on like you know getting getting a broad level of coverage um on the things that you care about. We are collecting traces in production. So when you deploy Phoenix in production, this is the picture. The traces are coming into the system from an instrumented application or agent. We're collecting those traces and we're writing them to the database.

When we're running things in harbor, you can see [clears throat] that this this thing disappears, right? the OTOP ingest, the AI agent, which is the user's agent, those disappear and we just seed the database with data. So when you're running when we evaluate in harbor, um we don't we don't pretend or emulate an end user's application. Um we assume that the end user's application exists and has resulted in the data in the in the SQL life file. Yeah. So the question is around um Harbor is canonically used for evaluating coding agents. >> That's the canonical use case and we're doing something similar on the one hand but the unit that we're trying to evaluation to to evaluate here is not the coding agent in isolation because we don't work on cloud code, right? We don't we don't tweak cloud code. Uh we don't tweak codecs, right? The thing that we're trying to optimize for is this hybrid system which is cloud code plus our particular bespoke MCP server or cloud code plus our particular bespoke set of skills that we maintain or cloud code plus our um cloud code plus uh our CLI or cloud code um interacting with our APIs.

And that is a different problem, right? And the fact that that is the thing that we're trying to optimize this hybrid entity which includes cloud code, it includes the coding agent, but it's more than just the coding agent. It's all of the um supporting infrastructure around the coding agent which we maintain which helps cloud code be better at interacting with our product. Um, that's the reason that we've shoved all of this stuff into an environment together is because all of that stuff has to change um and be evaluated together um as a unit. I think it took a second to kind of conceptually land on and understand Harbor well enough to land on this picture. Um, and then it took a second. it didn't take super long and again like I think part of that is part of that is due to our advantage which I mentioned which is that Phoenix runs on a single node um and that makes it easy to dockerize right and so as as some of these other folks suggested it's like you know if you're working on a distributed system you know it's it's tougher your your life is going to be a little tougher right and so we had a certain advantage that made it easier for us to set this picture up um but I would say like the coding agents are Um, Harbor's tough in the sense that like there's no because the contract that Harbor tries to honor is all based on like scripts in certain directories. You know, I need to put my my task my task initiation script here. I need to put my

verifier script here. Um, there's no like llinters that the coding agents can rely on to make sure that that works. Um, the best that you can do uh is you can download Harbor's skills. Um and those will help you set up your project correctly. Yeah. So the question is uh I have a SQLite file here. Where do I put it? Right. Um cloud storage. Yeah. So basically um it's kind of a one-off process right now. It's like we have a collection of data that we generally use and if that thing works for the task at hand, I tend to reuse that file. And if it if I don't need the whole file, I'll say Claude, can you clean this up? >> [laughter] >> can you can you cut out the stuff I don't need um and push it to cloud storage for me? Um that data though, the data is the hard part, right? I think the data getting the data is the hard part um for for any of these tasks. And the more realistic your data, the better the system is going to run uh the better it's going to like uncover realistic issues for you. So it's always the hard part getting data