Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Well, let's begin. Welcome Well, let's begin. Welcome to our to our to our seminar on evaluating AI seminar on evaluating AI applications applications applications using Langfuse. My using Langfuse. My using Langfuse. My name is Annabel. I am an name is Annabel. I am an name is Annabel. I am an AI solutions engineer AI solutions engineer AI solutions engineer at Langfuse. at Langfuse. With me today is my With me today is my colleague Donnelly, who colleague Donnelly, who colleague Donnelly, who leads leads leads AI architecture at Langfuse and AI architecture at Langfuse and AI architecture at Langfuse and works with our works with our works with our largest largest largest customers, customers, customers, helping them helping them helping them release trusted release trusted release trusted agents into production. agents into production. agents into production. In my " In my " previous life," previous life," previous life," as I call it, because it as I call it, because it as I call it, because it feels like an feels like an feels like an eternity ago, I worked eternity ago, I worked eternity ago, I worked for a company that for a company that for a company that built AI agents built AI agents built AI agents to verify the to verify the to verify the compliance of compliance of compliance of medical devices. In medical devices. In medical devices. In this industry, it was this industry, it was this industry, it was crucial to conduct crucial to conduct crucial to conduct assessments and assessments and assessments and ensure that we ensure that we ensure that we had a data set had a data set had a data set validated by our validated by our validated by our customers. We also customers. We also customers. We also used used used many systems with many systems with many systems with human involvement. And human involvement. And human involvement. And I spent a lot of time I spent a lot of time I spent a lot of time at Langfuse at Langfuse . Somehow I decided: . Somehow I decided: . Somehow I decided: why not switch why not switch why not switch sides and start sides and start sides and start helping other helping other helping other teams teams teams developing agents, developing agents, developing agents, especially in especially in especially in regulated industries regulated industries . So about 7 . So about 7 . So about 7 months ago I months ago I months ago I joined the joined the joined the Langfuse team. Langfuse team. Langfuse team. Before we dive into the Before we dive into the Before we dive into the topic, we would like to topic, we would like to understand a little better who is gathered here today. We want gathered here today. We want to know which of you has to know which of you has to know which of you has already put AI agents already put AI agents already put AI agents into production? I into production? I into production? I mean direct mean direct mean direct participation in the participation in the participation in the development team, not development team, not development team, not just knowing that just knowing that just knowing that someone in the company someone in the company someone in the company did it. That is, you did it. That is, you did it. That is, you were personally were personally were personally part of the process of part of the process of part of the process of bringing the product to the bringing the product to the bringing the product to the world. Okay, that's good world. Okay, that's good . Perfectly. Many people . Perfectly. Many people . That's exactly what we were . That's exactly what we were . That's exactly what we were hoping for. And who among hoping for. And who among hoping for. And who among you has already worked with you has already worked with you has already worked with Langfuse? Okay, quite a Langfuse? Okay, quite a Langfuse? Okay, quite a few. What about few. What about any other any other monitoring and monitoring and evaluation tools? Okay, there

evaluation tools? Okay, there evaluation tools? Okay, there are some too. Surprisingly. And are some too. Surprisingly. And are some too. Surprisingly. And finally: who has finally: who has finally: who has completely solved the completely solved the completely solved the issue of evaluating AI issue of evaluating AI applications? I invite those who are applications? I invite those who are applications? I invite those who are confident confident confident to our stand to our stand — tell us how — tell us how — tell us how you did it. After all, you did it. After all, you did it. After all, it's actually quite it's actually quite it's actually quite difficult, and I think difficult, and I think difficult, and I think each of you each of you each of you has encountered this if you work has encountered this if you work has encountered this if you work with AI or just with AI or just with AI or just communicate with ChatGPT or communicate with ChatGPT or communicate with ChatGPT or Claude. This is quite difficult, Claude. This is quite difficult, Claude. This is quite difficult, because AI agents can because AI agents can because AI agents can make mistakes make mistakes , even when , even when , even when system metrics system metrics system metrics look great. For look great. For look great. For example, example, example, look at this look at this look at this case. We have a case. We have a user support agent. user support agent. The user asks: " The user asks: " Can I get a Can I get a Can I get a refund?" refund?" refund?" The agent processes the request, The agent processes the request, The agent processes the request, checks the rules, and checks the rules, and checks the rules, and replies: “Yes, replies: “Yes, replies: “Yes, of course. I have issued a of course. I have issued a of course. I have issued a refund." And refund." And refund." And the user the user the user replies: "Thank you." replies: "Thank you." replies: "Thank you." And it will take a few days for And it will take a few days for And it will take a few days for the user the user the user to see if everything to see if everything to see if everything really worked. really worked. really worked. The status is green, from the The status is green, from the The status is green, from the point of view of traditional point of view of traditional point of view of traditional monitoring, nothing monitoring, nothing monitoring, nothing went wrong. There are also went wrong. There are also went wrong. There are also no problems with no problems with no problems with delays. But if delays. But if delays. But if you dig deeper into what you dig deeper into what you dig deeper into what actually happened, the actually happened, the actually happened, the agent didn't agent didn't agent didn't return anything. So, return anything. So, return anything. So, the tool was not the tool was not the tool was not called; it did not called; it did not called; it did not work, and the agent work, and the agent work, and the agent lied to lied to lied to the user that he the user that he the user that he did it. We'll did it. We'll only find out in a few weeks when the user when the user when the user comes back and says, " comes back and says, " Hey, by the way, I Hey, by the way, I Hey, by the way, I never got my never got my never got my refund." And that's the refund." And that's the refund." And that's the problem we problem we problem we have with these have with these have with these non-deterministic non-deterministic non-deterministic systems. They systems. They systems. They may tell us they are may tell us they are may tell us they are performing actions, but they performing actions, but they performing actions, but they don't actually don't actually don't actually do them. That's why you do them. That's why you do them. That's why you need to find an need to find an need to find an approach and a way to approach and a way to approach and a way to constantly look for constantly look for constantly look for signals of where signals of where signals of where something is going wrong and something is going wrong and something is going wrong and ensure it doesn't ensure it doesn't ensure it doesn't happen again. Or happen again. Or happen again. Or maybe it won't even maybe it won't even maybe it won't even happen at all.

happen at all. happen at all. Our Our Our founders founders founders faced this same problem 3 years faced this same problem 3 years faced this same problem 3 years ago when they were at the ago when they were at the ago when they were at the Y Combinator accelerator. Y Combinator accelerator. Y Combinator accelerator. They realized: okay They realized: okay , there's nothing , there's nothing , there's nothing like like like unit unit unit testing for AI testing for AI applications, and we applications, and we applications, and we need a solution. So need a solution. So need a solution. So they created LangFuse, they created LangFuse, they created LangFuse, which is now an which is now an open-source platform for open-source platform for continuously continuously continuously tracking, tracking, tracking, monitoring, and monitoring, and monitoring, and improving your improving your improving your AI systems. And while AI systems. And while AI systems. And while we started out as a we started out as a tool for tool for developers, developers, developers, our target audience now our target audience now our target audience now includes industry includes industry includes industry experts who experts who experts who work with AI work with AI engineers to make the engineers to make the engineers to make the system do what it's system do what it's system do what it's supposed to do. And also a new supposed to do. And also a new supposed to do. And also a new group of users group of users group of users over the last 8-12 over the last 8-12 over the last 8-12 months are months are months are coding agents who coding agents who coding agents who help our help our help our engineers work engineers work engineers work with and improve AI applications with and improve AI applications with and improve AI applications . . Langfuse is now Langfuse is now used by over used by over used by over 150,000 developers 150,000 developers 150,000 developers worldwide, thanks in large part to worldwide, thanks in large part to worldwide, thanks in large part to our our our open source open source open source code, of course. 21 code, of course. 21 Fortune 50 companies use Fortune 50 companies use us as part of their us as part of their us as part of their core core core technology stack technology stack technology stack when deploying when deploying when deploying AI applications, and AI applications, and over over 90 billion 90 billion 90 billion observations are fed into Langfuse instances every month, a observations are fed into Langfuse instances every month, a observations are fed into Langfuse instances every month, a trend that is growing trend that is growing trend that is growing as agents as agents as agents take on take on take on more steps and more steps and more steps and take on take on take on more tasks.

more tasks. more tasks. Companies like Ramp, Canva, Companies like Ramp, Canva, Companies like Ramp, Canva, Khan Academy, and SumUp Khan Academy, and SumUp Khan Academy, and SumUp work with us, and work with us, and work with us, and we've we've we've certainly spent a certainly spent a certainly spent a lot of time with them over the past 3 years. lot of time with them over the past 3 years. lot of time with them over the past 3 years. One thing we realized One thing we realized One thing we realized while working with these while working with these while working with these companies is that those who have been companies is that those who have been companies is that those who have been successful in successful in successful in producing reliable producing reliable producing reliable agents and continuously improving them agents and continuously improving them follow what we follow what we call the AI call the AI engineering cycle. engineering cycle. It is a set of It is a set of practices that practices that practices that help you help you help you continuously continuously continuously improve your improve your improve your system through a system through a system through a systematic approach. systematic approach. The basis for this The basis for this is always tracing. is always tracing. is always tracing. Tracing is the Tracing is the Tracing is the recording of all the steps of recording of all the steps of recording of all the steps of your agent's work your agent's work your agent's work in the form of a in the form of a in the form of a kind of log, kind of log, kind of log, but ideally in a form but ideally in a form but ideally in a form accessible for accessible for accessible for human inspection. human inspection. human inspection. Monitoring is about Monitoring is about Monitoring is about identifying the identifying the identifying the most interesting most interesting most interesting moments. That is, what is moments. That is, what is moments. That is, what is going wrong, what is working going wrong, what is working going wrong, what is working well, where you should well, where you should well, where you should look more closely look more closely , and an understanding , and an understanding , and an understanding of how the system of how the system of how the system works on average? works on average? works on average? Creating Creating Creating datasets is about finding datasets is about finding datasets is about finding representative representative representative examples of what your examples of what your examples of what your program should be able program should be able program should be able to handle and to handle and how it should how it should how it should do it. Then you do it. Then you do it. Then you conduct conduct conduct experiments, experiments, experiments, trying out different trying out different trying out different variations of your variations of your variations of your system to system to system to see how it see how it see how it works, how works, how works, how things are progressing, things are progressing, things are progressing, and of course, evaluate the and of course, evaluate the and of course, evaluate the results. This is not a results. This is not a results. This is not a one-time action, you are one-time action, you are one-time action, you are constantly closing constantly closing constantly closing this loop, striving this loop, striving this loop, striving to improve AI to improve AI agents by training agents by training agents by training on real data and feeding on real data and feeding on real data and feeding it back into it back into it back into datasets. But datasets. But datasets. But let's look at let's look at let's look at each of these steps in each of these steps in each of these steps in a little more detail.

a little more detail. So, the So, the most important goal for tracing most important goal for tracing most important goal for tracing is to gain a is to gain a is to gain a deep understanding deep understanding deep understanding of what exactly of what exactly of what exactly your agent is doing. On the right side of your agent is doing. On the right side of your agent is doing. On the right side of the screen we see the screen we see the screen we see a trace, this is a snapshot a trace, this is a snapshot a trace, this is a snapshot from LangFuse, and in this from LangFuse, and in this from LangFuse, and in this case it's an IT case it's an IT support agent. That is, an support agent. That is, an support agent. That is, an agent who agent who agent who answers answers answers your your your parents' questions so you don't have to parents' questions so you don't have to . We . We first see the root first see the root first see the root trace, the trace, the trace, the total time and total time and total time and cost of the step, and cost of the step, and cost of the step, and then we get then we get then we get a breakdown of each a breakdown of each a breakdown of each individual step that individual step that individual step that the agent performs. the agent performs. Executes. That is, both the Executes. That is, both the generations, which are generations, which are generations, which are OpenAI chats, and the OpenAI chats, and the OpenAI chats, and the tool calls tool calls , which we see , which we see , which we see thanks to the thanks to the thanks to the tool icon next to them. tool icon next to them. tool icon next to them. So, we have a So, we have a So, we have a detailed overview of detailed overview of detailed overview of what's happening, and what's happening, and what's happening, and if the screen were wider if the screen were wider , we would see the input , we would see the input , we would see the input and output data, as well as and output data, as well as and output data, as well as all the relevant all the relevant all the relevant metadata. This becomes the metadata. This becomes the metadata. This becomes the basis for basis for basis for tracking cost tracking cost , latency, , latency, , latency, token count, and a token count, and a token count, and a log for your log for your log for your audit, so you audit, so you audit, so you can can can dive deep into the process dive deep into the process dive deep into the process and your and your code agents can code agents can code agents can work with it work with it work with it at scale, having a at scale, having a verifiable log of verifiable log of what actually what actually what actually happened.

happened. Monitoring is about Monitoring is about making sense of making sense of making sense of all this at scale, all this at scale, all this at scale, because, especially because, especially because, especially now, with the increasing now, with the increasing now, with the increasing level of automation, level of automation, level of automation, longer traces, longer traces, longer traces, and a large number of and a large number of use cases, it is use cases, it is no longer possible for a human to manually no longer possible for a human to manually no longer possible for a human to manually review all these review all these review all these logs. Instead, you need a logs. Instead, you need a logs. Instead, you need a way way way to get aggregate to get aggregate to get aggregate metrics of what's metrics of what's metrics of what's actually actually actually happening. And also happening. And also happening. And also to detect signals that to detect signals that to detect signals that something is going wrong. something is going wrong. something is going wrong. These could be, These could be, These could be, for example, for example, for example, cost and latency metrics, cost and latency metrics, cost and latency metrics, but also but also but also quality signals, like quality signals, like quality signals, like compliance with our compliance with our compliance with our security policies, or how security policies, or how security policies, or how many users are many users are many users are actually actually actually frustrated with our frustrated with our frustrated with our system, and how many system, and how many system, and how many are writing messages in are writing messages in are writing messages in all caps because all caps because all caps because they're maybe a little they're maybe a little they're maybe a little upset by the upset by the upset by the agent's actions. And so, gathering agent's actions. And so, gathering agent's actions. And so, gathering all this all this all this information, and then, information, and then, information, and then, of course, finding interesting of course, finding interesting of course, finding interesting points that you points that you points that you want to delve into. want to delve into. What you get from your What you get from your work environment work environment work environment through monitoring through monitoring through monitoring becomes one of the foundations becomes one of the foundations becomes one of the foundations that you can that you can that you can use to use to use to build your build your build your datasets. And datasets. And datasets. And datasets are, datasets are, datasets are, after all, a after all, a after all, a representation of what representation of what representation of what you think an you think an agent should be able to handle in a work agent should be able to handle in a work environment. So, these are environment. So, these are environment. So, these are all the cases all the cases all the cases it may encounter, as it may encounter, as it may encounter, as well as requests that are well as requests that are well as requests that are beyond the scope of beyond the scope of beyond the scope of the application that you the application that you the application that you want to correctly want to correctly want to correctly reject. You need to reject. You need to reject. You need to make sure that make sure that make sure that the agent is not trying to the agent is not trying to the agent is not trying to give investment give investment give investment advice or advice or advice or answer answer answer questions about the weather in questions about the weather in questions about the weather in New York City if they are New York City if they are New York City if they are really just a really just a really just a technical support agent.

technical support agent. technical support agent. For these elements, you For these elements, you For these elements, you can have input can have input can have input data as well as data as well as data as well as expected output expected output expected output data, which can data, which can data, which can be either a sample be either a sample be either a sample response, the response, the response, the expected behavior expected behavior expected behavior you want to see, you want to see, you want to see, or keywords or keywords or keywords that should be that should be that should be captured. This can then be captured. This can then be used as a used as a stable reference stable reference stable reference for offline improvements and for offline improvements and for offline improvements and iterations or iterations or iterations or as part of your as part of your as part of your CI/CD process. These data sets CI/CD process. These data sets are then used to are then used to conduct conduct conduct experiments. experiments. experiments. Experiments, if Experiments, if Experiments, if you think back to a you think back to a you think back to a university university university statistics course or statistics course or statistics course or something like that, are something like that, are something like that, are ultimately the idea of ​​changing ultimately the idea of ​​changing ultimately the idea of ​​changing something in something in something in a setting and a setting and a setting and observing what observing what observing what happens happens happens next. For example, next. For example, next. For example, changing the prompt, model changing the prompt, model , context, or , context, or , context, or tool description to tool description to tool description to see if it does a better job of capturing see if it does a better job of capturing what you actually what you actually wanted to capture, and wanted to capture, and wanted to capture, and ideally, isolating ideally, isolating ideally, isolating individual aspects. So individual aspects. So individual aspects. So if you want to if you want to if you want to test whether the test whether the test whether the same model works better same model works better same model works better with a different prompt, with a different prompt, with a different prompt, ideally you should ideally you should ideally you should only change the prompt, not only change the prompt, not only change the prompt, not five other things, five other things, five other things, otherwise it will be difficult for you otherwise it will be difficult for you otherwise it will be difficult for you to interpret what to interpret what to interpret what you are actually seeing. you are actually seeing. And finally, you And finally, you need assessments or need assessments or evaluation approaches so you evaluation approaches so you can measure the can measure the can measure the results of these results of these results of these experiments. After all, experiments. After all, experiments. After all, you don't want you don't want you don't want to check everything to check everything to check everything yourself—although, by yourself—although, by yourself—although, by the way, sometimes it's worth the way, sometimes it's worth the way, sometimes it's worth doing that—so you doing that—so you doing that—so you need need need scalable approaches scalable approaches , like LLM as a judge.

, like LLM as a judge. That is, using That is, using another LLM to another LLM to another LLM to check the performance of the check the performance of the check the performance of the main LLM, or main LLM, or code-based evaluation for code-based evaluation for classification tasks where an classification tasks where an exact match can be checked, or exact match can be checked, or deterministic deterministic deterministic behavior checking, behavior checking, behavior checking, such as whether text is such as whether text is such as whether text is capitalized capitalized capitalized . This is the . This is the . This is the AI ​​engineering cycle, and we AI ​​engineering cycle, and we AI ​​engineering cycle, and we see that many of see that many of see that many of our customers our customers have have different entry points into different entry points into different entry points into this process depending on the use case. If you this process depending on the use case. If you this process depending on the use case. If you are working are working are working in a in a in a low- low- low- risk environment, for example, and are risk environment, for example, and are risk environment, for example, and are only building an internal chat only building an internal chat agent, you may agent, you may not have any monitoring tools at all at first, and you monitoring tools at all at first, and you will start by will start by will start by tracking and tracking and tracking and adding monitors adding monitors adding monitors to see where to see where to see where users users users start to get nervous. start to get nervous. start to get nervous. But at the initial But at the initial But at the initial stage, you don't really stage, you don't really stage, you don't really care about the care about the care about the offline assessment part. offline assessment part. offline assessment part. You simply develop You simply develop You simply develop the system over time and the system over time and the system over time and add test add test add test cases as you cases as you cases as you go, but you go, but you go, but you can can can go into production right away. go into production right away. go into production right away. However, if you However, if you However, if you are building a are building a customer-facing solution, especially in customer-facing solution, especially in regulated industries regulated industries , you probably don't , you probably don't , you probably don't want to risk an want to risk an want to risk an agent doing agent doing agent doing something they something they something they shouldn't. So for shouldn't. So for shouldn't. So for regulated industries, regulated industries, regulated industries, we see several we see several emerging approaches, and the emerging approaches, and the foundation is always foundation is always foundation is always tracing. You tracing. You tracing. You always need to always need to always need to know what is know what is actually happening: what steps actually happening: what steps are being performed, what are being performed, what are being performed, what input and output input and output input and output the agent sees, and the agent sees, and the agent sees, and what tools it what tools it what tools it calls. And on top of calls. And on top of calls. And on top of that, teams in that, teams in that, teams in regulated industries regulated industries regulated industries have found several have found several have found several ways to build ways to build ways to build benchmark benchmark benchmark datasets that can be datasets that can be datasets that can be used for

used for used for iteration, improvement, iteration, improvement, iteration, improvement, and, as you may have heard and, as you may have heard , automated , automated , automated improvement. All of this improvement. All of this improvement. All of this requires a foundation in requires a foundation in requires a foundation in the form of a set of the form of a set of the form of a set of reference data, and there are reference data, and there are reference data, and there are several approaches to several approaches to several approaches to achieving this. achieving this. achieving this. For example, expert For example, expert For example, expert curation. That is, curation. That is, curation. That is, working with specialized working with specialized working with specialized experts within the experts within the experts within the company to company to company to ensure the ensure the ensure the correct formation of a correct formation of a correct formation of a set of reference set of reference set of reference data. Next, data. Next, data. Next, for example, shadow for example, shadow for example, shadow launches. You perform launches. You perform launches. You perform the process manually, but the process manually, but run the agent in parallel run the agent in parallel so that you can so that you can so that you can compare how the compare how the compare how the agent handled it and what agent handled it and what agent handled it and what steps it took. steps it took. steps it took. This way, you can This way, you can This way, you can measure accuracy and measure accuracy and measure accuracy and also build also build also build your own your own your own reference database. And reference database. And reference database. And finally, as we did finally, as we did finally, as we did at my previous at my previous at my previous company, run the company, run the company, run the system entirely with a system entirely with a system entirely with a human, and over time, human, and over time, human, and over time, separate the data into separate the data into separate the data into parts where you can parts where you can parts where you can confidently confidently confidently automate and automate and automate and those where you always those where you always those where you always want to see a human want to see a human want to see a human in the process. So, this in the process. So, this concludes the introduction. Now we concludes the introduction. Now we want to move on to a want to move on to a want to move on to a more practical more practical more practical example, and for that example, and for that example, and for that I invite I invite I invite Donnelly back on stage. And Donnelly back on stage. And Donnelly back on stage. And together we will look at together we will look at together we will look at how to track how to track how to track your your your agent's performance in LangChain, how agent's performance in LangChain, how agent's performance in LangChain, how to evaluate and to evaluate and to evaluate and improve your improve your improve your system, and how to continuously system, and how to continuously system, and how to continuously close this loop.

close this loop. close this loop. Hello everyone. As Hello everyone. As Hello everyone. As Annabelle said, I'm Annabelle said, I'm Annabelle said, I'm Donnelly, but you can Donnelly, but you can Donnelly, but you can call me Dawn. call me Dawn. call me Dawn. Donnelly is easy for Donnelly is easy for Donnelly is easy for everyone to remember. everyone to remember. everyone to remember. Only my mother calls Only my mother calls Only my mother calls me Donnelly when I'm in me Donnelly when I'm in me Donnelly when I'm in trouble. Um, yeah, that's good trouble. Um, yeah, that's good , that's great. A few , that's great. A few , that's great. A few laughs, you laughs, you laughs, you know, a good way to know, a good way to know, a good way to lighten the mood lighten the mood . So, this is the first . So, this is the first . So, this is the first assessment. It went assessment. It went assessment. It went perfectly. So, perfectly. So, perfectly. So, I'm going to show you an example I'm going to show you an example I'm going to show you an example that we've built for that we've built for that we've built for a couple of financial a couple of financial a couple of financial institutions that we institutions that we institutions that we work with, work with, work with, because they're because they're because they're at the stage where they've at the stage where they've at the stage where they've built their first built their first built their first prototype, their first prototype, their first demo, but then the demo, but then the demo, but then the question comes up question comes up question comes up from management or from management or from management or teams, "How do we teams, "How do we teams, "How do we measure these measure these measure these customer-facing customer-facing customer-facing agent solutions that agent solutions that agent solutions that we're building?" How many of we're building?" How many of we're building?" How many of you here work in you here work in you here work in financial financial financial services or finance? services or finance? services or finance? Yes, we're in New York, Yes, we're in New York, Yes, we're in New York, so of course, all of you. Are so of course, all of you. Are so of course, all of you. Are there other industries besides there other industries besides there other industries besides financial services that are financial services that are financial services that are regulated, regulated, regulated, like like like healthcare or something healthcare or something healthcare or something like that? Okay, cool. like that? Okay, cool. like that? Okay, cool. Perfectly. I think it will be Perfectly. I think it will be Perfectly. I think it will be very close to you. very close to you. So, essentially, what I So, essentially, what I have here is a typical have here is a typical have here is a typical bank assistant bank assistant . Every one of us who . Every one of us who . Every one of us who uses uses uses banking banking banking services services services has probably seen this at our has probably seen this at our has probably seen this at our bank, right?

bank, right? However, everything that However, everything that happens behind happens behind happens behind the scenes is very the scenes is very the scenes is very important for the important for the important for the bank or institution itself bank or institution itself bank or institution itself to provide you with the to provide you with the to provide you with the most accurate most accurate most accurate information possible, as information possible, as information possible, as there is financial there is financial there is financial liability, liability, liability, legal legal legal liability, and a liability, and a liability, and a bunch of risks for the bunch of risks for the bunch of risks for the bank when they bank when they bank when they implement such implement such implement such LLM-based solutions, LLM-based solutions, LLM-based solutions, which as we all know which as we all know which as we all know are inherently are inherently are inherently non-deterministic. non-deterministic. So, we want to So, we want to make sure that make sure that make sure that we reduce we reduce we reduce unpredictability and unpredictability and unpredictability and can reproduce can reproduce can reproduce the results as much as the results as much as possible to possible to find out where find out where find out where errors occur and errors occur and errors occur and where the agent produces where the agent produces where the agent produces results that we results that we results that we don't want, so that we don't want, so that we don't want, so that we can then can then can then improve those points. improve those points. In this example, I In this example, I want to show you want to show you want to show you a few things. a few things. First, I can First, I can First, I can represent, you know, represent, you know, represent, you know, several different types of several different types of several different types of customers, because there will customers, because there will customers, because there will always be always be always be regular users regular users like me in the bank. I like me in the bank. I like me in the bank. I only have a few dollars in the bank. only have a few dollars in the bank. only have a few dollars in the bank. But there are also But there are also But there are also ultra-high ultra-high ultra-high net worth clients and the like net worth clients and the like net worth clients and the like . So, we can . So, we can . So, we can model different model different model different types of personalities here. This is types of personalities here. This is types of personalities here. This is important because we important because we important because we want to know if want to know if want to know if the agent is adhering to the agent is adhering to the agent is adhering to all all all the restrictions we have set for the restrictions we have set for the restrictions we have set for these types of these types of these types of users. Another users. Another users. Another thing, if you notice thing, if you notice thing, if you notice here, let me here, let me here, let me zoom in.

zoom in. zoom in. We also have We also have We also have PII identification enabled, and PII identification enabled, and PII identification enabled, and any data any data any data considered considered considered personal is personal is personal is automatically automatically automatically masked before masked before masked before being stored in LangFuse. being stored in LangFuse. This is a very important This is a very important requirement, especially for requirement, especially for requirement, especially for financial institutions financial institutions financial institutions that do not want that do not want that do not want to store customers' to store customers' credit card or credit card or social social social security numbers in the surveillance system security numbers in the surveillance system . They want . They want . They want to store it in their to store it in their to store it in their accounting systems accounting systems accounting systems and have maximum and have maximum and have maximum control over their own control over their own control over their own data. So that's also data. So that's also data. So that's also enabled, and I'll enabled, and I'll enabled, and I'll show you some show you some show you some examples of how it examples of how it examples of how it works. We works. We works. We will also see whether the will also see whether the will also see whether the agent adheres to the agent adheres to the controls we have established, how controls we have established, how they are evaluated, and whether the they are evaluated, and whether the quality criteria we have defined are met. So , we measure all of this. , we measure all of this. , we measure all of this. I'll show you a few I'll show you a few I'll show you a few examples and then examples and then examples and then take you behind take you behind take you behind the scenes of the product the scenes of the product the scenes of the product to show you how to to show you how to to show you how to set it up and what the set it up and what the set it up and what the full full full workflow looks like.

workflow looks like. Let's assume that in Let's assume that in this case I am this case I am this case I am acting as Anna. I am acting as Anna. I am acting as Anna. I am asking about fees asking about fees asking about fees for international for international for international transfers. Very simple transfers. Very simple transfers. Very simple question, just a question, just a question, just a knowledge base. In a few knowledge base. In a few knowledge base. In a few seconds, the system seconds, the system seconds, the system will return and will return and will return and provide results from provide results from provide results from our knowledge base, and our knowledge base, and our knowledge base, and here they are. As you can see, here they are. As you can see, here they are. As you can see, here I am showing here I am showing here I am showing information that I would information that I would information that I would never show to a never show to a never show to a regular customer, regular customer, regular customer, but I am doing it for but I am doing it for but I am doing it for demonstration purposes. demonstration purposes. But I note here the But I note here the tools that are tools that are tools that are invoked, the invoked, the invoked, the data sources that data sources that data sources that are used, and are used, and are used, and some of the some of the some of the agent configuration details agent configuration details agent configuration details that provide such a that provide such a that provide such a response. If I response. If I response. If I said then, "Okay, said then, "Okay, said then, "Okay, look, I lost look, I lost look, I lost my card. Here's my my card. Here's my my card. Here's my number. Is it number. Is it number. Is it blocked?" I'm blocked?" I'm blocked?" I'm giving my giving my giving my credit card number in credit card number in credit card number in plain text, plain text, plain text, right? So this is right? So this is right? So this is obviously not very obviously not very obviously not very good. So, you want to good. So, you want to good. So, you want to make sure that make sure that make sure that even if you can't even if you can't even if you can't control what control what control what users users users are inputting into the system, you are inputting into the system, you are inputting into the system, you should track should track should track it when you're it when you're it when you're setting up setting up setting up tools and tools and tools and creating assessments.

creating assessments. Now we will see in Now we will see in the trace itself that the the trace itself that the the trace itself that the credit credit credit card number will be card number will be card number will be masked. If I masked. If I masked. If I go here to LangFuse go here to LangFuse go here to LangFuse and open my and open my and open my trace, you'll trace, you'll trace, you'll see that the see that the see that the question now reads question now reads question now reads "my card number—," "my card number—," "my card number—," and it went into and it went into and it went into LangFuse completely LangFuse completely LangFuse completely masked. This is because masked. This is because I have a I have a I have a masking feature that masking feature that masking feature that automatically hides automatically hides automatically hides any any any sensitive sensitive sensitive information I identify, so it is information I identify, so it is information I identify, so it is not stored in not stored in not stored in plain text on plain text on plain text on my system. So, my system. So, my system. So, to show you what a to show you what a to show you what a typical typical typical interaction looks like: at the top I interaction looks like: at the top I interaction looks like: at the top I see a prompt that see a prompt that see a prompt that the user submitted. the user submitted. But as you know, there are a lot of a lot of processes happening behind the scenes that we don't processes happening behind the scenes that we don't processes happening behind the scenes that we don't necessarily necessarily necessarily show to end show to end show to end users, but users, but users, but they are they are they are critical for us as critical for us as critical for us as platform platform platform teams and teams and teams and developers to developers to developers to understand the understand the understand the agent's behavior, its agent's behavior, its agent's behavior, its performance, and the performance, and the performance, and the quality of the results. I quality of the results. I quality of the results. I see the entire tree of see the entire tree of see the entire tree of steps he steps he steps he took. Here is the took. Here is the took. Here is the Anthropic LLM challenge he Anthropic LLM challenge he Anthropic LLM challenge he did and the did and the did and the tools available. Here he tools available. Here he tools available. Here he called up a list of called up a list of called up a list of tools, tools, tools, listed the bills, and listed the bills, and listed the bills, and so on, and these are the so on, and these are the results. But more results. But more results. But more importantly, here at the top importantly, here at the top importantly, here at the top I see all the scores that I see all the scores that I see all the scores that I defined for this I defined for this I defined for this particular particular particular usage scenario. You usage scenario. You usage scenario. You see that they are see that they are see that they are automatically automatically automatically calculated every time a calculated every time a calculated every time a new new new interaction occurs or a new interaction occurs or a new interaction occurs or a new trace is trace is trace is captured by the system.

captured by the system. captured by the system. I can record things I can record things , such as whether , such as whether the information provided to the information provided to the client meets banking the client meets banking the client meets banking standards; If standards; If standards; If a client asks about a client asks about a client asks about the weather, I the weather, I the weather, I will not provide that will not provide that will not provide that information, it is not information, it is not information, it is not compliant. compliant. Also any types of Also any types of failures or the presence of failures or the presence of failures or the presence of fuses that fuses that fuses that blocked any blocked any blocked any responses that should have been responses that should have been responses that should have been provided to the client. provided to the client. One example is someone might come in and someone might come in and someone might come in and ask, "Can I ask, "Can I ask, "Can I invest my invest my invest my savings in savings in savings in Bitcoin?" Yes? So, this is something that Bitcoin?" Yes? So, this is something that Bitcoin?" Yes? So, this is something that the bank itself the bank itself the bank itself should not provide should not provide should not provide information about, because information about, because information about, because it, you know, has certain it, you know, has certain it, you know, has certain consequences. And the agent, you consequences. And the agent, you consequences. And the agent, you know, is doing it know, is doing it know, is doing it right, but right, but right, but at the same time, he at the same time, he at the same time, he labels it as labels it as labels it as investment advice. investment advice. And that requires And that requires the person to either the person to either the person to either talk to talk to talk to a counselor or a counselor or a counselor or do something else, and do something else, and do something else, and we use an we use an we use an assessment to do that. And assessment to do that. And assessment to do that. And if I go back if I go back if I go back to my tracing, this is to my tracing, this is to my tracing, this is how I see it. how I see it. how I see it. I have a barrier to entry here I have a barrier to entry here I have a barrier to entry here that that that suggests that the primary suggests that the primary suggests that the primary risk is risk is risk is investment advice, investment advice, investment advice, and in the assessment here I see and in the assessment here I see and in the assessment here I see that the client's intent that the client's intent that the client's intent was indeed to was indeed to was indeed to receive receive receive investment advice investment advice . So this way I . So this way I . So this way I can very easily can very easily can very easily filter out exactly filter out exactly filter out exactly those cases. But those cases. But those cases. But anyway, I anyway, I anyway, I wanted to show you a kind of wanted to show you a kind of wanted to show you a kind of simple simple simple flow of this process.

flow of this process. flow of this process. One last point One last point One last point before moving on to the before moving on to the before moving on to the end-to-end process of end-to-end process of dataset creation and evaluation—is that dataset creation and evaluation—is that we also we also we also support support support multimodality. multimodality. Let's say, instead of customers writing to the customers writing to the customers writing to the chat, they call chat, they call chat, they call your contact your contact center, we can center, we can center, we can go here, say " go here, say " I want to handle this I want to handle this I want to handle this call", and the system call", and the system call", and the system will convert the speech will convert the speech will convert the speech to text, after which the to text, after which the to text, after which the text will be evaluated for text will be evaluated for text will be evaluated for any any any violations. And I can violations. And I can violations. And I can see here that see here that see here that the customer requested to the customer requested to the customer requested to block their block their block their card, and I can actually card, and I can actually card, and I can actually see the trace here. see the trace here. I can even I can even see a link see a link see a link to the media file that to the media file that to the media file that was sent to was sent to was sent to the client, which is the the client, which is the the client, which is the audio recording of the call audio recording of the call audio recording of the call the client made, as well the client made, as well the client made, as well as the transcript and the as the transcript and the as the transcript and the assistant's response. assistant's response. So this is about So this is about tracing, how can I tracing, how can I tracing, how can I capture all these capture all these capture all these actions. Now that I'm actions. Now that I'm actions. Now that I'm trying to think about trying to think about trying to think about how I would create or how I would create or how I would create or implement a framework implement a framework implement a framework for evaluating the for evaluating the for evaluating the interactions my interactions my interactions my customers customers customers would send through would send through would send through my system. Because, as we my system. Because, as we my system. Because, as we all know, customers are all know, customers are all know, customers are unpredictable. We unpredictable. We unpredictable. We may design a may design a may design a system or solution for a system or solution for a system or solution for a specific purpose, but specific purpose, but specific purpose, but our customers will always our customers will always our customers will always find very find very find very clever ways clever ways clever ways to use the to use the to use the system differently. We system differently. We system differently. We also cannot also cannot also cannot predict every predict every predict every outcome, especially with outcome, especially with outcome, especially with LLM-based solutions, LLM-based solutions, LLM-based solutions, that will arise that will arise that will arise when we release them when we release them when we release them into the real world for into the real world for customer use. That's when

customer use. That's when we start to we start to we start to rely on rely on rely on building our building our building our evaluation criteria. And evaluation criteria. And evaluation criteria. And for us, the foundation of this for us, the foundation of this for us, the foundation of this is creating is creating is creating datasets. So, what is a datasets. So, what is a datasets. So, what is a dataset? A dataset? A dataset? A dataset is essentially... dataset is essentially... let's move on to let's move on to let's move on to this, the general this, the general this, the general version. A dataset version. A dataset is essentially your is essentially your is essentially your list of questions and list of questions and list of questions and expected expected expected answers that answers that answers that allow you to measure allow you to measure allow you to measure the quality of the results of the quality of the results of the quality of the results of your LLM or AI your LLM or AI solution. If you solution. If you solution. If you have experience in have experience in have experience in software development software development software development , I , I , I suggest treating suggest treating suggest treating this as your test this as your test this as your test scripts. And I don't think scripts. And I don't think anyone here who anyone here who anyone here who has released has released has released software to software to software to production would ship production would ship production would ship something without going through something without going through something without going through quality control, unless quality control, unless quality control, unless you merge the code you merge the code you merge the code directly into the master branch directly into the master branch , which I'm sure , which I'm sure , which I'm sure many of you many of you many of you have done, but it's not have done, but it's not have done, but it's not best practice. best practice. best practice. Um, I know, I know what I Um, I know, I know what I Um, I know, I know what I did, but it doesn't matter. did, but it doesn't matter. But at the same time, it's a But at the same time, it's a way to get way to get way to get evidence that you've evidence that you've evidence that you've walked the agent through walked the agent through walked the agent through a series of milestones or requirements a series of milestones or requirements that measure everything that measure everything that measure everything you want to you want to you want to track. And it's worth track. And it's worth track. And it's worth noting that noting that noting that the expected result the expected result the expected result doesn't necessarily have to doesn't necessarily have to doesn't necessarily have to be the exact be the exact be the exact answer answer answer the agent will provide.

the agent will provide. Maybe you want to Maybe you want to measure an measure an measure an agent's intent, what it should agent's intent, what it should agent's intent, what it should do, or you want do, or you want do, or you want to evaluate its to evaluate its to evaluate its behavior. There are behavior. There are behavior. There are different ways to measure the different ways to measure the different ways to measure the quality of results, and quality of results, and quality of results, and so you typically so you typically so you typically get different get different get different sets of data to sets of data to sets of data to measure measure measure specific aspects specific aspects specific aspects or areas of the quality of or areas of the quality of or areas of the quality of your your your agent fleet. You see, here agent fleet. You see, here agent fleet. You see, here I have a I have a red red red teaming set that looks for teaming set that looks for hint injection attacks and hint injection attacks and any attempts to any attempts to any attempts to bypass the bypass the bypass the barriers we have set up to barriers we have set up to barriers we have set up to get information get information get information that shouldn't be given out. The that shouldn't be given out. The other set I have other set I have here is specifically here is specifically here is specifically for for for when I want when I want when I want to know which cases to know which cases to know which cases have been appealed and where have been appealed and where have been appealed and where those disputes are still those disputes are still those disputes are still pending. So the point is pending. So the point is pending. So the point is that I can that I can that I can create create create specific specific specific datasets that very closely datasets that very closely datasets that very closely match not only the match not only the use case but also the use case but also the technical requirements for technical requirements for technical requirements for that agent. This is the one time that agent. This is the one time that agent. This is the one time when I will say that I when I will say that I when I will say that I don't regret it at all don't regret it at all don't regret it at all , because I believe that , because I believe that , because I believe that we humans remain we humans remain we humans remain very important. So very important. So very important. So here you still here you still here you still need your need your domain experts domain experts working alongside the working alongside the development teams development teams to define these to define these to define these parameters. There are two parameters. There are two parameters. There are two main ways you main ways you main ways you can create can create can create your reference datasets your reference datasets your reference datasets . You either have a . You either have a . You either have a team—and I've seen team—and I've seen team—and I've seen variations of both variations of both variations of both approaches. You either approaches. You either approaches. You either have a team that have a team that have a team that already has a spreadsheet somewhere

already has a spreadsheet somewhere already has a spreadsheet somewhere . This is all . This is all . This is all we want to we want to we want to test on test on test on my agent, just my agent, just my agent, just download it. And it's download it. And it's download it. And it's quite simple. You quite simple. You quite simple. You can go here and can go here and can go here and say, "Okay, great say, "Okay, great . I want to create a . I want to create a . I want to create a test dataset test dataset ." And then I want to ." And then I want to ." And then I want to download the CSV file. download the CSV file. download the CSV file. You can You can You can upload a CSV and it will upload a CSV and it will upload a CSV and it will exactly recreate everything exactly recreate everything exactly recreate everything you had you had you had before. One more thing before. One more thing before. One more thing I want to I want to I want to mention: everything I mention: everything I mention: everything I show you can show you can show you can do agentically through do agentically through do agentically through our CLI, NCP, or our APIs. our CLI, NCP, or our APIs. our CLI, NCP, or our APIs. I just think it's I just think it's I just think it's better for me to show better for me to show better for me to show it in the interface than it in the interface than it in the interface than to ask Claude to do it to ask Claude to do it to ask Claude to do it from start to finish from start to finish from start to finish without errors. OK? without errors. OK? without errors. OK? So, that's all. But So, that's all. But So, that's all. But then I can also then I can also then I can also go in here and say I go in here and say I go in here and say I want to add want to add want to add items to my items to my items to my dataset that dataset that dataset that are already part of my are already part of my are already part of my productive productive productive traces. traces. Let's say you saw how I was already collecting a how I was already collecting a how I was already collecting a lot of activity from lot of activity from lot of activity from my app. I can my app. I can my app. I can go in here and say, go in here and say, go in here and say, "I want to add all of this "I want to add all of this "I want to add all of this to my dataset to my dataset , and then select the , and then select the , and then select the dataset that I dataset that I dataset that I need." So this need." So this need." So this way you can way you can way you can use use use your own results your own results your own results for a dataset for a dataset for a dataset by looking at the by looking at the by looking at the productive productive productive traces that traces that traces that your agent produces your agent produces your agent produces if you have determined that if you have determined that if you have determined that these are good examples these are good examples these are good examples of what of what of what future future future output or output or output or traces should be. Now, traces should be. Now, traces should be. Now, the important thing here the important thing here the important thing here is the process that I is the process that I is the process that I always emphasize, which is always emphasize, which is always emphasize, which is error analysis, error analysis, error analysis, right? So,

right? So, right? So, error analysis is error analysis is error analysis is essentially the idea that I essentially the idea that I essentially the idea that I want to review want to review want to review my traces. And I my traces. And I my traces. And I think Annabel think Annabel think Annabel also also also spoke very eloquently spoke very eloquently spoke very eloquently about this when she was on the about this when she was on the about this when she was on the other side as a other side as a other side as a LangFuse user, which LangFuse user, which LangFuse user, which she had to, and we she had to, and we she had to, and we talked about talked about talked about earlier today, spend days earlier today, spend days literally literally literally looking at looking at looking at tracebacks. That was tracebacks. That was tracebacks. That was her job—to study her job—to study her job—to study the tracing, the the tracing, the the tracing, the structure, what the structure, what the structure, what the source data was, and all source data was, and all source data was, and all that. And it's still a very that. And it's still a very that. And it's still a very important process important process important process because when you look because when you look because when you look at the trace, you at the trace, you at the trace, you start to understand start to understand start to understand how the agent is behaving: is how the agent is behaving: is how the agent is behaving: is it performing all the it performing all the it performing all the necessary steps, is it necessary steps, is it necessary steps, is it invoking all the invoking all the invoking all the tools it should be tools it should be tools it should be invoking, and is invoking, and is invoking, and is it producing the desired it producing the desired it producing the desired output. This, you know, is output. This, you know, is output. This, you know, is very, very important. very, very important. very, very important. But how to simplify But how to simplify But how to simplify this process? Because, this process? Because, this process? Because, obviously, it is impossible for me obviously, it is impossible for me obviously, it is impossible for me as a human as a human to review thousands of to review thousands of traces when traces when traces when the system is already running. the system is already running. So we usually So we usually want to take a want to take a want to take a representative representative representative sample of sample of sample of our product our product our product system traces so that an expert system traces so that an expert system traces so that an expert can review them and can review them and can review them and give feedback, right? It's give feedback, right? It's give feedback, right? It's like, "Hey, I have all like, "Hey, I have all like, "Hey, I have all this activity. I'll this activity. I'll this activity. I'll just take 10% or 20% and just take 10% or 20% and just take 10% or 20% and I can say I want to I can say I want to I can say I want to focus on what focus on what the customer is talking about, you know, the customer is talking about, you know, disputes or disputes or credit card information."

credit card information." Any filter Any filter Any filter you want, you you want, you you want, you can apply. And can apply. And can apply. And then you can then you can then you can create what's create what's create what's called an called an called an annotation queue for them, right? In this annotation queue for them, right? In this annotation queue for them, right? In this queue, you're essentially queue, you're essentially queue, you're essentially telling an expert in a telling an expert in a telling an expert in a particular field, particular field, particular field, let's say me, "Hey, you're a let's say me, "Hey, you're a let's say me, "Hey, you're a risk expert. risk expert. I need you to I need you to look at look at look at the output that the agent the output that the agent the output that the agent is giving us or our is giving us or our is giving us or our clients and give me clients and give me clients and give me your feedback." And the point is your feedback." And the point is your feedback." And the point is that it should be that it should be that it should be very, very easy for very, very easy for very, very easy for an expert, because they have a an expert, because they have a an expert, because they have a main job, they main job, they main job, they can't spend can't spend can't spend hours reviewing. hours reviewing. So the typical thing I So the typical thing I recommend is to give recommend is to give recommend is to give them a few them a few them a few rating options, you know, yes, rating options, you know, yes, rating options, you know, yes, no, true, false, a no, true, false, a no, true, false, a scale of 1 to 10, scale of 1 to 10, scale of 1 to 10, whatever. And then whatever. And then whatever. And then they can also they can also they can also add their comment. add their comment. So, let's assume the So, let's assume the SME error mode, SME error mode, SME error mode, say for example say for example , in this case , in this case , in this case there is no quoting. And there is no quoting. And there is no quoting. And then I'll add a then I'll add a then I'll add a comment. The link is comment. The link is comment. The link is missing. It missing. It missing. It needs to be added in needs to be added in needs to be added in all such cases, all such cases, all such cases, right? You will go right? You will go right? You will go through this process and through this process and through this process and see how simple it was see how simple it was see how simple it was for me.

for me. for me. What did it take me? What did it take me? What did it take me? Maybe 10 seconds. So, Maybe 10 seconds. So, Maybe 10 seconds. So, let's say I have, I don't let's say I have, I don't let's say I have, I don't know, 100 of these, and know, 100 of these, and know, 100 of these, and each one takes 30 seconds. each one takes 30 seconds. each one takes 30 seconds. What is this? 50 minutes? No, no, What is this? 50 minutes? No, no, What is this? 50 minutes? No, no, even less, right? I'm doing even less, right? I'm doing even less, right? I'm doing poorly poorly poorly in math today. But in math today. But in math today. But anyway, you anyway, you anyway, you get the idea, right? get the idea, right? get the idea, right? You can say to You can say to You can say to your industry your industry your industry expert, "Hey, expert, "Hey, expert, "Hey, give me an hour of give me an hour of give me an hour of your time. I'll take a your time. I'll take a your time. I'll take a bunch of activity for you bunch of activity for you bunch of activity for you . Just . Just . Just give me your feedback. give me your feedback. give me your feedback. Just write some Just write some Just write some comments." And now, comments." And now, comments." And now, with WhisperFlow or something with WhisperFlow or something with WhisperFlow or something like that, they can like that, they can like that, they can just dictate just dictate just dictate it. They don't even it. They don't even it. They don't even need to need to need to print anything, right? So print anything, right? So print anything, right? So make it very make it very make it very simple. The second thing simple. The second thing simple. The second thing you do with this—I'll you do with this—I'll you do with this—I'll mark it as mark it as mark it as done. This done. This done. This human annotation human annotation human annotation is now stored along is now stored along is now stored along with the trace as with the trace as with the trace as evidence. But then, as a evidence. But then, as a evidence. But then, as a second step, I want to second step, I want to second step, I want to take all of this take all of this take all of this information, all of these information, all of these information, all of these annotations that they annotations that they annotations that they gave us, and actually gave us, and actually gave us, and actually use LLM to use LLM to use LLM to find all of the find all of the find all of the failure modes or categories failure modes or categories failure modes or categories that are patterns in that are patterns in that are patterns in all of the other all of the other all of the other comments, comments, comments, right? That is, you are right? That is, you are right? That is, you are not asking a domain expert not asking a domain expert to invent categories . You just tell them, . You just tell them, . You just tell them, "Give me your "Give me your "Give me your unfiltered, unfiltered, unfiltered, raw raw raw comments on these comments on these comments on these traces." And then you'll traces." And then you'll traces." And then you'll use LLM to use LLM to use LLM to figure out the categories, figure out the categories, figure out the categories, because that's what they're because that's what they're because that's what they're great at. They are great at great at. They are great at matching matching patterns, they are great at patterns, they are great at patterns, they are great at extracting information extracting information extracting information from text. This way from text. This way from text. This way you simplify you simplify you simplify the task for the the task for the the task for the expert and expert and expert and use LLMs for what use LLMs for what use LLMs for what they do they do they do best. Now that best. Now that best. Now that you have all these you have all these you have all these categories, this is where categories, this is where categories, this is where we can move on to we can move on to we can move on to our failure modes.

our failure modes. And these failure modes, And these failure modes, if I go here and if I go here and if I go here and look at my look at my look at my experiments, this is what experiments, this is what experiments, this is what you'll see when I you'll see when I you'll see when I run the evaluation, run the evaluation, run the evaluation, you see all these things that you see all these things that you see all these things that I'm measuring. I'm measuring. I'm measuring. Maybe let Maybe let Maybe let me open it here. me open it here. So now, after So now, after my expert has my expert has my expert has given me given me given me input, I've identified the input, I've identified the input, I've identified the most common most common most common categories that categories that categories that appear in my appear in my appear in my corpus of data based on the corpus of data based on the expert's work, I can expert's work, I can define my define my define my vectors or dimensions of, vectors or dimensions of, vectors or dimensions of, you know, quality. you know, quality. Let's just Let's just call it that. We call it that. We call it that. We just call it just call it just call it estimates, right? But estimates, right? But estimates, right? But I like to I like to I like to make up fancy make up fancy make up fancy words to describe words to describe words to describe things. But you see things. But you see things. But you see here how one of them here how one of them here how one of them was citing was citing was citing experts, expected experts, expected experts, expected source, source, source, correctness, correctness, correctness, formal register, which formal register, which formal register, which means that, you know, in means that, you know, in means that, you know, in some languages ​​you some languages ​​you some languages ​​you want to address want to address want to address your clients in a your clients in a your clients in a more formal more formal more formal tone, if the language tone, if the language tone, if the language matches the one matches the one the client addressed, should the client addressed, should include unsolicited include unsolicited include unsolicited messages, etc.

messages, etc. So now that So now that I have a base of categories I have a base of categories by which I'm going to by which I'm going to by which I'm going to measure the quality of measure the quality of measure the quality of my output, I can my output, I can my output, I can decide, "Okay, cool decide, "Okay, cool . So how am I going to . So how am I going to . So how am I going to evaluate this, right? evaluate this, right? evaluate this, right? What type of evaluation is this? What type of evaluation is this? What type of evaluation is this? What type of evaluator is this What type of evaluator is this ?" And for that, I can ?" And for that, I can ?" And for that, I can either come here and either come here and either come here and create my own create my own create my own library of library of library of evaluators, right? So that evaluators, right? So that evaluators, right? So that when when when the traces come in, they the traces come in, they the traces come in, they automatically automatically automatically calculate these calculate these calculate these estimates for me. But estimates for me. But estimates for me. But at the same time, I can at the same time, I can at the same time, I can calculate scores calculate scores calculate scores directly in the directly in the directly in the client application client application when something happens when something happens when something happens online in online in online in real time, and real time, and real time, and then reflect it then reflect it then reflect it in the system. But in the system. But in the system. But essentially, I want to have a essentially, I want to have a essentially, I want to have a way to evaluate the way to evaluate the agent's performance both agent's performance both using LLM-based scores or using LLM-based scores or using LLM-based scores or estimators estimators estimators , and using , and using , and using deterministic deterministic deterministic estimators. Another estimators. Another estimators. Another lesson we lesson we lesson we learned while learned while learned while working with this client is working with this client is that that that the more you can the more you can the more you can use a use a use a mix—a mix—a balanced balanced balanced mix, depending mix, depending mix, depending on the type of categories on the type of categories on the type of categories you are defining— you are defining— between LLM estimators and between LLM estimators and between LLM estimators and deterministic deterministic deterministic estimators, the estimators, the estimators, the better. Because if better. Because if better. Because if you rely 100% you rely 100% on LLM alone as a judge, on LLM alone as a judge, on LLM alone as a judge, there is a non-zero there is a non-zero there is a non-zero chance that chance that chance that some results some results some results will be inaccurate, will be inaccurate, will be inaccurate, because that is the nature of LLM.

because that is the nature of LLM. But if you factor in But if you factor in some deterministic some deterministic some deterministic checks that checks that checks that will always be the same will always be the same will always be the same for the same for the same for the same input, input, input, no matter what no matter what no matter what happens, you happens, you happens, you can get a can get a can get a mixed assessment of the mixed assessment of the mixed assessment of the quality of your agent. quality of your agent. So, this is what we would So, this is what we would normally like to normally like to normally like to see. And you see see. And you see see. And you see here, when I try to here, when I try to here, when I try to define a new define a new define a new evaluator, I can evaluator, I can evaluator, I can choose, "I want choose, "I want choose, "I want this to be a new this to be a new this to be a new LLM-based evaluator" or "I LLM-based evaluator" or "I LLM-based evaluator" or "I want this to be a want this to be a want this to be a code evaluator," and I code evaluator," and I code evaluator," and I can define all the can define all the can define all the details of how I'm going details of how I'm going details of how I'm going to evaluate a to evaluate a to evaluate a particular particular particular agent activity. So that's it, and agent activity. So that's it, and agent activity. So that's it, and if I want to have a if I want to have a if I want to have a code evaluator, you code evaluator, you code evaluator, you see it's just see it's just see it's just code. This is a function, this is code. This is a function, this is code. This is a function, this is code. It's very simple, code. It's very simple, code. It's very simple, it's just a regular it's just a regular it's just a regular expression. It performs an expression. It performs an expression. It performs an exact match. exact match. exact match. But it will give me the But it will give me the But it will give me the same result same result same result 100% of the time. It won't 100% of the time. It won't 100% of the time. It won't be, say, 95% or 80% be, say, 95% or 80% , it will always be 100% the , it will always be 100% the same result same result same result for the same for the same for the same input data. So, input data. So, input data. So, again, the lesson here is that again, the lesson here is that again, the lesson here is that you want to you want to you want to have a combination of both have a combination of both have a combination of both methods. So, that's on the methods. So, that's on the methods. So, that's on the evaluation side.

evaluation side. evaluation side. So now we've So now we've So now we've talked about what talked about what talked about what Annabel said, and Annabel said, and Annabel said, and maybe let maybe let maybe let me go back me go back me go back here to the here to the here to the AI ​​development cycle. So, we AI ​​development cycle. So, we AI ​​development cycle. So, we talked here about talked here about talked here about creating creating creating datasets. We touched on datasets. We touched on datasets. We touched on how we can how we can how we can then implement then implement then implement quality measurements of quality measurements of quality measurements of these agents when these agents when these agents when conducting conducting conducting experiments. And experiments. And experiments. And maybe let maybe let maybe let me come back me come back me come back here. So, essentially, here. So, essentially, here. So, essentially, an experiment an experiment an experiment is based on a is based on a is based on a data set. These are my test data set. These are my test data set. These are my test cases. These are my cases. These are my cases. These are my tests that I will tests that I will tests that I will run to run to run to test my test my test my agents. And now I agents. And now I agents. And now I will say: “You know what? This will say: “You know what? This will say: “You know what? This agent today agent today agent today works with this works with this works with this system prompt, system prompt, system prompt, this model, these this model, these this model, these tools. What tools. What tools. What happens if I happens if I happens if I change the model? What change the model? What change the model? What happens if I happens if I happens if I make changes to the make changes to the make changes to the system prompt? What system prompt? What system prompt? What happens if I happens if I happens if I suddenly give him suddenly give him suddenly give him access to new access to new access to new tools to tools to tools to work with new work with new work with new data? So you data? So you data? So you want to be able to want to be able to want to be able to prototype this prototype this prototype this before you before you before you put it into put it into put it into production, rather than production, rather than production, rather than just relying just relying just relying on intuition, which is what we all on intuition, which is what we all on intuition, which is what we all usually do, usually do, usually do, right? It's like, "Oh, yeah, it right? It's like, "Oh, yeah, it right? It's like, "Oh, yeah, it looks looks looks pretty good." Let's pretty good." Let's pretty good." Let's just release it to the just release it to the just release it to the market." But this is a market." But this is a market." But this is a systematic way systematic way systematic way to do it right.

to do it right. to do it right. And in a few minutes I'll And in a few minutes I'll And in a few minutes I'll show you how you can show you how you can show you how you can incorporate this into your incorporate this into your incorporate this into your CI/CD pipeline so you CI/CD pipeline so you CI/CD pipeline so you don't have to be don't have to be don't have to be aware of every step. aware of every step. aware of every step. So every time I So every time I So every time I make a change to the agent, a make a change to the agent, a make a change to the agent, a CI job is run CI job is run to to to run run run experiments on the experiments on the experiments on the dataset. It dataset. It dataset. It will evaluate your will evaluate your will evaluate your agent with the new agent with the new agent with the new configuration and configuration and configuration and give you a give you a give you a pass/ pass/ fail score on all the fail score on all the fail score on all the scores and scores and scores and criteria I criteria I criteria I mentioned earlier so that mentioned earlier so that mentioned earlier so that you have a consistent you have a consistent you have a consistent way to evaluate and way to evaluate and way to evaluate and measure the measure the measure the performance of performance of performance of these agents. these agents. these agents. Of course? Okay, Of course? Okay, Of course? Okay, great. So, these are great. So, these are great. So, these are experiments. This is experiments. This is experiments. This is great. So, we great. So, we great. So, we touched on the topic of touched on the topic of touched on the topic of experiments. We experiments. We experiments. We touched on the topic of touched on the topic of touched on the topic of datasets and datasets and datasets and human annotation human annotation to bring people to bring people to bring people and their expertise into the and their expertise into the and their expertise into the process. Now, how do process. Now, how do process. Now, how do I put all this I put all this I put all this together? If I come together? If I come together? If I come here and I have my here and I have my here and I have my own prompt, these are my own prompt, these are my own prompt, these are my system prompts that system prompts that system prompts that I have a version of here in LangFuse, I have a version of here in LangFuse, I have a version of here in LangFuse, you can do that you can do that you can do that or here in LangFuse, you know or here in LangFuse, you know , we don't... that's normal. , we don't... that's normal. , we don't... that's normal. We don't even We don't even We don't even charge for it, so it's all charge for it, so it's all charge for it, so it's all good. Many of our good. Many of our good. Many of our customers do this: customers do this: customers do this: they have another they have another versioning system, they versioning system, they have versions in GitHub, have versions in GitHub, have versions in GitHub, GitLab, or GitLab, or GitLab, or whatever. But you should whatever. But you should whatever. But you should consider this as a consider this as a consider this as a trigger to run trigger to run trigger to run automation to automation to automation to perform the assessment perform the assessment perform the assessment fully automatically fully automatically . So, it . So, it . So, it goes like this: I've goes like this: I've goes like this: I've already defined already defined already defined automation here. And if automation here. And if automation here. And if you look at you look at you look at the configuration, it the configuration, it the configuration, it reacts to reacts to any changes in the prompts any changes in the prompts any changes in the prompts that I make here. And

that I make here. And that I make here. And then the action runs a CI then the action runs a CI task on my GitHub, task on my GitHub, task on my GitHub, this is it here. I this is it here. I hope I don't hope I don't see a unicorn from see a unicorn from see a unicorn from GitHub. So, this is this GitHub. So, this is this GitHub. So, this is this guy right here. So, guy right here. So, guy right here. So, I already have a I already have a I already have a CI set up that is CI set up that is CI set up that is waiting for any waiting for any waiting for any changes to changes to changes to my agent configuration. my agent configuration. my agent configuration. Let me move Let me move Let me move this closer here. And this closer here. And this closer here. And then I can go in and then I can go in and then I can go in and see my see my see my prompt. Let's prompt. Let's prompt. Let's just take a look at just take a look at just take a look at this one. I have five this one. I have five this one. I have five versions of the prompt. You versions of the prompt. You versions of the prompt. You can see how the prompt can see how the prompt can see how the prompt has evolved over time. has evolved over time. The first version looked like this The first version looked like this , and then the , and then the , and then the second one was a slight second one was a slight second one was a slight change. Then the third change. Then the third change. Then the third version, it was, version, it was, version, it was, you know, we tried you know, we tried you know, we tried something different. This is for something different. This is for something different. This is for development. Then the development. Then the development. Then the fourth version, we fourth version, we fourth version, we added more added more added more details and also details and also details and also passed other passed other passed other parameters to the parameters to the parameters to the agent configuration. agent configuration. agent configuration. But now I can But now I can But now I can say, "You know what? say, "You know what? say, "You know what? This is a system prompt This is a system prompt This is a system prompt that I have in production right now that I have in production right now that I have in production right now . I want . I want . I want to test again to test again to test again with version four with version four with version four because I noticed some because I noticed some because I noticed some regressions." So I can regressions." So I can regressions." So I can come here and come here and come here and click "move click "move click "move to production." And again to production." And again to production." And again , I'm showing you this , I'm showing you this , I'm showing you this as a human, you know, as a as a human, you know, as a as a human, you know, as a primitive human, primitive human, primitive human, just by clicking a just by clicking a just by clicking a mouse. All of this can be mouse. All of this can be mouse. All of this can be done via API and CLI.

done via API and CLI. done via API and CLI. So you don't even So you don't even So you don't even have to do it have to do it have to do it manually. You can manually. You can manually. You can simply call the API and simply call the API and simply call the API and say "push this say "push this say "push this version of the prompt with the version of the prompt with the version of the prompt with the production tag" and production tag" and production tag" and it will run the it will run the it will run the process for you. I process for you. I process for you. I have a production tag. I'm have a production tag. I'm have a production tag. I'm going to save and going to save and going to save and move to production. move to production. move to production. Use the new Use the new Use the new configuration. In a configuration. In a configuration. In a few seconds, we few seconds, we few seconds, we should have a new CI should have a new CI job job job running here, and running here, and running here, and it will start it will start it will start the experiment to the experiment to the experiment to run my run my run my assessment. Okay? Here, assessment. Okay? Here, assessment. Okay? Here, please. Thanks, GitHub please. Thanks, GitHub . Thank you very much. So, . Thank you very much. So, . Thank you very much. So, perfect. Now perfect. Now perfect. Now I have this task I have this task I have this task that runs that runs that runs automatically. So automatically. So automatically. So every time I every time I every time I push any push any push any changes to the changes to the changes to the agent configuration, they agent configuration, they agent configuration, they go into CI. It go into CI. It go into CI. It will run will run will run assessment stages to assessment stages to assessment stages to let me know if let me know if let me know if I pass or I pass or I pass or fail any of the fail any of the fail any of the previously defined previously defined previously defined criteria. This criteria. This criteria. This way, I get way, I get way, I get evidence and evidence and evidence and confidence about confidence about confidence about whether I can safely whether I can safely whether I can safely go into production go into production go into production with this new with this new with this new configuration. Is the configuration. Is the agent's performance actually agent's performance actually getting worse or getting worse or getting worse or better, or is better, or is better, or is he possibly violating he possibly violating he possibly violating any any any restrictions I have set? This restrictions I have set? This restrictions I have set? This will take a few will take a few will take a few minutes, but let minutes, but let minutes, but let me show you the one me show you the one I ran I ran I ran earlier to earlier to earlier to demonstrate the demonstrate the demonstrate the result. I can result. I can result. I can go here and see go here and see go here and see the result, and it the result, and it the result, and it shows what shows what shows what prompt was prompt was prompt was sent, the version, the sent, the version, the sent, the version, the tags, and says the tags, and says the tags, and says the validation was validation was validation was successful. Truth? So, successful. Truth? So, successful. Truth? So, perfect. And these are the perfect. And these are the perfect. And these are the evaluators by which I evaluators by which I evaluators by which I measured my measured my measured my agent. I have the agent. I have the agent. I have the main metrics here:

main metrics here: main metrics here: completeness of the source completeness of the source completeness of the source data, data, data, language consistency, correctness, and language consistency, correctness, and language consistency, correctness, and absence of absence of absence of unwanted upsells unwanted upsells . I can also . I can also . I can also define minimum define minimum define minimum values. I can values. I can values. I can set a set a set a minimum threshold minimum threshold minimum threshold for this particular for this particular for this particular assessment. If it does not assessment. If it does not assessment. If it does not pass this threshold, it will not pass this threshold, it will not pass this threshold, it will not be considered to have be considered to have be considered to have passed the test passed the test passed the test for for for production implementation. Of course, production implementation. Of course, production implementation. Of course, in this case I'm not in this case I'm not crazy enough to crazy enough to just say, "Okay, just say, "Okay, just say, "Okay, cool. You passed. cool. You passed. cool. You passed. Merge this and push it to the Merge this and push it to the Merge this and push it to the main branch." I'm just main branch." I'm just main branch." I'm just leaving it as is, but leaving it as is, but leaving it as is, but honestly, you could honestly, you could honestly, you could create a complete create a complete create a complete end-to-end process end-to-end process end-to-end process that would create that would create that would create PRs itself and wait for PRs itself and wait for PRs itself and wait for validation, or, if validation, or, if validation, or, if you know, like everyone you know, like everyone you know, like everyone does now, just does now, just does now, just push PRs to push PRs to push PRs to main immediately if they pass main immediately if they pass main immediately if they pass the bar. But at least the bar. But at least the bar. But at least you have evidence here, you have evidence here, you have evidence here, right? So you don't right? So you don't right? So you don't send PR send PR send PR based solely on based solely on based solely on intuition or what intuition or what intuition or what you think the agent you think the agent you think the agent does well. You does well. You does well. You actually have a actually have a actually have a documentary trail, documentary trail, documentary trail, so to speak, and so to speak, and so to speak, and evidence of why you evidence of why you evidence of why you made that made that made that decision about this decision about this decision about this new new new agent configuration. Ideally. So agent configuration. Ideally. So agent configuration. Ideally. So now I want to show now I want to show now I want to show you a few more things you a few more things you a few more things as a real-world as a real-world as a real-world use case, okay?

use case, okay? Because we talked a lot Because we talked a lot about the technical aspects of about the technical aspects of about the technical aspects of LangFuse, about how LangFuse, about how LangFuse, about how to set up the whole to set up the whole to set up the whole process from start to process from start to process from start to finish, but I also finish, but I also finish, but I also understand that understand that understand that organizations are increasingly organizations are increasingly focusing on focusing on tokenomics, right? tokenomics, right? I think in 2025 and I think in 2025 and early 2026 everyone early 2026 everyone early 2026 everyone was doing " was doing " token maximization." Now everyone token maximization." Now everyone is doing " is doing " is doing " token valuation" or whatever they token valuation" or whatever they token valuation" or whatever they call it now. So call it now. So , now everyone , now everyone , now everyone is trying to extract is trying to extract is trying to extract as much as much as much value as possible from their value as possible from their value as possible from their tokens. So tokens. So tokens. So let me let me let me show you a few show you a few show you a few ways you ways you ways you can can can demonstrate this to demonstrate this to demonstrate this to your executives, your executives, your executives, leaders, or anyone leaders, or anyone leaders, or anyone else, or maybe else, or maybe else, or maybe you want to create this you want to create this you want to create this for business for business stakeholders so stakeholders so stakeholders so they feel more they feel more they feel more confident that the confident that the confident that the agent solution agent solution agent solution you're building is actually you're building is actually you're building is actually delivering delivering delivering business value. It's not just business value. It's not just business value. It's not just "oh, this is a cool thing "oh, this is a cool thing "oh, this is a cool thing we're doing now." we're doing now." we're doing now." I have a dashboard here I have a dashboard here where I'm trying where I'm trying to gather some to gather some to gather some metrics that I can metrics that I can metrics that I can report to the business report to the business report to the business based on the data that I based on the data that I based on the data that I collect from my collect from my collect from my agent, okay?

agent, okay? The first one—and this is where The first one—and this is where you can you can you can argue with argue with argue with me about the logic behind me about the logic behind me about the logic behind this, but, this, but, this, but, you know, every you know, every you know, every organization is different. organization is different. But I want to show But I want to show here that in the last 7 here that in the last 7 here that in the last 7 days, this agent days, this agent days, this agent has handled 1,400 has handled 1,400 has handled 1,400 interactions that didn't interactions that didn't interactions that didn't require require require human intervention, human intervention, human intervention, right? This is great. This right? This is great. This right? This is great. This means that I means that I means that I save human save human save human resources that resources that resources that are spent on are spent on are spent on talking to customers, talking to customers, talking to customers, and if you calculate, and if you calculate, and if you calculate, how much did it how much did it how much did it cost me? It cost me? It cost me? It cost me less cost me less cost me less than $7, right? than $7, right? than $7, right? In the last 7 days. In the last 7 days. In the last 7 days. If you take any If you take any If you take any contacts—I used to contacts—I used to contacts—I used to be a be a be a contact center employee 20 contact center employee 20 contact center employee 20 years ago—do you know years ago—do you know years ago—do you know how much you would how much you would how much you would pay such a person? pay such a person? pay such a person? $15–20 an $15–20 an $15–20 an hour, right? But this hour, right? But this hour, right? But this person won't handle 1,400 person won't handle 1,400 person won't handle 1,400 calls in one calls in one calls in one hour. So you already hour. So you already hour. So you already see at least see at least see at least some strategic some strategic some strategic business benefit from business benefit from business benefit from using this using this using this solution, right? This is the solution, right? This is the solution, right? This is the first. Second—you first. Second—you first. Second—you can start to can start to can start to identify areas where identify areas where identify areas where your agent is your agent is your agent is making making making mistakes, right?

mistakes, right? Remember, we Remember, we defined categories, defined categories, defined categories, or "failure modes," or "failure modes," or "failure modes," as we call them in as we call them in as we call them in the industry, when an agent the industry, when an agent the industry, when an agent doesn't give me the doesn't give me the doesn't give me the results or results or results or answers I need. Here are the ones answers I need. Here are the ones answers I need. Here are the ones I recorded in I recorded in I recorded in order of occurrence order of occurrence order of occurrence over the last 7 days. over the last 7 days. You see, right now, " You see, right now, " outside authority," outside authority," outside authority," "informal style," "informal style," "informal style," and "dispute not and "dispute not and "dispute not resolved" are my resolved" are my resolved" are my top three problems, top three problems, top three problems, and their number and their number and their number is growing over time. So, is growing over time. So, is growing over time. So, instead of instead of instead of going through thousands going through thousands going through thousands or hundreds of thousands of or hundreds of thousands of or hundreds of thousands of records trying records trying records trying to figure out which to figure out which to figure out which agent action is causing the agent action is causing the agent action is causing the most errors most errors most errors or problems. I can or problems. I can or problems. I can go here, narrow down my go here, narrow down my go here, narrow down my search, and already know that search, and already know that search, and already know that this failure mode is this failure mode is this failure mode is basic: outside basic: outside basic: outside authority, authority, authority, informal style, and informal style, and informal style, and dispute not dispute not dispute not resolved. Now I resolved. Now I resolved. Now I can go here in can go here in can go here in my tracing and my tracing and my tracing and apply a simple apply a simple apply a simple filter by selecting " filter by selecting " dispute not dispute not dispute not resolved". I can resolved". I can resolved". I can save this as a save this as a save this as a view so I don't view so I don't view so I don't have to have to have to filter every time.

filter every time. filter every time. You see how it You see how it You see how it filters from the top, and filters from the top, and filters from the top, and even from the side I can even from the side I can even from the side I can select "dispute select "dispute select "dispute not resolved" not resolved" not resolved" equals zero. This equals zero. This equals zero. This shows all cases shows all cases shows all cases where a customer where a customer where a customer had a dispute had a dispute had a dispute or requested a or requested a or requested a product appeal, product appeal, product appeal, but the agent did not but the agent did not but the agent did not resolve it. So, it either resolve it. So, it either resolve it. So, it either had to go to a had to go to a had to go to a human operator, or it human operator, or it human operator, or it just hung, the session just hung, the session just hung, the session closed, and nothing closed, and nothing closed, and nothing happened. But now happened. But now happened. But now I can spend my I can spend my I can spend my time not looking through time not looking through time not looking through thousands of entries, but thousands of entries, but thousands of entries, but narrowing it down to a narrowing it down to a narrowing it down to a few dozen or few dozen or few dozen or hundreds. And now I can hundreds. And now I can hundreds. And now I can conduct more conduct more conduct more targeted targeted targeted research, as well as research, as well as research, as well as find areas where find areas where find areas where my agent is my agent is my agent is making making making mistakes. So, this is the first thing mistakes. So, this is the first thing . Second, if . Second, if . Second, if I go here, to the I go here, to the I go here, to the dashboard, dashboard, dashboard, I can say, " I can say, " Okay, how many times Okay, how many times Okay, how many times do customers do customers do customers like or dislike me like or dislike me ?" I can very easily ?" I can very easily ?" I can very easily record this as an record this as an record this as an assessment. You see here assessment. You see here assessment. You see here that I that I that I can see likes and can see likes and can see likes and dislikes every time. And I can dislikes every time. And I can dislikes every time. And I can just show it on the just show it on the just show it on the dashboard where dashboard where dashboard where it's very easy it's very easy it's very easy to perceive. Any to perceive. Any to perceive. Any business person, any business person, any business person, any user, user, user, any person in general, can any person in general, can any person in general, can see this and say, see this and say, "Okay, most of "Okay, most of "Okay, most of them are likes, that's cool, them are likes, that's cool, them are likes, that's cool, but what about the dislikes but what about the dislikes ? Let's ? Let's ? Let's focus on those, focus on those, focus on those, you know, 18% of them. What you know, 18% of them. What you know, 18% of them. What happened?" And another happened?" And another happened?" And another point concerns point concerns point concerns security issues. Here's the security issues. Here's the security issues. Here's the good news: good news: good news: most of our

most of our most of our agent interactions agent interactions agent interactions involve no involve no involve no risk in risk in risk in communicating with communicating with communicating with customers. But we customers. But we customers. But we see here that in 89 see here that in 89 see here that in 89 cases in the last 7 cases in the last 7 cases in the last 7 days, a client requested days, a client requested days, a client requested investment advice investment advice investment advice and the agent was able to give or and the agent was able to give or and the agent was able to give or attempted to give attempted to give attempted to give investment advice. investment advice. So we can So we can focus on those focus on those focus on those cases, and again cases, and again cases, and again , the same thing. I can , the same thing. I can , the same thing. I can come in here and say, come in here and say, come in here and say, let me let me let me go back to my go back to my go back to my default view default view . And now instead of a . And now instead of a . And now instead of a numerical score, I numerical score, I numerical score, I can choose can choose can choose categorical scores categorical scores , add a condition, and , add a condition, and , add a condition, and then say: what then say: what then say: what is the is the is the security risk? And then the security risk? And then the security risk? And then the security risk, security risk, security risk, let's say, let's say, let's say, investment advice. investment advice. And now I can And now I can see specific see specific see specific examples where examples where examples where investment advice investment advice investment advice was identified as a was identified as a was identified as a reason for the reason for the reason for the designation, and that the designation, and that the designation, and that the agent should not agent should not agent should not discuss it with discuss it with discuss it with the client. And you can see the client. And you can see the client. And you can see this even from the this even from the this even from the previous previous previous example that I example that I example that I showed you. So, this is what showed you. So, this is what I wanted to show: I wanted to show: I wanted to show: first, the nature of the first, the nature of the first, the nature of the integrated solution integrated solution that you are implementing that you are implementing that you are implementing primarily for primarily for primarily for working with clients, working with clients, working with clients, which obviously has which obviously has which obviously has a lot of critical a lot of critical a lot of critical implications in terms of implications in terms of liability, liability, financial implications financial implications financial implications and reputation, and reputation, and reputation, because you don't want because you don't want because you don't want to provide information to provide information to provide information that is inaccurate or that is inaccurate or that is inaccurate or fabricated to your fabricated to your fabricated to your clients. And how clients. And how clients. And how can you follow a

can you follow a can you follow a systematic process systematic process systematic process to then provide to then provide to then provide evidence and evidence and evidence and confirm that confirm that confirm that the results of the results of the results of these agents' work are indeed these agents' work are indeed these agents' work are indeed consistent with the consistent with the constraints and constraints and controls you have set. And with that, controls you have set. And with that, controls you have set. And with that, I'll hand it over to I'll hand it over to I'll hand it over to Annabel again so we Annabel again so we Annabel again so we can wrap up and can wrap up and can wrap up and then maybe then maybe then maybe answer a few answer a few answer a few questions at the end. questions at the end. Thank you. Thank you. So what we see So what we see some companies or some companies or teams trying to do is this. teams trying to do is this. teams trying to do is this. Please correct Please correct Please correct my agent, don't my agent, don't my agent, don't make mistakes. make mistakes. make mistakes. And, probably, there is little that And, probably, there is little that And, probably, there is little that can be advised that is less can be advised that is less can be advised that is less effective than this. effective than this. effective than this. Even if your agent Even if your agent Even if your agent is connected via MCP is connected via MCP is connected via MCP or CLI to your data or CLI to your data or CLI to your data in LangSmith, or to in LangSmith, or to any other any other any other tool you tool you tool you use. This is something that use. This is something that , as we see, , as we see, , as we see, creates more creates more creates more clutter than anything that clutter than anything that clutter than anything that actually actually actually helps you. But of helps you. But of helps you. But of course, we also strongly course, we also strongly course, we also strongly support support support working with your working with your working with your agents to agents to agents to write code write code write code to improve to improve to improve your applications. your applications. your applications. So we wanted to So we wanted to So we wanted to share this as a share this as a share this as a visualization to make it visualization to make it visualization to make it easier for you easier for you easier for you to remember. So, to remember. So, to remember. So, we have two axes. The first we have two axes. The first is the time spent: is the time spent: is the time spent: a lot of a lot of a lot of time spent or very little, and the time spent or very little, and the time spent or very little, and the second is a second is a second is a low-quality system or a low-quality system or a low-quality system or a high- high- high- quality system. And you don't want to quality system. And you don't want to quality system. And you don't want to end up at the bottom. If end up at the bottom. If end up at the bottom. If you fully you fully you fully automate the process automate the process automate the process and just let and just let and just let the agent act on the the agent act on the the agent act on the principle of "fix everything principle of "fix everything , do it for me".

, do it for me". , do it for me". This will not This will not This will not end well. You end well. You end well. You will get poor will get poor will get poor quality, and yes, you won't quality, and yes, you won't quality, and yes, you won't have to have to have to do anything, but your do anything, but your do anything, but your customers wo customers wo customers wo n't be very happy either. n't be very happy either. n't be very happy either. At the same time, you also At the same time, you also At the same time, you also no longer want to go the no longer want to go the no longer want to go the completely manual completely manual completely manual route. You no longer route. You no longer route. You no longer want want want to deal with to deal with to deal with the tools yourself. This is something that the tools yourself. This is something that the tools yourself. This is something that can be can be can be automated. You don't automated. You don't automated. You don't want to find want to find want to find all types of errors and all types of errors and all types of errors and review all the data review all the data review all the data yourself. This can be yourself. This can be yourself. This can be automated. But automated. But automated. But at the same time, for certain at the same time, for certain at the same time, for certain things, it's worth adding things, it's worth adding things, it's worth adding your own vision and your own vision and your own vision and engaging industry engaging industry engaging industry experts to experts to experts to create create create datasets to determine datasets to determine datasets to determine what problem what problem what problem your your your application should solve and in what application should solve and in what application should solve and in what direction it should direction it should direction it should develop. And for develop. And for develop. And for that, you want that, you want that, you want to be somewhere here. to be somewhere here. to be somewhere here. Where there is a perfect Where there is a perfect Where there is a perfect balance between balance between balance between looking at the data, looking at the data, looking at the data, using your using your using your taste and expert taste and expert taste and expert knowledge, and knowledge, and knowledge, and working with your working with your working with your agent to agent to agent to automate everything automate everything possible: possible: possible: setting up setting up setting up tools, tools, tools, detecting errors, detecting errors, detecting errors, conducting conducting conducting evaluations, and evaluations, and evaluations, and perhaps running 20 perhaps running 20 perhaps running 20 experiments experiments experiments instead of the one instead of the one instead of the one you could have come up with on your you could have come up with on your you could have come up with on your own. And so, own. And so, own. And so, just avoid just avoid just avoid getting in there in the bottom getting in there in the bottom getting in there in the bottom left corner, and left corner, and left corner, and instead instead instead try to be an AI try to be an AI engineer with good engineer with good engineer with good taste who brings taste who brings taste who brings that taste in, making sure the that taste in, making sure the that taste in, making sure the emails are emails are emails are great, and then you great, and then you great, and then you can can automate that automate that cycle more and more over time. If we were in cycle more and more over time. If we were in cycle more and more over time. If we were in your shoes and had to your shoes and had to your shoes and had to start start start tomorrow, we would do tomorrow, we would do tomorrow, we would do exactly that. We would exactly that. We would exactly that. We would install the install the install the LangFuse skill, which is, in fact, an LangFuse skill, which is, in fact, an LangFuse skill, which is, in fact, an agent skill; This is a agent skill; This is a agent skill; This is a set of markdown files that set of markdown files that set of markdown files that helps your helps your helps your coding agent coding agent coding agent understand how understand how understand how to work with LangFuse and how to to work with LangFuse and how to to work with LangFuse and how to implement these implement these implement these best practices.

best practices. best practices. It is connected to It is connected to It is connected to our documentation, our documentation, our documentation, to all of our to all of our integration pages, and to integration pages, and to our academy, where our academy, where our academy, where we implement we implement we implement best practices for best practices for best practices for working with working with working with datasets, datasets, datasets, tooling, tooling, tooling, writing writing writing estimators, and everything estimators, and everything estimators, and everything else. So this is a great else. So this is a great else. So this is a great starting point to starting point to starting point to do automatically do automatically do automatically the things that should be the things that should be the things that should be automated, but automated, but automated, but also maybe to also maybe to also maybe to pick up a pick up a pick up a dataset that's worth dataset that's worth dataset that's worth looking at manually. looking at manually. looking at manually. You should start You should start You should start collecting all the collecting all the collecting all the ground truth against ground truth against ground truth against which you can which you can which you can evaluate your evaluate your evaluate your agent—perhaps from agent—perhaps from agent—perhaps from historical data, from a historical data, from a historical data, from a production production production environment, or from environment, or from environment, or from samples that you samples that you samples that you test with test with test with your experts. And your experts. And your experts. And if you work in a if you work in a if you work in a regulated field, regulated field, regulated field, define your define your define your hard limits hard limits hard limits so that you can be so that you can be so that you can be sure that these sure that these sure that these things definitely shouldn't things definitely shouldn't things definitely shouldn't happen. For example, happen. For example, happen. For example, for financial for financial for financial services, an agent services, an agent services, an agent should never provide should never provide should never provide investment advice. investment advice. investment advice. So make sure So make sure So make sure you factor this into you factor this into you factor this into your your your assessment kit so that assessment kit so that assessment kit so that ideally it never ideally it never ideally it never happens, or happens, or happens, or at least so that you at least so that you at least so that you know about it know about it know about it if it does. if it does. And we have several And we have several resources for you. resources for you. resources for you. The repository that Donnelly The repository that Donnelly The repository that Donnelly showed you and showed you and showed you and used is used is used is also also also available online. You available online. You available online. You can view can view can view it yourself. You it yourself. You it yourself. You can take can take can take whatever you want from there and whatever you want from there and whatever you want from there and use it for use it for use it for your own cases. This is, your own cases. This is, your own cases. This is, of course, fictional data of course, fictional data , not something that , not something that , not something that was actually taken from was actually taken from was actually taken from our clients. And our clients. And our clients. And I think this session I think this session I think this session will also be recorded, will also be recorded, will also be recorded, so you can so you can so you can watch it again.

watch it again. watch it again. We also We also We also recommend LangFuse recommend LangFuse recommend LangFuse Academy, which has a detailed Academy, which has a detailed Academy, which has a detailed description of how to description of how to description of how to approach AI approach AI engineering, with all the engineering, with all the engineering, with all the steps described steps described steps described again. We also have a again. We also have a self- self- study workshop that study workshop that study workshop that walks you through walks you through walks you through all the steps so you all the steps so you all the steps so you understand everything understand everything understand everything once and then can once and then can once and then can safely move on safely move on safely move on to your own to your own to your own application. application. application. You You You will have the opportunity will have the opportunity will have the opportunity to meet with us in the coming days. to meet with us in the coming days. First, Donnelly will also give a First, Donnelly will also give a talk on talk on creating creating creating deployment barriers for AI deployment barriers for AI agents on Wednesday. So agents on Wednesday. So agents on Wednesday. So come on in if come on in if come on in if you want to learn how you want to learn how you want to learn how to make sure that AI to make sure that AI models that go into models that go into models that go into production actually production actually production actually pass pass pass quality control. We will also have a quality control. We will also have a quality control. We will also have a booth, the ninth booth, the ninth booth, the ninth booth. I haven't looked booth. I haven't looked where it is yet, but I'm where it is yet, but I'm where it is yet, but I'm sure you'll sure you'll sure you'll find it. There you find it. There you find it. There you will be able to meet the will be able to meet the will be able to meet the team and also team and also team and also receive receive receive personalized personalized personalized stickers. You stickers. You stickers. You take a picture of your take a picture of your take a picture of your face, AI generates face, AI generates face, AI generates stickers with you and stickers with you and stickers with you and LangFuse, maybe with LangFuse, maybe with LangFuse, maybe with jokes, and you can jokes, and you can jokes, and you can take them home and take them home and take them home and stick them on everything you stick them on everything you stick them on everything you own. And finally, we own. And finally, we own. And finally, we also have a also have a also have a joint race. So joint race. So joint race. So if you want to get if you want to get if you want to get outside the conference outside the conference center and meet the center and meet the center and meet the team, we have team, we have team, we have two founders here and a two founders here and a two founders here and a few other few other few other team members, and we all team members, and we all team members, and we all love to run, so love to run, so love to run, so we decided to invite we decided to invite we decided to invite anyone who wants to anyone who wants to anyone who wants to join. So join. So join. So come to this come to this come to this event, and we will be happy event, and we will be happy event, and we will be happy to use the to use the to use the remaining time for your remaining time for your remaining time for your questions. We also questions. We also questions. We also have regular stickers.

have regular stickers. have regular stickers. It's just a It's just a It's just a LangFuse sticker, but I think LangFuse sticker, but I think LangFuse sticker, but I think it's still pretty it's still pretty it's still pretty cool. So if cool. So if cool. So if you want, you can you want, you can you want, you can come and get them. We are come and get them. We are come and get them. We are ready to answer ready to answer ready to answer any questions, any questions, any questions, get to know us and get to know us and get to know us and don't hesitate. I'll don't hesitate. I'll don't hesitate. I'll give the floor to give the floor to give the floor to Donnelly. Donnelly. Donnelly. Yes, LangChain itself has a Yes, LangChain itself has a Yes, LangChain itself has a tight dependency tight dependency tight dependency on ClickHouse for its on ClickHouse for its on ClickHouse for its repositories. This is because during the repositories. This is because during the product evolution, we started product evolution, we started with Postgres, but it couldn't with Postgres, but it couldn't with Postgres, but it couldn't handle handle handle the scale we the scale we the scale we reached—billions of reached—billions of reached—billions of events per day. So we events per day. So we events per day. So we switched to ClickHouse. switched to ClickHouse. switched to ClickHouse. So, regarding the So, regarding the So, regarding the data plane—yes, data plane—yes, data plane—yes, it's ClickHouse. But, perhaps it's ClickHouse. But, perhaps , a clarification for you: , a clarification for you: , a clarification for you: what are your needs what are your needs what are your needs regarding data storage, regarding data storage, regarding data storage, because we also provide because we also provide because we also provide the ability to the ability to the ability to export data. You export data. You export data. You can export can export can export them to S3. You can them to S3. You can them to S3. You can use the API, CLI, use the API, CLI, use the API, CLI, or MCP if you want to or MCP if you want to or MCP if you want to retrieve data from retrieve data from retrieve data from LangFuse. So, it's LangFuse. So, it's LangFuse. So, it's completely up completely up completely up to you. Yes. Yes. Yes, to you. Yes. Yes. Yes, to you. Yes. Yes. Yes, you can list you can list you can list this 100% on Amazon. We this 100% on Amazon. We this 100% on Amazon. We also have an option for also have an option for also have an option for self- self- self- deployment. You deployment. You deployment. You can can can use our use our use our cloud offering, cloud offering, cloud offering, which is a fully which is a fully which is a fully ready-made SaaS solution ready-made SaaS solution ready-made SaaS solution where you don't where you don't where you don't need to do anything. She need to do anything. She need to do anything. She works for Amazon. But works for Amazon. But works for Amazon. But if you want to if you want to if you want to deploy deploy deploy the infrastructure the infrastructure the infrastructure yourself, because you have, yourself, because you have, yourself, because you have, I don't know, some I don't know, some I don't know, some easy-to instances that easy-to instances that easy-to instances that you want you want you want to use, we to use, we to use, we also have a also have a also have a deployment model for deployment model for deployment model for that, and it that, and it that, and it comes with comes with comes with all the components already.

all the components already. all the components already. Yes, an extremely Yes, an extremely Yes, an extremely important question. important question. important question. So, the question So, the question So, the question was about was about was about the reliability of the reliability of the reliability of the assessments by different the assessments by different the assessments by different experts, and experts, and experts, and perhaps the consistency of the perhaps the consistency of the perhaps the consistency of the LLMs as assessors to LLMs as assessors to LLMs as assessors to make sure they make sure they make sure they really capture what really capture what really capture what you want. This is also you want. This is also you want. This is also part of the part of the part of the LangFuse functionality that deals with LangFuse functionality that deals with LangFuse functionality that deals with this and helps you, this and helps you, this and helps you, well, uh, there's a special well, uh, there's a special well, uh, there's a special term for this. term for this. term for this. Off the top of my head, Off the top of my head, Off the top of my head, but ultimately it's but ultimately it's but ultimately it's about the process of about the process of about the process of ensuring that the ensuring that the ensuring that the LLM assesses as a LLM assesses as a LLM assesses as a human would. And what human would. And what human would. And what you do, essentially, is you do, essentially, is you do, essentially, is this: this: this: you test a sample, you test a sample, you test a sample, ideally a ideally a representative representative representative sample yourself, sample yourself, sample yourself, then you run LLM then you run LLM then you run LLM as a judge, and then you as a judge, and then you as a judge, and then you optimize LLM so that optimize LLM so that optimize LLM so that its ratings its ratings its ratings match the ratings of match the ratings of match the ratings of people, and ideally having people, and ideally having people, and ideally having multiple such multiple such multiple such raters, that's the raters, that's the raters, that's the academic approach. academic approach. academic approach. So a practical So a practical So a practical approach is to approach is to have one have one have one person mark up a person mark up a person mark up a portion of the data, portion of the data, portion of the data, optimize the work optimize the work optimize the work against that benchmark, and against that benchmark, and against that benchmark, and then ideally then ideally just periodically just periodically just periodically check whether check whether check whether your sample remains your sample remains representative. And representative. And especially if you're especially if you're especially if you're using this using this using this to work with to work with to work with real data, real data, real data, ultimately it's better to start ultimately it's better to start ultimately it's better to start with having a little with having a little with having a little more more more false false false positives, meaning positives, meaning positives, meaning capturing a little capturing a little capturing a little more than you need to more than you need to more than you need to but not but not but not missing anything, and then missing anything, and then missing anything, and then fine-tune fine-tune fine-tune it over time. Yes.