How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

My name is Lori Voss. My name is Lori Voss. I am the head of the department. I am the head of the department. I am the head of the department. connections with connections with connections with by developers at Arise AI. by developers at Arise AI. by developers at Arise AI. Uh, once upon a time I Uh, once upon a time I Uh, once upon a time I I was doing JavaScript in npm, I was doing JavaScript in npm, I was doing JavaScript in npm, Inc., so you might Inc., so you might Inc., so you might Do you remember me from Do you remember me from Do you remember me from at that time. Uh, but at that time. Uh, but at that time. Uh, but now I'm thinking about now I'm thinking about now I'm thinking about artificial additives artificial additives artificial additives intelligence and that intelligence and that intelligence and that how to start them and how how to start them and how how to start them and how make them work. make them work. make them work. Uh, this slide was in Uh, this slide was in Uh, this slide was in me 15 minutes ago. me 15 minutes ago. me 15 minutes ago. I hope you like it. I hope you like it. I hope you like it. already seen. If not, already seen. If not, already seen. If not, so now is the time so now is the time so now is the time begin. Uh, this is begin. Uh, this is begin. Uh, this is moderately practical moderately practical moderately practical seminar. Here I am. seminar. Here I am. seminar. Here I am. created for you created for you created for you agent. He lies in agent. He lies in agent. He lies in therefore repositories. Uh, therefore repositories. Uh, therefore repositories. Uh, you can play you can play you can play with tips and with tips and with tips and to perform actual to perform actual to perform actual evaluations. Uh, but he evaluations. Uh, but he evaluations. Uh, but he has control points has control points has control points throughout. throughout. throughout. So, if you So, if you So, if you get lost, you can get lost, you can get lost, you can just switch just switch just switch to the desired to the desired to the desired control point and, control point and, control point and, uh, go to that one uh, go to that one uh, go to that one parts where he already parts where he already parts where he already works. Uh, you works. Uh, you works. Uh, you you will need Node 22 or you will need Node 22 or you will need Node 22 or later version. All later version. All later version. All these examples will be on these examples will be on these examples will be on Typescript. Um, since you Typescript. Um, since you Typescript. Um, since you you know it's me, me you know it's me, me you know it's me, me will use will use will use JavaScript, if I can. Um, JavaScript, if I can. Um, JavaScript, if I can. Um, and, um, you're going to and, um, you're going to and, um, you're going to to install and to install and to install and launch Phoenix on launch Phoenix on launch Phoenix on to your laptop. Um, to your laptop. Um, to your laptop. Um, on conference Wi-Fi it is on conference Wi-Fi it is on conference Wi-Fi it is it can be difficult, it can be difficult, it can be difficult, but I tried it but I tried it but I tried it this morning, and this morning, and this morning, and everything went well enough everything went well enough everything went well enough quickly. So, um, UVX– quickly. So, um, UVX– This is magic. Um, you don't This is magic. Um, you don't This is magic. Um, you don't will need will need will need run none run none run none Python in this thing.

Python in this thing. Python in this thing. All Python will be All Python will be All Python will be launched for you from launched for you from launched for you from your JavaScript name, your JavaScript name, your JavaScript name, which is good. Um, I think, which is good. Um, I think, which is good. Um, I think, what is this all for what is this all for what is this all for settings. Um, settings. Um, settings. Um, so that we so that we so that we we are going to do we are going to do we are going to do today, this, um, oh yeah today, this, um, oh yeah , sorry, one more , sorry, one more , sorry, one more setup step. setup step. setup step. After installation After installation After installation Phoenix you should Phoenix you should Phoenix you should open a browser on open a browser on open a browser on localhost 606 to see, localhost 606 to see, localhost 606 to see, What does he look like? Um, What does he look like? Um, What does he look like? Um, now he will be now he will be now he will be empty, but we empty, but we empty, but we we are going to perform we are going to perform we are going to perform teams that teams that teams that will fill it will fill it will fill it data. Um, you too data. Um, you too data. Um, you too will be needed will be needed will be needed anthropic key. IN anthropic key. IN anthropic key. IN in all my examples in all my examples in all my examples used by Claude used by Claude . Uh, you can. . Uh, you can. . Uh, you can. to do all this, to do all this, to do all this, using any using any which supplier which supplier which supplier LLM in the world, but to me LLM in the world, but to me LLM in the world, but to me had to choose had to choose had to choose example, that's why I chose example, that's why I chose example, that's why I chose the one I use the one I use the one I use daily. So, this one daily. So, this one daily. So, this one uses Claude. Ahem, uses Claude. Ahem, who needs it who needs it help with help with help with settings? settings? settings? Raise your hands. Raise your hands. Raise your hands. Just shout at Just shout at Just shout at me. Everything is fine. me. Everything is fine. OK. So, the first 10. OK. So, the first 10. How are you doing? How are you doing?

We're just getting started. We're just getting started. Okay, great. Okay, great. Yes. So, the first 10 Yes. So, the first 10 minutes is a theory that minutes is a theory that minutes is a theory that will give you some time, will give you some time, will give you some time, to make up for to make up for to make up for lost from lost from lost from settings while I settings while I settings while I I say. So, let's I say. So, let's I say. So, let's Let's start with the theory. Ahem, Let's start with the theory. Ahem, Let's start with the theory. Ahem, here is the situation in which, here is the situation in which, here is the situation in which, I suppose I suppose I suppose most of you. You most of you. You most of you. You already created an application, already created an application, already created an application, which contains an LLM. Uh, which contains an LLM. Uh, which contains an LLM. Uh, maybe a chatbot, maybe a chatbot, maybe a chatbot, maybe an agent. You maybe an agent. You maybe an agent. You tried it tried it tried it several times, and he several times, and he several times, and he looks pretty looks pretty looks pretty good. Uh, then you good. Uh, then you good. Uh, then you he was sent, and he was sent, and he was sent, and someone used someone used someone used its unexpected its unexpected its unexpected way, and it broke. way, and it broke. way, and it broke. Uh, or you changed Uh, or you changed Uh, or you changed request to fix request to fix request to fix one thing. Uh, and you have one thing. Uh, and you have one thing. Uh, and you have this is annoying this is annoying this is annoying the feeling that you the feeling that you the feeling that you broke something else. Uh-huh broke something else. Uh-huh , and there is none , and there is none , and there is none way to find out. way to find out. way to find out. That's what it's about. That's what it's about. That's what it's about. today's problem today's problem . This very problem . This very problem . This very problem the appraisers decide. the appraisers decide. the appraisers decide. How to find out if How to find out if How to find out if is your agent good As is your agent good As is your agent good As to find out if yours to find out if yours to find out if yours the changes improved the changes improved the changes improved situation or situation or situation or worsened? Uh, and how? worsened? Uh, and how? worsened? Uh, and how? prevent deterioration prevent deterioration when you enter when you enter when you enter changes, uh, no changes, uh, no changes, uh, no reading each output reading each output reading each output independently? My independently? My independently? My the promise is the promise is the promise is because until the end because until the end because until the end today in today in today in you will be working you will be working you will be working agent tracked agent tracked agent tracked from beginning to end from beginning to end from beginning to end with code evaluations from with code evaluations from with code evaluations from with the LLM judge's assessments and with the LLM judge's assessments and with the LLM judge's assessments and custom custom custom the assessment you the assessment you the assessment you wrote independently.

wrote independently. wrote independently. Uh, and you check, Uh, and you check, Uh, and you check, can you trust can you trust can you trust to your judge. You to your judge. You to your judge. You will spend will spend will spend experiment to experiment to experiment to to prove that your to prove that your to prove that your The changes worked. Uh, The changes worked. Uh, The changes worked. Uh, and you run all this and you run all this and you run all this as a set of tests with npm- as a set of tests with npm- tests so you can tests so you can tests so you can paste this into npmci. Uh, paste this into npmci. Uh, paste this into npmci. Uh, this is what I this is what I this is what I most worried. most worried. most worried. So this will be the last one, So this will be the last one, So this will be the last one, What will we do? I see, What will we do? I see, What will we do? I see, that some people that some people that some people were late. So were late. So were late. So Allow me very much Allow me very much Allow me very much come back soon come back soon come back soon to the slide to the slide to the slide settings to settings to settings to they are not lost they are not lost they are not lost fully. Good. fully. Good. fully. Good. Remember this one Remember this one Remember this one repository, because it repository, because it repository, because it you will need it. As you will need it. As you will need it. As Does Wi-Fi work for people? Does Wi-Fi work for people? Does Wi-Fi work for people? Is he...or you? Is he...or you? Is he...or you? Are you getting seats? Are you getting seats? Are you getting seats? Perfectly. Good. I did Perfectly. Good. I did Perfectly. Good. I did this is on the World this is on the World this is on the World exhibition, and, uh, every exhibition, and, uh, every exhibition, and, uh, every once, me too once, me too once, me too happened to be happened to be happened to be repository, full repository, full repository, full product images, product images, product images, which looked very which looked very which looked very great in demo version, great in demo version, great in demo version, but he was 400 but he was 400 but he was 400 megabytes of images megabytes of images megabytes of images products, and he products, and he products, and he just experienced just experienced just experienced complete failure.

complete failure. complete failure. No one could No one could No one could download even download even download even repository. So, I repository. So, I repository. So, I learned his lesson and learned his lesson and learned his lesson and today he has today he has today he has no images. Uh, no images. Uh, so let's go so let's go Let's look at the basics. What Let's look at the basics. What Let's look at the basics. What What is eval? The shortest What is eval? The shortest What is eval? The shortest version, the one I version, the one I version, the one I I use it every time, I use it every time, I use it every time, when I talk to when I talk to when I talk to someone who already someone who already someone who already some time ago some time ago some time ago engineer, consists of engineer, consists of engineer, consists of because, uh, there are such because, uh, there are such because, uh, there are such things that are called things that are called things that are called tracings, and they, tracings, and they, tracings, and they, In fact, there are such things as In fact, there are such things as In fact, there are such things as which are called eval, and which are called eval, and which are called eval, and they are essentially tests, they are essentially tests, they are essentially tests, and eval is used and eval is used and eval is used to run tests on to run tests on to run tests on your traces. THERE ARE- your traces. THERE ARE- e, the difference between eval and e, the difference between eval and e, the difference between eval and module test module test module test is that, uh is that, uh , eval are intended for , eval are intended for , eval are intended for exits that never exits that never exits that never are not absolutely are not absolutely are not absolutely the same twice, the same twice, the same twice, that is, that is, that is, non-deterministic non-deterministic non-deterministic exits that you exits that you exits that you you get from LLM. Ahem, you get from LLM. Ahem, you get from LLM. Ahem, the tracks record that the tracks record that the tracks record that did your AI, with did your AI, with did your AI, with incredible level incredible level incredible level details. So details. So details. So every step of the agent, every step of the agent, every step of the agent, every call every call every call tool, each tool, each tool, each LLM call, incoming and LLM call, incoming and LLM call, incoming and source data on source data on source data on every step. Track every step. Track every step. Track consists of, consists of, consists of, what do we call what do we call what do we call intervals (span). Everyone intervals (span). Everyone intervals (span). Everyone the gap is one the gap is one the gap is one step in execution. You step in execution. You step in execution. You can you imagine this can you imagine this can you imagine this just like a line in just like a line in just like a line in magazine. Hmm, but a whole magazine. Hmm, but a whole magazine. Hmm, but a whole the agent's move is the agent's move is the agent's move is the interval in which the interval in which the interval in which other nested other nested other nested gaps. Uh, and everyone gaps. Uh, and everyone gaps. Uh, and everyone of them writes their of them writes their of them writes their input data, yours input data, yours input data, yours original data, how much original data, how much original data, how much It took time, It took time, It took time, how many tokens were there how many tokens were there how many tokens were there used, uh, a used, uh, a used, uh, a sometimes how much is it sometimes how much is it sometimes how much is it it cost. Therefore, it cost. Therefore, it cost. Therefore, the trails tell the story the trails tell the story the trails tell the story you, what happened, and you, what happened, and you, what happened, and the estimates say whether the estimates say whether the estimates say whether it was good, and it it was good, and it it was good, and it the entire mental model the entire mental model the entire mental model for today. This is very for today. This is very for today. This is very far from far from far from rocketry.

rocketry. It seems to me that this It seems to me that this not a set of tests. Ahem, not a set of tests. Ahem, not a set of tests. Ahem, but it is supplied that way but it is supplied that way but it is supplied that way most AI functions. most AI functions. most AI functions. You create it, You create it, You create it, try a few try a few try a few times, does it look times, does it look times, does it look Is he right? Yes, it is. Is he right? Yes, it is. Is he right? Yes, it is. looks good. You looks good. You looks good. You supply it. Uh, supply it. Uh, supply it. Uh, and he probably and he probably and he probably looks good on looks good on looks good on three attempts. Hmm, but three attempts. Hmm, but three attempts. Hmm, but three attempts–that’s not three attempts–that’s not three attempts–that’s not a set of tests. AND a set of tests. AND a set of tests. AND ordinary ordinary ordinary the correction is the correction is the correction is unit tests. Hmm, unit tests. Hmm, unit tests. Hmm, but unit tests but unit tests but unit tests they don't work here, so they don't work here, so they don't work here, so that one and the same that one and the same that one and the same the query will produce the query will produce the query will produce different text different text different text answers every time, answers every time, answers every time, when you launch when you launch when you launch your program. Uh, your program. Uh, your program. Uh, so two absolutely so two absolutely so two absolutely different answers different answers different answers can be both can be both can be both correct. Thousand correct. Thousand correct. Thousand different answers different answers different answers can be both can be both can be both correct. Uh, correct. Uh, correct. Uh, just differently just differently just differently correct. So correct. So correct. So there is no expectation there is no expectation there is no expectation line against which line against which line against which it would be possible it would be possible it would be possible affirm. Uh, affirm. Uh, affirm. Uh, therefore people are prone therefore people are prone therefore people are prone to rely on to rely on to rely on human judgment. human judgment. human judgment. They are looking at They are looking at They are looking at lines being output, lines being output, lines being output, and they think, "Well, this and they think, "Well, this and they think, "Well, this looks good. Uh, looks good. Uh, looks good. Uh, but it's not but it's not but it's not scalable. This is not scalable. This is not scalable. This is not takes into account regressions, and takes into account regressions, and takes into account regressions, and this definitely doesn't work in this definitely doesn't work in this definitely doesn't work in illegible illegible illegible integration where there is no integration where there is no integration where there is no people who would people who would people who would watched this. watched this. watched this. Uh, in practice it is Uh, in practice it is Uh, in practice it is looks like this: you looks like this: you looks like this: you change system settings change system settings change system settings requests to fix requests to fix requests to fix tone, and the tone gets better tone, and the tone gets better . Uh, but now the bot . Uh, but now the bot . Uh, but now the bot invents functions invents functions invents functions product. Uh, and you don't product. Uh, and you don't product. Uh, and you don't you will find out while you will find out while you will find out while the client does not the client does not the client does not will do. Every change will do. Every change will do. Every change request may be affected request may be affected request may be affected for each type of input for each type of input for each type of input data that is sent data that is sent data that is sent your users. Uh, your users. Uh, your users. Uh, and you can't and you can't and you can't test all of this test all of this test all of this manually. So you finally- manually. So you finally- "You're playing a game after all."

"You're playing a game after all." "You're playing a game after all." kill the old man "where are you kill the old man "where are you kill the old man "where are you fix one thing, fix one thing, fix one thing, and something else breaks and something else breaks , and you fix it, , and you fix it, , and you fix it, and something else breaks and something else breaks . Uh, grades . Uh, grades . Uh, grades fix this, fix this, fix this, giving you a number, giving you a number, giving you a number, which you can which you can which you can to track, to track, to track, compare and compare and compare and respond. This has respond. This has respond. This has more important more important more important every month, because, uh, every month, because, uh, every month, because, uh, every few weeks every few weeks every few weeks someone is releasing a new one someone is releasing a new one someone is releasing a new one model, and you think:" model, and you think:" model, and you think:" Oh, I want to go to Oh, I want to go to Oh, I want to go to new model ". Uh, but new model ". Uh, but new model ". Uh, but you can't know, in you can't know, in you can't know, in why the new model will be why the new model will be why the new model will be better, and in what way? better, and in what way? better, and in what way? worse if you have worse if you have worse if you have there is no set of grades, there is no set of grades, there is no set of grades, which you can which you can which you can verify. Uh, and verify. Uh, and verify. Uh, and that's exactly what we will do that's exactly what we will do that's exactly what we will do today. We will begin today. We will begin today. We will begin with, uh, Claude 4.5, and then with, uh, Claude 4.5, and then with, uh, Claude 4.5, and then let's update it to Claude, eh let's update it to Claude, eh -er, Haiku 5.5, er, which was -er, Haiku 5.5, er, which was -er, Haiku 5.5, er, which was released approximately released approximately released approximately a week ago that a week ago that a week ago that made this made this made this demonstration demonstration demonstration extremely light extremely light extremely light to create. I to create. I to create. I I thought I wanted to I thought I wanted to I thought I wanted to update the model for update the model for update the model for this demonstration. this demonstration. this demonstration. So I'm going So I'm going So I'm going to make it to make it to make it demonstration. Uh, demonstration. Uh, demonstration. Uh, so with the help of so with the help of so with the help of you run assessments you run assessments you run assessments package, can package, can package, can compare numbers, and to compare numbers, and to compare numbers, and to at the end of the day you know, eh at the end of the day you know, eh -uh, can you -uh, can you -uh, can you update your model.

update your model. update your model. Uh, and there are several types Uh, and there are several types Uh, and there are several types grades. The first type– grades. The first type– this is code evaluation. this is code evaluation. this is code evaluation. Code evaluation is fast, Code evaluation is fast, Code evaluation is fast, free and a little free and a little free and a little fragile. She is very fragile. She is very fragile. She is very similar to modular similar to modular similar to modular tests. These are ordinary tests. These are ordinary tests. These are ordinary functions where you, by functions where you, by functions where you, by essentially, you claim essentially, you claim essentially, you claim line. There are many line. There are many line. There are many methods that can be methods that can be methods that can be to use to to use to to use to to make them to make them to make them more flexible than more flexible than more flexible than direct modular direct modular direct modular test that claims test that claims test that claims base line. But, according to base line. But, according to base line. But, according to In fact, that's what they are. In fact, that's what they are. In fact, that's what they are. do. So the things that do. So the things that do. So the things that you can you can you can to protest for to protest for to protest for through evaluation through evaluation through evaluation Kodi code, these are things like, Kodi code, these are things like, Kodi code, these are things like, like: was there a conclusion like: was there a conclusion like: was there a conclusion parsed as JSON? Or parsed as JSON? Or parsed as JSON? Or Is it less than 500 words? Or Is it less than 500 words? Or Is it less than 500 words? Or it was mentioned there, it was mentioned there, it was mentioned there, What was I asking about? What was I asking about? What was I asking about? Their advantage lies in Their advantage lies in Their advantage lies in because they because they because they are performed by are performed by are performed by milliseconds, and their milliseconds, and their milliseconds, and their launch nothing launch nothing launch nothing costs. Hmm, but there is costs. Hmm, but there is costs. Hmm, but there is many valid many valid many valid answers about which answers about which answers about which line matching is not possible line matching is not possible line matching is not possible to tell you. Hmm, and to tell you. Hmm, and to tell you. Hmm, and for this you for this you for this you need a second view need a second view need a second view evaluations, namely LLM–this evaluations, namely LLM–this evaluations, namely LLM–this Judge's assessment. Hmm, exactly. Judge's assessment. Hmm, exactly. Judge's assessment. Hmm, exactly. thinking about it thinking about it thinking about it most people when most people when most people when they are thinking about they are thinking about they are thinking about evaluations. In this evaluations. In this evaluations. In this in case you in case you in case you use use use second LLM for assessment second LLM for assessment second LLM for assessment the conclusion of the first LLM. Hmm, the conclusion of the first LLM. Hmm, the conclusion of the first LLM. Hmm, you rate it by you rate it by you rate it by the so-called rubric.

the so-called rubric. In the world of assessment, there are In the world of assessment, there are a bunch of terms, a bunch of terms, a bunch of terms, borrowed from borrowed from borrowed from machine learning, machine learning, machine learning, what is the meaning of it to me what is the meaning of it to me what is the meaning of it to me incomprehensible. One of incomprehensible. One of incomprehensible. One of of them are "traces", one is " of them are "traces", one is " evals", and the other is "rubric". I evals", and the other is "rubric". I evals", and the other is "rubric". I I think that "rubric" I think that "rubric" I think that "rubric" simply means "rules". simply means "rules". simply means "rules". Why is there a reason for this? Why is there a reason for this? Why is there a reason for this? special word? IN special word? IN special word? IN in any case, in any case, L&M judges understand L&M judges understand meaning, not just meaning, not just meaning, not just lines. So the question is lines. So the question is lines. So the question is like: is this accurate? like: is this accurate? like: is this accurate? Does this match? Does this match? Does this match? source documents? source documents? source documents? Is the tone correct for Is the tone correct for Is the tone correct for client? The hook in client? The hook in client? The hook in because since because since because since LLM judge in itself LLM judge in itself LLM judge in itself is not an LLM, it has all is not an LLM, it has all is not an LLM, it has all Disadvantages of LLM Disadvantages of LLM . So he can be . So he can be . So he can be unpredictable. unpredictable. unpredictable. He never happens. He never happens. He never happens. the same twice. Uh, the same twice. Uh, the same twice. Uh, and he can allow and he can allow and he can allow errors. Uh, they errors. Uh, they errors. Uh, they also cost also cost also cost money because you money because you money because you using LLM. Uh-huh using LLM. Uh-huh , and they can , and they can , and they can to be sure to be wrong. to be sure to be wrong. to be sure to be wrong. So you have to them So you have to them So you have to them check. And this is one check. And this is one check. And this is one of the things we of the things we of the things we we are going to do we are going to do we are going to do today. Then there is today. Then there is today. Then there is third kind third kind third kind evaluation, this evaluation, this evaluation, this human evaluation. human evaluation. human evaluation. Someone who knows Someone who knows Someone who knows subject area, subject area, subject area, who can read who can read who can read result and evaluate result and evaluate result and evaluate him as an expert. Um, him as an expert. Um, him as an expert. Um, People are golden People are golden People are golden quality standard. Hmm, quality standard. Hmm, quality standard. Hmm, they are also slow they are also slow they are also slow and expensive. Um, and them and expensive. Um, and them and expensive. Um, and them gets bored, and they gets bored, and they gets bored, and they begin begin begin to review. So, uh, to review. So, uh, to review. So, uh, you want you want you want use all use all use all three at the same time. This is not three at the same time. This is not three at the same time. This is not competing approaches.

competing approaches. competing approaches. These are complementary approaches. These are complementary approaches. These are complementary approaches. And we'll talk a little. And we'll talk a little. And we'll talk a little. about how you about how you about how you use them use them use them additionally later. additionally later. additionally later. So what is your assessment? So what is your assessment? So what is your assessment? Do you use it when? Do you use it when? Do you use it when? Use the code, Use the code, Use the code, when is the answer when is the answer when is the answer deterministic. That is deterministic. That is deterministic. That is things like formats, things like formats, things like formats, lengths, required lengths, required lengths, required fields and group phrases. fields and group phrases. fields and group phrases. Um, but Um, but Um, but use a judge use a judge use a judge LLM when you need it LLM when you need it LLM when you need it understanding. Is he understanding. Is he understanding. Is he exact? Is he exact? Is he exact? Is he justified? Is he justified? Is he justified? Is he useful? Um, and useful? Um, and useful? Um, and use people use people use people for those kinds of failures, for those kinds of failures, for those kinds of failures, which you have not seen before which you have not seen before which you have not seen before saw. Things you saw. Things you saw. Things you didn't expect. And so that didn't expect. And so that didn't expect. And so that check if yours check if yours check if yours LLM judges evaluate LLM judges evaluate LLM judges evaluate right. Um, all of this right. Um, all of this right. Um, all of this applies to any applies to any applies to any LLM programs, but LLM programs, but LLM programs, but agents, in particular, agents, in particular, agents, in particular, complicate all this, complicate all this, complicate all this, because agents because agents because agents more complicated. One more complicated. One more complicated. One a simple call to LLM is a simple call to LLM is a simple call to LLM is relatively simple. You relatively simple. You relatively simple. You give him the input give him the input give him the input data, you get data, you get data, you get source data. Um, but source data. Um, but source data. Um, but agent prone agent prone agent prone to combine a whole bunch to combine a whole bunch to combine a whole bunch steps together. Therefore, steps together. Therefore, steps together. Therefore, one of the things that can one of the things that can one of the things that can to go wrong, it's very to go wrong, it's very to go wrong, it's very early stage of the chain, early stage of the chain, early stage of the chain, the agent makes a mistake, the agent makes a mistake, the agent makes a mistake, and it cascades and it cascades and it cascades permeates all others permeates all others permeates all others your agent's steps, your agent's steps, your agent's steps, and each of them and each of them and each of them gradually getting lost gradually getting lost gradually getting lost from the rails. Hmm, and one of from the rails. Hmm, and one of from the rails. Hmm, and one of the biggest problems— the biggest problems— This is a definite mistake.

This is a definite mistake. This is a definite mistake. A sure mistake A sure mistake A sure mistake worse than obvious worse than obvious worse than obvious error. Uh, me error. Uh, me error. Uh, me talked about cascading talked about cascading talked about cascading failure. Uh, one of failure. Uh, one of failure. Uh, one of examples of this— examples of this— let's say you wrote let's say you wrote let's say you wrote report for agent with report for agent with report for agent with financial analysis, financial analysis, financial analysis, which is that type which is that type which is that type the agent I gave you the agent I gave you the agent I gave you I will show you today. Uh, I will show you today. Uh, I will show you today. Uh, and you ask him and you ask him and you ask him provide a report on Tesla. provide a report on Tesla. provide a report on Tesla. He is looking for a Tesla and He is looking for a Tesla and He is looking for a Tesla and instead of instead of instead of get results get results get results about the automobile about the automobile about the automobile company, he gets company, he gets company, he gets results about results about results about inventor 18 inventor 18 inventor 18 century. He writes century. He writes century. He writes beautiful, perfect beautiful, perfect beautiful, perfect formatted report formatted report formatted report about the inventor 18 about the inventor 18 about the inventor 18 century. Uh, and none century. Uh, and none century. Uh, and none from your checks not from your checks not from your checks not passes because you passes because you passes because you Do you think he is okay? Do you think he is okay? Do you think he is okay? formatted. He formatted. He formatted. He in a great tone. All in a great tone. All in a great tone. All grammatically correct grammatically correct . Uh, that's almost it. . Uh, that's almost it. . Uh, that's almost it. not the right topic. Uh, and this not the right topic. Uh, and this not the right topic. Uh, and this the type of failure that, the type of failure that, the type of failure that, uh, sneaky for uh, sneaky for uh, sneaky for agents. Agents also agents. Agents also agents. Agents also may experience may experience may experience failures in reverse failures in reverse failures in reverse direction. Um, Anthropic, uh- direction. Um, Anthropic, uh- Who created Claude? Who created Claude? Who created Claude? conducted a benchmark under conducted a benchmark under conducted a benchmark under called Ta2 bench, which called Ta2 bench, which called Ta2 bench, which simulates conversations with simulates conversations with simulates conversations with service service service customers, and they, uh, customers, and they, uh, customers, and they, uh, gave him this task gave him this task , where the user , where the user , where the user asks to change his asks to change his asks to change his flight. Uh, and the test, flight. Uh, and the test, flight. Uh, and the test, according to the test, according to the test, according to the test, the user had not the user had not the user had not take a test, uh, or, take a test, uh, or, take a test, uh, or, rather, the agent should not rather, the agent should not rather, the agent should not to pass because, uh, to pass because, uh, to pass because, uh, the person was in the economy the person was in the economy class, but you don't class, but you don't class, but you don't allowed to change allowed to change allowed to change in places in economy in places in economy class. Uh, what is Cloud?

class. Uh, what is Cloud? class. Uh, what is Cloud? discovered, this is what discovered, this is what discovered, this is what they can they can they can to upgrade to upgrade to upgrade person to the first. person to the first. person to the first. You are allowed You are allowed You are allowed to change one's place in to change one's place in to change one's place in first class, and first class, and first class, and then they lowered then they lowered then they lowered her class back after her class back after her class back after how she changed how she changed how she changed your place back to your place back to your place back to economy class. Uh, and economy class. Uh, and economy class. Uh, and passed, and did what passed, and did what passed, and did what what the user requested, what the user requested, what the user requested, and the test said "no" that and the test said "no" that and the test said "no" that you didn't pass because you didn't pass because you didn't pass because you shouldn't have you shouldn't have you shouldn't have to be able to do this to be able to do this to be able to do this make. Uh, but make. Uh, but make. Uh, but the agent was smarter the agent was smarter the agent was smarter for the rules by which for the rules by which for the rules by which he was. Uh, so you he was. Uh, so you he was. Uh, so you should expect that should expect that uh, when you uh, when you uh, when you create an estimate, create an estimate, create an estimate, sometimes an agent will come up with sometimes an agent will come up with sometimes an agent will come up with problem solving, problem solving, problem solving, which you didn't expect which you didn't expect . Therefore, your tests . Therefore, your tests . Therefore, your tests must be very must be very must be very flexible. Here's how flexible. Here's how flexible. Here's how usually looks like usually looks like usually looks like evaluation result evaluation result . He has a score, a mark, and . He has a score, a mark, and . He has a score, a mark, and explanation. Ball is explanation. Ball is explanation. Ball is the number that usually the number that usually the number that usually is zero or is zero or is zero or units. The label is units. The label is units. The label is just a word that just a word that just a word that can read can read can read man and what man and what man and what corresponds to the value corresponds to the value corresponds to the value ball. For example, " ball. For example, " "enrolled" and "not" "enrolled" and "not" "enrolled" and "not" enrolled", " enrolled", " subject to" and "not subject to" and "not subject to" and "not "subject" is what "subject" is what "subject" is what can be done. AND can be done. AND can be done. AND judges with LLM add judges with LLM add judges with LLM add the third thing is an explanation, the third thing is an explanation, the third thing is an explanation, why did you get it why did you get it why did you get it such an assessment, why do you such an assessment, why do you such an assessment, why do you got exactly this got exactly this got exactly this label. And this label. And this label. And this the most valuable part the most valuable part the most valuable part work of a judge with an LLM.

work of a judge with an LLM. Because it is in Because it is in explanations explanations explanations is there value is there value . It tells you, . It tells you, . It tells you, what was wrong, what was wrong, what was wrong, and what was missing, and and what was missing, and and what was missing, and often says that often says that often says that correct in the query, correct in the query, correct in the query, so that everything is fine so that everything is fine so that everything is fine right. It right. It right. It especially useful, especially useful, especially useful, when you spend when you spend when you spend evaluation in large evaluation in large evaluation in large scale, because you scale, because you scale, because you you can get, you can get, you can get, you know, 50, 100, a thousand you know, 50, 100, a thousand you know, 50, 100, a thousand explanations from your explanations from your explanations from your LLM, and you can LLM, and you can LLM, and you can ask for another LLM ask for another LLM ask for another LLM read all these read all these read all these explanation and say: explanation and say: explanation and say: "It seems that in these "It seems that in these "It seems that in these explanations are certain explanations are certain explanations are certain regularity". regularity". regularity". It seems we have It seems we have It seems we have systematic systematic systematic problem. And you can problem. And you can problem. And you can just hand them over just hand them over just hand them over to his agent, and he to his agent, and he to his agent, and he says, "Okay, I see 50 says, "Okay, I see 50 says, "Okay, I see 50 routes that do that routes that do that routes that do that exactly, with the same exactly, with the same exactly, with the same explanation of what explanation of what explanation of what It went wrong. I can It went wrong. I can It went wrong. I can "fix it." So, "fix it." So, "fix it." So, this is a cycle that we this is a cycle that we this is a cycle that we we will create we will create we will create today. We will today. We will today. We will trace your trace your trace your agent. We are going agent. We are going agent. We are going read that he read that he read that he did. We are going did. We are going did. We are going write evaluations (eval) write evaluations (eval) write evaluations (eval) for what it has for what it has for what it has value. Uh, we value. Uh, we value. Uh, we we are going we are going we are going check if possible check if possible check if possible trust these estimates. trust these estimates. trust these estimates. Uh, then we Uh, then we Uh, then we we are going to change we are going to change we are going to change agent and we agent and we agent and we we are going to prove, we are going to prove, we are going to prove, that the change worked. that the change worked. that the change worked. Then we will block Then we will block Then we will block this, so that it remains this, so that it remains this, so that it remains corrected, corrected, corrected, by placing this in CI. by placing this in CI. by placing this in CI. So, uh, who else So, uh, who else So, uh, who else need help with need help with need help with settings? Who settings? Who settings? Who was late and was late and was late and needs needs needs settings?

settings? settings? Raise your hands. Good Raise your hands. Good , let me , let me , let me return to return to return to settings screen settings screen settings screen and see if and see if and see if can you do this can you do this right. This is magical. right. This is magical. URL. If you have URL. If you have URL. If you have it is not there, then all that it is not there, then all that it is not there, then all that you need to do, you need to do, you need to do, this is to read the readme file this is to read the readme file this is to read the readme file and, uh, launch and, uh, launch and, uh, launch instructions, or instructions, or instructions, or just, you know, just, you know, just, you know, insert the readme file into insert the readme file into insert the readme file into your agent, and your your agent, and your your agent, and your the agent will read the agent will read the agent will read instructions and instructions and instructions and will do everything for you. will do everything for you. will do everything for you. One of the strange things about One of the strange things about One of the strange things about creating educational creating educational creating educational benefits in 2026 benefits in 2026 benefits in 2026 is that we is that we is that we we just say, "Well, I we just say, "Well, I we just say, "Well, I could show you, could show you, could show you, how to do all this, or how to do all this, or how to do all this, or I could just I could just I could just tell you how tell you how tell you how tell your tell your tell your agent to do all this agent to do all this agent to do all this ". Uh, and that's what we ". Uh, and that's what we ". Uh, and that's what we we are making a bigger one we are making a bigger one we are making a bigger one part of the time. Good, part of the time. Good, part of the time. Good, now I assume that now I assume that now I assume that everyone has this URL- everyone has this URL- address. Steeply. So address. Steeply. So address. Steeply. So let's start with the first one let's start with the first one let's start with the first one step. Step one– step. Step one– this is tracing. First this is tracing. First this is tracing. First than you can do something than you can do something than you can do something to evaluate, you need to evaluate, you need to evaluate, you need get data for get data for get data for launch of assessments. And this launch of assessments. And this launch of assessments. And this what is what is what is tracing. Today tracing. Today tracing. Today we will we will we will use Phoenix.

use Phoenix. use Phoenix. I hope this is it. I hope this is it. I hope this is it. no one was surprised. no one was surprised. no one was surprised. Uh, Phoenix is Uh, Phoenix is Uh, Phoenix is platform platform platform observation of observation of observation of with the help of artificial with the help of artificial with the help of artificial intelligence with intelligence with intelligence with open source from open source from open source from Aris. Uh, he's collecting. Aris. Uh, he's collecting. Aris. Uh, he's collecting. tracing from your tracing from your tracing from your AI application AI in any AI application AI in any which framework, which framework, which framework, which one are you which one are you which one are you use for use for use for its creation. He its creation. He its creation. He saves your grades saves your grades saves your grades next to these next to these next to these tracings. Uh, and tracings. Uh, and tracings. Uh, and he provides you he provides you he provides you interface interface interface user to user to user to to dig in everything to dig in everything to dig in everything this. He also this. He also this. He also executes data sets executes data sets executes data sets and experiments, about and experiments, about and experiments, about who we are today who we are today who we are today Let's talk. Uh, you too Let's talk. Uh, you too Let's talk. Uh, you too launch it on launch it on launch it on to your laptop. So, to your laptop. So, to your laptop. So, uh, you can keep it uh, you can keep it uh, you can keep it its completely its completely its completely private for yourself, private for yourself, private for yourself, or you can or you can or you can run it on run it on run it on server and server and server and share it with share it with share it with with your team. So with your team. So with your team. So now I'm going now I'm going now I'm going dangerously dangerously dangerously check if I can check if I can check if I can actually...Oh, that's right actually...Oh, that's right actually...Oh, that's right small. Let's do it. small. Let's do it. small. Let's do it. this is bigger. Okay, so this is bigger. Okay, so this is bigger. Okay, so today we are working today we are working today we are working over Typescript. Uh, you over Typescript. Uh, you over Typescript. Uh, you you will see a bunch you will see a bunch you will see a bunch Python examples, and Python examples, and Python examples, and Fenix ​​also has great Fenix ​​also has great Fenix ​​also has great Python SDK. Uh, but I Python SDK. Uh, but I Python SDK. Uh, but I like JavaScript, like JavaScript, like JavaScript, so we will so we will so we will use use use TypeScript. Uh, TypeScript. Uh, TypeScript. Uh, repository is repository is repository is small set small set small set scripts. Instrumentation.ts scripts. Instrumentation.ts scripts. Instrumentation.ts configures configures configures tracing. Agent.ts is tracing. Agent.ts is tracing. Agent.ts is the agent himself. Prompts.ts the agent himself. Prompts.ts the agent himself. Prompts.ts contains hints. Uh, contains hints. Uh, contains hints. Uh, Where are all these things? That's it.

Where are all these things? That's it. Where are all these things? That's it. all. Um, and you will be all. Um, and you will be all. Um, and you will be edit tooltips edit tooltips edit tooltips in this folder. THERE ARE, in this folder. THERE ARE, in this folder. THERE ARE, sorry, in this sorry, in this sorry, in this file. Um, and the evals folder- file. Um, and the evals folder- this is the place to this is the place to this is the place to our evals are coming. Queries.json our evals are coming. Queries.json -this is our test data. -this is our test data. -this is our test data. Um, and I hope... Um, and I hope... Um, and I hope... every step today every step today every step today will be just one npm- will be just one npm- team. Um, and each team. Um, and each team. Um, and each of them is independent. of them is independent. of them is independent. So, if you So, if you So, if you are you behind, you can are you behind, you can are you behind, you can go directly to go directly to go directly to next. Um, and next. Um, and next. Um, and use use use the initial data that I the initial data that I the initial data that I placed in placed in placed in repository, eh, so that you repository, eh, so that you repository, eh, so that you did not lag behind. Therefore, did not lag behind. Therefore, did not lag behind. Therefore, what we create what we create what we create today, or, actually, today, or, actually, today, or, actually, what I have already created what I have already created what I have already created for you, this is for you, this is for you, this is financial analyst. financial analyst. financial analyst. It works like this: you It works like this: you It works like this: you give him stock give him stock give him stock ticker and something on which ticker and something on which ticker and something on which you can concentrate, you can concentrate, you can concentrate, for example, for example, for example, supply supply supply Tesla cars. Um, and Tesla cars. Um, and Tesla cars. Um, and he will leave, and he he will leave, and he he will leave, and he will search the Internet will search the Internet will search the Internet real-time data real-time data , uh, and write you a report , uh, and write you a report , uh, and write you a report about this. Um, there are goals. about this. Um, there are goals. about this. Um, there are goals. startups built startups built startups built on this idea of ​​providing on this idea of ​​providing on this idea of ​​providing you know, you know, you know, financial analysis financial analysis financial analysis whether you should whether you should whether you should buy shares. Um, buy shares. Um, buy shares. Um, but ours intentionally but ours intentionally but ours intentionally simple, because I simple, because I simple, because I something was needed, something was needed, something was needed, which would allow which would allow which would allow mistakes so that we can mistakes so that we can mistakes so that we can fix them. Um, fix them. Um, today is not about how today is not about how create an agent. I create an agent. I create an agent. I just created for you just created for you just created for you agent. I suppose agent. I suppose agent. I suppose What do you already have somewhere?

What do you already have somewhere? What do you already have somewhere? an agent works, and you an agent works, and you an agent works, and you need help with need help with need help with its reliability. Our its reliability. Our its reliability. Our The agent works in two shifts. The agent works in two shifts. The agent works in two shifts. course. He has a move. course. He has a move. course. He has a move. research, during research, during research, during which he is looking for in which he is looking for in which he is looking for in Internet, and then Internet, and then Internet, and then progress of writing the report, progress of writing the report, progress of writing the report, where he spends where he spends where he spends research and writes research and writes research and writes report on it. I report on it. I report on it. I created it for created it for created it for using the SDK agent using the SDK agent using the SDK agent Claude. If you haven't already Claude. If you haven't already Claude. If you haven't already used, then this used, then this used, then this Anthropics framework for Anthropics framework for Anthropics framework for agents. This is the same one. agents. This is the same one. agents. This is the same one. an engine that works an engine that works an engine that works under the code clawed, that under the code clawed, that under the code clawed, that will become relevant will become relevant will become relevant later, especially later, especially later, especially if you if you if you using the code using the code using the code clawed to work with him, clawed to work with him, clawed to work with him, because two versions because two versions because two versions cloud code, like cloud code, like cloud code, like usually try usually try usually try cooperate with one another cooperate with one another cooperate with one another one, and this forces them one, and this forces them one, and this forces them to deceive the assessment that to deceive the assessment that to deceive the assessment that not very useful. He not very useful. He not very useful. He provides you with provides you with provides you with tools for tools for tools for using the web using the web searches and conversations that searches and conversations that searches and conversations that are conducted during are conducted during are conducted during all moves. There is also all moves. There is also all moves. There is also many other options, with many other options, with many other options, with How compatible we are. Versel How compatible we are. Versel How compatible we are. Versel AI SDK, Mastra, Langchain, SDK AI SDK, Mastra, Langchain, SDK AI SDK, Mastra, Langchain, SDK OpenAI agent. Uh, in Phoenix OpenAI agent. Uh, in Phoenix OpenAI agent. Uh, in Phoenix there are adapters and there are adapters and there are adapters and toolkit for toolkit for toolkit for every possible every possible every possible framework that you framework that you framework that you only you can only you can only you can imagine, and each imagine, and each imagine, and each LLM provider, LLM provider, LLM provider, which you only which you only which you only You can imagine.

You can imagine. You can imagine. Wherever you build Wherever you build Wherever you build his agent, Phoenix his agent, Phoenix his agent, Phoenix will work with him, and will work with him, and will work with him, and toolkit toolkit toolkit will take place will take place will take place automatically. Uh, but automatically. Uh, but automatically. Uh, but I had to I had to I had to choose one for choose one for choose one for demonstrations, so I demonstrations, so I demonstrations, so I chose this one. Uh, chose this one. Uh, chose this one. Uh, tracing takes tracing takes tracing takes about five lines. about five lines. about five lines. Uh, first we Uh, first we Uh, first we call register from call register from call register from Phoenix hotel package, which Phoenix hotel package, which Phoenix hotel package, which configures configures configures tracing and tracing and tracing and directs him to directs him to directs him to Phoenix. One of the advantages Phoenix. One of the advantages of this industry is to of this industry is to because everyone chose because everyone chose because everyone chose almost the same almost the same almost the same standard standard standard tools. Uh, tools. Uh, tools. Uh, everything is built on everything is built on everything is built on open telemetry open telemetry open telemetry in the industry, and in addition in the industry, and in addition in the industry, and in addition to the open to the open to the open telemetry exists telemetry exists telemetry exists standard called standard called standard called open conclusion, open conclusion, open conclusion, who arize cares who arize cares who arize cares about, uh. And everyone already about, uh. And everyone already about, uh. And everyone already built-in built-in built-in tools in tools in tools in your frameworks; ago your frameworks; ago your frameworks; ago we are compatible with everyone we are compatible with everyone we are compatible with everyone these frameworks. these frameworks. these frameworks. Everything you need Everything you need Everything you need to do, this to do, this to do, this use this use this use this registered registered registered command to say:" command to say:" command to say:" Hey, that one Hey, that one Hey, that one toolkit, which toolkit, which toolkit, which you have already built in you have already built in you have already built in your framework, your framework, your framework, turn it on and turn it on and turn it on and send here. " In send here. " In send here. " In in this case you in this case you in this case you just point it just point it just point it on your laptop, and this on your laptop, and this on your laptop, and this everything you need everything you need everything you need make. Every call make. Every call make. Every call agent who does agent who does agent who does our agent, will now be our agent, will now be our agent, will now be be tracked. Hmm, oh, be tracked. Hmm, oh, be tracked. Hmm, oh, I already said that. So I already said that. So I already said that. So advantage of this advantage of this advantage of this is that we is that we is that we no need to write no need to write no need to write log line for log line for log line for each individual each individual each individual call. We don't call. We don't call. We don't need to write need to write need to write tracing for tracing for tracing for each individual action each individual action each individual action our agent. He our agent. He our agent. He just do it just do it just do it automatically. Sometimes automatically. Sometimes automatically. Sometimes we will need we will need we will need add an additional add an additional add an additional information, and information, and information, and framework is framework is framework is absolutely supports.

absolutely supports. absolutely supports. Hmm, but we don't Hmm, but we don't Hmm, but we don't need to do this need to do this need to do this for each individual for each individual for each individual actions. Hmm, in the version actions. Hmm, in the version actions. Hmm, in the version TypeScript has one problem TypeScript has one problem , which you should , which you should , which you should to know, namely: manually to know, namely: manually to know, namely: manually you must you must you must use use use tool manually tool manually tool manually instrument that will return instrument that will return instrument that will return you a copy you a copy you a copy tool, and you tool, and you tool, and you must must must use use use copy of the instrument, copy of the instrument, copy of the instrument, because the cloud SDK because the cloud SDK because the cloud SDK agent is an ES module, and agent is an ES module, and agent is an ES module, and export ES modules export ES modules export ES modules is read. So is read. So is read. So the toolkit is not the toolkit is not the toolkit is not can do the same, can do the same, can do the same, as in Python, where it as in Python, where it as in Python, where it replaces yours replaces yours replaces yours tool and just tool and just tool and just changes everything. You changes everything. You changes everything. You need need need use use use the copy he gave you the copy he gave you the copy he gave you provides, in which this provides, in which this provides, in which this The tool has been replaced. The tool has been replaced. The tool has been replaced. Hmm, it won't break, Hmm, it won't break, Hmm, it won't break, he just does nothing he just does nothing he just does nothing will record if you will record if you will record if you do not use do not use do not use the copy he gave you the copy he gave you the copy he gave you provides. So, everywhere in provides. So, everywhere in provides. So, everywhere in this repository we this repository we this repository we import the agent SDK import the agent SDK import the agent SDK Claude from our file Claude from our file Claude from our file tools, not tools, not tools, not directly from directly from directly from package. Now, if you package. Now, if you package. Now, if you open agent.ts, you open agent.ts, you open agent.ts, you you will see our you will see our you will see our agent parameters. We agent parameters. We agent parameters. We let's start let's start let's start use Claude use Claude use Claude Haiku F 4.5 because it Haiku F 4.5 because it Haiku F 4.5 because it fast, cheap and fast, cheap and fast, cheap and quite useful. He quite useful. He quite useful. He makes mistakes that makes mistakes that makes mistakes that pleasant and pleasant and pleasant and predictable even predictable even predictable even with a relatively simple with a relatively simple with a relatively simple agent. Unlike agent. Unlike agent. Unlike from Frontier models that from Frontier models that from Frontier models that they will definitely be anyway they will definitely be anyway they will definitely be anyway make mistakes, but make mistakes, but make mistakes, but you need enough you need enough you need enough complex agent and complex agent and complex agent and quite complicated quite complicated quite complicated set of tasks. And this set of tasks. And this set of tasks. And this makes demo version difficult makes demo version difficult . If you haven't yet . If you haven't yet . If you haven't yet used it used it , then allowed , then allowed , then allowed tools provide tools provide tools provide him a web search to him a web search to him a web search to he could find he could find he could find current financial current financial current financial data. And the permission mode"

data. And the permission mode" data. And the permission mode" don't ask " don't ask " means that he is not means that he is not means that he is not will be interactive. will be interactive. will be interactive. He is not He is not He is not will stop and not will stop and not will stop and not will ask you questions, will ask you questions, will ask you questions, uh, because there uh, because there uh, because there "no one will tell" "no one will tell" "no one will tell" yes ". Um, this option yes ". Um, this option yes ". Um, this option will become relevant, uh, will become relevant, uh, will become relevant, uh, for failure mode for failure mode for failure mode later. Um, and the others? later. Um, and the others? later. Um, and the others? three options, as I already three options, as I already three options, as I already remembered, not remembered, not remembered, not allow your allow your allow your laptop to the agent. SDK laptop to the agent. SDK laptop to the agent. SDK cloud agent, uh, cloud agent, uh, cloud agent, uh, works on the same works on the same works on the same engine, which is also cloud-based engine, which is also cloud-based engine, which is also cloud-based code. Uh, so if you code. Uh, so if you code. Uh, so if you do it in the cloud do it in the cloud do it in the cloud in Kodi, he usually in Kodi, he usually in Kodi, he usually uses options, uses options, uses options, which you gave to your which you gave to your which you gave to your copies of Claude's code, not copies of Claude's code, not copies of Claude's code, not its cloud copies its cloud copies its cloud copies code. Uh, what, how? code. Uh, what, how? code. Uh, what, how? usually does things usually does things usually does things extremely extremely extremely non-deterministic, non-deterministic, non-deterministic, which is not useful. Uh, which is not useful. Uh, which is not useful. Uh, so I, uh, added it there so I, uh, added it there so I, uh, added it there a few lines of code, a few lines of code, a few lines of code, which guarantee that which guarantee that which guarantee that cloud, uh, SDK Claude cloud, uh, SDK Claude cloud, uh, SDK Claude works in his own works in his own works in his own isolated version, e isolated version, e -e, cloud SDK and not -e, cloud SDK and not -e, cloud SDK and not steals your copies, steals your copies, steals your copies, you know, keys and you know, keys and you know, keys and other things from your other things from your other things from your data. Uh, yes, uh, bye data. Uh, yes, uh, bye data. Uh, yes, uh, bye things, none of this things, none of this things, none of this does not only apply to does not only apply to does not only apply to Phoenix. Uh, it's simple.

Phoenix. Uh, it's simple. Phoenix. Uh, it's simple. danger danger danger using using using cloud SDK at the same time cloud SDK at the same time cloud SDK at the same time using using using cloud code. Uh, in cloud code. Uh, in cloud code. Uh, in this SDK one turn- this SDK one turn- this is one query call this is one query call . So, you give him . So, you give him . So, you give him hint. Um, he hint. Um, he hint. Um, he streams streams streams message while message while message while The agent is working. He The agent is working. He The agent is working. He thinks that causes thinks that causes thinks that causes tool. He tool. He tool. He reads the result. Um, reads the result. Um, reads the result. Um, he can cause he can cause he can cause another tool, and another tool, and another tool, and when he finishes, when he finishes, when he finishes, you get you get you get message about message about message about result from result from result from final text final text final text and identifier and identifier and identifier session. session. session. Session ID Session ID -this is how we -this is how we -this is how we Let's continue the conversation. Let's continue the conversation. Let's continue the conversation. If we pass If we pass If we pass its back as a resume, its back as a resume, its back as a resume, next move next move next move remembers everything that remembers everything that remembers everything that saw the previous move. saw the previous move. saw the previous move. Um, so this is how we Um, so this is how we Um, so this is how we we can spend we can spend we can spend research, and then research, and then research, and then say: "Hey, write say: "Hey, write say: "Hey, write report on this report on this report on this research "because research "because research "because he knows what it is he knows what it is he knows what it is is in the same is in the same is in the same session. Um, and he can session. Um, and he can session. Um, and he can find research, find research, find research, which I just created which I just created which I just created previous move. Then previous move. Then previous move. Then we combine two we combine two we combine two walk. Um, so we walk. Um, so we walk. Um, so we we get two moves in we get two moves in we get two moves in one trace. Um, one trace. Um, one trace. Um, first move first move first move research, then research, then research, then the flow of writing, the flow of writing, the flow of writing, restoring the session restoring the session restoring the session the course of the research, so that the course of the research, so that the course of the research, so that the writer could the writer could the writer could to see everything that to see everything that to see everything that the researcher found. Em the researcher found. Em , and then we , and then we , and then we let's wrap it all up let's wrap it all up let's wrap it all up tracing agent. tracing agent. tracing agent. Um, the reason why we Um, the reason why we Um, the reason why we we do this, um, consists we do this, um, consists we do this, um, consists in that it gives us in that it gives us in that it gives us one parent one parent one parent the gap that the gap that the gap that covers all data.

covers all data. covers all data. If we didn't do this If we didn't do this If we didn't do this did, in our did, in our did, in our We would get Phoenix We would get Phoenix We would get Phoenix two, uh, two, uh, two, uh, two, uh, two, uh, two, uh, gaps between brothers gaps between brothers gaps between brothers and sisters. Hmm, this is and sisters. Hmm, this is and sisters. Hmm, this is it would be nice, but it would be nice, but it would be nice, but a little harder to read. THERE ARE- a little harder to read. THERE ARE- uh so i just uh uh so i just uh uh so i just uh I'm generating data now, I'm generating data now, I'm generating data now, to make it easier for yourself to make it easier for yourself to make it easier for yourself life later. Now, life later. Now, life later. Now, if you open if you open if you open prompts.ts, ts, you will see, prompts.ts, ts, you will see, prompts.ts, ts, you will see, what, uh, there's a bunch of them what, uh, there's a bunch of them what, uh, there's a bunch of them tips. They tips. They tips. They intentionally bad. Uh, intentionally bad. Uh, intentionally bad. Uh, there is one line there is one line there is one line instructions with instructions with instructions with research, one research, one research, one a string of instructions from a string of instructions from a string of instructions from writing. Uh, me too writing. Uh, me too writing. Uh, me too did it intentionally, did it intentionally, did it intentionally, because it gives us because it gives us because it gives us many opportunities many opportunities many opportunities for improvement for improvement for improvement later. Uh, if you later. Uh, if you later. Uh, if you programming, now programming, now programming, now it's time to start it's time to start it's time to start to experiment with to experiment with to experiment with things. Don't be shy. things. Don't be shy. things. Don't be shy. change them. change them. change them. Add things like this, Add things like this, Add things like this, how to not make mistakes how to not make mistakes how to not make mistakes ", or I'll give you 100 ", or I'll give you 100 ", or I'll give you 100 dollars in tips. Uh, dollars in tips. Uh, dollars in tips. Uh, supposedly it helps. supposedly it helps. supposedly it helps. Uh, I want everyone in Uh, I want everyone in Uh, I want everyone in received a room received a room received a room a little different agent, a little different agent, a little different agent, uh, so that we can get uh, so that we can get uh, so that we can get different experiences, uh, what different experiences, uh, what different experiences, uh, what will make a comparison will make a comparison will make a comparison notes more interesting. notes more interesting. So now, if you So now, if you adhered to adhered to adhered to instructions, everything instructions, everything instructions, everything installed, and everything installed, and everything installed, and everything works well, uh, you works well, uh, you works well, uh, you you can run this.

you can run this. you can run this. You should run npm You should run npm You should run npm run agent dash, uh, Tesla, a run agent dash, uh, Tesla, a run agent dash, uh, Tesla, a then growth outlook (in then growth outlook (in then growth outlook (in in quotes). Um, you can in quotes). Um, you can in quotes). Um, you can do it on a Tesla, you do it on a Tesla, you do it on a Tesla, you you can do it on you can do it on you can do it on to some other to some other to some other companies, if you want companies, if you want . Um, this will take a minute- . Um, this will take a minute- two, because he two, because he two, because he really need really need really need search the Internet, and search the Internet, and search the Internet, and you are connected to Wi-Fi you are connected to Wi-Fi you are connected to Wi-Fi conferences. Um, and conferences. Um, and conferences. Um, and the model will conduct the model will conduct the model will conduct several rounds several rounds several rounds considerations. Hmm, what considerations. Hmm, what considerations. Hmm, what happening here, happening here, happening here, while it works for while it works for while it works for you, this is what the agent you, this is what the agent you, this is what the agent receives a request for receives a request for receives a request for research. He research. He research. He decides what to do decides what to do decides what to do need to search, uh, need to search, uh, need to search, uh, what is the challenge what is the challenge what is the challenge tool. He tool. He tool. He reads the results, e- reads the results, e- uh, decides whether uh, decides whether uh, decides whether he has enough he has enough he has enough results, or to him results, or to him results, or to him need to search more need to search more need to search more time. And that's what it does. time. And that's what it does. time. And that's what it does. his agent. We are not his agent. We are not his agent. We are not indicated how much indicated how much indicated how much times he needs times he needs times he needs search the Internet. search the Internet. search the Internet. We did not specify what he We did not specify what he We did not specify what he need to search in need to search in need to search in Internet. Agent Internet. Agent Internet. Agent makes these decisions makes these decisions makes these decisions independently. Uh, so independently. Uh, so independently. Uh, so all our agents all our agents all our agents will accept something will accept something will accept something different solutions regarding different solutions regarding different solutions regarding these things. They these things. They these things. They will use will use will use slightly different search engines slightly different search engines slightly different search engines terms. Uh, they, terms. Uh, they, terms. Uh, they, will probably be looking for will probably be looking for will probably be looking for different amounts different amounts different amounts times. Um, and then, like times. Um, and then, like times. Um, and then, like only this is done, only this is done, only this is done, the queue begins the queue begins the queue begins writing, and all that writing, and all that writing, and all that turns into turns into turns into report. So, the path that report. So, the path that report. So, the path that chooses an agent, is chooses an agent, is chooses an agent, is non-deterministic by non-deterministic by non-deterministic by by its nature. Ahem, by its nature. Ahem, by its nature. Ahem, both reports can both reports can both reports can to be normal, to be normal, to be normal, if you run them, if you run them, if you run them, if you run them if you run them if you run them twice. Um, or one twice. Um, or one twice. Um, or one could be great, but could be great, but could be great, but another one can be another one can be another one can be wrong, and you don't wrong, and you don't wrong, and you don't you can, but you can't you can, but you can't you can, but you can't can you predict can you predict can you predict this, um, weekend this, um, weekend this, um, weekend result from input result from input result from input data. Um, that's why data. Um, that's why data. Um, that's why you need grades.

you need grades. Hmm, who already has one? Hmm, who already has one? real report? Someone real report? Someone ? All ? All ? All still being done. still being done. still being done. Okay, we have time. Okay, we have time. Okay, we have time. Let me be Let me be Let me be brave. And in fact brave. And in fact brave. And in fact I will try to run it. I will try to run it. I will try to run it. him/herself. One him/herself. One him/herself. One from the problems with which, as from the problems with which, as from the problems with which, as we will see, we will see, we will see, the agent will encounter, the agent will encounter, the agent will encounter, is that we is that we is that we told him told him told him write a report, but we write a report, but we write a report, but we didn't tell him where didn't tell him where didn't tell him where write a report. And therefore write a report. And therefore write a report. And therefore he assumes that he he assumes that he he assumes that he should, um, should, um, should, um, write a report in write a report in write a report in file. That's it. So, you file. That's it. So, you file. That's it. So, you You can see mine. You can see mine. You can see mine. I need permission. I need permission. I need permission. to write a file to write a file to write a file financial report. financial report. financial report. Do you want me to Do you want me to Do you want me to displayed the report here? displayed the report here? displayed the report here? This is completely useless. This is completely useless. This is completely useless. The agent does not understand that The agent does not understand that The agent does not understand that he works in he works in he works in environment without environment without environment without permissions, and what I wanted permissions, and what I wanted so that the report is actually so that the report is actually so that the report is actually was brought out in was brought out in was brought out in command line. Uh, command line. Uh, command line. Uh, and that's because mine and that's because mine and that's because mine The request is ambiguous. The request is ambiguous. The request is ambiguous. Hmm, but sometimes he Hmm, but sometimes he Hmm, but sometimes he doesn't do this. doesn't do this. doesn't do this. Sometimes he just Sometimes he just Sometimes he just understands and gives out understands and gives out understands and gives out report to you. Uh, this is it. report to you. Uh, this is it. report to you. Uh, this is it. non-determinism, non-determinism, non-determinism, which we are talking about. which we are talking about. which we are talking about. So let's So let's So let's let's see what Phoenix let's see what Phoenix let's see what Phoenix pulled out of everything pulled out of everything pulled out of everything this. Here is my Phoenix 101.

this. Here is my Phoenix 101. this. Here is my Phoenix 101. Here is my financial Here is my financial Here is my financial report. This is what it looks like report. This is what it looks like report. This is what it looks like tracing. So, you tracing. So, you tracing. So, you you can see how you can see how you can see how agent works agent works agent works financial statements, and financial statements, and financial statements, and you can see two you can see two you can see two agent claude calls. agent claude calls. One is this One is this research report. research report. research report. You can see You can see You can see search for tools search for tools search for tools inside it, so that inside it, so that inside it, so that to find out if there is something that to find out if there is something that to find out if there is something that available to him. Then available to him. Then available to him. Then you can see that you can see that you can see that he finds he finds he finds web tool web tool search. So, he search. So, he search. So, he performs three different web performs three different web -search. You can -search. You can -search. You can see incoming and see incoming and see incoming and source data. Uh, source data. Uh, source data. Uh, what kind of request, what what kind of request, what what kind of request, what output result output result output result he received from he received from he received from every call every call every call tool. Uh, and tool. Uh, and tool. Uh, and then you can see then you can see uh, what's up with this? uh, what's up with this? uh, what's up with this? case everything case everything case everything it went bad. Um, he it went bad. Um, he it went bad. Um, he tried to write down tried to write down tried to write down this, wrote a report, and this, wrote a report, and this, wrote a report, and then tried then tried then tried write it down write it down write it down file on my file on my file on my computer and said, computer and said, computer and said, that it can't that it can't that it can't work. Um, so, uh, work. Um, so, uh, work. Um, so, uh, What is important is that we don't What is important is that we don't What is important is that we don't had to add had to add had to add no logging no logging no logging to our agent. to our agent. to our agent. The important thing is that all this The important thing is that all this The important thing is that all this reflected all this reflected all this reflected all this very useful very useful very useful information about what information about what information about what happens, without happens, without happens, without none of ours none of ours none of ours interventions. We are not interventions. We are not interventions. We are not they said that your they said that your they said that your tools must tools must tools must to be instrumented to be instrumented . We didn't say that your . We didn't say that your . We didn't say that your web search queries web search queries web search queries must be must be must be instrumented. All instrumented. All instrumented. All this happens as this happens as this happens as part of the cloud SDK part of the cloud SDK part of the cloud SDK agent, because it is already agent, because it is already agent, because it is already instrumented for instrumented for instrumented for us. Um, and you can us. Um, and you can us. Um, and you can click on any click on any some place in Phoenix, and some place in Phoenix, and some place in Phoenix, and you can see you can see you can see a huge number a huge number a huge number data that data that data that is happening. Um, uh, and is happening. Um, uh, and is happening. Um, uh, and the reason why all this the reason why all this the reason why all this happened, reason, happened, reason, happened, reason, why is he trying why is he trying why is he trying burn to disk, and burn to disk, and burn to disk, and then it fails, then it fails, then it fails, is that we is that we is that we they set him up they set him up they set him up settings" is not settings" is not settings" is not to ask ". We to ask ". We to ask ". We told him that you told him that you told him that you you will work without

you will work without you will work without supervision. Um, don't bet. supervision. Um, don't bet. supervision. Um, don't bet. I have no questions. I have no questions. I have no questions. He started, not He started, not He started, not putting no putting no putting no questions, and then something questions, and then something questions, and then something it went bad. Hmm, but it went bad. Hmm, but it went bad. Hmm, but you wouldn't know that, you wouldn't know that, you wouldn't know that, if we couldn't if we couldn't if we couldn't look at look at look at tracing, right? tracing, right? tracing, right? You would just get You would just get You would just get mysterious conclusion, where mysterious conclusion, where mysterious conclusion, where it is about that it is about that it is about that Something wrong. Something wrong. Something wrong. Tracing is a place, Tracing is a place, Tracing is a place, where you can see, where you can see, where you can see, what a challenge what a challenge what a challenge the tool is not the tool is not the tool is not succeeds. Tracing succeeds. Tracing —this is the place where you —this is the place where you —this is the place where you you can see that he you can see that he you can see that he trying to do trying to do trying to do something he doesn't like something he doesn't like something he doesn't like permission granted. Um, and permission granted. Um, and permission granted. Um, and this is the type that this is the type that this is the type that is in span. Uh, is in span. Uh, is in span. Uh, The span type tells you, The span type tells you, The span type tells you, What type of step is this? What type of step is this? What type of step is this? So, this is a tool So, this is a tool So, this is a tool agent, agent, call agent, agent, call agent, agent, call instrument, LLM, etc. instrument, LLM, etc. instrument, LLM, etc. Um, and in the challenge Um, and in the challenge Um, and in the challenge tool and web tool and web search etc. you can search etc. you can search etc. you can to see that our to see that our to see that our input data input data input data actually JSON. Um, and this actually JSON. Um, and this actually JSON. Um, and this will be our first will be our first will be our first assessment, uh, because assessment, uh, because assessment, uh, because this gives us some this gives us some this gives us some structured data. structured data. structured data. Um, you too Um, you too Um, you too you get things like this, you get things like this, you get things like this, as time and quantity as time and quantity as time and quantity tokens. Um, and, uh, tokens. Um, and, uh, tokens. Um, and, uh, often your agent often your agent often your agent will record, will record, will record, How much does it cost? Ahem, How much does it cost? Ahem, when we are a little when we are a little we write eval, these are fields, we write eval, these are fields, we write eval, these are fields, which we are going to which we are going to which we are going to extract and evaluate.

extract and evaluate. So, uh, you've done it. So, uh, you've done it. one, you won one, one, you won one, one, you won one, you took one test you took one test you took one test for now. Uh, this data for now. Uh, this data for now. Uh, this data not enough to not enough to not enough to to do some to do some to do some conclusions. So now I conclusions. So now I conclusions. So now I I want you to I want you to I want you to ran npm run batch uh, ran npm run batch uh, ran npm run batch uh, who will complete 15 who will complete 15 who will complete 15 requests in a row. Uh, requests in a row. Uh, requests in a row. Uh, this will take some time. this will take some time. this will take some time. So I will speak, So I will speak, So I will speak, while you're doing it. Uh, while you're doing it. Uh, while you're doing it. Uh, it starts on it starts on it starts on five requests at a time. five requests at a time. five requests at a time. Optimistically, this Optimistically, this Optimistically, this takes three or four takes three or four takes three or four minutes on Wi-Fi. Uh, minutes on Wi-Fi. Uh, minutes on Wi-Fi. Uh, so start it now. so start it now. so start it now. Uh, if from some Uh, if from some Uh, if from some reasons it won't work, reasons it won't work, reasons it won't work, Don't worry. Don't worry. Don't worry. Initial project, Initial project, Initial project, which you downloaded which you downloaded which you downloaded previously, called previously, called previously, called Phoenix 101 golden. Uh, and in Phoenix 101 golden. Uh, and in Phoenix 101 golden. Uh, and in it is my 16 it is my 16 it is my 16 launches that I launches that I launches that I did earlier on did earlier on did earlier on reliable Wi-Fi. So you reliable Wi-Fi. So you reliable Wi-Fi. So you can you can you can you use them use them use them instead, if instead, if instead, if can't start can't start can't start your package project. your package project. your package project. Uh, here are a few Uh, here are a few Uh, here are a few examples of queries. THERE ARE- examples of queries. THERE ARE- Well, there are obvious ones, such as Well, there are obvious ones, such as Well, there are obvious ones, such as Apple, Nvidia and Amazon. AND Apple, Nvidia and Amazon. AND Apple, Nvidia and Amazon. AND then there are some extreme ones then there are some extreme ones then there are some extreme ones cases I added cases I added cases I added there, so that everything there, so that everything there, so that everything failed. Uh, so failed. Uh, so , one asks , one asks , one asks Apple vs. Apple Apple vs. Apple Apple vs. Apple Microsoft, which is more difficult, Microsoft, which is more difficult, Microsoft, which is more difficult, than just compare than just compare than just compare one company. Uh, one company. Uh, one company. Uh, one concerns Rivian, one concerns Rivian, one concerns Rivian, which is not public which is not public which is not public by the company. So the data by the company. So the data by the company. So the data much less than much less than much less than about, uh, public about, uh, public about, uh, public companies. Uh, Coca-Cola— companies. Uh, Coca-Cola— this is a completely different type this is a completely different type this is a completely different type companies than companies than companies than technological technological technological company. Uh, and ZZQ company. Uh, and ZZQ company. Uh, and ZZQ not at all not at all not at all a real ticker. I a real ticker. I a real ticker. I just invented it. THERE ARE- just invented it. THERE ARE- uh, and this is in order to uh, and this is in order to uh, and this is in order to to see what he does to see what he does to see what he does agent when he sees agent when he sees agent when he sees ticker symbol that ticker symbol that ticker symbol that he doesn't recognize. Or

he doesn't recognize. Or he doesn't recognize. Or Is he inventing a company? Is he inventing a company? Is he inventing a company? Does he guess? Does he guess? Does he guess? What company do you own? What company do you own? What company do you own? mean, does it say, mean, does it say, mean, does it say, what I hope that what I hope that what I hope that ticker does not exist? Uh, ticker does not exist? Uh, ticker does not exist? Uh, one of the requests one of the requests one of the requests regarding, uh, Tesla, regarding, uh, Tesla, regarding, uh, Tesla, but the main focus but the main focus but the main focus is devoted to whether is devoted to whether is devoted to whether is it worth buying it is it worth buying it is it worth buying it now? Uh, because now? Uh, because now? Uh, because that's how real people are that's how real people are that's how real people are send requests send requests send requests agents. They do agents. They do agents. They do This is slang. Uh, and they This is slang. Uh, and they This is slang. Uh, and they not often exactly not often exactly not often exactly clarify what exactly clarify what exactly clarify what exactly they wanted. Uh, so they wanted. Uh, so they wanted. Uh, so these are all requests that these are all requests that these are all requests that look look look realistic. Hmm, so what? realistic. Hmm, so what? realistic. Hmm, so what? you need to do, you need to do, you need to do, this, as I said, this, as I said, this, as I said, write down what it means write down what it means write down what it means "okay" before "okay" before "okay" before to measure. You are not to measure. You are not to measure. You are not can you say that can you say that can you say that your agent is not working, your agent is not working, your agent is not working, if you haven't specified, if you haven't specified, if you haven't specified, what does " mean what does " mean work". So, for work". So, for work". So, for our analyst, as our analyst, as our analyst, as looks good looks good looks good report? Um, he should report? Um, he should report? Um, he should use use use real current data. real current data. real current data. He probably has He probably has He probably has to be about the right thing to be about the right thing to be about the right thing company. He must company. He must company. He must to adhere to, to adhere to, to adhere to, what is he really what is he really what is he really discovered. Um, and for mine too discovered. Um, and for mine too discovered. Um, and for mine too goals, I want him to goals, I want him to goals, I want him to told us that we told us that we told us that we should be done. I want, should be done. I want, should be done. I want, so that he speaks " so that he speaks " to buy, to hold to buy, to hold to buy, to hold or sell these or sell these or sell these shares". I'm not just shares". I'm not just shares". I'm not just I want him to give it to me.

I want him to give it to me. I want him to give it to me. analysis. Hmm, like this analysis. Hmm, like this analysis. Hmm, like this the requirements are sound the requirements are sound the requirements are sound obviously, as soon as obviously, as soon as obviously, as soon as you will write them down, but before you will write them down, but before you will write them down, but before they can do this they can do this they can do this to be simple to be simple to be simple vague, or vague, or vague, or different people in your different people in your different people in your teams can have teams can have teams can have different ideas about it different ideas about it what should I do what should I do what should I do good financial good financial good financial analyst. Um, that's why they analyst. Um, that's why they analyst. Um, that's why they record converts record converts record converts vague disappointment vague disappointment vague disappointment on specific things, on specific things, on specific things, which you can which you can which you can check, and we check, and we check, and we we are going we are going we are going convert each of convert each of convert each of these requirements for assessment these requirements for assessment these requirements for assessment today. Definition today. Definition today. Definition "good" is not "good" is not "good" is not exclusively, um, exclusively, um, exclusively, um, engineering work. engineering work. engineering work. If you determine, If you determine, If you determine, what is good for what is good for what is good for your product, you your product, you your product, you need to talk need to talk need to talk with non-technical with non-technical with non-technical interested interested interested parties in your parties in your parties in your organizations, organizations, organizations, because they are because they are because they are experts in experts in experts in subject area, subject area, subject area, who will know how who will know how who will know how looks good. IN looks good. IN looks good. IN you, as an engineer, you, as an engineer, you, as an engineer, will be specific will be specific will be specific assumptions about assumptions about assumptions about what, uh, assumptions and what, uh, assumptions and what, uh, assumptions and prejudice against prejudice against prejudice against what he does, uh, what he does, uh, what he does, uh, a good agent and what a good agent and what a good agent and what gives a good gives a good gives a good result, and you don't result, and you don't result, and you don't always always always agree with agree with agree with experts in experts in experts in subject area. subject area. subject area. So, uh, people who So, uh, people who So, uh, people who work with clients, work with clients, work with clients, must be must be must be present when you present when you present when you determine how determine how determine how your agent looks like. THERE ARE- your agent looks like. THERE ARE- uh, that's also great, uh uh, that's also great, uh if you don't have if you don't have if you don't have users, you users, you users, you you can make them up. THERE ARE- you can make them up. THERE ARE- Well, you can.

Well, you can. Well, you can. to use LLM for to use LLM for to use LLM for generation of multitude generation of multitude generation of multitude test queries and test queries and test queries and intentionally them intentionally them intentionally them vary. So vary. So vary. So research research research financial financial financial Tesla's performance is Tesla's performance is Tesla's performance is one of the ways one of the ways one of the ways to ask what to ask what to ask what happens with happens with happens with Tesla shares, and yours Tesla shares, and yours Tesla shares, and yours current situation in current situation in current situation in Tesla is third. Uh, then Tesla is third. Uh, then Tesla is third. Uh, then you can ask you can ask you can ask person to view person to view person to view list because list because list because queries generated queries generated queries generated LLM is useful LLM is useful LLM is useful by sowing method by sowing method by sowing method your initials your initials your initials data. They are not data. They are not data. They are not ideal, mainly ideal, mainly ideal, mainly way because they way because they way because they relatively relatively relatively conservative. They conservative. They conservative. They don't come in like that don't come in like that don't come in like that far away, how far away far away, how far away far away, how far away usually do usually do usually do human users. human users. human users. Human users Human users Human users they say: "Tell me they say: "Tell me what's wrong with what's wrong with what's wrong with to my health to my health to my health child", instead of child", instead of to ask you about to ask you about to ask you about financial financial financial analytics, and your analytics, and your analytics, and your the agent must have up to the agent must have up to the agent must have up to this is some kind of this is some kind of this is some kind of relation. But relation. But relation. But synthetic data is synthetic data is synthetic data is great way great way great way start when you start when you start when you have already started, and you have have already started, and you have have already started, and you have real users real users real users with real traffic. with real traffic. with real traffic. Real traffic is Real traffic is Real traffic is gold standard for gold standard for gold standard for agent testing. You agent testing. You agent testing. You you want as much as possible you want as much as possible you want as much as possible turn it on faster turn it on faster turn it on faster to your dataset to your dataset . One more question, . One more question, . One more question, before we before we before we Let's consider the data that, Let's consider the data that, Let's consider the data that, I hope now I hope now I hope now coming to your coming to your coming to your laptops. I laptops. I laptops. I reviewed the tracing reviewed the tracing reviewed the tracing manually, because you have to manually, because you have to manually, because you have to understand what it is understand what it is understand what it is does, but you also does, but you also does, but you also you can, as I said you can, as I said , entrust it to your , entrust it to your , entrust it to your to the encoding agent. IN to the encoding agent. IN to the encoding agent. IN Phoenix is ​​a very good CLI.

Phoenix is ​​a very good CLI. Phoenix is ​​a very good CLI. And if you run And if you run And if you run setup command in your setup command in your setup command in your project, it will provide project, it will provide project, it will provide to your agent to your agent to your agent coding, he coding, he coding, he will make your will make your will make your encoding agent encoding agent encoding agent add trace to add trace to add trace to your program. And this your program. And this your program. And this will work as long as will work as long as will work as long as your program does not your program does not your program does not will be installed will be installed will be installed tracing, and you don't tracing, and you don't tracing, and you don't will have to will have to will have to to guess how to guess how to guess how add register command add register command add register command and all that other stuff. It and all that other stuff. It and all that other stuff. It will work every will work every will work every in language. Um, that's true. in language. Um, that's true. in language. Um, that's true. steeply. So, step steeply. So, step steeply. So, step the second is reading the second is reading the second is reading your data. Who has your data. Who has your data. Who has some data on this some data on this some data on this moment? Tell me. moment? Tell me. moment? Tell me. hands. One person. hands. One person. hands. One person. Anyone else? Oh yes. Steeply. Anyone else? Oh yes. Steeply. Anyone else? Oh yes. Steeply. OK. Several people. OK. Several people. OK. Several people. Fantasy. Um, you Fantasy. Um, you Fantasy. Um, you must read must read must read your traces, first your traces, first your traces, first than to write one than to write one than to write one assessment. Um, read. assessment. Um, read. assessment. Um, read. a dozen or more of a dozen or more of a dozen or more of them from the beginning to them from the beginning to them from the beginning to end. Um, what was on end. Um, what was on end. Um, what was on entrance? What was on entrance? What was on entrance? What was on exit? Um, what exactly? exit? Um, what exactly? exit? Um, what exactly? Yes? This sounds like Yes? This sounds like Yes? This sounds like manual and manual and manual and old-fashioned process, old-fashioned process, old-fashioned process, but this is incredible but this is incredible but this is incredible a valuable thing that you a valuable thing that you a valuable thing that you you can do. Um, you can do. Um, you can do. Um, when you create when you create when you create agent, um, you agent, um, you agent, um, you required manually required manually required manually check that check that check that happens to happens to happens to find things you don't find things you don't find things you don't expected–unknown expected–unknown expected–unknown unknown, which you do not unknown, which you do not unknown, which you do not expected. Um, expected. Um, expected. Um, Anthropic team Anthropic team Anthropic team sits down regularly and sits down regularly and sits down regularly and reads transcripts reads transcripts reads transcripts manually evaluated, um, and manually evaluated, um, and manually evaluated, um, and created a pile created a pile created a pile tools tools tools internally, so that internally, so that internally, so that It's easy to do. Um, It's easy to do. Um, It's easy to do. Um, if you if you if you automate, first automate, first automate, first than you will understand how than you will understand how than you will understand how your agent is crashing, you your agent is crashing, you your agent is crashing, you eventually eventually eventually you will measure what you will measure what you will measure what easy to measure, easy to measure, easy to measure, instead of that instead of that instead of that It's actually important. Ahem It's actually important. Ahem , therefore, you should , therefore, you should , therefore, you should read the tracks, and we read the tracks, and we read the tracks, and we we are going to do this we are going to do this we are going to do this make. But reading make. But reading make. But reading no trails no trails no trails scalable.

scalable. scalable. Read 16 tracks yourself Read 16 tracks yourself –it is absolutely –it is absolutely –it is absolutely normally. Um, but 1600 normally. Um, but 1600 normally. Um, but 1600 the tracks, uh, will give you the tracks, uh, will give you the tracks, uh, will give you a bunch of really great ones a bunch of really great ones a bunch of really great ones data, and this is also data, and this is also data, and this is also will take a week of your will take a week of your will take a week of your life. Um, but life. Um, but life. Um, but it turns out that it turns out that it turns out that encoding agents– encoding agents– aces in reading routes. Em aces in reading routes. Em , so you can , so you can , so you can to direct your to direct your to direct your encoding agent on encoding agent on encoding agent on your tracks and make your tracks and make your tracks and make to look for him strangely to look for him strangely to look for him strangely things in your tracks. things in your tracks. things in your tracks. Make them write Make them write Make them write code. Um, but they don't code. Um, but they don't code. Um, but they don't will always find what will always find what will always find what What did you expect? Ago What did you expect? Ago What did you expect? Ago manual check as manual check as manual check as sampling method– sampling method– always a great idea. always a great idea. So, um, if you So, um, if you use if use if use if you get, uh, you get, uh, you get, uh, agent who does agent who does agent who does something for you, for you something for you, for you something for you, for you need to install need to install need to install Phoenix plugin or Phoenix plugin or Phoenix plugin or Phoenix skill, depending Phoenix skill, depending Phoenix skill, depending depending on which agent depending on which agent depending on which agent coding you coding you coding you use. It use. It use. It was the last step in was the last step in was the last step in readme file for readme file for readme file for settings if settings if settings if You still have it. Um, You still have it. Um, You still have it. Um, maybe you already have it maybe you already have it maybe you already have it there are. Um, for the cloud there are. Um, for the cloud there are. Um, for the cloud code we have a plugin, code we have a plugin, code we have a plugin, which consists of which consists of which consists of two teams. Uh, two teams. Uh, two teams. Uh, there are also plugins for there are also plugins for there are also plugins for cursor and codecs. THERE ARE- cursor and codecs. THERE ARE- Well, for any Well, for any Well, for any another agent another agent another agent encoding is a file encoding is a file encoding is a file skills. Um, plugin skills. Um, plugin skills. Um, plugin your agent connects your agent connects your agent connects encoding to your encoding to your encoding to your Phoenix. Uh, and he already Phoenix. Uh, and he already Phoenix. Uh, and he already points to localhost 606, points to localhost 606, points to localhost 606, so nothing so nothing so nothing not to configure not to configure not to configure necessary. Um, and myself necessary. Um, and myself necessary. Um, and myself Phoenix will provide your Phoenix will provide your Phoenix will provide your agent skill under agent skill under agent skill under called Phoenix error analysis called Phoenix error analysis called Phoenix error analysis skill with instructions skill with instructions skill with instructions regarding the correct regarding the correct regarding the correct error analysis. Um, error analysis. Um, installation has installation has take about 10 take about 10 take about 10 seconds. Uh, and I don't seconds. Uh, and I don't seconds. Uh, and I don't I'm going to wait until I'm going to wait until I'm going to wait until you will install it.

you will install it. you will install it. Um, now open it. Um, now open it. Um, now open it. your encoding agent your encoding agent your encoding agent and ask him and ask him and ask him perform analysis perform analysis perform analysis errors. Uh, if he errors. Uh, if he errors. Uh, if he You have it. Who has it, has it? You have it. Who has it, has it? You have it. Who has it, has it? who is installed who is installed who is installed skill with skill with skill with last time? last time? last time? Fantastically. Me again Fantastically. Me again Fantastically. Me again I will stab you. Or I will stab you. Or I will stab you. Or can you please can you please can you please your agent your agent your agent coding something coding something coding something make? make? make? Okay, great. Um, Okay, great. Um, Okay, great. Um, The hint is on the slide. The hint is on the slide. The hint is on the slide. Tell him where Tell him where Tell him where is located in Phoenix. is located in Phoenix. is located in Phoenix. Tell him Tell him Tell him read trace read trace . Come up with . Come up with . Come up with categories of failures. Uh, categories of failures. Uh, categories of failures. Uh, count them and count them and count them and write down each write down each write down each category back to its category back to its category back to its tracing in Phoenix as tracing in Phoenix as tracing in Phoenix as annotation. Annotation- annotation. Annotation- it's just a line it's just a line it's just a line text that a person text that a person text that a person can read, can read, can read, added to added to added to tracing. Um, this is tracing. Um, this is tracing. Um, this is method of addition method of addition method of addition additional additional additional information to information to information to tracing so that you tracing so that you tracing so that you or your agent could or your agent could or your agent could understand that understand that understand that is happening. Um, yours. is happening. Um, yours. is happening. Um, yours. the plugin will ask the plugin will ask the plugin will ask permission to permission to permission to using using using Phoenix tools. Phoenix tools. Phoenix tools. So you must So you must So you must to say "yes". Um, to say "yes". Um, to say "yes". Um, when I launched it, when I launched it, when I launched it, it took about 9 it took about 9 it took about 9 minutes. Um, that's why I minutes. Um, that's why I minutes. Um, that's why I I'm not going yet. I'm not going yet. I'm not going yet. to cling to you. Um, uh, to cling to you. Um, uh, to cling to you. Um, uh, also, if he also, if he also, if he will ask to write down will ask to write down will ask to write down notes in notes in notes in repository, uh, this repository, uh, this repository, uh, this normally. Simply normally. Simply normally. Simply let him let him let him to burn things to disk to burn things to disk . Now let's do it again.

. Now let's do it again. . Now let's do it again. let's look at Phoenix. Em let's look at Phoenix. Em , I am going to be very , I am going to be very , I am going to be very dangerously dangerously try try remember yours remember yours remember yours syntax for searching. syntax for searching. Ha. Okay, so that's all. Ha. Okay, so that's all. my startup, um, this my startup, um, this my startup, um, this financial reports. Um, financial reports. Um, financial reports. Um, we can press we can press we can press on some of them. on some of them. on some of them. Let's see if Let's see if Let's see if can we find can we find can we find interesting. Um, ZZQ, what am I interesting. Um, ZZQ, what am I interesting. Um, ZZQ, what am I brought out for ZZQ? ZZZQ is not brought out for ZZQ? ZZZQ is not brought out for ZZQ? ZZZQ is not valid or active valid or active valid or active financial ticker. financial ticker. financial ticker. Hooray. The agent did what Hooray. The agent did what Hooray. The agent did what what we expected. Um, what we expected. Um, what we expected. Um, How about Coca-Cola? On- How about Coca-Cola? On- first of all first of all first of all starts with starts with starts with apologies. She works.

apologies. She works. apologies. She works. in limited mode and in limited mode and in limited mode and cannot record cannot record cannot record files directly. files directly. files directly. We will have to do this. We will have to do this. We will have to do this. exactly correct. Um, ah exactly correct. Um, ah exactly correct. Um, ah then it comes to then it comes to then it comes to conclusion. Coca-Cola presents conclusion. Coca-Cola presents conclusion. Coca-Cola presents a high-quality a high-quality a high-quality investment with low investment with low investment with low dividend level. Em dividend level. Em , KO—this is great , KO—this is great , KO—this is great long-term long-term long-term receiving tool receiving tool receiving tool income for income for income for investors who do not investors who do not investors who do not risk-averse. It risk-averse. It risk-averse. It It will be fun. It will be fun. It will be fun. Raise your hands, who Raise your hands, who Raise your hands, who thinks this is a report, on thinks this is a report, on thinks this is a report, on who can who can who can to work? Okay, I to work? Okay, I to work? Okay, I already prepared you. already prepared you. already prepared you. Um, it doesn't say Um, it doesn't say Um, it doesn't say buy, sell buy, sell buy, sell or keep So there or keep So there or keep So there it's just said that it is it's just said that it is it's just said that it is a good investment. a good investment. a good investment. It doesn't say that you It doesn't say that you It doesn't say that you should be done. Um, I should be done. Um, I should be done. Um, I I am going to install I am going to install I am going to install standard of action for these standard of action for these standard of action for these reports, which has reports, which has reports, which has tell me whether tell me whether tell me whether should I buy it? should I buy it? should I buy it? sell or hold sell or hold sell or hold shares. I am not going. shares. I am not going. shares. I am not going. leave it leave it leave it ambiguous. Um, ambiguous. Um, ambiguous. Um, a whole bunch of them a whole bunch of them a whole bunch of them do different things in do different things in do different things in this set. Um, how? this set. Um, how? this set. Um, how? you see, this one had you see, this one had you see, this one had same problem. He same problem. He same problem. He tried to write down tried to write down tried to write down on the disk, and that was it, and on the disk, and that was it, and on the disk, and that was it, and it was, and it wasn't it was, and it wasn't it was, and it wasn't succeeded. Hmm, uh, succeeded. Hmm, uh, succeeded. Hmm, uh, Rivian tracking, I Rivian tracking, I Rivian tracking, I noted that Rivian was noted that Rivian was noted that Rivian was interesting. Here it is. Uh, interesting. Here it is. Uh, interesting. Here it is. Uh, for some reason it for some reason it for some reason it full of emojis. Uh, full of emojis. Uh, full of emojis. Uh, it opens with it opens with it opens with question: "Would you like to?"

question: "Would you like to?" question: "Would you like to?" Yes. Would you like to Yes. Would you like to Yes. Would you like to create a report file? create a report file? create a report file? We've seen this before, We've seen this before, We've seen this before, when I launched it. when I launched it. when I launched it. Uh, it's asking me. Uh, it's asking me. Uh, it's asking me. to do something, and it to do something, and it to do something, and it cannot ask. I cannot ask. I cannot ask. I told him that told him that told him that told him that he told him that he told him that he this is not allowed, and this is not allowed, and this is not allowed, and it still asks it still asks it still asks to do something. Uh, everyone to do something. Uh, everyone to do something. Uh, everyone these things are, uh, different these things are, uh, different these things are, uh, different ways to fail. ways to fail. ways to fail. Many of them are Many of them are Many of them are failures. Many of them failures. Many of them failures. Many of them It's perfectly normal. Uh, It's perfectly normal. Uh, but then, uh, there's Tesla. Tesla is particularly exciting . Uh, Tesla in particular—it . Uh, Tesla in particular—it . Uh, Tesla in particular—it what is worth buying what is worth buying what is worth buying now. So here, uh, now. So here, uh, now. So here, uh, it is written that it is written that it is written that analyst consensus analyst consensus -buy. Uh, -buy. Uh, -buy. Uh, it is written that 50% say it is written that 50% say it is written that 50% say "buy", 52.6%—" "buy", 52.6%—" keep" and 13.1%—" keep" and 13.1%—" to sell", which in total to sell", which in total to sell", which in total is 116% is 116% analysts who analysts who analysts who They say something. And it is They say something. And it is They say something. And it is it is not clear what it is it is not clear what it is it is not clear what it is mathematics was mathematics was mathematics was absolutely absolutely absolutely incorrect. He incorrect. He incorrect. He performed two different web performed two different web -search. One where he is -search. One where he is -search. One where he is found that 50% of people found that 50% of people found that 50% of people they say something, another, where they say something, another, where they say something, another, where he found that 13% of people he found that 13% of people he found that 13% of people They say something. He took They say something. He took They say something. He took these two web searches and these two web searches and these two web searches and combined them together, combined them together, combined them together, although they were from although they were from although they were from different data sets, different data sets, different data sets, and got 116%. But he and got 116%. But he and got 116%. But he confidently recorded that confidently recorded that confidently recorded that 116% of analysts something 116% of analysts something 116% of analysts something they say. This is the one.

they say. This is the one. they say. This is the one. type of agency type of agency type of agency failure, what are you failure, what are you failure, what are you You see. Uh, this is You see. Uh, this is You see. Uh, this is sure mistake, sure mistake, sure mistake, done, uh, bad done, uh, bad done, uh, bad mathematics. Um, if mathematics. Um, if mathematics. Um, if you will visit these websites you will visit these websites -searches, you can -searches, you can -searches, you can see specific see specific see specific the places they come from the places they come from the places they come from received the data, and how received the data, and how received the data, and how they, uh, combined them they, uh, combined them they, uh, combined them together. Um, so when together. Um, so when together. Um, so when you find failure, you find failure, you find failure, you should ask, you should ask, you should ask, why did she fail? why did she fail? why did she fail? Incorrect Incorrect Incorrect the answer is the answer is the answer is symptom. This is not symptom. This is not symptom. This is not reason. Um, for reason. Um, for reason. Um, for with the help of traces you with the help of traces you with the help of traces you you can see what you can see what you can see what was actual was actual was actual respond. Or respond. Or respond. Or they got bad ones they got bad ones they got bad ones search results? It search results? It search results? It there would be a search failure there would be a search failure . This is not what happened. . This is not what happened. . This is not what happened. here. Every result here. Every result here. Every result search was search was search was normal. Uh, or normal. Uh, or normal. Uh, or were in it were in it were in it correct data, but correct data, but correct data, but wrong wrong wrong conclusion? Then this conclusion? Then this conclusion? Then this error in reasoning. error in reasoning. error in reasoning. This, uh, what's here This, uh, what's here This, uh, what's here happened. Isn't that right? happened. Isn't that right? happened. Isn't that right? He saw two different He saw two different He saw two different data fragments and data fragments and data fragments and I was wrong. I was wrong. I was wrong. about them. Um, sometimes he about them. Um, sometimes he about them. Um, sometimes he completely fictional completely fictional completely fictional numbers. This is a hallucination. numbers. This is a hallucination. . Um, and sometimes he . Um, and sometimes he . Um, and sometimes he responds to responds to responds to a question that is not that a question that is not that a question that is not that the question you the question you the question you put. Therefore, put. Therefore, put. Therefore, volume problem. volume problem. volume problem. Every reason has Every reason has Every reason has different solution. Um, but different solution. Um, but different solution. Um, but if you don't know if you don't know if you don't know reasons you don't know reasons you don't know reasons you don't know decision. That's why decision. That's why decision. That's why reading traces reading traces reading traces useful. Um, there really is.

useful. Um, there really is. useful. Um, there really is. great way great way great way to do this, namely: to do this, namely: to do this, namely: let's go back let's go back let's go back to the slides. Um, to the slides. Um, to the slides. Um, first you should first you should first you should write notes, and write notes, and write notes, and then categories. So then categories. So then categories. So review all your review all your review all your traces and just traces and just traces and just write in human write in human write in human in language what you see in language what you see . This is bad because it . This is bad because it . This is bad because it does this. This is good, does this. This is good, does this. This is good, because it does that. Um, this is because it does that. Um, this is because it does that. Um, this is is called is called is called open coding open coding . Um, another term, . Um, another term, . Um, another term, borrowed from borrowed from borrowed from machine learning. machine learning. machine learning. You're just saying, You're just saying, You're just saying, What did you see? What did you see? What did you see? After you do this After you do this After you do this did, um, you can did, um, you can did, um, you can to do what to do what to do what called axial called axial called axial coding, where you coding, where you coding, where you viewing all viewing all viewing all notes that you notes that you notes that you wrote to themselves, and wrote to themselves, and wrote to themselves, and group them by group them by group them by categories. As categories. As categories. As engineer, very, very engineer, very, very engineer, very, very seductively seductively seductively try try try to guess in advance, to guess in advance, to guess in advance, what will be what will be what will be categories, come up with categories, come up with categories, come up with categories and just categories and just categories and just to place things in to place things in to place things in categories from the very categories from the very categories from the very beginning; uh, because beginning; uh, because beginning; uh, because it's less work. But in it's less work. But in it's less work. But in it will be worse for you, it will be worse for you, it will be worse for you, because you are not because you are not because you are not you will know you will know you will know in advance, what in advance, what in advance, what will be your nodes will be your nodes will be your nodes refusals. Er, that is refusals. Er, that is refusals. Er, that is categories that you categories that you categories that you are you going to choose, are you going to choose, are you going to choose, will be or too will be or too will be or too narrow, or too narrow, or too narrow, or too wide, or simply wide, or simply wide, or simply not the things that not the things that not the things that actually went not actually went not actually went not Yes. Uh, while we Yes. Uh, while we Yes. Uh, while we Let's talk about this, I Let's talk about this, I Let's talk about this, I talked about how talked about how talked about how you are composing, uh, you are composing, uh, you are composing, uh, Kod valves, LLM valves Kod valves, LLM valves Kod valves, LLM valves for evaluation and for evaluation and for evaluation and human assessments. Uh, human assessments. Uh, human assessments. Uh, this is, uh, a model, this is, uh, a model, this is, uh, a model, banned in the world banned in the world banned in the world security, which security, which security, which called a model called a model called a model Swiss cheese. THERE ARE Swiss cheese. THERE ARE -e, every level -e, every level -e, every level the protection you have, the protection you have, the protection you have, will have holes. Each will have holes. Each will have holes. Each your rating will have your rating will have your rating will have blind spot. What are you

blind spot. What are you blind spot. What are you want to do this, uh- want to do this, uh- e, to compose as much as possible e, to compose as much as possible e, to compose as much as possible more marks one by one more marks one by one more marks one by one one so that mistakes one so that mistakes one so that mistakes it was very difficult it was very difficult it was very difficult to go through all of to go through all of to go through all of them. Um, because everyone has them. Um, because everyone has them. Um, because everyone has There are holes in them, but these There are holes in them, but these There are holes in them, but these holes in more than one holes in more than one holes in more than one place. So, if you place. So, if you place. So, if you you can, uh, if you have you can, uh, if you have you can, uh, if you have enough ratings, enough ratings, enough ratings, if you have enough if you have enough if you have enough different types different types different types checks, they checks, they checks, they will be will be will be accumulate alone accumulate alone accumulate alone on one and will be on one and will be on one and will be much more reliable much more reliable . Uh, code evaluations . Uh, code evaluations . Uh, code evaluations can detect can detect can detect problems problems problems formatting, but formatting, but formatting, but they will miss they will miss they will miss problems with problems with problems with meaning. Judges LM meaning. Judges LM meaning. Judges LM will discover will discover will discover gaps in gaps in gaps in considerations, but they considerations, but they considerations, but they will miss some will miss some will miss some barely noticeable barely noticeable barely noticeable hallucinations. People hallucinations. People hallucinations. People will often be found will often be found will often be found barely noticeable things, barely noticeable things, barely noticeable things, but they can't but they can't but they can't view each view each view each a separate trajectory. a separate trajectory. a separate trajectory. So no single one So no single one So no single one the method, uh, won't be able to the method, uh, won't be able to the method, uh, won't be able to discover everything. Uh, so discover everything. Uh, so discover everything. Uh, so today we today we today we we create several we create several we create several small estimates, and small estimates, and small estimates, and not one giant not one giant not one giant super rating, which super rating, which super rating, which is also another one is also another one is also another one failure mode. When failure mode. When failure mode. When people for the first time people for the first time people for the first time are faced with are faced with are faced with estimates, they estimates, they estimates, they think: "I'll think about it think: "I'll think about it think: "I'll think about it about everything that can go about everything that can go about everything that can go not so, and I'll squeeze it in not so, and I'll squeeze it in not so, and I'll squeeze it in one big LLM as one big LLM as one big LLM as judge, and he will find judge, and he will find judge, and he will find all". And this, uh, very all". And this, uh, very all". And this, uh, very unreliable and very unreliable and very unreliable and very a complicated way not a complicated way not a complicated way not solve the problem.

solve the problem. So, oops. Uh, agent So, oops. Uh, agent can cause can cause can cause agent, he agent, he agent, he going to do going to do going to do a whole bunch of things. Here a whole bunch of things. Here a whole bunch of things. Here what does it look like what does it look like what does it look like result of work result of work result of work agent. I already agent. I already agent. I already showed you in showed you in showed you in Phoenix. Uh, we already Phoenix. Uh, we already Phoenix. Uh, we already showed you different showed you different showed you different things that he things that he things that he came up with. Uh, in this came up with. Uh, in this came up with. Uh, in this case, uh, my case, uh, my case, uh, my encoding agent encoding agent encoding agent looked through all my looked through all my looked through all my tracing for me. tracing for me. He found, uh, like, uh- e, open coding e, open coding , and AIA encoding, e- , and AIA encoding, e- oh, and he found it oh, and he found it oh, and he found it category, uh, which one category, uh, which one category, uh, which one called a leak called a leak called a leak data. Uh, leaked data. Uh, leaked data. Uh, leaked data is how it data is how it data is how it decided to name that one decided to name that one decided to name that one thing when he thing when he thing when he constantly asks us constantly asks us constantly asks us give him permission give him permission give him permission to do things that you to do things that you to do things that you he doesn't have permission.

he doesn't have permission. he doesn't have permission. Uh, I mean, uh, it's a bunch Uh, I mean, uh, it's a bunch Uh, I mean, uh, it's a bunch times, so that this times, so that this times, so that this specific thing specific thing specific thing happened. Uh, there is also happened. Uh, there is also happened. Uh, there is also a bunch of other things, he a bunch of other things, he a bunch of other things, he also found also found also found hallucinations. He hallucinations. He hallucinations. He found, uh, errors found, uh, errors found, uh, errors reasoning, as in reasoning, as in reasoning, as in Tesla, where he had Tesla, where he had Tesla, where he had too many percent, too many percent, too many percent, he added. Um, if you he added. Um, if you he added. Um, if you processed this data themselves processed this data themselves processed this data themselves and your coding agent and your coding agent and your coding agent created his own created his own created his own analysis, you will get analysis, you will get analysis, you will get another analysis and another another analysis and another another analysis and another set of marks. Ago set of marks. Ago set of marks. Ago that all our agents that all our agents that all our agents different. Who already different. Who already different. Who already entrusted to his entrusted to his entrusted to his to the coding agent to the coding agent to the coding agent complete the analysis? complete the analysis? Is anyone still working? Is anyone still working? Good. Well, we Good. Well, we Good. Well, we let's continue. let's continue. let's continue. I hope you I hope you I hope you get some, but get some, but get some, but if not, you can if not, you can if not, you can simply simply simply use my use my use my initial data. As initial data. As initial data. As only you will have only you will have only you will have data, you can data, you can data, you can count categories count categories . The most common failure . The most common failure . The most common failure will be where you are will be where you are will be where you are looking for first looking for first , but the most common , but the most common , but the most common failure is not necessarily failure is not necessarily failure is not necessarily the first one that you the first one that you the first one that you are you going to fix are you going to fix . What you need . What you need . What you need to do, to turn to do, to turn to do, to turn attention to frequency and attention to frequency and attention to frequency and weight. So, if you weight. So, if you weight. So, if you Do you know that 40% Do you know that 40% cases he cases he cases he hallucinates, this hallucinates, this hallucinates, this problem. If in 1% problem. If in 1% cases he gives cases he gives cases he gives to your user to your user to your user instructions for instructions for instructions for creating a bomb, you creating a bomb, you creating a bomb, you immediately this immediately this immediately this correct, even correct, even correct, even if this happens if this happens if this happens only in 1% of cases, only in 1% of cases, only in 1% of cases, because it is absolutely because it is absolutely because it is absolutely not what it should be not what it should be not what it should be to deal with finance to deal with finance to deal with finance analyst. Uh, you analyst. Uh, you analyst. Uh, you multiply the frequency by multiply the frequency by multiply the frequency by seriousness, so that seriousness, so that seriousness, so that get yours get yours get yours priority. Good, priority. Good, priority. Good, I hope yours I hope yours I hope yours agents will catch up agents will catch up agents will catch up us, but I will continue.

us, but I will continue. us, but I will continue. Step three is Step three is Step three is Kodi valves. Valves Kodi valves. Valves Kodi valves. Valves Kodi is the simplest. There Kodi is the simplest. There Kodi is the simplest. There no model, no no model, no no model, no API call, there is only API call, there is only API call, there is only TypeScript. Uh, TypeScript. Uh, TypeScript. Uh, the simplest useful the simplest useful the simplest useful the assessment I could the assessment I could the assessment I could to think of this or that to think of this or that to think of this or that really in the report really in the report really in the report the action is mentioned, about the action is mentioned, about the action is mentioned, about which we asked. which we asked. which we asked. Our agent must Our agent must Our agent must analyze analyze analyze a specific company. a specific company. a specific company. If he does this, If he does this, If he does this, he must remember he must remember he must remember ticker of the company you ticker of the company you ticker of the company you he analyzes. Uh, he analyzes. Uh, he analyzes. Uh, if I ask about Tesla if I ask about Tesla , and there they will write about , and there they will write about , and there they will write about Microsoft, what they really are Microsoft, what they really are Microsoft, what they really are did at least did at least did at least once, um, this once, um, this once, um, this completely useless completely useless completely useless report. So either a report on report. So either a report on report. So either a report on the ticker is in the report, or the ticker is in the report, or the ticker is in the report, or he is not there. AND he is not there. AND he is not there. AND use LLM use LLM use LLM to check this to check this to check this would be completely unnecessary. would be completely unnecessary. would be completely unnecessary. Uh, to do this Uh, to do this Uh, to do this check, the first thing check, the first thing check, the first thing we need to do, we need to do, we need to do, is to get our data is to get our data is to get our data from Phoenix so we can from Phoenix so we can from Phoenix so we can perform an analysis. Here perform an analysis. Here perform an analysis. Here how is it done? We how is it done? We how is it done? We we request intervals ( we request intervals ( span) for our span) for our span) for our project, and project, and project, and parent ID null parent ID null parent ID null means that it is only means that it is only means that it is only spans that are not spans that are not spans that are not have a parent have a parent have a parent element. So, this is element. So, this is element. So, this is upper gaps upper gaps upper gaps equal. Those that I equal. Those that I equal. Those that I showed you in my showed you in my showed you in my on the tracks. Repository on the tracks. Repository on the tracks. Repository launches this in launches this in launches this in a little helper a little helper a little helper called get agent runs, called get agent runs, called get agent runs, which returns each which returns each which returns each identifier identifier identifier launch interval, launch interval, launch interval, its input data and its input data and its input data and source data, plus source data, plus source data, plus research turns output data, research turns output data, research turns output data, which we will need which we will need which we will need later. And then this later. And then this later. And then this ticker check. Create ticker check. Create ticker check. Create Evaluator comes from Evaluator comes from Evaluator comes from Phoenix evals package. You Phoenix evals package. You Phoenix evals package. You you can just provide you can just provide you can just provide his usual his usual his usual function. You don't function. You don't function. You don't need to do it yourself need to do it yourself need to do it yourself create a function create a function create a function appraiser. Everything you need appraiser. Everything you need appraiser. Everything you need need to do this need to do this need to do this tell him what it is tell him what it is tell him what it is code evaluation so as not to code evaluation so as not to code evaluation so as not to LLM was involved.

LLM was involved. LLM was involved. The function gets The function gets The function gets input data and input data and input data and source data. She source data. She source data. She separates tickers, about separates tickers, about separates tickers, about which we asked. which we asked. which we asked. She checks whether She checks whether She checks whether each of them each of them each of them displayed in displayed in displayed in report. And returns report. And returns report. And returns label and rating. AND label and rating. AND label and rating. AND if something is missing, if something is missing, if something is missing, she also indicates, she also indicates, she also indicates, what a ticker. So what a ticker. So what a ticker. So failure is explained failure is explained failure is explained by itself. Hmm, by itself. Hmm, by itself. Hmm, structured input structured input structured input the data I the data I the data I said before, as we said before, as we said before, as we sending incoming sending incoming sending incoming data to our data to our data to our tool in tool in tool in JSON format, this is JSON format, this is JSON format, this is helps us here, helps us here, helps us here, because it is significantly because it is significantly because it is significantly simplifies writing simplifies writing simplifies writing this Kodi val, because this Kodi val, because this Kodi val, because this is already JSON. So we are already this is already JSON. So we are already this is already JSON. So we are already we know what a ticker is, we know what a ticker is, we know what a ticker is, and what is the report description, and what is the report description, and what is the report description, which one does he have which one does he have which one does he have record. So, if record. So, if record. So, if you can do you can do you can do your login details your login details your login details structured, it structured, it structured, it will make it much easier will make it much easier will make it much easier work with your work with your work with your estimates. But if not, estimates. But if not, estimates. But if not, you will have to you will have to you will have to to involve another LLM, to involve another LLM, to involve another LLM, to read yours to read yours to read yours input data, input data, input data, determine what it is determine what it is determine what it is input data, and then input data, and then input data, and then pass them on to your pass them on to your pass them on to your to the appraiser. Then we to the appraiser. Then we to the appraiser. Then we we launch it and we launch it and we launch it and register register register results back to results back to results back to Phoenix. So, the script Phoenix. So, the script Phoenix. So, the script starts the evaluator starts the evaluator starts the evaluator at each launch at each launch at each launch agent, and then agent, and then agent, and then causes log annotations causes log annotations causes log annotations spans that attach spans that attach spans that attach every result to every result to every result to its span. This will be the one its span. This will be the one its span. This will be the one the very template for the very template for the very template for every assessment we make every assessment we make every assessment we make We are launching today.

We are launching today. We are launching today. You get runs, You get runs, You get runs, evaluate them, and then evaluate them, and then evaluate them, and then register register register results back to results back to results back to Phoenix. So, the team Phoenix. So, the team Phoenix. So, the team here-npm run eval code. here-npm run eval code. here-npm run eval code. Let's see, Let's see, Let's see, Can I force it? Can I force it? Can I force it? work. Okay, this is it. work. Okay, this is it. work. Okay, this is it. it worked, and it worked, and it worked, and almost started almost started almost started instantly, because it instantly, because it instantly, because it valves Code. They don't valves Code. They don't valves Code. They don't you have to wait until you have to wait until you have to wait until LLM will do something. They LLM will do something. They LLM will do something. They simply inform, simply inform, simply inform, what turned out to be 15 what turned out to be 15 what turned out to be 15 fast downloads fast downloads fast downloads and two failures. One—Tesla and two failures. One—Tesla and two failures. One—Tesla Tesla is missing, but Tesla is missing, but Tesla is missing, but one—Amazon is missing one—Amazon is missing one—Amazon is missing Amazon. So, this is Amazon. So, this is Amazon. So, this is the stupidest check the stupidest check which we could do which we could do . Is LLM really . Is LLM really . Is LLM really wrote a report about wrote a report about wrote a report about what are we talking about what are we talking about what are we talking about asked? Write asked? Write asked? Write report, and it is already report, and it is already report, and it is already found some found some found some errors. He found errors. He found errors. He found times when he wrote times when he wrote times when he wrote nothing. I know, nothing. I know, nothing. I know, for example, that for example, that for example, that problem with Amazon problem with Amazon problem with Amazon is that he is that he is that he asked about asked about asked about AWS profitability, and AWS profitability, and AWS profitability, and the agent focused on the agent focused on the agent focused on AWS and wrote a report, AWS and wrote a report, AWS and wrote a report, which was only about AWS and which was only about AWS and which was only about AWS and about what AWS does, and about what AWS does, and about what AWS does, and not about what he does not about what he does not about what he does Amazon, you know, shopping Amazon, you know, shopping Amazon, you know, shopping company. So this is a test, company. So this is a test, company. So this is a test, which is worth doing, and which is worth doing, and which is worth doing, and its launch its launch its launch absolutely nothing absolutely nothing absolutely nothing costs. So Kod valves costs. So Kod valves —These are not toys. So, —These are not toys. So, —These are not toys. So, does it parse JSON output?

does it parse JSON output? does it parse JSON output? Does it exceed Does it exceed Does it exceed length limit? Or length limit? Or length limit? Or he avoids phrases that he avoids phrases that he avoids phrases that you never want you never want you never want to see how the model to see how the model to see how the model AI languages? Hmm, and this is not AI languages? Hmm, and this is not AI languages? Hmm, and this is not must be must be must be line checking. Kodval line checking. Kodval line checking. Kodval simply means that simply means that simply means that evaluation is evaluation is evaluation is deterministic. Therefore, deterministic. Therefore, deterministic. Therefore, uh, your Kodval can uh, your Kodval can uh, your Kodval can query your database query your database query your database data to data to data to make sure that the condition make sure that the condition make sure that the condition correct. He can correct. He can correct. He can to make sure that to make sure that to make sure that tool calls tool calls tool calls did what they had to do did what they had to do did what they had to do do. Uh, he can do. Uh, he can do. Uh, he can call the API and twice call the API and twice call the API and twice check if the price check if the price check if the price shares, which, in my opinion, shares, which, in my opinion, shares, which, in my opinion, agent, happens, agent, happens, agent, happens, is really the stock price. is really the stock price. is really the stock price. Uh, whatever gives Uh, whatever gives Uh, whatever gives the same answer the same answer the same answer for the same incoming for the same incoming for the same incoming data every time, maybe data every time, maybe data every time, maybe become Kodi val. So yours become Kodi val. So yours become Kodi val. So yours Kodi valves can Kodi valves can Kodi valves can to be enough to be enough to be enough complicated. Uh, but complicated. Uh, but complicated. Uh, but one thing you need one thing you need one thing you need you should be careful, this you should be careful, this you should be careful, this evaluation evaluation evaluation result, not result, not result, not way. Uh, here it is. way. Uh, here it is. way. Uh, here it is. a tempting mistake, huh a tempting mistake, huh -e, check if -e, check if -e, check if the agent performed certain the agent performed certain the agent performed certain steps. Did you call? steps. Did you call? steps. Did you call? he first he first he first search tool, and search tool, and search tool, and then this other one then this other one then this other one tool? Er, Anthropic tool? Er, Anthropic tool? Er, Anthropic advises that this is too much advises that this is too much advises that this is too much tough approach. Uh, tough approach. Uh, tough approach. Uh, as we saw earlier, as we saw earlier, as we saw earlier, the agent can be the agent can be the agent can be creative. He can creative. He can creative. He can to do things to do things to do things in an unexpected way.

in an unexpected way. in an unexpected way. You don't want to. You don't want to. You don't want to. to assume that he to assume that he to assume that he will always do the right thing will always do the right thing will always do the right thing what you would do. THERE ARE- what you would do. THERE ARE- uh, so you don't want to uh, so you don't want to uh, so you don't want to to evaluate the completed to evaluate the completed to evaluate the completed way. Do you want way. Do you want way. Do you want to assess whether it was to assess whether it was to assess whether it was the result is good. Ahem the result is good. Ahem , our verification , our verification , our verification ticket, no matter how ticket, no matter how ticket, no matter how the agent found his the agent found his the agent found his data, right? Her data, right? Her data, right? Her it is simply important that in it is simply important that in it is simply important that in the report was correct the report was correct the report was correct company, and this company, and this company, and this correct level correct level correct level for many estimates. for many estimates. for many estimates. Later we Later we Later we let's talk about let's talk about let's talk about assessments at the level assessments at the level assessments at the level trajectories where we trajectories where we trajectories where we we really want we really want we really want evaluate the path, but it evaluate the path, but it evaluate the path, but it completely different, uh, completely different, uh, completely different, uh, case. So, this is case. So, this is case. So, this is leads us to leads us to leads us to Step Four: LLM as Step Four: LLM as Step Four: LLM as judge, uh, and we judge, uh, and we judge, uh, and we let's start with the built-in ones let's start with the built-in ones let's start with the built-in ones Phoenix ratings. Hmm, the code is not Phoenix ratings. Hmm, the code is not Phoenix ratings. Hmm, the code is not can check if there is can check if there is can check if there is ticker in the report, but it ticker in the report, but it ticker in the report, but it can't tell you, can't tell you, can't tell you, Is the analysis good? THERE ARE Is the analysis good? THERE ARE -Uh, are the numbers accurate? Or -Uh, are the numbers accurate? Or -Uh, are the numbers accurate? Or Is the report in the report complete? Is the report in the report complete? Is the report in the report complete? Is it a recommendation? Is it a recommendation? Is it a recommendation? well-founded? well-founded? well-founded? This is a question about This is a question about This is a question about the value of the report, not the value of the report, not the value of the report, not text content text content text content report. And for those who report. And for those who report. And for those who need, uh, something that need, uh, something that need, uh, something that has a judgment. Um, has a judgment. Um, has a judgment. Um, LLM evaluation LLM evaluation LLM evaluation consists of three consists of three consists of three parts. He has parts. He has parts. He has assessment model, assessment model, assessment model, which is the LM that conducts which is the LM that conducts which is the LM that conducts evaluation. Um, in evaluation. Um, in evaluation. Um, in there is a section that is there is a section that is there is a section that is a hint that indicates a hint that indicates what quality looks like.

what quality looks like. what quality looks like. And then he has the data And then he has the data , that is, examples that , that is, examples that , that is, examples that he evaluates. Phoenix he evaluates. Phoenix he evaluates. Phoenix holds these three parts holds these three parts holds these three parts separately so you can separately so you can separately so you can combine and combine and combine and combine. You can combine. You can combine. You can to try the same ones to try the same ones to try the same ones rules with different rules with different rules with different models. You can models. You can models. You can to try that to try that to try that the evaluation itself with the evaluation itself with the evaluation itself with different rules. Em different rules. Em , good news for you , good news for you , good news for you is that is that is that Phoenix supplies a bunch Phoenix supplies a bunch Phoenix supplies a bunch built-in assessments. built-in assessments. built-in assessments. So you don't always So you don't always So you don't always need to write need to write need to write own assessments, own assessments, own assessments, at least not right away. at least not right away. at least not right away. Um, there is a correctness, Um, there is a correctness, Um, there is a correctness, which checks whether which checks whether which checks whether The answer is correct. Um, yes. The answer is correct. Um, yes. The answer is correct. Um, yes. loyalty, which loyalty, which loyalty, which checks whether checks whether checks whether she answers she answers she answers source material. source material. source material. Um, we'll see. Um, we'll see. Um, we'll see. the difference between these the difference between these the difference between these two per second. Um, two per second. Um, two per second. Um, there is completeness, there is completeness, there is completeness, hallucination, denial, hallucination, denial, hallucination, denial, personal information personal information personal information and toxicity. and toxicity. and toxicity. Many things that Many things that Many things that need to check need to check need to check to many agents, to many agents, to many agents, built-in. Um, and for built-in. Um, and for built-in. Um, and for agents are estimates for agents are estimates for agents are estimates for using using using tools. Is he tools. Is he tools. Is he chose the right one chose the right one chose the right one tool? Is he tool? Is he tool? Is he passed the correct passed the correct passed the correct arguments? Hmm, is it arguments? Hmm, is it arguments? Hmm, is it did it do something did it do something did it do something reasonable with reasonable with reasonable with result? Hmm, these result? Hmm, these result? Hmm, these grades are not magic grades are not magic . It's just, uh, . It's just, uh, . It's just, uh, tips that we already tips that we already tips that we already wrote, wrote, wrote, protested in battle and protested in battle and protested in battle and convinced that they convinced that they convinced that they working for you. So working for you. So let's find let's find judge model. Hmm, uh, judge model. Hmm, uh, judge model. Hmm, uh, Phoenix evaluations in Typescript, e- Phoenix evaluations in Typescript, e- e, use Versel e, use Versel e, use Versel AI SDK for interaction with AI SDK for interaction with AI SDK for interaction with models. So be models. So be which model, which which model, which which model, which supports AI SDK, can supports AI SDK, can supports AI SDK, can to be a judge Uh, in to be a judge Uh, in to be a judge Uh, in in this case I in this case I in this case I I use Claude Sonnet I use Claude Sonnet I use Claude Sonnet 5.5. Uh, the agent is working.

5.5. Uh, the agent is working. 5.5. Uh, the agent is working. on Haiku. As I said, on Haiku. As I said, on Haiku. As I said, it's cheap and fast. it's cheap and fast. it's cheap and fast. Our judge works on Our judge works on Our judge works on Sonnet, because you want to Sonnet, because you want to so that your judge is so that your judge is so that your judge is more powerful than more powerful than more powerful than the agent himself. So give it the agent himself. So give it the agent himself. So give it give him a Sonnet or give him a Sonnet or give him a Sonnet or him Opus. Uh, or give him Opus. Uh, or give him Opus. Uh, or give he, uh, even better, he, uh, even better, he, uh, even better, give him a model from give him a model from , uh, completely different , uh, completely different , uh, completely different supplier. If in supplier. If in supplier. If in you have the OpenAI key, you have the OpenAI key, you have the OpenAI key, ask OpenAI ask OpenAI ask OpenAI evaluate Claude's work. THERE ARE- evaluate Claude's work. THERE ARE- uh, because Claude has that, uh, because Claude has that, uh, because Claude has that, what is known as, uh, what is known as, uh, what is known as, uh, bias of the same bias of the same bias of the same models. So Claude, how models. So Claude, how models. So Claude, how usually, like usually, like usually, like result of others result of others result of others There are more versions of Claude, There are more versions of Claude, There are more versions of Claude, than the result of OpenAI or than the result of OpenAI or than the result of OpenAI or some other some other some other supplier. Hmm, supplier. Hmm, supplier. Hmm, the last line here, the last line here, the last line here, if you are interested, if you are interested, if you are interested, suppresses tracing suppresses tracing suppresses tracing on the judge because we on the judge because we on the judge because we enabled tracing enabled tracing enabled tracing all over the board. Judge all over the board. Judge all over the board. Judge himself is a copy of Claude. himself is a copy of Claude. So by default, So by default, and actually in mine and actually in mine and actually in mine test data you test data you test data you you'll see what I forgot you'll see what I forgot you'll see what I forgot turn on, uh, turn on, uh, turn on, uh, suppression suppression suppression telemetry. So my telemetry. So my telemetry. So my traces are filled traces are filled traces are filled because the judge because the judge because the judge starts itself and starts itself and starts itself and sees everyone sees everyone sees everyone tracing from the judge, tracing from the judge, tracing from the judge, which you don't need.

which you don't need. So every evaluator in So every evaluator in this repository this repository this repository gets it gets it gets it settings settings settings telemetry so that the judge telemetry so that the judge telemetry so that the judge did not get to did not get to did not get to our agent. our agent. Let's start with Let's start with correctness. Uh, what? correctness. Uh, what? correctness. Uh, what? is the obvious first is the obvious first is the obvious first choice. Is this report choice. Is this report choice. Is this report correct? This is one correct? This is one correct? This is one import and one line import and one line import and one line settings. You don't settings. You don't settings. You don't nothing needed nothing needed nothing needed do. You just do. You just do. You just you say, "Hey, I you say, "Hey, I you say, "Hey, I I'm running the evaluator. I'm running the evaluator. I'm running the evaluator. correctness". correctness". correctness". Run it on Run it on Run it on my files and my files and my files and let's see how it goes let's see how it goes let's see how it goes will go. Then you will go. Then you will go. Then you call evaluate, and you call evaluate, and you call evaluate, and you you get a label, you get a label, you get a label, assessment and explanation. assessment and explanation. assessment and explanation. And the script runs And the script runs And the script runs its on everyone its on everyone its on everyone agent launch and agent launch and agent launch and records the results records the results records the results back to Phoenix. Uh, yes. back to Phoenix. Uh, yes. back to Phoenix. Uh, yes. just like we did just like we did just like we did ticker check. Uh, ticker check. Uh, I have this too. I have this too. launch. This will take launch. This will take launch. This will take a little time, because a little time, because a little time, because will need will need will need to do some to do some to do some LLM assignments. Uh, LLM assignments. Uh, LLM assignments. Uh, if it works then if it works then if it works then works. If not, in works. If not, in works. If not, in I already have them built in.

I already have them built in. I already have them built in. functions. So you functions. So you functions. So you you can see, uh, in you can see, uh, in you can see, uh, in my column my column my column annotations results annotations results annotations results different assessments that I different assessments that I different assessments that I already spent. In this already spent. In this already spent. In this case it shows case it shows case it shows you, uh, efficiency, but you, uh, efficiency, but you, uh, efficiency, but you can see, uh, you can see, uh, you can see, uh, correctness, correctness, correctness, loyalty, effectiveness loyalty, effectiveness loyalty, effectiveness for a person, uh, yes for a person, uh, yes for a person, uh, yes mention sticker. We mention sticker. We mention sticker. We let's talk about that let's talk about that let's talk about that all of this is like that. But all of this is like that. But all of this is like that. But first you will see first you will see first you will see correctness. Uh, correctness. Uh, correctness. Uh, correctness in all correctness in all correctness in all in these cases in these cases in these cases is equal to zero. Uh, is equal to zero. Uh, all 16 reports were all 16 reports were recognized recognized recognized incorrect. AND incorrect. AND incorrect. AND the reason why they the reason why they the reason why they were recognized were recognized were recognized incorrect, incorrect, incorrect, is that is that is that Claude evaluates Claude evaluates Claude evaluates correctness on correctness on correctness on based on the fact that he based on the fact that he based on the fact that he knows. This is the only thing that knows. This is the only thing that knows. This is the only thing that available to him. AND available to him. AND available to him. AND Claude's knowledge on the subject Claude's knowledge on the subject Claude's knowledge on the subject moment limited moment limited moment limited around January. around January. So, my agent So, my agent talks about prices for talks about prices for talks about prices for shares in October, and Claude shares in October, and Claude shares in October, and Claude panics and says, "You panics and says, "You panics and says, "You talk about things, talk about things, talk about things, which will take place in which will take place in which will take place in future. It, future. It, future. It, obviously, obviously, obviously, hallucination. This report hallucination. This report hallucination. This report there can be no way there can be no way there can be no way correct." So ours correct." So ours correct." So ours built-in check built-in check built-in check not correct not correct not correct every report passes every report passes every report passes for this reason. Hmm, what for this reason. Hmm, what for this reason. Hmm, what not very useful. Hmm, not very useful. Hmm, not very useful. Hmm, therefore, the assessment therefore, the assessment therefore, the assessment half half half incorrect, incorrect, incorrect, half correct half correct . Seven of the 16 explanations, . Seven of the 16 explanations, . Seven of the 16 explanations, according to my data according to my data , complain that , complain that , complain that the judge cannot the judge cannot the judge cannot check fresh check fresh check fresh data. Hmm, that is, the assessment data. Hmm, that is, the assessment data. Hmm, that is, the assessment definitely wrong. Hmm definitely wrong. Hmm , but many of them , but many of them , but many of them also indicate also indicate also indicate something real. Such something real. Such something real. Such things like numbers that don't things like numbers that don't things like numbers that don't are composed, or given are composed, or given , which contradict each other

, which contradict each other , which contradict each other one. Hmm, so one. Hmm, so one. Hmm, so zero points for us zero points for us zero points for us says nothing. says nothing. says nothing. He tells us He tells us He tells us in general, that this assessment, in general, that this assessment, in general, that this assessment, probably not probably not probably not the right choice the right choice the right choice evaluations. Hmm, what we evaluations. Hmm, what we evaluations. Hmm, what we need, this is a question, need, this is a question, need, this is a question, to which the judge, uh, can to which the judge, uh, can to which the judge, uh, can actually answer actually answer . So, uh, I already said . So, uh, I already said . So, uh, I already said This is for you. Good. Instead This is for you. Good. Instead This is for you. Good. Instead this we this we this we we will use we will use we will use loyalty. Loyalty- loyalty. Loyalty- to put it to put it to put it much better much better much better question. This means question. This means question. This means to say that, to say that, to say that, given the data, you... given the data, you... Did your report make sense? Did your report make sense? Did your report make sense? Is it based on Is it based on Is it based on response to data, response to data, response to data, which were given to us? It which were given to us? It which were given to us? It works a lot works a lot works a lot better because it can better because it can better because it can work with data, work with data, work with data, this is exactly the same data, this is exactly the same data, this is exactly the same data, that both the agent and that both the agent and that both the agent and to check whether to check whether to check whether he receives a report, he receives a report, he receives a report, which makes sense. Therefore, which makes sense. Therefore, which makes sense. Therefore, this is a question that this is a question that this is a question that the judge can actually the judge can actually the judge can actually answer. AND answer. AND answer. AND the context is already in the context is already in the context is already in the track. Every web the track. Every web search has its own search has its own search has its own own range, as own range, as own range, as we saw. Um, every one we saw. Um, every one we saw. Um, every one the study concludes the study concludes the study concludes own notes own notes own notes agent that he agent that he agent that he found. So mine found. So mine found. So mine first version, mine first version, mine first version, mine first grade, uh, first grade, uh, first grade, uh, gave the judge, uh, only gave the judge, uh, only gave the judge, uh, only research notes research notes , and then I read , and then I read , and then I read explanation, and the judge explanation, and the judge explanation, and the judge marked the numbers as marked the numbers as marked the numbers as fictional, uh, because fictional, uh, because fictional, uh, because my version of the assessment is not my version of the assessment is not my version of the assessment is not I told him that I I told him that I I told him that I searched the Internet and searched the Internet and searched the Internet and actually found these actually found these actually found these data. Er, that is, er, in data. Er, that is, er, in data. Er, that is, er, in the version you see, the version you see, the version you see, we gave him full we gave him full we gave him full context. Uh, ours context. Uh, ours context. Uh, ours assistant get agent runs assistant get agent runs assistant get agent runs going to pull out going to pull out going to pull out like, uh, raw like, uh, raw like, uh, raw search results, search results, search results, and notes so that he and notes so that he and notes so that he could see that could see that could see that found found found research agent research agent research agent and how he found these

and how he found these and how he found these data. Uh, that is, how do I data. Uh, that is, how do I data. Uh, that is, how do I I already said, loyalty I already said, loyalty I already said, loyalty requires three things. requires three things. requires three things. She accepts incoming calls. She accepts incoming calls. She accepts incoming calls. data, that is, what data, that is, what data, that is, what user requested user requested . She accepts the Exodus, . She accepts the Exodus, . She accepts the Exodus, which is a report, takes which is a report, takes which is a report, takes context, that is context, that is context, that is search results and search results and search results and notes that I notes that I notes that I just spoke. Uh, and just spoke. Uh, and just spoke. Uh, and you launch it, you launch it, you launch it, using that one using that one using that one the same template as the same template as the same template as earlier. You create earlier. You create earlier. You create appraiser, appraiser appraiser, appraiser appraiser, appraiser every launch, every launch, every launch, record the results record the results . Uh, like I said, . Uh, like I said, . Uh, like I said, this is slower than this is slower than this is slower than correctness, because correctness, because correctness, because The evaluator reads each The evaluator reads each The evaluator reads each search. Uh, who? search. Uh, who? search. Uh, who? got some got some got some results from your e results from your e -e-grades so far? Again -e-grades so far? Again -e-grades so far? Again anyway, you're faster anyway, you're faster anyway, you're faster Wi-Fi connection than in Wi-Fi connection than in Wi-Fi connection than in all others. Anyone else? all others. Anyone else? Cool. Uh, so Cool. Uh, so let's take a look let's take a look let's take a look Phoenix on my reports Phoenix on my reports Phoenix on my reports about reliability. THERE ARE- about reliability. THERE ARE- uh, I got a lot uh, I got a lot uh, I got a lot zeros relative to zeros relative to zeros relative to reliability, but not reliability, but not reliability, but not all. Uh, come on. all. Uh, come on. all. Uh, come on. let's read the explanation let's read the explanation let's read the explanation Microsoft. This is what I Microsoft. This is what I Microsoft. This is what I read what happened read what happened read what happened the most interesting. Uh, how?

the most interesting. Uh, how? the most interesting. Uh, how? you see, next to you see, next to you see, next to each of them you each of them you each of them you you get an explanation. you get an explanation. you get an explanation. Uh, so in this Uh, so in this Uh, so in this case answer case answer case answer adds a few adds a few adds a few statements that statements that statements that context not context not context not supports. Azure is supports. Azure is supports. Azure is cloudy cloudy cloudy supplier number supplier number supplier number two. The lag gives two. The lag gives two. The lag gives three years of visibility, and three years of visibility, and three years of visibility, and there are implicit goals for 202 there are implicit goals for 202 there are implicit goals for 202 financial year. Uh, financial year. Uh, this tells me that this tells me that he doesn't like it he doesn't like it he doesn't like it uh, this thing. She uh, this thing. She uh, this thing. She tells me why tells me why tells me why considers this thing considers this thing considers this thing untrue. Um, she untrue. Um, she untrue. Um, she complains that, uh, complains that, uh, complains that, uh, some goals 2027 some goals 2027 some goals 2027 fabricated, because they fabricated, because they fabricated, because they none none none search results. THERE ARE search results. THERE ARE -eh, Nvidia is also interesting, -eh, Nvidia is also interesting, -eh, Nvidia is also interesting, if I can get it if I can get it if I can get it find. Right. IN find. Right. IN find. Right. IN this one, she says that this one, she says that this one, she says that the answer mixes two the answer mixes two the answer mixes two different data points. It different data points. It different data points. It error of reasoning, error of reasoning, error of reasoning, which agent, which which agent, which which agent, which judge, uh, exactly judge, uh, exactly judge, uh, exactly found. Uh, she is found. Uh, she is found. Uh, she is says that contextual says that contextual says that contextual this information this information this information fabricated. It fabricated. It fabricated. It much more useful much more useful much more useful results. We have, uh- results. We have, uh- e, 14 out of 16 reports, e, 14 out of 16 reports, e, 14 out of 16 reports, marked as marked as marked as false, but false, but false, but explanation is much explanation is much explanation is much more useful. Uh, and she more useful. Uh, and she more useful. Uh, and she actually does what actually does what actually does what what we wanted. Uh, so what we wanted. Uh, so what we wanted. Uh, so for agents you too for agents you too for agents you too can you evaluate can you evaluate can you evaluate tool calls tool calls . Uh, we saw how the web . Uh, we saw how the web -search covers every -search covers every -search covers every the track. Phoenix also has the track. Phoenix also has the track. Phoenix also has built-in assessment for built-in assessment for built-in assessment for all of them. Choice all of them. Choice all of them. Choice the tool asks,

the tool asks, the tool asks, did the agent choose did the agent choose did the agent choose correct correct correct tool, uh, or tool, uh, or tool, uh, or searched when he didn't searched when he didn't searched when he didn't It was necessary. Challenge It was necessary. Challenge It was necessary. Challenge the tool asks, the tool asks, the tool asks, did he transmit did he transmit did he transmit reasonable arguments reasonable arguments reasonable arguments these tools, these tools, these tools, such as search such as search such as search request that actually request that actually request that actually concerning...company concerning...company . And the processing . And the processing . And the processing answers answers answers the tool asks, the tool asks, the tool asks, did he do something did he do something did he do something smart, uh, with smart, uh, with smart, uh, with as a result of this as a result of this as a result of this call. Uh, this is call. Uh, this is call. Uh, this is solutions that can solutions that can solutions that can lead to lead to lead to fracture, make or fracture, make or fracture, make or destroy the agent. THERE ARE- destroy the agent. THERE ARE- oh, and all of them oh, and all of them oh, and all of them occur before occur before occur before how the report will actually be how the report will actually be how the report will actually be read. Uh, and we read. Uh, and we read. Uh, and we we already know that our we already know that our we already know that our the agent sometimes does the agent sometimes does the agent sometimes does strange choice strange choice strange choice tools, tools, tools, for example, he for example, he for example, he trying to save trying to save trying to save report to file. Uh, so report to file. Uh, so report to file. Uh, so we can see, we can see, we can see, does ours catch this? does ours catch this? does ours catch this? built-in assessment. Uh, built-in assessment. Uh, built-in assessment. Uh, I am not going. I am not going. I am not going. run this one, but you run this one, but you run this one, but you you can run that one. you can run that one. you can run that one. Uh, choice Uh, choice Uh, choice the tool takes the tool takes the tool takes three things. What was asked? three things. What was asked? three things. What was asked? from the agent, who from the agent, who from the agent, who tools he could tools he could tools he could would use and would use and would use and which he chose. Uh, and which he chose. Uh, and which he chose. Uh, and script separately script separately script separately passes through passes through passes through each agent's move and each agent's move and each agent's move and every study and every study and every study and trace recording and trace recording and trace recording and collects ranges collects ranges collects ranges tools under tools under tools under each of them. Uh, each of them. Uh, each of them. Uh, trace records, trace records, trace records, what tools what tools what tools called the agent, but called the agent, but called the agent, but not what he could not what he could not what he could cause. So we cause. So we cause. So we must themselves must themselves must themselves write this list, write this list, write this list, uh, in tool selection.ts. Uh, uh, in tool selection.ts. Uh, uh, in tool selection.ts. Uh, then it's, uh, the same one then it's, uh, the same one then it's, uh, the same one scheme, like any other scheme, like any other scheme, like any other another launch. Uh, another launch. Uh, another launch. Uh, so, npm run eval tools, uh, and so, npm run eval tools, uh, and so, npm run eval tools, uh, and then everything is ready. Uh, then everything is ready. Uh, then everything is ready. Uh, this should take this should take this should take about 15 seconds.

about 15 seconds. So, let's So, let's see you in Phoenix see you in Phoenix see you in Phoenix on Tool Selection. on Tool Selection. Tool selection. It Tool selection. It It helps me. Good It helps me. Good . So, these are the cases, . So, these are the cases, . So, these are the cases, when it was decided, when it was decided, when it was decided, what is my choice what is my choice what is my choice the tool was the tool was the tool was wrong with wrong with wrong with for some reason. Um, for some reason. Um, for some reason. Um, user requested user requested user requested brief report on brief report on brief report on based on the previous based on the previous based on the previous research. Record research. Record research. Record it to a file by it to a file by it to a file by using write is using write is using write is plausible and plausible and plausible and appropriate because appropriate because appropriate because write exists and write exists and write exists and safe. Um, however, safe. Um, however, safe. Um, however, LLM did the second LLM did the second LLM did the second excessive call excessive call excessive call bash. Perfectly. Um, yes, bash. Perfectly. Um, yes, bash. Perfectly. Um, yes, definitely failed. Um, definitely failed. Um, the judge failed six, the judge failed six, who tried twice. who tried twice. who tried twice. He passed seven. He passed seven. He passed seven. Six of the seven that Six of the seven that Six of the seven that tried once.

tried once. tried once. Um, a total of 13 of my Um, a total of 13 of my Um, a total of 13 of my 16 writing attempts 16 writing attempts 16 writing attempts tried to preserve tried to preserve tried to preserve file. Um, and just one of file. Um, and just one of file. Um, and just one of these explanations these explanations these explanations said that said that said that the user never the user never the user never asked to save the file. asked to save the file. asked to save the file. So, um, this judge So, um, this judge So, um, this judge is wrong? Is the judge is wrong? Is the judge , um, I accurately estimated that , um, I accurately estimated that , um, I accurately estimated that tool call tool call tool call was correct? Um, was correct? Um, was correct? Um, our tip for our tip for our tip for the letter read: " the letter read: " Write a concise Write a concise Write a concise financial report on financial report on financial report on based on your based on your based on your research above. research above. research above. Our tip is here Our tip is here Our tip is here really ambiguous really ambiguous . We said: "Write," . We said: "Write," . We said: "Write," we didn't say where it was we didn't say where it was we didn't say where it was write. We didn't say write. We didn't say write. We didn't say Why write this? Em Why write this? Em , we didn't say, for , we didn't say, for , we didn't say, for Who will it be? Who will it be? Who will it be? audience. "Therefore, audience. "Therefore, audience. "Therefore, tool selection tool selection tool selection knows this, looks at it knows this, looks at it knows this, looks at it input and output input and output input and output further, it seems further, it seems further, it seems reasonable that the agent reasonable that the agent reasonable that the agent will try to write here will try to write here will try to write here to disk. "So, again to disk. "So, again to disk. "So, again well, what you well, what you well, what you you order the agent you order the agent you order the agent to do, and what you to do, and what you to do, and what you instruct the appraiser instruct the appraiser instruct the appraiser to do, interact to do, interact to do, interact with each other. Uh, you with each other. Uh, you with each other. Uh, you you can't necessarily you can't necessarily you can't necessarily to be sure that to be sure that to be sure that tool call tool call tool call will be correct, therefore will be correct, therefore will be correct, therefore what in this case what in this case what in this case tool call tool call tool call refers to refers to refers to tool call, tool call, tool call, the appraiser takes sides the appraiser takes sides the appraiser takes sides agent. It's like no, agent. It's like no, agent. It's like no, The agent is right.

The agent is right. The agent is right. The agent gave us The agent gave us The agent gave us instructions, uh, agent instructions, uh, agent instructions, uh, agent received instructions received instructions received instructions and did everything possible and did everything possible , this is, in fact, a position , this is, in fact, a position , this is, in fact, a position appraiser. Uh, appraiser. Uh, results results evaluation also evaluation also evaluation also live in the highway. Uh, live in the highway. Uh, so if you so if you look at the tracks in look at the tracks in look at the tracks in Phoenix, uh, you Phoenix, uh, you Phoenix, uh, you you can see them you can see them you can see them here, tied to here, tied to here, tied to what route they take what route they take what route they take received, to which received, to which received, to which gap in the track they gap in the track they gap in the track they tied. So, you tied. So, you tied. So, you you see that the effectiveness you see that the effectiveness you see that the effectiveness and correctness and correctness and correctness tied to the root. tied to the root. tied to the root. Uh, while the choice Uh, while the choice Uh, while the choice tool tool tool is attached to is attached to is attached to specific, er, specific, er, specific, er, turn, which turn, which turn, which made this call made this call made this call tool. Uh, you tool. Uh, you tool. Uh, you you can filter you can filter you can filter their routes for any their routes for any what assessment? Uh, you what assessment? Uh, you what assessment? Uh, you you can sort by you can sort by you can sort by assessment to find assessment to find assessment to find your worst your worst your worst runs. You can runs. You can runs. You can to go to go to go directly from directly from directly from failed assessment on failed assessment on failed assessment on step that her step that her step that her caused. Uh, from caused. Uh, from caused. Uh, from evaluations to evaluations to evaluations to evaluations to evaluations to evaluations to reasons for one reasons for one reasons for one mouse click. Uh, mouse click. Uh, mouse click. Uh, so, everything..." this so, everything..." this so, everything..." this built-in assessment. Ahem, built-in assessment. Ahem, built-in assessment. Ahem, and the lesson here is and the lesson here is and the lesson here is because the choice because the choice because the choice correct assessment correct assessment correct assessment surpasses her surpasses her surpasses her settings. Choice settings. Choice settings. Choice correctness was correctness was correctness was absolutely absolutely absolutely incorrect assessment incorrect assessment . She doesn't mean anything to us.

. She doesn't mean anything to us. . She doesn't mean anything to us. said. She just said. She just said. She just told us that told us that told us that there was a lot of things there was a lot of things there was a lot of things wrong. Choice wrong. Choice wrong. Choice the tool really the tool really the tool really told us something, but told us something, but told us something, but loyalty told us loyalty told us loyalty told us quite a few. Ahem, quite a few. Ahem, quite a few. Ahem, so check so check so check correct assessment correct assessment correct assessment surpasses her surpasses her surpasses her settings, and settings, and settings, and knowledge of what knowledge of what knowledge of what the question is really you the question is really you the question is really you you bet, it's real you bet, it's real you bet, it's real the basis of correct the basis of correct the basis of correct execution, that execution, that execution, that leads us to leads us to leads us to the fifth step, which the fifth step, which the fifth step, which consists of writing consists of writing consists of writing own assessment. Exactly own assessment. Exactly own assessment. Exactly here the estimate becomes here the estimate becomes here the estimate becomes specific for specific for specific for your program. Um, and your program. Um, and your program. Um, and again, this is when again, this is when again, this is when people think about people think about people think about evaluations, that's what it's about evaluations, that's what it's about evaluations, that's what it's about the majority thinks the majority thinks the majority thinks people, but I wanted people, but I wanted people, but I wanted make sure that you make sure that you make sure that you first understood first understood first understood foundations. You don't foundations. You don't foundations. You don't needed immediately needed immediately needed immediately to go to to go to to go to writing your own writing your own writing your own rubrics. You can rubrics. You can rubrics. You can write code evaluations, write code evaluations, write code evaluations, to get rid of a pile to get rid of a pile to get rid of a pile things. You can things. You can things. You can write, um, you write, um, you write, um, you can you can you can you use use use built-in assessment to built-in assessment to built-in assessment to make a pile make a pile make a pile obvious things, not obvious things, not obvious things, not inventing your own inventing your own inventing your own tool that tool that tool that calls the evaluator calls the evaluator calls the evaluator correctness, um, um. correctness, um, um. correctness, um, um. Um, what I wanted to Um, what I wanted to Um, what I wanted to my report did, this my report did, this my report did, this gave me a bunch, gave me a bunch, gave me a bunch, Recommendation regarding Recommendation regarding Recommendation regarding sale or sale or sale or maintenance. None maintenance. None maintenance. None built-in assessment, built-in assessment, built-in assessment, who would check this, who would check this, who would check this, because that would be very because that would be very because that would be very specifically. So we specifically. So we specifically. So we Let's write our own. Um, Let's write our own. Um, Let's write our own. Um, each estimate each estimate each estimate the clue I saw the clue I saw the clue I saw and which works well, and which works well, and which works well, has five different has five different has five different parts, and we will parts, and we will parts, and we will to consider them in to consider them in to consider them in in turn. Part One in turn. Part One in turn. Part One is that you is that you is that you give the judge a role and give the judge a role and give the judge a role and context. Um, the thing that context. Um, the thing that context. Um, the thing that you constantly you constantly you constantly you will see, this is what you will see, this is what you will see, this is what you are an expert blah blah you are an expert blah blah blah blah. This is actually blah blah. This is actually blah blah. This is actually works. That's why everyone works. That's why everyone works. That's why everyone it is written there.

it is written there. it is written there. A little helps A little helps A little helps tell the agent that this tell the agent that this tell the agent that this expert. Um, to say expert. Um, to say expert. Um, to say him, that he is an expert, him, that he is an expert, him, that he is an expert, unlike, unlike, unlike, which simply tells him, which simply tells him, which simply tells him, that he is working on that he is working on that he is working on something else, too, something else, too, something else, too, seems to have seems to have seems to have meaning when they meaning when they meaning when they make a comparison make a comparison . Um, it's just one of those . Um, it's just one of those . Um, it's just one of those strange things about LLM. strange things about LLM. strange things about LLM. You need to say You need to say You need to say him that he really him that he really him that he really smart, and he becomes smart, and he becomes smart, and he becomes confident and confident and confident and tries. Um, uh, tries. Um, uh, tries. Um, uh, but what really matters but what really matters , so this is the context. Um, , so this is the context. Um, , so this is the context. Um, this is a financial report. this is a financial report. this is a financial report. Um, we're waiting for the report, Um, we're waiting for the report, Um, we're waiting for the report, which analyzes which analyzes which analyzes financial data and financial data and financial data and tells us... tells us... real things. Hmm, and real things. Hmm, and real things. Hmm, and exactly what we exactly what we exactly what we I'm interested in whether I'm interested in whether I'm interested in whether will you be able to act on will you be able to act on will you be able to act on based on this. So, uh, based on this. So, uh, based on this. So, uh, this gives us clear this gives us clear this gives us clear criteria that we criteria that we criteria that we include in our include in our include in our rubric. Hmm, right here rubric. Hmm, right here rubric. Hmm, right here many people don't many people don't many people don't they do enough they do enough they do enough efforts. So don't write efforts. So don't write efforts. So don't write good answer, which good answer, which good answer, which is useful and accurate, is useful and accurate, is useful and accurate, because, uh, the judge doesn't because, uh, the judge doesn't because, uh, the judge doesn't knows what is useful and what is not knows what is useful and what is not knows what is useful and what is not knows what it means knows what it means knows what it means accuracy in this accuracy in this accuracy in this context. Hmm, judge. context. Hmm, judge. context. Hmm, judge. cannot accept cannot accept cannot accept a "yes" or "no" decision a "yes" or "no" decision a "yes" or "no" decision based on this.

based on this. based on this. Write down exactly what Write down exactly what Write down exactly what makes the report effective. THERE ARE- makes the report effective. THERE ARE- oh, so that he would give oh, so that he would give oh, so that he would give specific specific specific recommendation. He recommendation. He recommendation. He names specific names specific names specific risks and looks risks and looks risks and looks forward, not just forward, not just forward, not just back. These are things that back. These are things that back. These are things that the judge can the judge can the judge can to look and to look and to look and actually check. actually check. actually check. Uh, and pay attention, Uh, and pay attention, Uh, and pay attention, that we are not just that we are not just that we are not just we make these things up we make these things up we make these things up blindly. Hmm, yours blindly. Hmm, yours blindly. Hmm, yours the criteria follow the criteria follow the criteria follow from your analysis from your analysis from your analysis errors. Uh, just errors. Uh, just errors. Uh, just summarizes data without summarizes data without summarizes data without interpretations are interpretations are interpretations are the problem we the problem we the problem we saw when reading saw when reading saw when reading our traces. We our traces. We our traces. We we're putting it in there, so we're putting it in there, so we're putting it in there, so what is this problem that what is this problem that what is this problem that we are trying we are trying we are trying to avoid. Hmm, in this to avoid. Hmm, in this to avoid. Hmm, in this the whole point first the whole point first the whole point first read data. read data. read data. Specific criteria Specific criteria Specific criteria give you agreed give you agreed give you agreed judgment, and you judgment, and you judgment, and you get specific get specific get specific criteria, knowing criteria, knowing criteria, knowing in advance, that in advance, that in advance, that happens with happens with happens with data. to look like this data. to look like this because you yourself do it because you yourself do it because you yourself do it read. Third read. Third read. Third part of the good part of the good part of the good rubrics—clear rubrics—clear rubrics—clear mark the data. mark the data. mark the data. Wrap each Wrap each Wrap each fragment of data in fragment of data in fragment of data in custom XML tag. Claude custom XML tag. Claude custom XML tag. Claude learned to work with learned to work with learned to work with billions and billions and billions and billions of XML tags, and billions of XML tags, and billions of XML tags, and he is really good with he is really good with he is really good with works with them. Hmm, if works with them. Hmm, if works with them. Hmm, if you will look at the code you will look at the code you will look at the code our own sections, our own sections, our own sections, then you will see that we then you will see that we then you will see that we we do. uh, ours we do. uh, ours we do. uh, ours own tips, uh, own tips, uh, own tips, uh, wrap everything in tags wrap everything in tags wrap everything in tags XML. Uh, and double ones XML. Uh, and double ones XML. Uh, and double ones curly braces—this curly braces—this curly braces—this template variables that template variables that template variables that Phoenix fills, uh, Phoenix fills, uh, Phoenix fills, uh, incoming and incoming and incoming and source data source data source data each range. THERE ARE- each range. THERE ARE- uh, the reason why you do this uh, the reason why you do this uh, the reason why you do this do, is to do, is to do, is to because you need because you need because you need clear boundaries between what clear boundaries between what clear boundaries between what are the input data that are the input data that are the input data that is the way out, and what is is the way out, and what is is the way out, and what is your instructions.

your instructions. your instructions. Uh, that helps. Uh, that helps. Uh, that helps. prevent prevent prevent implementation implementation implementation tips, but also tips, but also tips, but also helps, uh, the agent helps, uh, the agent helps, uh, the agent just get confused. THERE ARE just get confused. THERE ARE -e, in particular, this -e, in particular, this -e, in particular, this helps you helps you helps you to understand, uh, if to understand, uh, if to understand, uh, if you include you include you include examples that we examples that we examples that we we'll see, mark it we'll see, mark it we'll see, mark it example as an example, and example as an example, and example as an example, and not given; otherwise he not given; otherwise he not given; otherwise he very confused with very confused with very confused with about this. Uh, me about this. Uh, me about this. Uh, me I like to include I like to include I like to include marked examples, marked examples, marked examples, but in reality it is but in reality it is but in reality it is controversial controversial controversial procedure, uh, procedure, uh, procedure, uh, controversial practice controversial practice , uh, in the world of evaluation. THERE ARE , uh, in the world of evaluation. THERE ARE -eh, the Criteria say -eh, the Criteria say -eh, the Criteria say judges, what do you have on judges, what do you have on judges, what do you have on I mean abstractly, but I mean abstractly, but I mean abstractly, but examples show examples show examples show exactly what you have exactly what you have exactly what you have in mind. As a person, in mind. As a person, in mind. As a person, It's much easier for me. It's much easier for me. It's much easier for me. to do something when to do something when to do something when someone showed me someone showed me someone showed me an example of how this an example of how this an example of how this should work than should work than should work than if they would just give if they would just give if they would just give me a list me a list me a list instructions on how to instructions on how to instructions on how to work. Uh, and, uh, work. Uh, and, uh, work. Uh, and, uh, many LLMs too, many LLMs too, many LLMs too, They probably think so. THERE ARE They probably think so. THERE ARE -uh, so when you -uh, so when you -uh, so when you test yours test yours test yours the column, uh, what are we the column, uh, what are we the column, uh, what are we let's do it on let's do it on let's do it on next step, uh, next step, uh, next step, uh, test it with test it with test it with with and without examples with and without examples with and without examples examples and examples and examples and look which one look which one look which one works better, uh, works better, uh, works better, uh, because sometimes, uh, because sometimes, uh, because sometimes, uh, examples will help examples will help examples will help she better do it she better do it she better do it work. And sometimes, uh, work. And sometimes, uh, work. And sometimes, uh, example will make it example will make it example will make it excessively approximate.

excessively approximate. excessively approximate. She will look at She will look at She will look at your examples and will tell your examples and will tell :"Well, that's not exactly :"Well, that's not exactly :"Well, that's not exactly similar to my example " similar to my example " , or she will try , or she will try , or she will try to do each to do each to do each the result is similar to the result is similar to the result is similar to your example your example your example result. Uh, what? result. Uh, what? result. Uh, what? less useful. Uh, me less useful. Uh, me less useful. Uh, me conducted this very test conducted this very test conducted this very test on your own. Uh, so in on your own. Uh, so in on your own. Uh, so in the rubric that I the rubric that I the rubric that I used, uh, used, uh, used, uh, it was exciting in it was exciting in it was exciting in very boring way, very boring way, very boring way, so I included it. Uh, so I included it. Uh, so I included it. Uh, I threw mine away. I threw mine away. I threw mine away. examples. And regarding examples. And regarding examples. And regarding examples, judge examples, judge examples, judge agreed with me, uh agreed with me, uh , 15, 15 and 15 times out of 16 in , 15, 15 and 15 times out of 16 in , 15, 15 and 15 times out of 16 in three different attempts. three different attempts. three different attempts. Uh, and if I rejected Uh, and if I rejected Uh, and if I rejected examples, he examples, he examples, he agreed with me 16 agreed with me 16 agreed with me 16 times twice, all 16 times twice, all 16 times twice, all 16 examples and 15 times examples and 15 times examples and 15 times once. Uh, me too once. Uh, me too once. Uh, me too It was very interesting how It was very interesting how It was very interesting how discarding examples discarding examples discarding examples improved the situation. THERE ARE improved the situation. THERE ARE -e, and specific -e, and specific -e, and specific the problem that arose the problem that arose , was that in , was that in , was that in one of the reports is not one of the reports is not one of the reports is not was, uh, wasn't there was, uh, wasn't there was, uh, wasn't there list of risks. So list of risks. So as a person, I as a person, I as a person, I read the report, and there read the report, and there read the report, and there it was written that it was written that it was written that Microsoft is a purchase, and Microsoft is a purchase, and Microsoft is a purchase, and it did not include any it did not include any it did not include any risks, and I thought: risks, and I thought: risks, and I thought: "Well, it doesn't matter."

"Well, it doesn't matter." "Well, it doesn't matter." basis for a lawsuit." basis for a lawsuit." basis for a lawsuit." It included a purchase. It included a purchase. It included a purchase. Uh, that's why I marked it. Uh, that's why I marked it. Uh, that's why I marked it. its like, uh, a foundation its like, uh, a foundation its like, uh, a foundation for a lawsuit. Uh, but in for a lawsuit. Uh, but in for a lawsuit. Uh, but in my examples were my examples were my examples were it is said that you it is said that you it is said that you should include should include should include risks. And therefore the agent, risks. And therefore the agent, risks. And therefore the agent, uh, judge LLM, looked uh, judge LLM, looked uh, judge LLM, looked looked at the rules looked at the rules looked at the rules for example, and said, for example, and said, for example, and said, what should be here what should be here what should be here risks. So it's not risks. So it's not risks. So it's not grounds for a claim, grounds for a claim, grounds for a claim, because there are no risks. So because there are no risks. So because there are no risks. So it is really open it is really open it is really open the question of whether the question of whether the question of whether my man...Judgment my man...Judgment my man...Judgment was better or was better or was better or worse than that of judges. worse than that of judges. worse than that of judges. The judge was very The judge was very The judge was very strict, and I was a little strict, and I was a little strict, and I was a little loose, when I, like loose, when I, like loose, when I, like man, gave himself man, gave himself man, gave himself golden data. golden data. golden data. Excuse me, can I? Excuse me, can I? Excuse me, can I? ask a question? ask a question? ask a question? Of course. Of course. Of course. So what is your hypothesis? So what is your hypothesis? So what is your hypothesis? regarding that without regarding that without regarding that without examples she examples she examples she agreed with you, agreed with you, agreed with you, so she thought so she thought so she thought Is Microsoft effective? Is Microsoft effective? Yes. Yes. This is your hypothesis that This is your hypothesis that This is your hypothesis that just because you don't just because you don't just because you don't said that there is a risk said that there is a risk said that there is a risk a necessary element a necessary element , she considered it , she considered it , she considered it good and effective good and effective good and effective element? I have on element? I have on element? I have on mean that it is not mean that it is not mean that it is not hypothesis, as in hypothesis, as in hypothesis, as in case of the LLM judge, who case of the LLM judge, who case of the LLM judge, who comes from comes from comes from explanation. So explanation. So explanation. So he says it was he says it was he says it was an active element, or an active element, or an active element, or wrong?

wrong? Yes. So, as I already Yes. So, as I already said, in this said, in this said, in this case really case really case really It is unclear whether I It is unclear whether I It is unclear whether I did you do the right thing? did you do the right thing? did you do the right thing? the judge did the judge did the judge did right. Judge right. Judge right. Judge decided very strictly decided very strictly decided very strictly interpret interpret interpret rubric, and I, when rubric, and I, when rubric, and I, when marked these things as marked these things as marked these things as effective for humans or effective for humans or effective for humans or no, uh, it wasn't like that no, uh, it wasn't like that no, uh, it wasn't like that strict. Uh, here it is. strict. Uh, here it is. strict. Uh, here it is. example of an effective example of an effective example of an effective example. Uh, in it example. Uh, in it example. Uh, in it there are specific numbers. IN there are specific numbers. IN there are specific numbers. IN there is growth there is growth there is growth income. Uh, he has income. Uh, he has income. Uh, he has assessment compared to assessment compared to assessment compared to your sector. He your sector. He your sector. He calls a specific calls a specific calls a specific risk. And he risk. And he risk. And he ends ends ends recommendation that you recommendation that you ...it actually could be ...it actually could be ...it actually could be consequence, including consequence, including consequence, including cases when you cases when you cases when you change your mind. Hmm, and here it is change your mind. Hmm, and here it is change your mind. Hmm, and here it is what does it look like what does it look like what does it look like inoperative option inoperative option . He is not wrong. . He is not wrong. . He is not wrong. Everything in it is true. Everything in it is true. Everything in it is true. He's just not very He's just not very He's just not very vain. Investors vain. Investors vain. Investors must take into account must take into account must take into account various factors various factors various factors factions, it's not factions, it's not factions, it's not recommendation. Ahem, recommendation. Ahem, recommendation. Ahem, now that the judge now that the judge now that the judge saw both, saw both, saw both, we hope that he we hope that he we hope that he there will be more there will be more there will be more consistent in reporting, consistent in reporting, consistent in reporting, which are located somewhere which are located somewhere which are located somewhere in the middle. Um, and whether in the middle. Um, and whether in the middle. Um, and whether is that really true, you is that really true, you is that really true, you you will find out, you will find out, you will find out, protesting. Ahem, protesting. Ahem, protesting. Ahem, fifth part fifth part fifth part rubric creation rubric creation rubric creation is that you is that you is that you limit the answer.

limit the answer. limit the answer. You ask her about You ask her about You ask her about things that are effective things that are effective things that are effective or incapacitated. or incapacitated. or incapacitated. Previously you need Previously you need Previously you need was to say to withdraw was to say to withdraw was to say to withdraw line, and the line has line, and the line has line, and the line has to be a word of action to be a word of action to be a word of action or incapacitated. Ahem, or incapacitated. Ahem, or incapacitated. Ahem, you no longer you no longer you no longer need this need this need this do. what, um, do. what, um, do. what, um, modern appraisers modern appraisers modern appraisers use use use tool calls tool calls , um, to get your , um, to get your , um, to get your result. So, these, result. So, these, result. So, these, um, options, um, options, um, options, available for this available for this available for this tool call, tool call, tool call, defined in the call defined in the call defined in the call instrument, not in instrument, not in instrument, not in your column. Hmm, what your column. Hmm, what your column. Hmm, what makes them much makes them much makes them much more reliable. So more reliable. So more reliable. So now it's your turn now it's your turn now it's your turn write a column. Ahem, write a column. Ahem, write a column. Ahem, evaluationability.ts is open. evaluationability.ts is open. evaluationability.ts is open. Uh, there's my column there. Uh, there's my column there. Uh, there's my column there. with all five with all five with all five in parts. Uh, you too in parts. Uh, you too in parts. Uh, you too can you do it can you do it can you do it with your own. You can with your own. You can with your own. You can change the criteria. You change the criteria. You change the criteria. You you can change you can change you can change examples. Uh, what? examples. Uh, what? examples. Uh, what? will force you to act on will force you to act on will force you to act on based on one of these based on one of these based on one of these reports? What did you see in reports? What did you see in reports? What did you see in in your own footsteps? in your own footsteps? in your own footsteps? Write it down and it should Write it down and it should Write it down and it should become your column. become your column. become your column. Uh, it'll take three- Uh, it'll take three- four minutes. Whom four minutes. Whom four minutes. Whom managed to catch up with managed to catch up with managed to catch up with me yet? Who is on me yet? Who is on me yet? Who is on now? Perfectly.

now? Perfectly. now? Perfectly. I have several hands. Uh, I have several hands. Uh, I have several hands. Uh, as I said, as I said, as I said, last time I last time I last time I conducted this seminar, conducted this seminar, conducted this seminar, everyone was completely everyone was completely everyone was completely confused after 200- confused after 200- megabyte megabyte megabyte repository. So that one repository. So that one repository. So that one the fact that someone is the fact that someone is the fact that someone is I got it, just I got it, just I got it, just great for me. Uh, great for me. Uh, great for me. Uh, this is how you do it this is how you do it this is how you do it connect. You connect. You connect. You create an estimator create an estimator create an estimator classification, which classification, which classification, which performs all performs all performs all plumbing. Uh, you plumbing. Uh, you plumbing. Uh, you you give it a name, you you give it a name, you you give it a name, you give him a model. Uh-huh give him a model. Uh-huh , you give him your , you give him your , you give him your template. Uh, and how am I? template. Uh, and how am I? template. Uh, and how am I? said, uh, out of the zone said, uh, out of the zone said, uh, out of the zone you give it to him you give it to him you give it to him options. So you options. So you options. So you you give him a choice, you give him a choice, you give him a choice, which is effective with which is effective with which is effective with with a score of one and no with a score of one and no with a score of one and no effective with a score of zero effective with a score of zero . Uh, so let's go. . Uh, so let's go. . Uh, so let's go. let's look at let's look at let's look at practicality in practicality in practicality in Phoenix. Here it is Phoenix. Here it is Phoenix. Here it is It's so cold that I'm freezing. It's so cold that I'm freezing. It's so cold that I'm freezing. My fingers went numb, and I My fingers went numb, and I My fingers went numb, and I I can't normally I can't normally I can't normally print. OK.

print. OK. So, the report analyzes So, the report analyzes JPM rate risk and JPM rate risk and JPM rate risk and credit growth, e credit growth, e -e, and predictive -e, and predictive -e, and predictive results, but not results, but not results, but not gives recommendations gives recommendations gives recommendations regarding the purchase, regarding the purchase, regarding the purchase, maintenance, sales. maintenance, sales. maintenance, sales. Fantasy. He Fantasy. He Fantasy. He does exactly what it's supposed to do does exactly what it's supposed to do does exactly what it's supposed to do do. Uh, he do. Uh, he do. Uh, he complains about, uh, complains about, uh, complains about, uh, absence absence absence recommendations regarding recommendations regarding recommendations regarding specific specific specific recommendations. Uh, recommendations. Uh, before we before we let's consider all these let's consider all these let's consider all these recommendations, we recommendations, we recommendations, we should look at should look at should look at distribution distribution distribution recommendations. Uh, for recommendations. Uh, for recommendations. Uh, for according to my data, five of according to my data, five of according to my data, five of 16 reports were 16 reports were 16 reports were practical, and 11 were practical, and 11 were practical, and 11 were considered as not considered as not considered as not are subject to execution are subject to execution . Uh, and this is one of . Uh, and this is one of . Uh, and this is one of the most useful types the most useful types the most useful types ratings that you can ratings that you can ratings that you can get. Assessment where get. Assessment where get. Assessment where everything is perfect, no everything is perfect, no everything is perfect, no especially useful. EVA, especially useful. EVA, especially useful. EVA, where everything is wrong, where everything is wrong, where everything is wrong, is also not useful. is also not useful. is also not useful. EVA, where you have an equal EVA, where you have an equal EVA, where you have an equal distribution, which is distribution, which is distribution, which is the hill you are on the hill you are on the hill you are on You can get up. You You can get up. You You can get up. You suddenly you get, uh, suddenly you get, uh, suddenly you get, uh, some, you have, some, you have, some, you have, definitely a set definitely a set definitely a set examples where something examples where something examples where something actually works, actually works, actually works, a set of things where, uh, a set of things where, uh, a set of things where, uh, something is not working and you something is not working and you something is not working and you you can start you can start you can start to find out to find out to find out differences between them.

differences between them. differences between them. Uh-huh. So, if you Uh-huh. So, if you Uh-huh. So, if you click on these click on these click on these tracings, I don't tracings, I don't tracings, I don't I'm going to worry. I'm going to worry. I'm going to worry. about this. Rivian says about this. Rivian says that he never gives that he never gives that he never gives recommendations regarding recommendations regarding recommendations regarding purchase, maintenance, purchase, maintenance, purchase, maintenance, sale or any sale or any sale or any equivalent equivalent equivalent next step. For next step. For next step. For Nvidia, risk level Nvidia, risk level Nvidia, risk level investment thesis investment thesis investment thesis in conclusion there is in conclusion there is in conclusion there is moderate, this moderate, this moderate, this vague assessment, vague assessment, vague assessment, not a recommendation. AND not a recommendation. AND not a recommendation. AND for Meta, er, verdict for Meta, er, verdict for Meta, er, verdict is only that, is only that, is only that, what is the position what is the position what is the position stable. So this is it stable. So this is it stable. So this is it the same problem again and the same problem again and the same problem again and again. He again. He again. He refuses to give refuses to give refuses to give specific specific specific recommendations regarding recommendations regarding recommendations regarding buying, selling, buying, selling, buying, selling, maintenance. Um, if we maintenance. Um, if we uh, we can open uh, we can open uh, we can open the Phoenix query template. I the Phoenix query template. I the Phoenix query template. I I'm not going to do this. I'm not going to do this. I'm not going to do this. do it now. Hmm, but, do it now. Hmm, but, do it now. Hmm, but, uh, there's another rule uh, there's another rule uh, there's another rule for rubrics, which for rubrics, which for rubrics, which is that you is that you is that you should consider should consider should consider eval requests as code. Yours eval requests as code. Yours eval requests as code. Yours eval queries that eval queries that eval queries that coming to your coming to your coming to your user user user evaluations, extremely evaluations, extremely evaluations, extremely sensitive to minor sensitive to minor sensitive to minor changes in wording. changes in wording. changes in wording. Um, so, uh, one Um, so, uh, one Um, so, uh, one example with which I example with which I example with which I collided, creating collided, creating collided, creating it was, uh, I said it was, uh, I said it was, uh, I said you, what you should you, what you should you, what you should place XML tags place XML tags place XML tags around things. I around things. I around things. I forgot to post forgot to post forgot to post XML tags around your XML tags around your XML tags around your examples for examples for examples for beginning. Uh, and with mine beginning. Uh, and with mine beginning. Uh, and with mine examples like examples like examples like plain text, e- plain text, e- uh, I got the line uh, I got the line uh, I got the line useless things, uh, useless things, uh, useless things, uh, because he started because he started because he started to get confused between data to get confused between data to get confused between data and examples. And when?

and examples. And when? and examples. And when? I wrapped them with tags. I wrapped them with tags. I wrapped them with tags. XML, everything suddenly became XML, everything suddenly became XML, everything suddenly became practical, because it practical, because it practical, because it could understand what it was about could understand what it was about could understand what it was about I am speaking. Um, so I am speaking. Um, so I am speaking. Um, so There were no other changes. I There were no other changes. I There were no other changes. I every time I gave him those every time I gave him those every time I gave him those the same data, the same the same data, the same the same data, the same instructions, the same instructions, the same instructions, the same input data and those input data and those input data and those the original data itself. Uh-huh the original data itself. Uh-huh , just in the estimator , just in the estimator , just in the estimator became less became less became less to be confused about to be confused about to be confused about What's going on? Uh, What's going on? Uh, What's going on? Uh, so you need so you need so you need to consider their to consider their to consider their evaluations as code. You evaluations as code. You evaluations as code. You should them should them should them versioning, versioning, versioning, check and check and check and test. Uh, how am I? test. Uh, how am I? test. Uh, how am I? I already mentioned before, I already mentioned before, I already mentioned before, there is a trap you can fall into there is a trap you can fall into there is a trap you can fall into to get out of to get out of to get out of evaluators, namely evaluators, namely evaluators, namely try to write try to write try to write one evaluator who one evaluator who one evaluator who does everything for you. He does everything for you. He does everything for you. He is called god evaluator, is called god evaluator, is called god evaluator, because it is like this because it is like this because it is like this common scheme. Uh, common scheme. Uh, common scheme. Uh, you are trying you are trying you are trying to force him to force him to force him check accuracy, check accuracy, check accuracy, tone, fullness, tone, fullness, tone, fullness, formatting, formatting, formatting, politics all in one politics all in one politics all in one request. Uh, and this request. Uh, and this request. Uh, and this nightmare, because if it's not nightmare, because if it's not nightmare, because if it's not it turns out that it is not it turns out that it is not it turns out that it is not Is it possible? Was it? Is it possible? Was it? Is it possible? Was it? inaccurate? Was there a tone? inaccurate? Was there a tone? inaccurate? Was there a tone? wrong? Was there wrong? Was there wrong? Was there loyalty loyalty loyalty wrong? What about wrong? What about wrong? What about many, many many, many many, many the things you said the things you said the things you said this god appraiser this god appraiser this god appraiser assesses whether it is assesses whether it is assesses whether it is actually actually actually wrong, and you wrong, and you wrong, and you trying to decide trying to decide trying to decide this, giving him 15 marks this, giving him 15 marks which he can which he can which he can spit it out, and all that spit it out, and all that spit it out, and all that it is getting very difficult it is getting very difficult it is getting very difficult track. Um, track. Um, track. Um, best practice— best practice— to divide it, to divide it, to divide it, to share one's own to share one's own to share one's own appraisers on appraisers on appraisers on so many dimensions, so many dimensions, so many dimensions, How much do you test?

How much do you test? How much do you test? So, one for So, one for So, one for reliability, one reliability, one reliability, one for correctness, for correctness, for correctness, one for one for one for practicality. Um, you too practicality. Um, you too practicality. Um, you too should also know, should also know, should also know, which one of yours which one of yours which one of yours there are appraisers there are appraisers there are appraisers protective strip, and protective strip, and protective strip, and which is northern which is northern which is northern star. So, protective star. So, protective star. So, protective The stripe is what has The stripe is what has The stripe is what has block the release. block the release. block the release. So, if the agent So, if the agent So, if the agent invents the stock price, invents the stock price, invents the stock price, this should stop this should stop this should stop delivery. Um, but delivery. Um, but delivery. Um, but the northern stars—that's what the northern stars—that's what , what you strive for. , what you strive for. , what you strive for. So, offers other So, offers other So, offers other stocks worth buying stocks worth buying stocks worth buying to pay attention, was to pay attention, was to pay attention, was It would be good to have. It was It would be good to have. It was It would be good to have. It was It would be good to include. It would be good to include. It would be good to include. Your rating may Your rating may Your rating may to look for it, but she doesn't to look for it, but she doesn't to look for it, but she doesn't necessarily limits necessarily limits necessarily limits delivery on this. What delivery on this. What delivery on this. What leads us to leads us to leads us to step six, which is step six, which is step six, which is testing your testing your testing your tests. We created tests. We created tests. We created judge, and he says that judge, and he says that judge, and he says that some of our reports some of our reports some of our reports are not practical. Um, are not practical. Um, are not practical. Um, but can we but can we but can we trust the judge? How are we? trust the judge? How are we? trust the judge? How are we? we know that the judge we know that the judge we know that the judge right? Um, trickery, right? Um, trickery, right? Um, trickery, which one is needed which one is needed which one is needed remember, is to remember, is to remember, is to because what is your judge because what is your judge —this is a classifier in —this is a classifier in —this is a classifier in in the classical sense in the classical sense in the classical sense machine learning. THERE ARE machine learning. THERE ARE -he is looking at -he is looking at -he is looking at report and forecast report and forecast report and forecast a label as one that can a label as one that can a label as one that can to be effective or not. to be effective or not. to be effective or not. Forecasts, uh, can Forecasts, uh, can Forecasts, uh, can to be correct or to be correct or to be correct or incorrect. So, incorrect. So, incorrect. So, like any other like any other like any other classifier, you classifier, you classifier, you measure it, measure it, measure it, comparing it comparing it comparing it predictions with truth. THERE ARE- predictions with truth. THERE ARE- Well, the truth is this, Well, the truth is this, Well, the truth is this, what would an expert say- what would an expert say- man. You can man. You can man. You can judge, uh, you can judge, uh, you can judge, uh, you can to evaluate homework to evaluate homework to evaluate homework the judge's task. Uh, in the judge's task. Uh, in the judge's task. Uh, in in this case in this case in this case human expert human expert human expert I will be. I will I will be. I will I will be. I will to decide whether it is worth it to decide whether it is worth it to decide whether it is worth it buy, sell buy, sell buy, sell whether to hold shares. Uh, whether to hold shares. Uh, whether to hold shares. Uh, not necessarily that one not necessarily that one not necessarily that one the expert you are the expert you are the expert you are you want. Uh, but we you want. Uh, but we you want. Uh, but we We are doing everything possible.

We are doing everything possible. We are doing everything possible. It's simple. It's simple. It's simple. demonstration. Uh, how? demonstration. Uh, how? demonstration. Uh, how? to do this, very to do this, very to do this, very simply. You simply. You simply. You open open open tracing. Uh, you tracing. Uh, you tracing. Uh, you press the small press the small press the small pencil button and pencil button and pencil button and you can add you can add you can add annotations. So, I annotations. So, I annotations. So, I pre-created pre-created pre-created an annotation entitled " an annotation entitled " Effective for humans, Effective for humans, Effective for humans, uh, and I can uh, and I can uh, and I can set it as set it as set it as active or inactive, and I active or inactive, and I active or inactive, and I I can do it very quickly, uh, I can do it very quickly, uh, I can do it very quickly, uh, click on all click on all click on all on their routes and on their routes and on their routes and set this for set this for set this for each of them that I each of them that I each of them that I made in advance. THERE ARE made in advance. THERE ARE -uh, here you are -uh, here you are -uh, here you are attracted, uh, attracted, uh, attracted, uh, expert on expert on expert on subject area. subject area. subject area. You would throw them in You would throw them in You would throw them in Phoenix. You would say, Phoenix. You would say, Phoenix. You would say, that here are 10 tracks, here are 20 that here are 10 tracks, here are 20 that here are 10 tracks, here are 20 trails, here are 50 trails. trails, here are 50 trails. trails, here are 50 trails. Review and Review and Review and read the result read the result read the result and give me yours and give me yours and give me yours golden data set golden data set golden data set about whether these things about whether these things about whether these things good. Um, I sat down and good. Um, I sat down and good. Um, I sat down and correctly marked correctly marked correctly marked all 16 of their reports for all 16 of their reports for all 16 of their reports for the same criteria, the same criteria, the same criteria, which I gave to the judge. Um, and which I gave to the judge. Um, and which I gave to the judge. Um, and they lie in your they lie in your they lie in your the initial project, the initial project, the initial project, so that you don't have to so that you don't have to so that you don't have to to do this, because it to do this, because it to do this, because it it would take a lot of time it would take a lot of time . Um, what you . Um, what you . Um, what you create when create when create when Do it like I did.

Do it like I did. Do it like I did. said, it's called said, it's called said, it's called gold set gold set gold set data. Um, these are examples. data. Um, these are examples. , where, as far as is known , where, as far as is known , where, as far as is known humanity, you know that humanity, you know that humanity, you know that The answers are correct. The answers are correct. The answers are correct. Um, that's a lot of work, Um, that's a lot of work, Um, that's a lot of work, therefore you must therefore you must therefore you must to make this to make this to make this work was taken into account work was taken into account . So you must . So you must . So you must to establish for oneself to establish for oneself to establish for oneself the same specific the same specific the same specific criteria that you criteria that you criteria that you appointed judges. It appointed judges. It appointed judges. It there was a mistake that there was a mistake that there was a mistake that we talked before. we talked before. we talked before. Um, I'm not exactly Um, I'm not exactly Um, I'm not exactly understood the rules that understood the rules that understood the rules that I installed in my I installed in my I installed in my evaluation criteria. evaluation criteria. evaluation criteria. Right? I Right? I Right? I violated one of the violated one of the violated one of the their own rules. their own rules. their own rules. My rule said, My rule said, My rule said, which, uh, should include which, uh, should include which, uh, should include risks, and I said that risks, and I said that risks, and I said that this report is normal, this report is normal, this report is normal, although it...doesn't include although it...doesn't include although it...doesn't include risks. Uh, and in risks. Uh, and in risks. Uh, and in As a result, we As a result, we As a result, we The judge and the jury were divided, and The judge and the jury were divided, and The judge and the jury were divided, and it seems to me that it seems to me that it seems to me that the judge was actually the judge was actually the judge was actually more right than me. more right than me. more right than me. He was stricter. Uh-huh He was stricter. Uh-huh , so, uh, your enemy , so, uh, your enemy , so, uh, your enemy here is fatigue. Uh, if here is fatigue. Uh, if here is fatigue. Uh, if you will give yourself a hundred you will give yourself a hundred you will give yourself a hundred trails to evaluate trails to evaluate trails to evaluate off route 50, you will be off route 50, you will be off route 50, you will be It's very boring to evaluate. It's very boring to evaluate. It's very boring to evaluate. tracks, and you will become tracks, and you will become tracks, and you will become lazier about it. lazier about it. lazier about it. Uh, so it's better to have 50 Uh, so it's better to have 50 Uh, so it's better to have 50 really good really good really good marked, uh, marked, uh, marked, uh, examples than a hundred, examples than a hundred, examples than a hundred, where you just clicked where you just clicked where you just clicked and they said: "Ah, this and they said: "Ah, this and they said: "Ah, this "Looks good." Uh, "Looks good." Uh, "Looks good." Uh, the higher the quality of the data the higher the quality of the data the higher the quality of the data in your gold in your gold in your gold dataset, uh, so dataset, uh, so dataset, uh, so everything will be more reliable.

everything will be more reliable. Uh, and one of Uh, and one of the best things that the best things that the best things that you can do this you can do this you can do this constantly add to constantly add to constantly add to your golden one your golden one your golden one data set. When you data set. When you data set. When you you see complicated you see complicated you see complicated example when you example when you example when you Do you see something strange? Do you see something strange? Do you see something strange? takes place in takes place in takes place in production, find out production, find out , which should have been , which should have been , which should have been answer, and answer, and answer, and put it in your put it in your put it in your golden data set. golden data set. golden data set. Uh, a few good ones Uh, a few good ones Uh, a few good ones rules for human rules for human rules for human search for good search for good search for good gold set gold set gold set data consist of data consist of data consist of because you have to because you have to because you have to to write tests to write tests to write tests tasks that a person tasks that a person tasks that a person could pass. Uh, could pass. Uh, could pass. Uh, if your agent every time if your agent every time if your agent every time does not perform the task, does not perform the task, does not perform the task, it almost always will be it almost always will be it almost always will be faulty. task, faulty. task, faulty. task, not a broken agent. not a broken agent. not a broken agent. Um, and for everyone Um, and for everyone Um, and for everyone You have a task. You have a task. You have a task. be the answer, which, be the answer, which, be the answer, which, in your opinion, is in your opinion, is in your opinion, is good, which proves, good, which proves, good, which proves, that the task can be that the task can be that the task can be perform. Um, you perform. Um, you perform. Um, you should also be tested should also be tested should also be tested in both directions. in both directions. in both directions. If you check, If you check, If you check, for example, only that for example, only that for example, only that agent is looking for agent is looking for agent is looking for Internet when he Internet when he Internet when he should look in should look in should look in Internet, you Internet, you Internet, you accidentally teach accidentally teach accidentally teach look for your agent in look for your agent in look for your agent in Internet every time, Internet every time, Internet every time, constantly, even when constantly, even when constantly, even when This is not necessary. Um, and This is not necessary. Um, and This is not necessary. Um, and here is the rule here is the rule here is the rule machine learning, machine learning, machine learning, which is applied which is applied which is applied also to the judges. Not also to the judges. Not also to the judges. Not customize yours customize yours customize yours test set. Um, test set. Um, test set. Um, share your tags share your tags share your tags into two groups. One, into two groups. One, into two groups. One, which one are you which one are you which one are you use for use for use for settings of your settings of your settings of your grades, and one that you grades, and one that you grades, and one that you use for use for use for checking their checking their checking their work. Um, when in work. Um, when in work. Um, when in your golden one your golden one your golden one There are only 16 in the dataset There are only 16 in the dataset There are only 16 in the dataset answers, it does answers, it does answers, it does your test set your test set your test set very small Um, so very small Um, so very small Um, so what is the rule usually what is the rule usually what is the rule usually is: 75% is: 75% is: 75% customized set customized set , 25%in the test set , 25%in the test set . So in our . So in our . So in our their test set their test set their test set only four. Um, but only four. Um, but only four. Um, but if you were to 100, that if you were to 100, that if you were to 100, that It would be nice. Um, It would be nice. Um, It would be nice. Um, Phoenix datasets Phoenix datasets Phoenix datasets have built-in have built-in have built-in separation. So this is separation. So this is separation. So this is very simple. make.

very simple. make. very simple. make. Uh, look how it is Uh, look how it is Uh, look how it is looks. uh, looks. uh, looks. uh, first he creates first he creates first he creates golden data set with golden data set with golden data set with divisions. He divisions. He divisions. He finds every report finds every report finds every report with a human mark. Uh, and with a human mark. Uh, and with a human mark. Uh, and each of them becomes each of them becomes each of them becomes example in the set example in the set example in the set Phoenix data. Incoming Phoenix data. Incoming Phoenix data. Incoming The data is a report. The data is a report. The data is a report. Expected Expected Expected the result is a label the result is a label the result is a label human. We sort human. We sort human. We sort them for them for them for identifier identifier identifier range. Then range. Then range. Then every fourth every fourth every fourth the example falls into the example falls into the example falls into test division, and test division, and test division, and the rest is under development. Uh, the rest is under development. Uh, the rest is under development. Uh, sorting first sorting first sorting first means that the report means that the report means that the report always gets into always gets into always gets into one and the same one and the same one and the same division, how much division, how much division, how much times you don't times you don't times you don't launched. Uh, and launched. Uh, and launched. Uh, and identifier identifier identifier range connects range connects range connects each example with each example with each example with tracing from where tracing from where tracing from where he comes from. Uh, so he comes from. Uh, so he comes from. Uh, so you can always you can always you can always go to go to go to initial launch. initial launch. Then we launch Then we launch judge against the people. It judge against the people. It judge against the people. It the experiment we the experiment we the experiment we let's take a look let's take a look let's take a look later. Uh, but later. Uh, but later. Uh, but now, uh, now, uh, now, uh, experiment accepts experiment accepts experiment accepts data set. Needed data set. Needed data set. Needed task for task for task for launch on each launch on each launch on each example, uh, yes example, uh, yes example, uh, yes evaluators for evaluators for evaluators for evaluation evaluation evaluation results. results. results. The task here is ours The task here is ours The task here is ours judge, and the evaluator is judge, and the evaluator is judge, and the evaluator is tiny val Kodi, which tiny val Kodi, which tiny val Kodi, which checks whether checks whether checks whether the label corresponds the label corresponds the label corresponds Judges mark people. Uh-huh Judges mark people. Uh-huh , please note that , please note that , please note that we are only launching this we are only launching this we are only launching this in development in development in development division. Uh, you can.

division. Uh, you can. division. Uh, you can. run this run this run this independently, I launch independently, I launch independently, I launch npm run meta. Let's see, npm run meta. Let's see, npm run meta. Let's see, can we do it can we do it can we do it launch. He launch. He launch. He launch LM. So, uh, launch LM. So, uh, launch LM. So, uh, I assume that he I assume that he I assume that he does the same thing as does the same thing as does the same thing as last night. Ahem, last night. Ahem, last night. Ahem, the script should output the script should output the script should output a small table in a small table in a small table in ends. Uh, and in my 12 ends. Uh, and in my 12 ends. Uh, and in my 12 developer reports developer reports developer reports The judge and I every time The judge and I every time The judge and I every time did the same thing did the same thing did the same thing the most challenging thing the most challenging thing the most challenging thing great, but this is also great, but this is also great, but this is also only 12 reports. Uh, so only 12 reports. Uh, so only 12 reports. Uh, so take it as, take it as, take it as, you know, not yet you know, not yet you know, not yet signs of a problem. You signs of a problem. You signs of a problem. You will need a bigger one will need a bigger one will need a bigger one a set of tests when you a set of tests when you a set of tests when you will run this in will run this in will run this in produced. Uh, cool. produced. Uh, cool. . He actually . He actually . He actually completed it. Uh, completed it. Uh, completed it. Uh, the most interesting part- the most interesting part- these are the two ports that I these are the two ports that I these are the two ports that I I consider the most difficult I consider the most difficult I consider the most difficult for marking. Uh, for marking. Uh, so in my data so in my data Phoenix is ​​Coca-Cola and is Microsoft.

Phoenix is ​​Coca-Cola and is Microsoft. Uh, first one is Coca-Cola. Coca-Cola- this is the one that changed, this is the one that changed, this is the one that changed, uh, when I changed uh, when I changed uh, when I changed XML tags on, uh, from XML tags on, uh, from XML tags on, uh, from lack of XML tags lack of XML tags lack of XML tags to the presence of XML tags. to the presence of XML tags. to the presence of XML tags. Uh, right here, when he Uh, right here, when he Uh, right here, when he got examples, everything got examples, everything got examples, everything It got better. Um, for It got better. Um, for It got better. Um, for I am a border guard. I am a border guard. I am a border guard. the option turns out the option turns out the option turns out also border. IN also border. IN also border. IN KO report calls KO report calls KO report calls ideal option ideal option ideal option for conservative for conservative for conservative portfolios, portfolios, portfolios, oriented towards oriented towards oriented towards income, which sounds like income, which sounds like income, which sounds like recommendation. So, how recommendation. So, how recommendation. So, how man, I hesitated, man, I hesitated, man, I hesitated, should I consider this should I consider this should I consider this recommendation. recommendation. recommendation. In the end, I decided that In the end, I decided that In the end, I decided that no, but the judge too no, but the judge too no, but the judge too hesitates whether to hesitates whether to hesitates whether to to consider it to consider it to consider it sufficient sufficient sufficient recommendation. Hmm, ah recommendation. Hmm, ah recommendation. Hmm, ah the second was Microsoft, on the second was Microsoft, on the second was Microsoft, on which I am not going to which I am not going to which I am not going to to press, which is the opposite to press, which is the opposite . The report says . The report says . The report says buy with reasons, buy with reasons, buy with reasons, um, yes, hourly um, yes, hourly um, yes, hourly horizon. Hmm, and there horizon. Hmm, and there horizon. Hmm, and there never mentioned never mentioned never mentioned no risk. It no risk. It no risk. It the one I'm talking about the one I'm talking about the one I'm talking about spoke earlier. Um, in spoke earlier. Um, in spoke earlier. Um, in my columns my columns my columns specific ones are named specific ones are named specific ones are named risks as a criterion.

risks as a criterion. risks as a criterion. Um, and I decided that this Um, and I decided that this Um, and I decided that this okay, but the judge okay, but the judge okay, but the judge decided that it was not. There decided that it was not. There decided that it was not. There it was said that it was said that it was said that the risks are low. the risks are low. the risks are low. Risk section Risk section Risk section no. This gives all no. This gives all no. This gives all grounds for grounds for grounds for application, but not application, but not application, but not lot. So judge, lot. So judge, lot. So judge, who considered this who considered this who considered this material, arrived material, arrived material, arrived exactly my human exactly my human exactly my human judgment about whether judgment about whether judgment about whether is this report is this report is this report the basis for the basis for the basis for application, application, application, including including including ambiguity. Ahem, ambiguity. Ahem, ambiguity. Ahem, and this ambiguity and this ambiguity and this ambiguity still in our still in our still in our rubrics, because we still have it rubrics, because we still have it rubrics, because we still have it did not change. Uh, so did not change. Uh, so did not change. Uh, so This is very tempting. This is very tempting. This is very tempting. to fix. So I to fix. So I to fix. So I tried to add tried to add tried to add one sentence, uh, in one sentence, uh, in one sentence, uh, in which was stated, which was stated, which was stated, that the report is valid only that the report is valid only that the report is valid only if he answers if he answers if he answers all four all four all four criteria, because before criteria, because before criteria, because before it was ambiguous. it was ambiguous. it was ambiguous. I said that the report I said that the report I said that the report must contain all must contain all must contain all these things. I didn't say that. these things. I didn't say that. these things. I didn't say that. he must he must he must should. Uh, you should. Uh, you should. Uh, you you can change you can change you can change rubric and be more rubric and be more rubric and be more specific and specific and specific and to say that this to say that this to say that this absolutely absolutely absolutely must have must have must have contain all four of these contain all four of these contain all four of these criteria. Uh, and on criteria. Uh, and on criteria. Uh, and on development, uh, this development, uh, this development, uh, this Microsoft broke mine Microsoft broke mine Microsoft broke mine consent 12 out of 12 became consent 12 out of 12 became consent 12 out of 12 became by agreement 11 out of 12. Uh, and, uh by agreement 11 out of 12. Uh, and, uh , it also broke Apple, , it also broke Apple, , it also broke Apple, uh, because Apple doesn't uh, because Apple doesn't uh, because Apple doesn't it was enough, uh, it was enough, uh, it was enough, uh, big enough big enough big enough the risk section, which the risk section, which the risk section, which concerns the agent, concerns the agent, concerns the agent, judges. Uh, this is judges. Uh, this is judges. Uh, this is tells you that tells you that tells you that what can I change what can I change what can I change your column and your column and your column and to worsen one's to worsen one's to worsen one's assessment. Uh, I can.

assessment. Uh, I can. assessment. Uh, I can. make your assessment make your assessment make your assessment less useful. I can less useful. I can less useful. I can to make your free to make your free to make your free accessibility less accessibility less accessibility less similar to that similar to that similar to that a person would do. Uh, a person would do. Uh, a person would do. Uh, although I try to be although I try to be although I try to be specific. Uh, specific. Uh, specific. Uh, the script also prints the script also prints the script also prints accuracy and completeness. accuracy and completeness. accuracy and completeness. Um, accuracy, that's very Um, accuracy, that's very Um, accuracy, that's very important things, again important things, again important things, again yes, borrowed from yes, borrowed from yes, borrowed from machine world machine world machine world learning, or rather, learning, or rather, learning, or rather, from the world of statistics. from the world of statistics. from the world of statistics. Um, accuracy asks, Um, accuracy asks, Um, accuracy asks, when the judge says that it is when the judge says that it is when the judge says that it is not subject to measures, not subject to measures, not subject to measures, how often is this correct how often is this correct ? And the recall ? And the recall ? And the recall asks about all reports asks about all reports , which are not subject to , which are not subject to , which are not subject to events, how many events, how many events, how many did the judge find? These two did the judge find? These two did the judge find? These two the concepts sound very the concepts sound very the concepts sound very similar, but measured similar, but measured similar, but measured various things. Um, in various things. Um, in various things. Um, in in most cases in most cases in most cases you will need you will need you will need give priority give priority give priority recall, because recall, because recall, because the recall gives you the recall gives you the recall gives you that he reveals everything that he reveals everything , but it gives you , but it gives you , but it gives you high recall, high recall, high recall, reveals everything, but he reveals everything, but he reveals everything, but he will give you, uh, false will give you, uh, false will give you, uh, false anxiety. False anxiety. False anxiety. False Anxiety costs you Anxiety costs you Anxiety costs you a minute or two when you a minute or two when you a minute or two when you look at the report, look at the report, look at the report, which turns out to be which turns out to be which turns out to be normal. Uh, but normal. Uh, but normal. Uh, but missed failure, eh- missed failure, eh- eh, will give you a bad one eh, will give you a bad one eh, will give you a bad one a report that will reach a report that will reach a report that will reach client. Um, so you client. Um, so you client. Um, so you will have to exchange will have to exchange will have to exchange one on the other. If one on the other. If one on the other. If recall is increasing, recall is increasing, recall is increasing, uh, then, uh, accuracy uh, then, uh, accuracy uh, then, uh, accuracy will decrease, and vice versa.

will decrease, and vice versa. So for your application So for your application you need you need you need to decide whether to be to decide whether to be to decide whether to be accurate, uh, that is accurate, uh, that is accurate, uh, that is throw away some things, throw away some things, throw away some things, which were actually which were actually which were actually correct, uh, but, correct, uh, but, correct, uh, but, uh, you decided that uh, you decided that uh, you decided that they were not, better or not they were not, better or not they were not, better or not worse than getting worse than getting worse than getting false alarms when false alarms when false alarms when are you saying something are you saying something are you saying something important, but important, but important, but it turns out that it is not it turns out that it is not it turns out that it is not Yes. Um, the judges too Yes. Um, the judges too Yes. Um, the judges too have prejudices. THERE ARE- have prejudices. THERE ARE- oh, I already mentioned that oh, I already mentioned that oh, I already mentioned that one of them. Um, one one of them. Um, one one of them. Um, one of them–this of them–this of them–this partiality partiality partiality positions. Having two positions. Having two positions. Having two options, they options, they options, they tend to provide tend to provide tend to provide preference, depending on preference, depending on preference, depending on models, which one models, which one models, which one will be the first or will be the first or will be the first or the last one. It depends. the last one. It depends. the last one. It depends. depending on which way depending on which way depending on which way the model has been changed. the model has been changed. the model has been changed. Uh, but you should Uh, but you should Uh, but you should to change yours to change yours to change yours options in places to options in places to options in places to they are not always they are not always they are not always were considering these, uh, were considering these, uh, were considering these, uh, things in one order things in one order . Uh, they have . Uh, they have . Uh, they have partiality partiality partiality length. Uh, judges. length. Uh, judges. length. Uh, judges. prefer prefer prefer long answers, and long answers, and long answers, and also bias also bias also bias confidence. Judges confidence. Judges confidence. Judges prefer prefer prefer answers that answers that answers that sound free and sound free and sound free and confidently. Uh, and that one, confidently. Uh, and that one, confidently. Uh, and that one, which I already mentioned which I already mentioned which I already mentioned remembered, this is remembered, this is remembered, this is self-prejudice. self-prejudice. self-prejudice. A model that estimates A model that estimates A model that estimates own result, as own result, as own result, as rule, evaluates it rule, evaluates it rule, evaluates it above. Uh, so we above. Uh, so we above. Uh, so we Haiku was rated by Haiku was rated by Haiku was rated by using sonnet. Uh, and in using sonnet. Uh, and in using sonnet. Uh, and in you value ideals you value ideals you value ideals Haiku with the help of Haiku with the help of Haiku with the help of something from OpenAI. Uh, something from OpenAI. Uh, something from OpenAI. Uh, but before you get too but before you get too but before you get too Are you worried about Are you worried about Are you worried about as much as you can as much as you can as much as you can It is worth trusting the judge.

It is worth trusting the judge. It is worth trusting the judge. to note that the judges- to note that the judges- people also often people also often people also often disagree disagree disagree with each other. Um, if with each other. Um, if with each other. Um, if you will give two you will give two you will give two experts are the same experts are the same experts are the same results in one results in one results in one rubrics and measure, rubrics and measure, rubrics and measure, how often they how often they how often they agree, uh, for agree, uh, for agree, uh, for this exists this exists this exists standard score, which standard score, which standard score, which called Kappa called Kappa called Kappa Cohen, uh, and for Cohen, uh, and for Cohen, uh, and for this type of task this type of task this type of task he often makes he often makes he often makes only 0.2 or 0.3. Uh, only 0.2 or 0.3. Uh, only 0.2 or 0.3. Uh, that is, they that is, they that is, they agree with each other agree with each other agree with each other one not much one not much one not much more often than not more often than not more often than not accident. Uh, accident. Uh, accident. Uh, a bar for your a bar for your a bar for your human judgment- human judgment- this is h, the bar for this is h, the bar for this is h, the bar for your judgment is your judgment is your judgment is human human human productivity, and h productivity, and h productivity, and h human human human productivity, uh, productivity, uh, productivity, uh, not really like that not really like that not really like that and good when you and good when you and good when you you compare people you compare people you compare people with each other. So, with each other. So, with each other. So, having a judge who is not having a judge who is not having a judge who is not agree, you don't agree, you don't agree, you don't need a judge who need a judge who need a judge who agrees with you agrees with you agrees with you 100% of the time. Uh, yours. 100% of the time. Uh, yours. 100% of the time. Uh, yours. the judge never the judge never the judge never will agree with you on will agree with you on will agree with you on 100% of the time. You need 100% of the time. You need 100% of the time. You need the judge who the judge who the judge who is doing better, is doing better, is doing better, than another person who than another person who than another person who executes the judge, uh, executes the judge, uh, executes the judge, uh, because your judge because your judge because your judge may pass thousands may pass thousands may pass thousands examples and such examples and such examples and such way to help yourself. way to help yourself. way to help yourself. Er, one more principle Er, one more principle Er, one more principle regarding judges is regarding judges is regarding judges is because of failures because of failures because of failures must surrender must surrender must surrender fair. When fair. When fair. When something fails, something fails, something fails, it should be obvious that it should be obvious that it should be obvious that the agent did the agent did the agent did wrong. If wrong. If wrong. If look at look at look at failure and say: failure and say: failure and say: Well, actually, I Well, actually, I Well, actually, I it seems that this it seems that this it seems that this okay then okay then okay then the problem is probably in the problem is probably in the problem is probably in your assessment. Hmm, yes.

your assessment. Hmm, yes. your assessment. Hmm, yes. a famous example when a famous example when a famous example when Claude Opus 4.5 for the first time Claude Opus 4.5 for the first time Claude Opus 4.5 for the first time scored 42% in the benchmark scored 42% in the benchmark scored 42% in the benchmark called Corebench, which called Corebench, which called Corebench, which checks whether they can checks whether they can checks whether they can agents reproduce agents reproduce agents reproduce scientific articles. Anthropic scientific articles. Anthropic scientific articles. Anthropic were upset by this were upset by this were upset by this low score. So low score. So low score. So the researcher conducted the researcher conducted the researcher conducted actual transcripts actual transcripts actual transcripts and discovered that Claude and discovered that Claude and discovered that Claude was simply right about was simply right about to another. The test was expected to another. The test was expected to another. The test was expected six digits of precision, six digits of precision, six digits of precision, and Claude gave the correct one and Claude gave the correct one and Claude gave the correct one answer with two or answer with two or answer with two or four digits four digits four digits accuracy. Hmm, and the test accuracy. Hmm, and the test accuracy. Hmm, and the test showed that this showed that this showed that this wrong. Were wrong. Were wrong. Were ambiguous descriptions ambiguous descriptions ambiguous descriptions tasks. Some tasks. Some tasks. Some no task at all no task at all no task at all it was possible it was possible it was possible to reproduce. So Anthropic to reproduce. So Anthropic to reproduce. So Anthropic passed the test, passed the test, passed the test, corrected all the tests and corrected all the tests and corrected all the tests and said that these tests said that these tests said that these tests could be better. could be better. could be better. And magically they And magically they And magically they received 95% in the assessment. received 95% in the assessment. received 95% in the assessment. Hmm, that happens. Hmm, that happens. Hmm, that happens. constantly. SWEBench and all constantly. SWEBench and all constantly. SWEBench and all other, uh, benchmarks, other, uh, benchmarks, other, uh, benchmarks, which you see, which you see, which you see, constantly constantly constantly are overvalued and are overvalued and are overvalued and are being adjusted are being adjusted are being adjusted manually, and, uh, exactly manually, and, uh, exactly manually, and, uh, exactly so you see the version so you see the version so you see the version two, three, four two, three, four two, three, four versions of, uh, known versions of, uh, known versions of, uh, known benchmarks, uh, so benchmarks, uh, so benchmarks, uh, so that people constantly that people constantly that people constantly find that find that find that benchmarks actually benchmarks actually benchmarks actually not as good as not as good as not as good as could be, and yours could be, and yours could be, and yours assessments will lead to assessments will lead to assessments will lead to the same problem. Uh, the same problem. Uh, the same problem. Uh, so you need so you need so you need constantly constantly constantly to reassess one's to reassess one's to reassess one's a set of tests. Uh, what?

a set of tests. Uh, what? a set of tests. Uh, what? leads us to leads us to leads us to the seventh step, which is, the seventh step, which is, the seventh step, which is, uh, experiments. IN uh, experiments. IN uh, experiments. IN We have assessments that we We have assessments that we We have assessments that we we trust. Now we we trust. Now we we trust. Now we we are going we are going we are going use them, use them, use them, so that in fact so that in fact so that in fact improve the agent and improve the agent and improve the agent and to prove that to prove that to prove that correction correction correction It worked. Uh, me It worked. Uh, me It worked. Uh, me I'm going to go for a bit. I'm going to go for a bit. I'm going to go for a bit. faster because I faster because I faster because I worked in a very worked in a very worked in a very at a leisurely pace. IN at a leisurely pace. IN at a leisurely pace. IN starts me starts me starts me Time is running out. THERE ARE- Time is running out. THERE ARE- Well, that's what it is. Well, that's what it is. Well, that's what it is. problem. We know, problem. We know, problem. We know, that some reports are not that some reports are not that some reports are not effective, and we know effective, and we know effective, and we know Why. So we change Why. So we change Why. So we change request, and then what? As request, and then what? As request, and then what? As you know what this is you know what this is you know what this is Did it work? Uh, how are you? Did it work? Uh, how are you? Did it work? Uh, how are you? you know that you are not you know that you are not you know that you are not broke something that broke something that broke something that worked before? worked before? worked before? If you run this If you run this If you run this on several examples on several examples on several examples and estimate by eye, then and estimate by eye, then and estimate by eye, then will return to will return to will return to vibrations. Ambulance vibrations. Ambulance vibrations. Ambulance a change you don't a change you don't a change you don't measured, it's simple measured, it's simple measured, it's simple vibration. You vibration. You vibration. You necessary necessary necessary controlled controlled controlled comparison. You comparison. You comparison. You the same ones are needed the same ones are needed the same ones are needed input data, the same input data, the same input data, the same before and after assessment. It before and after assessment. It before and after assessment. It experiment. So, in experiment. So, in experiment. So, in Phoenix you can Phoenix you can Phoenix you can take a bunch of your own take a bunch of your own take a bunch of your own the grades you the grades you the grades you filtered, filtered, filtered, let's say, to let's say, to let's say, to impractical, impractical, impractical, select them and add select them and add select them and add to the data set. It to the data set. It to the data set. It will give you a set for will give you a set for will give you a set for testing, uh, regarding testing, uh, regarding testing, uh, regarding certain things. I am not certain things. I am not certain things. I am not I'm going to do this.

I'm going to do this. I'm going to do this. to do because I to do because I to do because I already did it here. Uh, already did it here. Uh, already did it here. Uh, as soon as you have as soon as you have as soon as you have data set, you data set, you data set, you you can create, you can create, you can create, turn it into turn it into turn it into experiment and experiment and experiment and launch your launch your launch your experiment only with experiment only with experiment only with data set, only with data set, only with data set, only with estimates that estimates that estimates that were unsuccessful, were unsuccessful, were unsuccessful, unlike everything unlike everything unlike everything dial every time, that, dial every time, that, dial every time, that, you know, if you have you know, if you have you know, if you have separation separation separation decent size decent size decent size and a set of tests and a set of tests and a set of tests decent size, decent size, decent size, this saves you this saves you this saves you thousands and thousands thousands and thousands thousands and thousands tests every time. Uh, tests every time. Uh, tests every time. Uh, now we are going now we are going now we are going to fix what to fix what to fix what the assessment revealed. Therefore, the assessment revealed. Therefore, if you still if you still You are programming, congratulations. You are programming, congratulations. You are programming, congratulations. Open prompts.ts. Open prompts.ts. Open prompts.ts. There is an improved There is an improved There is an improved everyone's version everyone's version everyone's version request. Or you can request. Or you can request. Or you can write your own write your own write your own own. Um, and each own. Um, and each own. Um, and each change corresponds change corresponds change corresponds conclusion. So, the judge conclusion. So, the judge conclusion. So, the judge said, uh, no said, uh, no said, uh, no recommendations, so we recommendations, so we recommendations, so we asked for a clear asked for a clear asked for a clear recommendation. There recommendation. There recommendation. There it was said that it was said that it was said that risks are mentioned risks are mentioned risks are mentioned without evidence. So we without evidence. So we without evidence. So we asked about the risks asked about the risks asked about the risks with confirmatory with confirmatory with confirmatory data. Loyalty data. Loyalty data. Loyalty discovered fictional discovered fictional discovered fictional numbers, so we numbers, so we numbers, so we said said said use only use only use only numbers that numbers that numbers that appear in your appear in your appear in your research. uh, ah research. uh, ah research. uh, ah reading traces reading traces reading traces revealed that the agent, uh, revealed that the agent, uh, revealed that the agent, uh, tried all the time tried all the time tried all the time save files. Uh, save files. Uh, save files. Uh, so we tell him so we tell him so we tell him respond with a report. We respond with a report. We respond with a report. We We don't guess here. We We don't guess here. We We don't guess here. We we come up with specific we come up with specific we come up with specific things for the agent in things for the agent in things for the agent in result result result previous previous previous results of our results of our results of our grades. Uh, that's how grades. Uh, that's how grades. Uh, that's how the scenario looks like the scenario looks like the scenario looks like experiment.

experiment. experiment. Experiment is a set Experiment is a set Experiment is a set data, tasks and data, tasks and data, tasks and appraisers. Set appraisers. Set appraisers. Set data is the one we data is the one we data is the one we just created. Uh, just created. Uh, just created. Uh, a task is a function, a task is a function, a task is a function, which simply launches which simply launches which simply launches agent on each agent on each agent on each example. Uh, in this example. Uh, in this example. Uh, in this case we case we case we we launch the whole we launch the whole we launch the whole agent, but we could agent, but we could agent, but we could launch it on launch it on launch it on a certain subset a certain subset a certain subset agent behavior agent behavior agent behavior or on a certain or on a certain or on a certain subtasks, if subtasks, if subtasks, if wanted. Uh, ours wanted. Uh, ours wanted. Uh, ours the task returns the task returns the task returns report and research. report and research. report and research. So every assessment So every assessment So every assessment receives a small receives a small receives a small a wrapper that selects a wrapper that selects a wrapper that selects She needs fields. Uh, She needs fields. Uh, She needs fields. Uh, and then loyalty and then loyalty and then loyalty receives research receives research receives research as a context. The same, as a context. The same, as a context. The same, as before, uh, Phoenix as before, uh, Phoenix as before, uh, Phoenix starts the task on starts the task on starts the task on each example. He each example. He each example. He everyone appreciates everyone appreciates everyone appreciates result and saves result and saves result and saves everything so that you can everything so that you can everything so that you can compare. If you compare. If you compare. If you want to run this, want to run this, want to run this, you can run you can run you can run npm run experiment, and it will take npm run experiment, and it will take npm run experiment, and it will take about four about four about four minutes. Uh, but I have minutes. Uh, but I have minutes. Uh, but I have time is running out, time is running out, time is running out, so I'm not going so I'm not going so I'm not going wait four wait four wait four minutes. Uh, me minutes. Uh, me minutes. Uh, me launched his launched his launched his in advance. How many in advance. How many in advance. How many examples for you examples for you examples for you necessary? Uh, this is necessary? Uh, this is necessary? Uh, this is really good really good really good question. Uh, from 10 to question. Uh, from 10 to question. Uh, from 10 to 20 examples give you 20 examples give you 20 examples give you rough idea rough idea rough idea about where about where about where Things are moving. It Things are moving. It Things are moving. It very useful in very useful in very useful in development. Uh, if you development. Uh, if you development. Uh, if you go to go to go to production, uh, you production, uh, you production, uh, you it will take hundreds it will take hundreds it will take hundreds examples. Hmm, this is examples. Hmm, this is examples. Hmm, this is will take a lot of time, will take a lot of time, will take a lot of time, but, uh, that's it but, uh, that's it but, uh, that's it worth it for the sake of worth it for the sake of worth it for the sake of accuracy. Um, for accuracy. Um, for accuracy. Um, for everyone, uh, so that everyone, uh, so that everyone, uh, so that get scales get scales get scales accuracy errors, you accuracy errors, you accuracy errors, you need to increase need to increase need to increase number of examples number of examples number of examples four times. So, uh four times. So, uh , transition from 25 , transition from 25 , transition from 25 examples up to 100 examples up to 100 examples up to 100 examples really examples really examples really very useful. Uh, and

very useful. Uh, and very useful. Uh, and not much effort not much effort not much effort need to attach, need to attach, need to attach, to get again to get again to get again so big so big so big jump. You jump. You jump. You will have to go will have to go will have to go up to 400 examples that up to 400 examples that up to 400 examples that starts to come after starts to come after starts to come after the limits of what you are the limits of what you are the limits of what you are you will never be that you will never be that you will never be that engage in. Uh, but engage in. Uh, but engage in. Uh, but the more, the better, the more, the better, the more, the better, uh, and the more uh, and the more uh, and the more production failures you production failures you production failures you add to your add to your add to your data set, so data set, so data set, so better. Uh, that's why I better. Uh, that's why I better. Uh, that's why I launched mine three times, launched mine three times, launched mine three times, and the reason why I do this and the reason why I do this and the reason why I do this did, consists in did, consists in did, consists in because the agent because the agent because the agent non-deterministic. non-deterministic. non-deterministic. Every launch of every Every launch of every Every launch of every for example, uh, maybe for example, uh, maybe for example, uh, maybe to mislead you into to mislead you into to mislead you into in any direction. in any direction. in any direction. Uh, and there are two useful ones Uh, and there are two useful ones Uh, and there are two useful ones ways of thinking about ways of thinking about ways of thinking about how the agent works, e- how the agent works, e- oh, when he oh, when he oh, when he starts several starts several starts several times. First times. First times. First called "pass at K", called "pass at K", called "pass at K", what does it mean, uh, or what does it mean, uh, or what does it mean, uh, or at least it works at least it works at least it works once in K attempts, and once in K attempts, and once in K attempts, and then there is the "pass to the K", which then there is the "pass to the K", which then there is the "pass to the K", which increases every time, increases every time, increases every time, when, uh, is it possible? when, uh, is it possible? when, uh, is it possible? it's every time I it's every time I it's every time I I run it, uh, K times. I run it, uh, K times. Therefore, with increasing K Therefore, with increasing K these two numbers these two numbers these two numbers diverge. Uh, what? diverge. Uh, what? diverge. Uh, what? the more attempts you give the more attempts you give , the greater , the greater , the greater probability probability probability successful "pass at K", successful "pass at K", successful "pass at K", because it works because it works because it works at least once.

at least once. at least once. The more attempts you make The more attempts you make The more attempts you make yes, the bigger yes, the bigger yes, the bigger probability of failure probability of failure probability of failure at least once, and at least once, and at least once, and then "pass to the K", "pass to then "pass to the K", "pass to then "pass to the K", "pass to the K" will decrease. Uh, the K" will decrease. Uh, the K" will decrease. Uh, coding assistant coding assistant coding assistant gets what he does gets what he does gets what he does that's right from the second that's right from the second that's right from the second attempts, that's normal. attempts, that's normal. attempts, that's normal. So "pass at the K" is So "pass at the K" is So "pass at the K" is normally. Uh, bot normally. Uh, bot normally. Uh, bot support, which support, which support, which wrong wrong wrong is performed every is performed every is performed every five attempts, that's it five attempts, that's it five attempts, that's it a huge problem. AND a huge problem. AND a huge problem. AND financial analyst, financial analyst, financial analyst, which gives inaccurate which gives inaccurate which gives inaccurate recommendations, there are also recommendations, there are also recommendations, there are also a huge problem a huge problem . Uh, so you want to " . Uh, so you want to " pass to the K". Do you want to pass to the K". Do you want to pass to the K". Do you want to to make it happen to make it happen to make it happen every time for them. Uh, every time for them. Uh, every time for them. Uh, when you improve when you improve when you improve agent, there is an order agent, there is an order agent, there is an order operations, based operations, based operations, based on how much on how much on how much it will be useful it will be useful it will be useful making changes. First making changes. First -this is data quality. If -this is data quality. If -this is data quality. If your basic data your basic data your basic data bad if they are difficult bad if they are difficult bad if they are difficult read if they are difficult read if they are difficult read if they are difficult interpret interpret interpret agent or judge, then agent or judge, then agent or judge, then nothing else that you nothing else that you nothing else that you do it, nothing do it, nothing do it, nothing other than that you other than that you other than that you you will adjust, it will not be you will adjust, it will not be you will adjust, it will not be useful. Just got it useful. Just got it useful. Just got it there is quality data, then you there is quality data, then you there is quality data, then you go to go to go to tips. It is with this tips. It is with this tips. It is with this we today we today we today mostly mostly mostly worked. Needed worked. Needed worked. Needed to improve things on to improve things on to improve things on hint levels, hint levels, hint levels, changing our changing our changing our the rubric, changing the rubric, changing the rubric, changing our agent. Then our agent. Then our agent. Then there is a choice of model. Choice there is a choice of model. Choice there is a choice of model. Choice model has a large model has a large model has a large influence, but it also influence, but it also influence, but it also changes a whole bunch changes a whole bunch changes a whole bunch things at the same time. AND things at the same time. AND things at the same time. AND also ordinary also ordinary also ordinary how to change the model how to change the model how to change the model is that you is that you is that you change it to a larger one change it to a larger one change it to a larger one model, and it does model, and it does model, and it does things more expensive. So things more expensive. So things more expensive. So this creates a compromise, this creates a compromise, this creates a compromise, which is not always which is not always which is not always I want to go. AND, I want to go. AND, I want to go. AND, finally, there is finally, there is finally, there is hyperparameters.

hyperparameters. hyperparameters. Hyperparameters are Hyperparameters are Hyperparameters are things like things like things like temperature, etc. temperature, etc. temperature, etc. They are so useless, They are so useless, They are so useless, that people probably have that people probably have that people probably have such a problem that such a problem that such a problem that modern models modern models modern models stopped allowing stopped allowing stopped allowing configure them. You configure them. You configure them. You you can't transfer you can't transfer you can't transfer temperature to temperature to temperature to models for modern models for modern models for modern cloud models. Ahem, cloud models. Ahem, cloud models. Ahem, if you are tempted, if you are tempted, if you are tempted, this is the least useful this is the least useful this is the least useful thing you can thing you can thing you can miss. Um, if miss. Um, if miss. Um, if we look at mine we look at mine we look at mine experiments, you experiments, you experiments, you you can look at you can look at you can look at your initial data your initial data your initial data and see the same and see the same and see the same experiments. Um, you experiments. Um, you experiments. Um, you you'll see that, uh, with you'll see that, uh, with you'll see that, uh, with improving my improving my improving my ran, my ran, my ran, my the result increased from 16 to 15 the result increased from 16 to 15 the result increased from 16 to 15 from 16 to 48 of 48, and from 5 of 16 to from 16 to 48 of 48, and from 5 of 16 to from 16 to 48 of 48, and from 5 of 16 to 48 of 48. And loyalty 48 of 48. And loyalty 48 of 48. And loyalty increased from 2 out of 16 to 3 out of 48. increased from 2 out of 16 to 3 out of 48. increased from 2 out of 16 to 3 out of 48. uh what's not so good uh what's not so good uh what's not so good isn't that right? We really isn't that right? We really isn't that right? We really it would be worth it it would be worth it it would be worth it improve fidelity with improve fidelity with improve fidelity with my superior my superior my superior request. Uh, important. request. Uh, important. request. Uh, important. that, or all that before that, or all that before that, or all that before passed, broke?

passed, broke? passed, broke? We can We can We can filter to filter to filter to distribution of aisles. I distribution of aisles. I distribution of aisles. I I'm not going to do this. I'm not going to do this. I'm not going to do this. in the interface in the interface in the interface user now. THERE ARE- user now. THERE ARE- uh, but all five uh, but all five uh, but all five reports that were reports that were reports that were passed earlier, when passed earlier, when passed earlier, when I improved my query. I improved my query. , were still , were still , were still passed. Uh, so passed. Uh, so passed. Uh, so practicality right away practicality right away practicality right away became ideal. Uh, became ideal. Uh, became ideal. Uh, but loyalty was but loyalty was but loyalty was bad. Loyalty, uh bad. Loyalty, uh , increased from 2 out of 16 to 3 out of 48. , increased from 2 out of 16 to 3 out of 48. , increased from 2 out of 16 to 3 out of 48. Uh, although in the query Uh, although in the query Uh, although in the query literally written literally written literally written use only use only use only numbers that numbers that numbers that appear in your appear in your appear in your research. Uh, so research. Uh, so research. Uh, so we fixed what we we fixed what we