The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft

OK. Since it's already OK. Since it's already 11:12, I want to start a 11:12, I want to start a 11:12, I want to start a conversation about our conversation about our conversation about our workshop. So, I'm workshop. So, I'm workshop. So, I'm going to do it as a going to do it as a going to do it as a guided workshop, guided workshop, guided workshop, so I'm going to go through so I'm going to go through so I'm going to go through all the examples here on the all the examples here on the all the examples here on the screen, but you screen, but you screen, but you can also try to can also try to can also try to follow along with me. So, follow along with me. So, follow along with me. So, the repository is there, the the repository is there, the the repository is there, the environment variables are there too. These environment variables are there too. These environment variables are there too. These environment variables environment variables environment variables give you access to give you access to give you access to a number of Foundry models a number of Foundry models a number of Foundry models that I deployed in the that I deployed in the that I deployed in the Microsoft Foundry project. Microsoft Foundry project. Microsoft Foundry project. They have a certain amount of They have a certain amount of They have a certain amount of tokens. It is tokens. It is tokens. It is possible that we will possible that we will possible that we will run out of run out of run out of tokens during this tokens during this tokens during this session. I found out about session. I found out about session. I found out about this workshop this workshop this workshop being accepted very being accepted very being accepted very recently, so I recently, so I recently, so I tried to get tried to get tried to get as many as many as many resources as possible in a short resources as possible in a short resources as possible in a short period of time. So in period of time. So in period of time. So in that .env file there that .env file there that .env file there are actually a few are actually a few are actually a few different different different environment variables that can be environment variables that can be environment variables that can be enabled or disabled by enabled or disabled by enabled or disabled by commenting. If commenting. If commenting. If we start we start we start getting 429 errors getting 429 errors , that is, request , that is, request , that is, request limit exceeded errors limit exceeded errors limit exceeded errors . So . So . So I hope there's I hope there's enough capacity for all of you to enough capacity for all of you to try these try these try these models, and we'll models, and we'll models, and we'll see.

see. see. My colleague My colleague My colleague Cedric is also with me. Cedric Cedric is also with me. Cedric Cedric is also with me. Cedric will be walking around the room while will be walking around the room while will be walking around the room while I speak, so he I speak, so he I speak, so he can help you with can help you with can help you with any questions you may have. any questions you may have. Regarding Regarding code setup: if you code setup: if you code setup: if you already have Python and UV on your machine, already have Python and UV on your machine, already have Python and UV on your machine, you can clone you can clone you can clone the repository and the repository and the repository and run "uv sync" and it will have run "uv sync" and it will have run "uv sync" and it will have everything you need everything you need everything you need to run the to run the to run the examples once you examples once you examples once you add that .env. add that .env. add that .env. Otherwise, you Otherwise, you Otherwise, you can create a GitHub can create a GitHub can create a GitHub Codespace. It's essentially VS Code Codespace. It's essentially VS Code Codespace. It's essentially VS Code in the browser, with everything already in the browser, with everything already in the browser, with everything already set up for you. set up for you. Who here uses Who here uses UV? Cedric wants to know. UV? Cedric wants to know. UV? Cedric wants to know. Okay. So, it's not Okay. So, it's not Okay. So, it's not number one yet. Well, another number one yet. Well, another number one yet. Well, another question: who question: who question: who uses Python? uses Python? uses Python? Okay, there are more Okay, there are more Okay, there are more people now. Okay. people now. Okay. people now. Okay. Almost everyone here Almost everyone here Almost everyone here uses Python. uses Python. uses Python. Yes. Oh, did you count Yes. Oh, did you count Yes. Oh, did you count those who use those who use those who use both? OK. Okay. both? OK. Okay. both? OK. Okay. Well, if you haven't Well, if you haven't Well, if you haven't taken a picture of this taken a picture of this taken a picture of this screen yet, do it screen yet, do it screen yet, do it now. If not, now. If not, now. If not, Cedric has a photo Cedric has a photo Cedric has a photo on his phone, he might be able on his phone, he might be able on his phone, he might be able to help if you to help if you to help if you need those need those need those links. So, the goal of links. So, the goal of links. So, the goal of this workshop is this workshop is to feel the difference between to feel the difference between to feel the difference between various top various top various top models, see models, see models, see how they how they how they differ, and differ, and differ, and understand what can be understand what can be understand what can be improved in them. I improved in them. I improved in them. I have prepared many have prepared many have prepared many different examples, as different examples, as different examples, as well as evaluations and well as evaluations and well as evaluations and optimizations that we optimizations that we optimizations that we can use can use can use to improve our to improve our to improve our work with models.

work with models. work with models. I think this is very I think this is very I think this is very important for all of us important for all of us important for all of us as AI engineers, because as AI engineers, because as AI engineers, because as you can see, there are as you can see, there are so many new so many new models coming out all the time. And, you know, the models coming out all the time. And, you know, the models coming out all the time. And, you know, the US government US government US government can restrict can restrict can restrict access to a access to a access to a particular model at any moment. So we particular model at any moment. So we particular model at any moment. So we need to have a choice, need to have a choice, need to have a choice, right? We need to be right? We need to be right? We need to be able to able to able to switch our AI switch our AI applications to another applications to another applications to another model so they model so they model so they can continue can continue can continue to work, or know to work, or know to work, or know how to improve them. So how to improve them. So how to improve them. So the goal of this class is to the goal of this class is to gain more gain more gain more experience working with experience working with experience working with different models. Here are the different models. Here are the different models. Here are the models I models I models I deployed to Microsoft Foundry deployed to Microsoft Foundry . I have GPT-5, it's the . I have GPT-5, it's the . I have GPT-5, it's the standard model standard model standard model that I use all the time that I use all the time that I use all the time . . . I also have a Kimmy K 2.6, it's a I also have a Kimmy K 2.6, it's a I also have a Kimmy K 2.6, it's a fairly new and pretty fairly new and pretty fairly new and pretty good model. good model. good model. I have a Deep Seek V4 Flash. And I have a Deep Seek V4 Flash. And I have a Deep Seek V4 Flash. And I also have a Mistral Large 3. These are the I also have a Mistral Large 3. These are the I also have a Mistral Large 3. These are the four models that we four models that we four models that we will compare in will compare in will compare in this workshop. As I this workshop. As I this workshop. As I said, to said, to said, to set up set up set up the workshop you the workshop you the workshop you need this need this need this repository. If repository. If repository. If you don't have Python on your you don't have Python on your you don't have Python on your computer, just computer, just computer, just click the "Code click the "Code " button and there will be a big " button and there will be a big " button and there will be a big "Create codespace" button.

"Create codespace" button. "Create codespace" button. Click on it and a Click on it and a Click on it and a codespace will be created. codespace will be created. So here is your codespace, I So here is your codespace, I already added the .env file there already added the .env file there already added the .env file there , but you need to , but you need to , but you need to follow the follow the follow the link I link I link I provided to get the provided to get the provided to get the data for the env. It leads data for the env. It leads data for the env. It leads to this file. You to this file. You to this file. You download it. download it. download it. Open it. And Open it. And Open it. And then you create a then you create a then you create a .env file with its contents, .env file with its contents, .env file with its contents, okay? He points okay? He points okay? He points to the project in Foundry to the project in Foundry to the project in Foundry where these where these where these four models are deployed. As you four models are deployed. As you four models are deployed. As you have seen, there are limits have seen, there are limits have seen, there are limits on the number of tokens. on the number of tokens. on the number of tokens. So, if we start to So, if we start to So, if we start to hit the limits, hit the limits, hit the limits, you can you can you can try other try other try other models that also models that also models that also deploy these four deploy these four deploy these four projects. And I also have a projects. And I also have a projects. And I also have a separate project separate project separate project specifically for Anthropic, specifically for Anthropic, specifically for Anthropic, simply because it has simply because it has simply because it has different different different terms of use. terms of use. terms of use. OK. So, yes, OK. So, yes, OK. So, yes, once you have everything once you have everything once you have everything set up, uv sync set up, uv sync set up, uv sync will sync all the will sync all the will sync all the packages needed for packages needed for packages needed for Python. Okay, this Python. Okay, this Python. Okay, this setup is setup is setup is complete. Now I'm complete. Now I'm complete. Now I'm going to walk through going to walk through going to walk through all of this myself here so all of this myself here so all of this myself here so you can, you know, you can, you know, you can, you know, follow me if follow me if follow me if you want, or just you want, or just you want, or just watch what I'm doing.

watch what I'm doing. So, as I said, So, as I said, we have four models, and we have four models, and we have four models, and you can you can you can compare them compare them compare them just just just by using by using by using benchmarks. You say, benchmarks. You say, "Oh, this one has good "Oh, this one has good "Oh, this one has good quality and quality and quality and safety records." You safety records." You safety records." You can also look at can also look at can also look at cost, cost, cost, bandwidth, etc. This bandwidth, etc. This bandwidth, etc. This gives you a rough gives you a rough gives you a rough idea of idea of idea of how good these how good these how good these models are, but you won't models are, but you won't models are, but you won't know for sure know for sure know for sure until you use the until you use the until you use the model in your model in your model in your specific scenario specific scenario , as there are many , as there are many , as there are many ways to ways to ways to apply them. So a apply them. So a apply them. So a particular benchmark won't particular benchmark won't particular benchmark won't necessarily tell you necessarily tell you necessarily tell you how the model how the model how the model will work in will work in will work in your particular case. That's why your particular case. That's why we have we have a lot of different a lot of different a lot of different scenarios in this repository, and we scenarios in this repository, and we scenarios in this repository, and we can test can test can test these models for each these models for each these models for each of them. So, let's start with the of them. So, let's start with the of them. So, let's start with the simplest thing, simplest thing, simplest thing, okay? Just one okay? Just one okay? Just one call to LLM, call to LLM, call to LLM, asking the model asking the model asking the model to perform a task. You to perform a task. You to perform a task. You can see that each of can see that each of can see that each of these files these files these files starts with a starts with a starts with a model variable, and by model variable, and by model variable, and by default it is GPT-3.5 default it is GPT-3.5 . This is kind of our . This is kind of our . This is kind of our baseline. This is baseline. This is baseline. This is obviously a very good obviously a very good obviously a very good model. But if we model. But if we model. But if we want to try a want to try a want to try a different model, we different model, we different model, we can simply can simply can simply uncomment this uncomment this uncomment this line and it line and it line and it will be used.

will be used. So, this file So, this file connects connects connects using the OpenAI Python SDK using the OpenAI Python SDK using the OpenAI Python SDK to the Foundry URL and to the Foundry URL and to the Foundry URL and passes the key from the passes the key from the passes the key from the environment. And the query environment. And the query environment. And the query we're asking here is: we're asking here is: we're asking here is: how many times does the letter how many times does the letter how many times does the letter "e" appear in "e" appear in "e" appear in the following sentence? the following sentence? the following sentence? The elderly gentleman eagerly entered The elderly gentleman eagerly entered The elderly gentleman eagerly entered the elevator. Count the elevator. Count the elevator. Count carefully. OK. So carefully. OK. So carefully. OK. So we send this to the we send this to the we send this to the OpenAI Response API and OpenAI Response API and OpenAI Response API and say, "Here, say, "Here, say, "Here, run this model run this model run this model with this query, and with this query, and with this query, and then let's look at the ' then let's look at the ' raw' output, okay raw' output, okay ?" We do the classic ?" We do the classic ?" We do the classic thing—ask LLM to do thing—ask LLM to do thing—ask LLM to do the math. And the math. And the math. And we'll see, you know, we'll see, you know, we'll see, you know, how well they how well they how well they handle it. handle it. handle it. So I'll write "uv run So I'll write "uv run So I'll write "uv run examples" and the name of that examples" and the name of that examples" and the name of that file. And all this is in the file. And all this is in the file. And all this is in the read me file. So we read me file. So we read me file. So we just go step by just go step by just go step by step. You can step. You can step. You can just copy and just copy and just copy and paste from there. paste from there. paste from there. OK. So GPT-3.5 OK. So GPT-3.5 OK. So GPT-3.5 actually actually actually does does does math very well, math very well, math very well, surprisingly well. My surprisingly well. My surprisingly well. My colleagues have colleagues have colleagues have been running been running been running a lot of financial a lot of financial a lot of financial tests on it lately, and tests on it lately, and tests on it lately, and they're like, "Oh my god, they're like, "Oh my god, they're like, "Oh my god, it's doing it's doing it's doing so many math so many math so many math calculations in its head calculations in its head ." So he does indeed ." So he does indeed ." So he does indeed correctly identify correctly identify correctly identify that there are 13 that there are 13 that there are 13 "e"s in this sentence, right?

"e"s in this sentence, right? "e"s in this sentence, right? So, let's So, let's So, let's try, now we'll try, now we'll try, now we'll just change it, just change it, just change it, okay? Okay, okay? Okay, okay? Okay, let's say we want to let's say we want to let's say we want to try Kimmy. So we try Kimmy. So we try Kimmy. So we go in and go in and go in and switch to Kimmy, switch to Kimmy, switch to Kimmy, and now the and now the Kimmy model will be used. And we Kimmy model will be used. And we test, "Okay, test, "Okay, test, "Okay, can Kimmy calculate as well can Kimmy calculate as well can Kimmy calculate as well as GPT- as GPT- 3.5?" And he can. He 3.5?" And he can. He 3.5?" And he can. He gets 13. Okay. gets 13. Okay. gets 13. Okay. You might be thinking, " You might be thinking, " Oh, that's too easy?" Oh, that's too easy?" Oh, that's too easy?" Let's try Let's try Let's try DeepSeek V4 Flash. And this DeepSeek V4 Flash. And this DeepSeek V4 Flash. And this time he gets time he gets time he gets six. Okay. So he six. Okay. So he six. Okay. So he missed seven "e missed seven "e " letters, right? So we " letters, right? So we " letters, right? So we can can can see right away that there's a clear see right away that there's a clear see right away that there's a clear difference between, you know, difference between, you know, difference between, you know, DeepSeek V4 and those first DeepSeek V4 and those first DeepSeek V4 and those first two models. And two models. And two models. And finally, we will try finally, we will try finally, we will try Mistral. And this one gives Mistral. And this one gives Mistral. And this one gives eight. So, we see eight. So, we see that there is a difference here. that there is a difference here. that there is a difference here. OK. When you OK. When you OK. When you see that and think, " see that and think, " Well, I want to be Well, I want to be Well, I want to be able to able to able to use all use all use all these models," then you these models," then you these models," then you start doing start doing start doing prompt engineering. prompt engineering. You say, "Okay, what You say, "Okay, what if I tell you to break if I tell you to break if I tell you to break the sentence down into words, the sentence down into words, the sentence down into words, count each count each count each word, summarize, word, summarize, word, summarize, show the flow of thought, show the flow of thought, show the flow of thought, right?" Let's right?" Let's right?" Let's try this, okay? And try this, okay? And try this, okay? And you know, we can you know, we can you know, we can fix our fix our fix our typos, typos, typos, although it doesn't although it doesn't although it doesn't matter. OK. So matter. OK. So matter. OK. So let's try it with let's try it with let's try it with Mistral and see if Mistral and see if Mistral and see if we can get a we can get a we can get a better answer better answer better answer with a little with a little with a little prompt engineering?

prompt engineering? Okay, and he did Okay, and he did it, right? So, it, right? So, it, right? So, you know, by breaking everything down you know, by breaking everything down you know, by breaking everything down , , , by applying a by applying a by applying a chain of reasoning, chain of reasoning, chain of reasoning, he was able to give the he was able to give the he was able to give the correct answer, correct answer, correct answer, right? So we can immediately right? So we can immediately right? So we can immediately see that there are obvious see that there are obvious see that there are obvious differences between differences between differences between the models, and if we the models, and if we the models, and if we change the prompts, we change the prompts, we change the prompts, we can get different can get different can get different results. OK. results. OK. results. OK. So, now let's So, now let's So, now let's move on to more move on to more move on to more interesting and interesting and interesting and realistic scenarios realistic scenarios . Consider RAG. RAG is . Consider RAG. RAG is . Consider RAG. RAG is still one of the main still one of the main uses. It could be uses. It could be argued that argued that argued that everything is actually RAG. everything is actually RAG. everything is actually RAG. Agents are also RAGs. Agents are also RAGs. Agents are also RAGs. So we have this...RAG— So we have this...RAG— this is when you ask an LLM this is when you ask an LLM this is when you ask an LLM to answer a to answer a to answer a question based on a question based on a question based on a certain context, certain context, certain context, right? And the thing that's right? And the thing that's right? And the thing that's important for RAG important for RAG important for RAG is that you is that you is that you want it to want it to want it to answer answer answer questions questions questions based on based on based on context. You don't context. You don't context. You don't want her to make want her to make want her to make something up just something up just something up just from her own weight, do from her own weight, do from her own weight, do you? You're trying to you? You're trying to you? You're trying to get her get her get her to stick to to stick to to stick to the context, right? the context, right? the context, right? And this is something that, to me, is And this is something that, to me, is And this is something that, to me, is very different in very different in very different in different models: smaller different models: smaller , less powerful models are , less powerful models are , less powerful models are much more likely much more likely much more likely to refer to their to refer to their to refer to their weights and pretend that it's weights and pretend that it's weights and pretend that it's part of the context. part of the context.

So what we have here is: So what we have here is: we have a dataset we have a dataset we have a dataset on carpenter bees, and on carpenter bees, and on carpenter bees, and we ask the question: we ask the question: we ask the question: what are the key what are the key what are the key differences in differences in differences in how honey bees how honey bees how honey bees and carpenter bees and carpenter bees and carpenter bees build their nests? build their nests? But this dataset only contains information about carpenter bees. So carpenter bees. So carpenter bees. So in reality, she in reality, she in reality, she should only be should only be should only be able to able to able to answer about answer about answer about carpenter bees. So a carpenter bees. So a carpenter bees. So a good answer in good answer in good answer in this case would be to this case would be to this case would be to provide information provide information provide information about carpenter bees and about carpenter bees and about carpenter bees and then say, " then say, " Sorry, I can't Sorry, I can't Sorry, I can't tell you about tell you about tell you about honey bees. I honey bees. I honey bees. I don't know." So we don't know." So we don't know." So we tell her, "Hey, your tell her, "Hey, your tell her, "Hey, your job is to answer job is to answer job is to answer based on sources, to based on sources, to based on sources, to cite cite cite sources. Here's your sources. Here's your sources. Here's your question, here's your question, here's your question, here's your sources." Okay, sources." Okay, sources." Okay, let's try this let's try this let's try this and see what we and see what we and see what we get. Okay, and get. Okay, and get. Okay, and she...Yeah, when she she...Yeah, when she she...Yeah, when she first does first does first does a search, a very basic a search, a very basic a search, a very basic search. Okay, so search. Okay, so search. Okay, so she did the search. she did the search. she did the search. Okay, and what do we see Okay, and what do we see : she says the : she says the : she says the dataset doesn't contain dataset doesn't contain dataset doesn't contain details about details about details about nest building by nest building by nest building by honey bees honey bees . So that's exactly what we . So that's exactly what we . So that's exactly what we really want really want really want to see. This is from GB55.

to see. This is from GB55. to see. This is from GB55. GB55 knows very well GB55 knows very well GB55 knows very well what it doesn't know. what it doesn't know. what it doesn't know. Now let's Now let's Now let's try try try switching. Kimmy's switching. Kimmy's switching. Kimmy's not bad either. So I won't not bad either. So I won't not bad either. So I won't show show show every model every time, every model every time, every model every time, just to just to just to save time. So, save time. So, save time. So, let's try Deep let's try Deep let's try Deep Seek, shall we? Since we have Seek, shall we? Since we have Seek, shall we? Since we have already seen that there is a already seen that there is a already seen that there is a certain difference here. certain difference here. certain difference here. Now Deep Seek, again Now Deep Seek, again Now Deep Seek, again , says that , says that , says that the sources provided do not contain the sources provided do not contain the sources provided do not contain information. Okay, information. Okay, information. Okay, so Deep Seek did well here so Deep Seek did well here so Deep Seek did well here . Now . Now . Now let's move on to Mistral. Uh, let's move on to Mistral. Uh, let's move on to Mistral. Uh, Mistral, sorry. Cedric Mistral, sorry. Cedric , how do we pronounce , how do we pronounce , how do we pronounce Mistral? It's French, Mistral? It's French, Mistral? It's French, right? A French right? A French right? A French company? company? company? Mistral. Mistral. Mistral. Mistral. Cedric will be my Mistral. Cedric will be my Mistral. Cedric will be my French French French translator. translator. translator. Whenever I need to Whenever I need to Whenever I need to say it, you'll say it, you'll say it, you'll just say just say just say it. Now Mistral, hmm? Yes it. Now Mistral, hmm? Yes , okay. Mistral. So, with the , okay. Mistral. So, with the , okay. Mistral. So, with the Mistral...We see that Mistral...We see that Mistral...We see that the Mistral actually the Mistral actually the Mistral actually responded with its responded with its responded with its weight, although this weight, although this weight, although this information was not in the information was not in the information was not in the sources. So this is an sources. So this is an sources. So this is an example of a model example of a model example of a model addressing her addressing her addressing her weight even though we didn't weight even though we didn't weight even though we didn't want her to, right? Now we want her to, right? Now we want her to, right? Now we can try to can try to can try to force her. Again force her. Again force her. Again , we can , we can , we can try try try using prompt using prompt engineering. We can engineering. We can engineering. We can say, wait, say, wait, say, wait, actually, for her to actually, for her to actually, for her to say: if say: if say: if the answer to some the answer to some the answer to some piece of information is piece of information is piece of information is not contained in the not contained in the not contained in the sources. Say " sources. Say " source not provided". I source not provided". I source not provided". I think that's how I think that's how I think that's how I got it to work.

got it to work. got it to work. Oh, I'll try that and not Oh, I'll try that and not Oh, I'll try that and not make up make up make up an answer. Okay, an answer. Okay, an answer. Okay, let's try this. let's try this. So, we try to So, we try to make her understand make her understand make her understand when she gives us when she gives us when she gives us information that is information that is information that is not in the scales. And sometimes not in the scales. And sometimes , if you force her to , if you force her to , if you force her to say, "Hey, for say, "Hey, for say, "Hey, for every piece of every piece of every piece of information you information you information you provide in your provide in your provide in your answer, you have to answer, you have to answer, you have to prove that you got it from prove that you got it from prove that you got it from sources, right?" Here I'm sources, right?" Here I'm sources, right?" Here I'm trying to get trying to get trying to get her to her to her to link each link each link each thing in the answer to the thing in the answer to the thing in the answer to the source, and maybe source, and maybe source, and maybe that will make her that will make her that will make her realize, "Oh, realize, "Oh, realize, "Oh, wait, that wait, that wait, that wasn't actually in wasn't actually in wasn't actually in my answers." my answers." Okay, it's still Okay, it's still working. She's working. She's working. She's going to think about it, going to think about it, going to think about it, she's she's she's thinking about it very carefully. Okay. And thinking about it very carefully. Okay. And thinking about it very carefully. Okay. And here, please. here, please. here, please. Details about Details about Details about honeybees— honeybees— source not provided, source not provided, source not provided, except for interactions with carpenter except for interactions with carpenter except for interactions with carpenter bees. bees. bees. So, she So, she So, she knew exactly when knew exactly when knew exactly when information was not information was not information was not provided in the sources. provided in the sources. provided in the sources. But she But she But she gave out the information anyway.

gave out the information anyway. gave out the information anyway. So, we are a little So, we are a little So, we are a little closer to the goal. closer to the goal. closer to the goal. But, as you can see, this is But, as you can see, this is But, as you can see, this is exactly the difference exactly the difference exactly the difference I see when working I see when working I see when working with RAG, and RAG is what with RAG, and RAG is what with RAG, and RAG is what I work with the most. I work with the most. I work with the most. This is something to This is something to This is something to evaluate when you evaluate when you evaluate when you are choosing models for are choosing models for are choosing models for RAG: are the models willing RAG: are the models willing RAG: are the models willing to admit what they to admit what they to admit what they don't know, and don't know, and don't know, and how likely are how likely are how likely are they to they to they to make it up? Okay, make it up? Okay, make it up? Okay, now let's move on to now let's move on to now let's move on to inserting images. inserting images. inserting images. Many of these models are Many of these models are Many of these models are multimodal, so multimodal, so multimodal, so they accept they accept they accept images as images as images as input. And there are a lot of input. And there are a lot of input. And there are a lot of really cool really cool really cool situations that can be situations that can be situations that can be implemented using implemented using image input. So here image input. So here we have some files we have some files we have some files locally and we're just locally and we're just locally and we're just going to send them to LLM and going to send them to LLM and going to send them to LLM and ask some ask some ask some questions about those questions about those questions about those files, right? I'll show files, right? I'll show files, right? I'll show you these files. you these files. you these files. Examples, Examples, Examples, image input. Okay. image input. Okay. OK. So the files OK. So the files we have here are we have here are we have here are this picture. This is a tour, this picture. This is a tour, this picture. This is a tour, not a unicorn. This not a unicorn. This not a unicorn. This picture. These are the picture. These are the picture. These are the crocodiles I crocodiles I crocodiles I saw in Costa Rica and saw in Costa Rica and saw in Costa Rica and which I was very afraid of. And which I was very afraid of. And which I was very afraid of. And we also have this we also have this we also have this picture, which is a picture, which is a picture, which is a list of plants. This is list of plants. This is list of plants. This is essentially a screenshot of a essentially a screenshot of a essentially a screenshot of a table with very table with very table with very small font. So small font. So , we're asking each of the , we're asking each of the , we're asking each of the language models language models language models to interpret these to interpret these to interpret these images, right? So images, right? So images, right? So first we ask, " first we ask, " Oh, is that a unicorn?" Yes Oh, is that a unicorn?" Yes ? It should be easy ? It should be easy ? It should be easy to answer that it is not a to answer that it is not a to answer that it is not a unicorn. Perfectly.

unicorn. Perfectly. unicorn. Perfectly. Then we ask, "Hey Then we ask, "Hey , are those alligators or , are those alligators or , are those alligators or crocodiles?" This is a little more crocodiles?" This is a little more crocodiles?" This is a little more complicated. They are actually complicated. They are actually complicated. They are actually crocodiles, crocodiles, crocodiles, judging by their judging by their judging by their snouts. I spent snouts. I spent snouts. I spent a lot of time a lot of time a lot of time reading the reading the reading the Wikipedia page about Wikipedia page about Wikipedia page about crocodiles, you have to crocodiles, you have to crocodiles, you have to look at their look at their look at their snouts and then it becomes snouts and then it becomes snouts and then it becomes clear. Fortunately, clear. Fortunately, clear. Fortunately, GPT-5.5 also "stuck" on GPT-5.5 also "stuck" on GPT-5.5 also "stuck" on Wikipedia, so it knows: Wikipedia, so it knows: Wikipedia, so it knows: if it sees such a if it sees such a if it sees such a face, then these are face, then these are face, then these are crocodiles. And then crocodiles. And then crocodiles. And then regarding the cheapest regarding the cheapest regarding the cheapest plant—it plant—it plant—it correctly found correctly found correctly found the cheapest plant the cheapest plant the cheapest plant in this screenshot of a in this screenshot of a in this screenshot of a table with fairly table with fairly table with fairly small text. It's small text. It's small text. It's impressive that GPT-5.5 was able to impressive that GPT-5.5 was able to impressive that GPT-5.5 was able to do this. Now do this. Now do this. Now let's try with other let's try with other let's try with other models. models. models. We cannot We cannot We cannot use DeepSeek for these tasks because use DeepSeek for these tasks because use DeepSeek for these tasks because it only accepts it only accepts it only accepts text. This is not a text. This is not a text. This is not a multimodal multimodal multimodal model. So we will model. So we will model. So we will only try Kimmy and only try Kimmy and only try Kimmy and Mistral. So, you know, Mistral. So, you know, Mistral. So, you know, she recognized the aurochs. she recognized the aurochs. she recognized the aurochs. Okay, great. Just Okay, great. Just Okay, great. Just checking if it checking if it checking if it works. OK. And as for works. OK. And as for works. OK. And as for crocodiles, she crocodiles, she crocodiles, she correctly answered correctly answered correctly answered that they were crocodiles. And that they were crocodiles. And she also coped with the cheapest plant. she also coped with the cheapest plant. So Kimmy is doing great too So Kimmy is doing great too So Kimmy is doing great too . And now . And now . And now finally let's move on to finally let's move on to finally let's move on to Mistral. Here. Let's take a look.

Mistral. Here. Let's take a look. Mistral. Here. Let's take a look. OK. She knows it's OK. She knows it's OK. She knows it's not a unicorn. She did not a unicorn. She did not a unicorn. She did n't recognize the aurochs. In n't recognize the aurochs. In n't recognize the aurochs. In this case, she this case, she this case, she wasn't sure. She wasn't sure. She wasn't sure. She thinks they are thinks they are thinks they are alligators. So it's alligators. So it's alligators. So it's harder for her, she thinks it's harder for her, she thinks it's harder for her, she thinks it's alligators. And for alligators. And for alligators. And for the plants, she chose the the plants, she chose the the plants, she chose the $2.30 option, $2.30 option, $2.30 option, which is significantly more expensive than the which is significantly more expensive than the which is significantly more expensive than the cheapest plant. cheapest plant. cheapest plant. She had to find the one that was She had to find the one that was She had to find the one that was 58 cents. Therefore, 58 cents. Therefore, 58 cents. Therefore, Mistral cannot Mistral cannot Mistral cannot process such process such process such images. Here's the images. Here's the images. Here's the thing: when it thing: when it thing: when it comes to comes to comes to multimodal multimodal multimodal understanding, you can't understanding, you can't change much with prompt engineering, because engineering, because engineering, because the model either understands the model either understands the model either understands the image or it doesn't. You the image or it doesn't. You the image or it doesn't. You could apply could apply could apply some prompt some prompt engineering if engineering if engineering if you work in a narrow you work in a narrow you work in a narrow field and can give field and can give field and can give hints on hints on hints on how to read a specific how to read a specific how to read a specific type of graph. But type of graph. But type of graph. But really, with input really, with input really, with input images, the key is images, the key is to understand whether to understand whether to understand whether your language model can your language model can your language model can process the process the process the type of images you are type of images you are type of images you are going to going to going to send it. Would that, send it. Would that, send it. Would that, you know, would that you know, would that you know, would that work in that work in that work in that scenario, right? Hmm, scenario, right? Hmm, scenario, right? Hmm, maybe Mistral is really maybe Mistral is really maybe Mistral is really good at good at good at certain types of certain types of imaging tasks, but not imaging tasks, but not this one, right? So, this one, right? So, this one, right? So, you need to understand you need to understand you need to understand what task you what task you what task you want to perform and whether want to perform and whether want to perform and whether the model will cope with it well the model will cope with it well the model will cope with it well ? What's also ? What's also ? What's also interesting about interesting about interesting about multimodality multimodality multimodality is that a lot of is that a lot of is that a lot of it also depends it also depends it also depends on language, right? I on language, right? I on language, right? I use use use English here.

English here. English here. Remember, all of these Remember, all of these Remember, all of these models are very models are very models are very Anglocentric, but Anglocentric, but Anglocentric, but if you if you if you switch to switch to switch to another language, for example, another language, for example, another language, for example, if I ask in if I ask in if I ask in Spanish whether it's Spanish whether it's Spanish whether it's caimanes or cocodrilos, it caimanes or cocodrilos, it caimanes or cocodrilos, it actually thinks, uh, actually thinks, uh, actually thinks, uh, it's giving the wrong it's giving the wrong it's giving the wrong answer because it's answer because it's answer because it's very confused in its very confused in its very confused in its head between these head between these head between these different words and different words and different words and how they how they how they relate to relate to relate to English. It becomes English. It becomes English. It becomes too difficult too difficult too difficult a task: a task: a task: to simultaneously understand the difference between to simultaneously understand the difference between to simultaneously understand the difference between alligators and alligators and alligators and crocodiles, and also crocodiles, and also crocodiles, and also remember that "caiman" remember that "caiman" remember that "caiman" means "alligator" means "alligator" means "alligator" in Spanish and is not the in Spanish and is not the in Spanish and is not the same as the caiman that same as the caiman that same as the caiman that we have, for example, in we have, for example, in we have, for example, in Florida. In any Florida. In any Florida. In any case, it is interesting case, it is interesting case, it is interesting to observe that as to observe that as to observe that as soon as we switch soon as we switch soon as we switch to Spanish, the model to Spanish, the model to Spanish, the model fails the task. Um fails the task. Um , so that's another , so that's another , so that's another point to keep point to keep point to keep in mind, especially in mind, especially in mind, especially since we're in since we're in since we're in California; Often California; Often California; Often when we develop when we develop when we develop apps here, they're apps here, they're apps here, they're bilingual because bilingual because bilingual because a lot of people, you know, a lot of people, you know, a lot of people, you know, 50% of them here speak 50% of them here speak 50% of them here speak Spanish. Um, so, Spanish. Um, so, Spanish. Um, so, you know, the results you know, the results you know, the results may vary may vary may vary depending on the depending on the depending on the language change. Okay, now language change. Okay, now language change. Okay, now let's move on to calling let's move on to calling let's move on to calling functions, also functions, also functions, also known as calling known as calling known as calling tools. So, tools. So, tools. So, tool calling tool calling tool calling is actually the basis of is actually the basis of is actually the basis of all agents. All all agents. All all agents. All agents are based on agents are based on agents are based on calling tools calling tools . And this is an ability that . And this is an ability that . And this is an ability that LLMs teach LLMs teach LLMs teach specifically during the specifically during the specifically during the learning process.

learning process. learning process. So they're taught to So they're taught to So they're taught to look at a look at a look at a function definition function definition function definition or multiple or multiple or multiple function definitions, function definitions, function definitions, parse the parse the parse the user's input, and say, user's input, and say, user's input, and say, "Okay, based on "Okay, based on "Okay, based on this input, I want this input, I want this input, I want to call this function to call this function to call this function with these arguments and, with these arguments and, with these arguments and, well, just return the well, just return the well, just return the result." So, in result." So, in result." So, in this example, I this example, I this example, I define a function define a function define a function using a JSON schema. using a JSON schema. using a JSON schema. This is what LLMs understand. This is what LLMs understand. This is what LLMs understand. So I say, “Okay, So I say, “Okay, So I say, “Okay, here’s my function. It is here’s my function. It is here’s my function. It is called "create a called "create a called "create a calendar event." calendar event." calendar event." It has this description, and It has this description, and It has this description, and here are the parameters here are the parameters here are the parameters it accepts. It it accepts. It it accepts. It accepts a name, which is a accepts a name, which is a accepts a name, which is a string. Here's the description, right? string. Here's the description, right? And all of this will be And all of this will be sent to LLM. So sent to LLM. So sent to LLM. So we send this we send this we send this full JSON schema to LLM and full JSON schema to LLM and full JSON schema to LLM and say, “Hey, here’s a say, “Hey, here’s a say, “Hey, here’s a function. You have to function. You have to function. You have to decide whether you will decide whether you will decide whether you will trigger it and how trigger it and how trigger it and how , right? So, , right? So, , right? So, even though it even though it even though it looks like an object, it's looks like an object, it's looks like an object, it's actually prompt actually prompt engineering. Everything here engineering. Everything here is really my is really my is really my decision about what decision about what decision about what would be best for the LLM, would be best for the LLM, would be best for the LLM, you know? What type should you know? What type should you know? What type should I use? I use? I use? What description should be What description should be What description should be added here? What should added here? What should added here? What should the names of these the names of these the names of these parameters be? Because LLM parameters be? Because LLM parameters be? Because LLM will decide will decide will decide based on all this based on all this based on all this information how it information how it information how it will call this function. Uh will call this function. Uh , so, next we have a , so, next we have a , so, next we have a message from message from message from the user. So the user. So the user. So the user says, "Oh, the user says, "Oh, the user says, "Oh, can you add something to can you add something to can you add something to my calendar? This is a my calendar? This is a platform team sync. Me, platform team sync. Me, Sarah from Ange, Marcus, Sarah from Ange, Marcus, Sarah from Ange, Marcus, new new new product manager Priya. Tomorrow product manager Priya. Tomorrow product manager Priya. Tomorrow at 1:30 PM. It's virtual on at 1:30 PM. It's virtual on at 1:30 PM. It's virtual on Microsoft Teams, okay? So Microsoft Teams, okay? So Microsoft Teams, okay? So then we say to LLM, " then we say to LLM, " then we say to LLM, " Hey, you're the calendar Hey, you're the calendar Hey, you're the calendar assistant. Today is assistant. Today is assistant. Today is Monday, June 29.

Monday, June 29. Monday, June 29. Use Use Use available tools available tools available tools to handle the to handle the to handle the user's request. And we user's request. And we user's request. And we send her the send her the send her the tools. So tools. So tools. So now LLM looks at the now LLM looks at the now LLM looks at the user request, user request, user request, looking at the looking at the looking at the tools. And in tools. And in tools. And in this case, she this case, she this case, she only has this only has this calendar event creation tool. And calendar event creation tool. And it's going to decide, " it's going to decide, " Hey, is there a Hey, is there a Hey, is there a tool here that I tool here that I tool here that I can use to can use to can use to respond respond respond to the user? Uh, and to the user? Uh, and to the user? Uh, and then we'll just then we'll just then we'll just walk around and walk around and walk around and see what see what see what tool calls tool calls tool calls it decides to it decides to it decides to make. Okay, let's make. Okay, let's make. Okay, let's go. So this is go. So this is go. So this is really a really a really a low-level low-level low-level tool call. tool call. tool call. When we create When we create When we create the agent, we'll the agent, we'll the agent, we'll abstract on top of abstract on top of abstract on top of that, but I wanted to that, but I wanted to that, but I wanted to start with the start with the start with the lowest-level lowest-level lowest-level tool call tool call . And the LLM just . And the LLM just . And the LLM just responds with the responds with the responds with the function name and function name and function name and the arguments that I the arguments that I the arguments that I have to send. The LLM itself have to send. The LLM itself have to send. The LLM itself doesn't call the function. doesn't call the function. doesn't call the function. That's our job. That's our job. That's our job. Okay, so what we Okay, so what we Okay, so what we see is that I've see is that I've see is that I've put the expected put the expected put the expected result on the left result on the left result on the left so you can so you can so you can compare it to the compare it to the compare it to the actual result. So GPT 5.5 actual result. So GPT 5.5 actual result. So GPT 5.5 added me as a added me as a added me as a participant, which I did participant, which I did participant, which I did n't really want. n't really want. n't really want. But otherwise it But otherwise it But otherwise it did a pretty did a pretty did a pretty good job. So let's good job. So let's good job. So let's try Deep Seek and try Deep Seek and try Deep Seek and see how it see how it see how it handles this handles this handles this . Because when we . Because when we . Because when we use a use a use a tool call, tool call, tool call, we want to see we want to see we want to see if it's calling the if it's calling the if it's calling the right right right tool? In this tool? In this tool? In this case, yes. It's case, yes. It's case, yes. It's calling the creation tool.

calling the creation tool. calendar events. And does calendar events. And does it pass the it pass the it pass the arguments that we arguments that we arguments that we expect, right? expect, right? expect, right? We expect some We expect some We expect some arguments. In this arguments. In this arguments. In this case, Deep Seek specified case, Deep Seek specified case, Deep Seek specified you as an attendee. And you as an attendee. And you as an attendee. And for the for the for the venue, it venue, it venue, it wrote virtual ( wrote virtual ( Microsoft Teams), which is Microsoft Teams), which is Microsoft Teams), which is not what I asked for at all, not what I asked for at all, not what I asked for at all, right? So it doesn't right? So it doesn't right? So it doesn't really respect really respect really respect my my my argument constraints. Let's argument constraints. Let's argument constraints. Let's try Mistral as well. try Mistral as well. try Mistral as well. And let's see how it And let's see how it And let's see how it handles that. When I handles that. When I handles that. When I see that, I start to see that, I start to see that, I start to think, "Okay, there's think, "Okay, there's think, "Okay, there's some ambiguity here, some ambiguity here, some ambiguity here, right? And, you know, if right? And, you know, if right? And, you know, if it called it with the it called it with the it called it with the wrong wrong wrong arguments, that arguments, that arguments, that means there's an means there's an means there's an ambiguity. ambiguity. ambiguity. I need to I need to I need to go back to my go back to my go back to my descriptions and say, you descriptions and say, you descriptions and say, you know, don't include know, don't include know, don't include me or you, just the me or you, just the me or you, just the specified attendees. specified attendees. You could have provided the first You could have provided the first and last names if and last names if and last names if they were there, right? So they were there, right? So they were there, right? So you start to see you start to see you start to see where you weren't where you weren't specific enough, and you specific enough, and you start to address that start to address that start to address that ambiguity, and ambiguity, and ambiguity, and then you evaluate all the then you evaluate all the then you evaluate all the different inputs and different inputs and different inputs and see how it see how it see how it works. Okay. That was works. Okay. That was works. Okay. That was basic basic basic tool call.

tool call. tool call. The next thing we The next thing we The next thing we want to do is we want to do is we want to do is we move to agents. move to agents. move to agents. Next is using Next is using tool calls in a loop, tool calls in a loop, right? So when we have an right? So when we have an right? So when we have an agent, it calls agent, it calls agent, it calls tools in a loop tools in a loop tools in a loop until it reaches a goal. until it reaches a goal. until it reaches a goal. We'll demonstrate We'll demonstrate We'll demonstrate this with the this with the this with the function "calculate." function "calculate." function "calculate." This calculator This calculator This calculator takes a mathematical takes a mathematical takes a mathematical expression and returns a expression and returns a expression and returns a result, right? And result, right? And result, right? And we have basic we have basic we have basic support for support for support for calculating these calculating these calculating these mathematical expressions mathematical expressions . We give it a . We give it a . We give it a text problem, text problem, text problem, like, "Oh, this like, "Oh, this like, "Oh, this store is having a sale. store is having a sale. store is having a sale. If I buy three If I buy three If I buy three shirts and pay shirts and pay shirts and pay such and such tax, such and such tax, such and such tax, how much will I how much will I how much will I spend in total?" There are spend in total?" There are spend in total?" There are a few a few a few different ways to do this. different ways to do this. different ways to do this. You can make multiple You can make multiple You can make multiple calls to this calls to this calls to this calculator, calculator, calculator, solving one solving one solving one small expression at small expression at small expression at a time, like, " a time, like, " do this, then this, then this, then do this, then this, then this, then do this, then this, then this, then this." Or, this." Or, this." Or, if the language model is if the language model is if the language model is really smart, it really smart, it really smart, it can combine all of this can combine all of this can combine all of this into one long expression and into one long expression and into one long expression and make one make one make one tool call. So tool call. So tool call. So this is where we're testing this is where we're testing this is where we're testing efficiency, right? efficiency, right? efficiency, right? Often when you Often when you Often when you use use use tool calls tool calls tool calls in a loop, the difference between in a loop, the difference between in a loop, the difference between the models is the models is the models is their efficiency.

their efficiency. their efficiency. Can they Can they Can they get to the goal faster? get to the goal faster? get to the goal faster? Can they get the Can they get the Can they get the same result in same result in same result in fewer fewer tool calls? So that's what tool calls? So that's what we're we're we're going to say. going to say. going to say. The only system The only system The only system hint is to not do hint is to not do hint is to not do the math in your head. the math in your head. the math in your head. Okay. Well, Okay. Well, Okay. Well, let's see. This is let's see. This is let's see. This is really a problem for really a problem for really a problem for GB55, because it's so GB55, because it's so GB55, because it's so proud of its proud of its proud of its math math math skills that it skills that it skills that it always tries to always tries to always tries to solve everything in its head. solve everything in its head. solve everything in its head. Well, seriously. Okay. Well, seriously. Okay. Well, seriously. Okay. So GB55 really does So GB55 really does So GB55 really does do all do all do all the calculations in one the calculations in one the calculations in one expression. So it expression. So it expression. So it only took one only took one only took one tool call tool call tool call to get the to get the to get the final result. final result. final result. That's good. That's good. That's, That's good. That's good. That's, That's good. That's good. That's, you know, fast. That you know, fast. That you know, fast. That saves us time. saves us time. saves us time. So now let's try it So now let's try it So now let's try it with, uh, DeepSeek. The only with, uh, DeepSeek. The only with, uh, DeepSeek. The only reason I reason I reason I keep skipping keep skipping keep skipping Kimmy is because Kimmy is because Kimmy is because it's actually a really it's actually a really it's actually a really good model, and it good model, and it good model, and it performs pretty much on par performs pretty much on par performs pretty much on par with GB55 on most of with GB55 on most of with GB55 on most of these problems. Okay. these problems. Okay. So now we see So now we see that DeepSeek goes through the that DeepSeek goes through the that DeepSeek goes through the process and does... process and does... what's that, four what's that, four what's that, four calls? It did calls? It did calls? It did four four four tool calls, right?

tool calls, right? tool calls, right? It still It still It still got the correct got the correct got the correct answer, but it answer, but it answer, but it took four calls. took four calls. took four calls. And each of those calls to the And each of those calls to the And each of those calls to the tool is a tool is a tool is a separate network separate network separate network request, right? So when request, right? So when request, right? So when we're evaluating two we're evaluating two we're evaluating two models, we want to models, we want to models, we want to look not only at the look not only at the look not only at the final answer, but final answer, but final answer, but also at how long it also at how long it also at how long it took to took to took to get it, right? By get it, right? By get it, right? By the way, Kimmy did something the way, Kimmy did something the way, Kimmy did something really interesting the other day. really interesting the other day. really interesting the other day. Let's see if Let's see if Let's see if she does it again she does it again she does it again now. Because Kimmy now. Because Kimmy now. Because Kimmy works pretty well most of the time works pretty well most of the time works pretty well most of the time , but , but , but sometimes she gets into sometimes she gets into sometimes she gets into weird loops when she's weird loops when she's weird loops when she's trying to trying to trying to test herself. Oh, okay, test herself. Oh, okay, test herself. Oh, okay, she did it this time she did it this time she did it this time . So, the other day . So, the other day . So, the other day when I was testing Kimmy, when I was testing Kimmy, when I was testing Kimmy, she got the she got the she got the correct answer correct answer correct answer on the first try, and on the first try, and on the first try, and then she decided then she decided then she decided to test her to test her to test her work with four more work with four more work with four more calls. I'm like, " calls. I'm like, " calls. I'm like, " No, it's fine." You're doing No, it's fine." You're doing No, it's fine." You're doing great." So that's what great." So that's what great." So that's what we're we're we're looking at here, okay? Exactly looking at here, okay? Exactly looking at here, okay? Exactly how many how many how many calls does it take calls does it take calls does it take to get to the end. to get to the end. Okay, now we Okay, now we can move on to the can move on to the can move on to the agent loops. Okay, agent loops. Okay, agent loops. Okay, I'm actually going through I'm actually going through I'm actually going through this pretty quickly this pretty quickly this pretty quickly now. Okay, good.

now. Okay, good. Okay. So now Okay. So now we have a base, right? We've we have a base, right? We've we have a base, right? We've seen the seen the seen the tool call in the loop. tool call in the loop. tool call in the loop. Now we can Now we can Now we can create a create a create a full-fledged agent. full-fledged agent. full-fledged agent. And for that, we And for that, we And for that, we usually usually usually use an use an agent framework. I've added agent framework. I've added three different frameworks here three different frameworks here three different frameworks here so you can choose so you can choose so you can choose the one that you the one that you the one that you like. They're LangChain, like. They're LangChain, like. They're LangChain, Pydantic AI, and Agent Framework. I'm Pydantic AI, and Agent Framework. I'm Pydantic AI, and Agent Framework. I'm sure there are other sure there are other sure there are other frameworks that I've frameworks that I've frameworks that I've missed, but missed, but missed, but hopefully you'll hopefully you'll hopefully you'll find one that find one that find one that works for you. So it's all, works for you. So it's all, works for you. So it's all, you know, the same you know, the same you know, the same idea. Okay, what exactly do idea. Okay, what exactly do idea. Okay, what exactly do we want to show? we want to show? we want to show? Somebody vote. Somebody vote. Somebody vote. LangChain. Okay, LangChain is LangChain. Okay, LangChain is LangChain. Okay, LangChain is still very still very still very popular, and popular, and popular, and hopefully someone from the hopefully someone from the hopefully someone from the LangChain team is LangChain team is LangChain team is here today at the here today at the here today at the conference. So, in conference. So, in conference. So, in LangChain, we're LangChain, we're LangChain, we're using LangChain using LangChain using LangChain V1, which is a good V1, which is a good V1, which is a good latest version latest version latest version if you haven't tried if you haven't tried if you haven't tried it in the it in the it in the last year. We're last year. We're last year. We're plugging into plugging into plugging into foundry models with LangChain. And foundry models with LangChain. And foundry models with LangChain. And what we have here what we have here is a is a is a travel agent, a classic travel agent, a classic travel agent, a classic example. So it has example. So it has example. So it has tools. We have a tools. We have a tools. We have a tool to tool to tool to search for flights, search for flights, search for flights, hotels, check the hotels, check the hotels, check the budget to see budget to see if the if the if the selected flights and selected flights and selected flights and hotels fit within the budget, and hotels fit within the budget, and hotels fit within the budget, and also to find also to find also to find things to do in the city. Then things to do in the city. Then things to do in the city. Then I have I have middleware so I middleware so I can see what can see what can see what tools are tools are tools are being used. And being used. And being used. And then we have the then we have the agent call directly. So the agent agent call directly. So the agent has a system prompt has a system prompt has a system prompt that says you're that says you're that says you're going to help going to help going to help users find users find users find flights and hotels.

flights and hotels. flights and hotels. It has certain rules It has certain rules like search for flights like search for flights like search for flights and hotels, and hotels, and hotels, prioritize direct prioritize direct prioritize direct flights, etc., always flights, etc., always flights, etc., always check the budget, and at check the budget, and at check the budget, and at the end the end the end suggest things to do suggest things to do suggest things to do based on the remaining based on the remaining based on the remaining budget. Then we have the budget. Then we have the user message. I user message. I need to fly from need to fly from need to fly from San Francisco to New San Francisco to New York next York next York next month for 3 nights. My month for 3 nights. My month for 3 nights. My total budget is 6600, total budget is 6600, total budget is 6600, I need direct I need direct I need direct flights and some flights and some flights and some fun. Okay. So fun. Okay. So fun. Okay. So we have all of that, and we have all of that, and we have all of that, and now we're going to run now we're going to run now we're going to run the agent. There's some the agent. There's some the agent. There's some extra code here that extra code here that extra code here that just logs the just logs the just logs the tool calls tool calls tool calls so we can so we can so we can see them. So let's see them. So let's see them. So let's try to do that. try to do that. Okay. This will log the Okay. This will log the tool calls tool calls that have happened. We're that have happened. We're that have happened. We're starting at GPT 55 here. What's starting at GPT 55 here. What's starting at GPT 55 here. What's interesting is that you'll interesting is that you'll interesting is that you'll see the first pass. see the first pass. see the first pass. Let me Let me Let me zoom in on that. The first zoom in on that. The first zoom in on that. The first pass, it says, is pass, it says, is pass, it says, is parallel. So parallel. So parallel. So one thing that you can one thing that you can one thing that you can do with LLM is that do with LLM is that do with LLM is that a lot of them a lot of them a lot of them support support support parallel parallel parallel tool calls, where if tool calls, where if tool calls, where if you give them a bunch of you give them a bunch of tool definitions, they tool definitions, they can offer can offer can offer multiple tools multiple tools multiple tools to call to call to call at the same time in one pass at the same time in one pass . That's really good from a . That's really good from a performance perspective, performance perspective, because on your server because on your server because on your server you can you can you can query, query, query, say, two say, two say, two different databases in parallel, different databases in parallel, different databases in parallel, right? So as long as right? So as long as right? So as long as they're independent they're independent they're independent of each other, that's a great of each other, that's a great of each other, that's a great feature of LLM: to say, " feature of LLM: to say, " feature of LLM: to say, " Hey, these two things can be Hey, these two things can be Hey, these two things can be done in parallel."

done in parallel." done in parallel." Why don't you Why don't you Why don't you do them? "So, uh, GPT do them? "So, uh, GPT do them? "So, uh, GPT 55 in the first step 55 in the first step 55 in the first step says, "Hey, find says, "Hey, find says, "Hey, find hotels in New York and hotels in New York and hotels in New York and find flights from San find flights from San Francisco to New Francisco to New York." Because it York." Because it York." Because it decided that there was decided that there was decided that there was no relationship between them. And no relationship between them. And no relationship between them. And then when the answer then when the answer then when the answer came in, with all came in, with all came in, with all the options, it the options, it the options, it checked the budget based on checked the budget based on the flight and hotel costs that were the flight and hotel costs that were returned in returned in returned in that query. Then that query. Then that query. Then it saw what it saw what it saw what budget was left over, and it budget was left over, and it budget was left over, and it ran the entertainment search ran the entertainment search ran the entertainment search with that remainder. So this is with that remainder. So this is probably the probably the probably the most efficient most efficient most efficient agent that we agent that we agent that we can have for this can have for this can have for this scenario. Let's scenario. Let's scenario. Let's see if see if see if we get similar we get similar we get similar results if results if results if we change the models. duh duh duh we change the models. duh duh duh And of course, here you And of course, here you And of course, here you can even can even can even change and change and change and compare both the compare both the compare both the models and the models and the models and the frameworks, right? frameworks, right? frameworks, right? Because each framework Because each framework Because each framework has a slightly different has a slightly different has a slightly different tool call cycle tool call cycle . We also call . We also call . We also call this "binding," right? this "binding," right? this "binding," right? So often you want So often you want So often you want to compare both the models and to compare both the models and to compare both the models and the bindings, and you'll see some the bindings, and you'll see some the bindings, and you'll see some interesting differences interesting differences interesting differences between them. Okay.

between them. Okay. between them. Okay. This time we're going to Kimmy, and This time we're going to Kimmy, and This time we're going to Kimmy, and Kimmy doesn't Kimmy doesn't Kimmy doesn't parallelize as parallelize as parallelize as well. I don't know well. I don't know well. I don't know if it's because Kimmy doesn't if it's because Kimmy doesn't if it's because Kimmy doesn't support parallel support parallel support parallel tool calls tool calls or if it just or if it just or if it just chose not to chose not to chose not to do it in this do it in this do it in this case. This time case. This time case. This time it did the same thing. it did the same thing. it did the same thing. In fact, it made In fact, it made In fact, it made even more even more even more calls. So it calls. So it calls. So it did the flight search did the flight search did the flight search and the hotel search. It and the hotel search. It and the hotel search. It did it in two did it in two did it in two steps, so it's less steps, so it's less steps, so it's less efficient because it didn't efficient because it didn't parallelize. Then parallelize. Then it decided to make it decided to make it decided to make two calls to two calls to two calls to check the budget, because it check the budget, because it seems to be checking seems to be checking seems to be checking another option. And another option. And another option. And then it did the then it did the then it did the final action. So this final action. So this final action. So this case took two case took two case took two extra extra extra tool calls and tool calls and tool calls and took longer because it did took longer because it did took longer because it did n't parallelize the n't parallelize the n't parallelize the first set of steps. first set of steps. first set of steps. Okay. Let's see Okay. Let's see Okay. Let's see how we handle how we handle how we handle DeepSeek. And you start DeepSeek. And you start DeepSeek. And you start to see why it's to see why it's to see why it's hard, because everyone asks hard, because everyone asks hard, because everyone asks me, "How do you rate me, "How do you rate me, "How do you rate agents?" Agents are agents?" Agents are agents?" Agents are the hardest to evaluate the hardest to evaluate the hardest to evaluate because there's because there's because there's so so so much going on in the process. There's much going on in the process. There's much going on in the process. There's so many so many so many different things going on different things going on different things going on during execution. during execution. during execution. It's like, is it calling the It's like, is it calling the It's like, is it calling the right right right tools? Is it tools? Is it tools? Is it calling them calling them calling them efficiently? Is efficiently? Is efficiently? Is it calling things it calling things it calling things in parallel? In this in parallel? In this in parallel? In this case, you know, it case, you know, it case, you know, it didn't do any didn't do any didn't do any parallelization, but parallelization, but parallelization, but at least it at least it at least it only did four only did four only did four tool calls. So, tool calls. So, tool calls. So, you know, pretty you know, pretty you know, pretty efficiently. And then efficiently. And then efficiently. And then Mistral, we'll often Mistral, we'll often Mistral, we'll often see that Mistral see that Mistral see that Mistral often forgets the often forgets the often forgets the last step. So last step. So last step. So we'll see. Well, you know, we'll see. Well, you know, we'll see. Well, you know, that's another problem with that's another problem with that's another problem with agents. Sometimes agents. Sometimes agents. Sometimes agents, when their agents, when their agents, when their context window context window context window fills up, they can fills up, they can fills up, they can forget some of the forget some of the forget some of the previous instructions previous instructions . They can be less . They can be less . They can be less attentive, you attentive, you attentive, you know, to what the know, to what the know, to what the initial steps were.

initial steps were. initial steps were. And sometimes you And sometimes you And sometimes you have to have to have to nudge them to nudge them to nudge them to remember, "Oh, remember, "Oh, remember, "Oh, wait, you have to, well, wait, you have to, well, wait, you have to, well, complete this complete this complete this final step here, final step here, final step here, right?" Okay, so it right?" Okay, so it right?" Okay, so it didn't do it didn't do it didn't do it in parallel. So we'll in parallel. So we'll in parallel. So we'll see that GPT-5.5 is very see that GPT-5.5 is very see that GPT-5.5 is very good at good at good at invoking invoking invoking tools in parallel. And the others, tools in parallel. And the others, tools in parallel. And the others, well, the others well, the others well, the others don't. Okay, so don't. Okay, so don't. Okay, so Mistral decided to just Mistral decided to just Mistral decided to just look up flights and look up flights and look up flights and hotels and hotels and hotels and do nothing else. Well, do nothing else. Well, do nothing else. Well, so it, it kind of so it, it kind of so it, it kind of gave up at that point gave up at that point . So this is an example of . So this is an example of . So this is an example of where, okay, we would have to where, okay, we would have to apply some apply some additional control additional control additional control to say, "Hey, are you to say, "Hey, are you to say, "Hey, are you sure you got sure you got sure you got everything, huh?" You could everything, huh?" You could everything, huh?" You could use something use something use something like a route loop. Jeff like a route loop. Jeff like a route loop. Jeff Huntley is here today. Huntley is here today. Huntley is here today. He created a route loop. And He created a route loop. And He created a route loop. And you know, you can, you know, you can, you know, you can, you can make you can make you can make agents be more agents be more agents be more aggressive, instead of aggressive, instead of aggressive, instead of just just just using a using a using a standard loop. standard loop. Okay. Okay, so this Okay. Okay, so this was an example agent was an example agent was an example agent that shows that we that shows that we that shows that we also have examples of also have examples of also have examples of agents in other agents in other agents in other frameworks. But you frameworks. But you frameworks. But you can see that the code is can see that the code is can see that the code is very similar. I very similar. I very similar. I tend to think that tend to think that tend to think that most agent most agent most agent frameworks are frameworks are frameworks are more similar than more similar than more similar than different now. But, different now. But, different now. But, you know, you can you know, you can you know, you can look at them.

look at them. Okay. So now we Okay. So now we can talk can talk can talk about evaluation, right? about evaluation, right? about evaluation, right? Up until now, we've been doing everything by Up until now, we've been doing everything by Up until now, we've been doing everything by just running them just running them just running them manually and manually and manually and watching the watching the watching the results. That's results. That's results. That's good to feel good to feel good to feel the difference that you can the difference that you can the difference that you can see, but when you're see, but when you're see, but when you're going to do going to do going to do something for production, something for production, something for production, you really need you really need you really need massive massive massive automated automated automated evaluations that you can evaluations that you can evaluations that you can reliably run and reliably run and reliably run and compare the compare the compare the results, right? results, right? So the first step that So the first step that we can do is we can do is we can do is just evaluate just evaluate just evaluate without, well, just without, well, just without, well, just using using using comparisons. strings, comparisons. strings, comparisons. strings, right? So the example right? So the example right? So the example I have here is a I have here is a I have here is a call to the calendar call to the calendar call to the calendar event creation tool event creation tool event creation tool , because we , because we , because we can, because we can, because we can, because we actually get actually get actually get the arguments back, the arguments back, the arguments back, just do a just do a just do a direct direct direct string comparison and say, " string comparison and say, " Hey, did Hey, did Hey, did it return 'virtual', it return 'virtual', it return 'virtual', right? We wanted ' right? We wanted ' virtual' for virtual' for virtual' for this, did this, did this, did it return 'virtual'?" it return 'virtual'?" it return 'virtual'?" So when you So when you So when you test a call to test a call to test a call to one tool, one tool, one tool, you can just you can just you can just check if check if check if it called the it called the it called the right right right tool and tool and tool and used the right used the right used the right arguments? So that's what arguments? So that's what arguments? So that's what we're going to do in we're going to do in we're going to do in this example. Um, and this example. Um, and this example. Um, and we created a bunch of we created a bunch of we created a bunch of test cases, test cases, test cases, right? Because when you right? Because when you right? Because when you run run run an evaluation, you an evaluation, you an evaluation, you need to have lots and need to have lots and lots of test lots of test lots of test cases. You also cases. You also cases. You also want it to be want it to be want it to be easy to add new easy to add new easy to add new cases, because when your cases, because when your cases, because when your app goes into app goes into app goes into production, you see production, you see users users users actually actually actually using it, and using it, and using it, and you're like, "Oh, okay.

you're like, "Oh, okay. you're like, "Oh, okay. Thanks. It's broken Thanks. It's broken ." No, and then you go ." No, and then you go ." No, and then you go and add these new and add these new user input options to your, user input options to your, you know, evaluations you know, evaluations you know, evaluations to make sure that to make sure that to make sure that you don't have deviations you don't have deviations , right? This is called , right? This is called , right? This is called "model drift" or, "model drift" or, "model drift" or, you know, deviation you know, deviation you know, deviation from it. So you don't from it. So you don't from it. So you don't want your want your want your estimates to deviate estimates to deviate estimates to deviate from what from what from what the users are inputting. So the users are inputting. So the users are inputting. So when they're in production when they're in production , you need a good , you need a good , you need a good system for system for system for adding new adding new user input to user input to your test your test your test cases. Here's what I cases. Here's what I cases. Here's what I do: I have a do: I have a do: I have a bunch of different, bunch of different, bunch of different, you know, cases here. you know, cases here. you know, cases here. For example, here's a For example, here's a user message, here's user message, here's a check, and these a check, and these a check, and these checks just checks just checks just do do do string comparisons, string comparisons, string comparisons, list comparisons, and we list comparisons, and we list comparisons, and we check different check different check different things, right? We things, right? We things, right? We check check check date calculations. I date calculations. I date calculations. I always check LLMs always check LLMs always check LLMs on date calculations, on date calculations, on date calculations, because they're usually because they're usually because they're usually pretty weak at that. pretty weak at that. pretty weak at that. So another tip: So another tip: So another tip: if you're working with if you're working with if you're working with dates, just give dates, just give dates, just give them a tool to them a tool to them a tool to calculate dates, okay calculate dates, okay ? Because it's too hard to ? Because it's too hard to ? Because it's too hard to do do do date calculations in your date calculations in your date calculations in your head. Especially in head. Especially in head. Especially in leap years. I've leap years. I've leap years. I've been evaluating LLMs on been evaluating LLMs on been evaluating LLMs on leap years, it leap years, it leap years, it was hard. Then we was hard. Then we was hard. Then we had, well, had, well, had, well, ambiguous ambiguous ambiguous durations. I did durations. I did durations. I did one in Spanish, right? one in Spanish, right? one in Spanish, right? Great test in Great test in Great test in different languages ​​that different languages ​​that different languages ​​that you're going to you're going to you're going to support your support your support your product. This one, I think, product. This one, I think, product. This one, I think, is written in real is written in real is written in real slang. Yo, can you slang. Yo, can you slang. Yo, can you book a big book a big book a big conference room?

conference room? conference room? Okay. So we have Okay. So we have Okay. So we have different edge different edge different edge cases here, right? cases here, right? cases here, right? Now let me Now let me Now let me run these run these run these assessments. Okay, and assessments. Okay, and assessments. Okay, and these assessments in this these assessments in this these assessments in this case I'll run for case I'll run for case I'll run for each of the models and each of the models and use cases. Often in use cases. Often in reality, when I reality, when I reality, when I do an assessment, do an assessment, do an assessment, I'll just run I'll just run I'll just run it on one model it on one model it on one model and then compare it to a and then compare it to a and then compare it to a previous run previous run previous run where maybe I where maybe I where maybe I used a different used a different used a different model. Well, you know, model. Well, you know, model. Well, you know, since we're since we're since we're comparing different comparing different comparing different models today, I'll run it models today, I'll run it models today, I'll run it for all of them so we can for all of them so we can for all of them so we can see—for each one see—for each one see—for each one of them, it makes of them, it makes of them, it makes a call to the tool, a call to the tool, a call to the tool, checks checks checks the arguments it receives, and decides the arguments it receives, and decides the arguments it receives, and decides whether it's a success or whether it's a success or whether it's a success or failure. As for failure. As for failure. As for the assessment: you don't the assessment: you don't the assessment: you don't want it to want it to want it to just be part of just be part of just be part of your pittest or your your pittest or your your pittest or your test suite, because that's test suite, because that's test suite, because that's slow and expensive. slow and expensive. slow and expensive. So, is there a question? So, is there a question? So, is there a question? I see you're just I see you're just I see you're just checking strings. Do checking strings. Do checking strings. Do you know of any you know of any you know of any frameworks that frameworks that frameworks that do this a little bit do this a little bit do this a little bit more elegant? more elegant?

You don't like You don't like my string validations. my string validations. my string validations. Regarding Regarding Regarding evaluation frameworks: first of all, evaluation frameworks: first of all, evaluation frameworks: first of all, most most monitoring providers have monitoring providers have built-in built-in built-in evaluation tools, evaluation tools, evaluation tools, right? If you right? If you right? If you use LangFuse, use LangFuse, use LangFuse, LangSmith, LangChain, or Arize— LangSmith, LangChain, or Arize— LangSmith, LangChain, or Arize— they all have their own they all have their own they all have their own built-in frameworks built-in frameworks . For Foundry, we have two . For Foundry, we have two . For Foundry, we have two different frameworks, and there are different frameworks, and there are different frameworks, and there are examples for them in the examples for them in the examples for them in the repository. There's the Azure AI repository. There's the Azure AI repository. There's the Azure AI Evaluation SDK, OpenAI Evals from the Evaluation SDK, OpenAI Evals from the Evaluation SDK, OpenAI Evals from the OpenAI package. So, there are OpenAI package. So, there are OpenAI package. So, there are a lot of frameworks, a lot of frameworks, a lot of frameworks, also DeepEval, and Promptfoo also DeepEval, and Promptfoo also DeepEval, and Promptfoo for JavaScript, if you're for JavaScript, if you're for JavaScript, if you're working with that language working with that language . And which one? . And which one? . And which one? Vali. Vali. Vali. Vali? Vali? Vali? Something like that. Something like that. Something like that. Mark-Vali, okay. Mark-Vali, okay. Mark-Vali, okay. V A L L Y V A L L Y V A L L Y V A L L Y, I got it. I have V A L L Y, I got it. I have V A L L Y, I got it. I have n't even n't even n't even used it. used it. used it. So, there are a lot of So, there are a lot of evaluation frameworks. I'll evaluation frameworks. I'll use a use a use a more advanced one soon. But more advanced one soon. But more advanced one soon. But regarding regarding regarding string validation—here's the thing. string validation—here's the thing. You don't really You don't really need a complex need a complex need a complex framework for string validation. Um. Because you're framework for string validation. Um. Because you're framework for string validation. Um. Because you're like... like... like... If you want it to If you want it to If you want it to do something do something do something specific... specific... specific... Yeah. Yeah. Yeah. I did I did I did the parallelization the parallelization the parallelization that you mentioned for that you mentioned for ...

... ... Yeah, yeah. For the tool calls, you're going to Yeah, yeah. For the tool calls, you're going to need something more need something more complex. I wanted to complex. I wanted to complex. I wanted to start with the basics, start with the basics, start with the basics, but yeah. Most of the time, but yeah. Most of the time, but yeah. Most of the time, we can't we can't we can't just do the basics, just do the basics, just do the basics, right? Oh, right? Oh, right? Oh, they did they did they did really well this time. Wow, that's really well this time. Wow, that's really well this time. Wow, that's cool. Usually, we'll cool. Usually, we'll cool. Usually, we'll have to have to have to move on to more move on to more move on to more complex evaluation complex evaluation . That's when we start . That's when we start . That's when we start bringing in the LLM as the judge. bringing in the LLM as the judge. That's because often the That's because often the input data is input data is input data is much more fuzzy, and much more fuzzy, and much more fuzzy, and we need to we need to we need to use the LLM use the LLM use the LLM to help to help to help analyze that analyze that analyze that uncertainty. uncertainty. uncertainty. There are a lot of There are a lot of There are a lot of frameworks that frameworks that frameworks that do that. In this do that. In this do that. In this example, I decided example, I decided example, I decided to write everything to write everything to write everything directly in the code directly in the code directly in the code so you can so you can so you can see it. So the idea of see it. So the idea of see it. So the idea of using the LLM as using the LLM as using the LLM as the judge is that the judge is that the judge is that you have to you have to you have to set up the model, set up the model, set up the model, put it in put it in put it in evaluation mode, and say, evaluation mode, and say, "Okay, you're the evaluator." "Okay, you're the evaluator." "Okay, you're the evaluator." Here's what you have to Here's what you have to Here's what you have to evaluate, and you evaluate, and you evaluate, and you have to say, does this have to say, does this have to say, does this pass or pass or pass or fail? Right? We're fail? Right? We're fail? Right? We're trying to get trying to get trying to get the LLM to say, "Does this the LLM to say, "Does this the LLM to say, "Does this pass or fail?" ". We pass or fail?" ". We pass or fail?" ". We also want also want also want to get a rationale from her to get a rationale from her to get a rationale from her , , , right? It's really, really right? It's really, really right? It's really, really helpful when she helpful when she helpful when she provides an explanation. So provides an explanation. So , we want to get , we want to get , we want to get the result: pass, the result: pass, the result: pass, fail, and fail, and fail, and the reason. Then we the reason. Then we the reason. Then we provide the query and provide the query and provide the query and the response to it. And the response to it. And the response to it. And for context, we for context, we for context, we also add also add also add additional information additional information , because it helps , because it helps , because it helps the model the model the model reason better. Sometimes, for a reason better. Sometimes, for a reason better. Sometimes, for a good evaluation, you good evaluation, you good evaluation, you really need really need really need access to more access to more access to more data.

data. data. For example, on For example, on tool calls: if you're tool calls: if you're evaluating the evaluating the evaluating the effectiveness of their effectiveness of their effectiveness of their use, you use, you use, you have to provide everything. have to provide everything. have to provide everything. You have to provide You have to provide You have to provide the definition of the the definition of the the definition of the tool calls. You tool calls. You tool calls. You have to provide have to provide have to provide a trace of the calls a trace of the calls a trace of the calls that were made, that were made, that were made, right? So to right? So to right? So to build good build good build good evaluators, you often evaluators, you often evaluators, you often need to provide not need to provide not need to provide not only the input and only the input and only the input and output, but everything output, but everything output, but everything that happened in that happened in that happened in the process, everything that was the process, everything that was the process, everything that was shown. So here we shown. So here we shown. So here we say to LLM, "Hey, we're say to LLM, "Hey, we're say to LLM, "Hey, we're going to going to going to use the use the GPT-5 judge model." It's a pretty GPT-5 judge model." It's a pretty GPT-5 judge model." It's a pretty good model for good model for good model for evaluation." And we're evaluation." And we're evaluation." And we're going to ask it to output going to ask it to output going to ask it to output JSON that we can JSON that we can JSON that we can look at." So look at." So look at." So let me let me let me run this LLM as a run this LLM as a run this LLM as a judge. But here's the judge. But here's the judge. But here's the thing: it's thing: it's thing: it's going to be even going to be even going to be even slower because slower because slower because every time we every time we every time we run an evaluation, we're run an evaluation, we're run an evaluation, we're simultaneously executing our simultaneously executing our simultaneously executing our own code that own code that own code that uses the model, uses the model, uses the model, and then and then and then we're using another we're using another we're using another LLM to evaluate LLM to evaluate LLM to evaluate that code, right? So that code, right? So that code, right? So each each each evaluation instance is evaluation instance is evaluation instance is actually two actually two actually two LLM calls. And often you're LLM calls. And often you're LLM calls. And often you're running multiple LLMs running multiple LLMs running multiple LLMs as judges at the same time. as judges at the same time. as judges at the same time. For example, you might have For example, you might have For example, you might have five five five different evaluators running.

different evaluators running. different evaluators running. Each one of them is a Each one of them is a Each one of them is a separate LLM call, separate LLM call, separate LLM call, right? That's the right? That's the right? That's the complexity of complexity of complexity of evaluation: you can't evaluation: you can't evaluation: you can't run run run it all the time. You can't it all the time. You can't it all the time. You can't necessarily necessarily necessarily run it on run it on run it on every CI or every CI or every CI or every commit, every commit, every commit, because it's quite a because it's quite a because it's quite a slow process. Because slow process. Because slow process. Because usually in usually in usually in production, production, production, for example, for RAG, for example, for RAG, for example, for RAG, when I evaluate RAG, when I evaluate RAG, when I evaluate RAG, you know, it usually you know, it usually you know, it usually takes at least takes at least takes at least 200 test cases, 200 test cases, 200 test cases, maybe 800. And that's, that's maybe 800. And that's, that's maybe 800. And that's, that's a lot of requests to the LLM, a lot of requests to the LLM, a lot of requests to the LLM, right? So it right? So it right? So it takes some time. Um, takes some time. Um, takes some time. Um, so you should so you should so you should think about how think about how think about how often you're going to, often you're going to, often you're going to, you know, run you know, run you know, run the evaluations. So you the evaluations. So you the evaluations. So you see it see it see it runs, it tests runs, it tests , it has a bunch of , it has a bunch of , it has a bunch of different test different test different test cases, different cases, different cases, different questions and questions and questions and answers. And it answers. And it answers. And it reviews the reviews the reviews the results. In this results. In this results. In this case, what I did was case, what I did was case, what I did was I kind of pre- I kind of pre- I kind of pre- ran the, um, ran the, um, ran the, um, questions and questions and questions and answers. That's also a answers. That's also a answers. That's also a common way common way common way to do it. Like, to do it. Like, to do it. Like, prepare, you know, prepare, you know, prepare, you know, get all your get all your get all your context for the context for the context for the evaluator evaluator evaluator in advance, because in advance, because in advance, because then you can then you can then you can change your change your change your evaluator regardless evaluator regardless evaluator regardless of your application, of your application, of your application, right? Um, and that's right? Um, and that's right? Um, and that's more efficient. Because more efficient. Because more efficient. Because often you realize often you realize often you realize that your evaluator is that your evaluator is that your evaluator is not working correctly, not working correctly, not working correctly, right? Like, that right? Like, that right? Like, that you wrote a bad you wrote a bad you wrote a bad evaluator. So that's what evaluator. So that's what evaluator. So that's what you need you need you need to figure out when you to figure out when you to figure out when you use LLM as a use LLM as a use LLM as a judge. You have to judge. You have to judge. You have to figure out, wait, is the figure out, wait, is the figure out, wait, is the problem in my problem in my problem in my application or is the problem in application or is the problem in application or is the problem in my judge, right? And you my judge, right? And you my judge, right? And you also want to also want to also want to calibrate your calibrate your calibrate your judges with judges with judges with human input. So it human input. So it human input. So it definitely definitely definitely helps me to work in the helps me to work in the helps me to work in the domain, you know, or domain, you know, or domain, you know, or collaborate with collaborate with collaborate with an expert who can an expert who can an expert who can help you help you help you mark up your data mark up your data mark up your data and say, you know, does and say, you know, does and say, you know, does it agree with your LLM it agree with your LLM it agree with your LLM as a judge? Was it as a judge? Was it as a judge? Was it actually a success or a actually a success or a

actually a success or a failure? So here we failure? So here we failure? So here we get the results get the results get the results back and we can see back and we can see , you know, the model and , you know, the model and , you know, the model and their scores, their scores, their scores, most of them most of them most of them got, like, got, like, got, like, two out of three. Mr. two out of three. Mr. two out of three. Mr. Owl got one out of Owl got one out of Owl got one out of three. You know, so they three. You know, so they three. You know, so they have some work to do have some work to do to be more to be more to be more grounded here. grounded here. So this is a generator So this is a generator for using LLM as a for using LLM as a for using LLM as a judge. As you can see, judge. As you can see, judge. As you can see, there are quite a few there are quite a few there are quite a few frameworks that, you frameworks that, you frameworks that, you know, have know, have know, have LLM as a judge built in, LLM as a judge built in, LLM as a judge built in, which makes it easier which makes it easier which makes it easier . But I want . But I want . But I want to show that you to show that you to show that you know, you can know, you can know, you can also do it also do it also do it yourself. Oh, and I also yourself. Oh, and I also yourself. Oh, and I also have to have to have to recommend, recommend, recommend, if you're interested in if you're interested in if you're interested in evaluation, hamel.dev. I don't evaluation, hamel.dev. I don't evaluation, hamel.dev. I don't know if he's at the know if he's at the know if he's at the conference this conference this conference this week, but he has the week, but he has the week, but he has the most amazing most amazing most amazing resources, and he resources, and he resources, and he teaches teaches teaches courses on evaluation all the time. And he has courses on evaluation all the time. And he has courses on evaluation all the time. And he has really good really good really good memes. I actually memes. I actually memes. I actually used them used them used them last night. So last night. So last night. So if you're ever in a if you're ever in a if you're ever in a conversation with conversation with conversation with someone about evals someone about evals and you need a and you need a and you need a meme about evaluations, uh, what did meme about evaluations, uh, what did meme about evaluations, uh, what did I do there last I do there last I do there last night? Yeah, building an LLM night? Yeah, building an LLM night? Yeah, building an LLM as a judge is hard. Uh as a judge is hard. Uh , so here's a really , so here's a really , so here's a really important resource. How important resource. How important resource. How can we trust an LLM can we trust an LLM judge? You have to judge? You have to judge? You have to calibrate your calibrate your calibrate your LLM judge. Okay.

LLM judge. Okay. Okay. So that was an LLM Okay. So that was an LLM as a judge. Now, uh, as a judge. Now, uh, as a judge. Now, uh, as you can see, as you can see, as you can see, building building building judge evaluators is judge evaluators is judge evaluators is hard, right? hard, right? hard, right? Like, you have to Like, you have to Like, you have to follow Hamel on follow Hamel on follow Hamel on Twitter, take his course, Twitter, take his course, Twitter, take his course, and learn all these and learn all these and learn all these best practices for best practices for best practices for writing, uh, a good writing, uh, a good writing, uh, a good evaluation. Uh, so we evaluation. Uh, so we evaluation. Uh, so we have this new have this new have this new tool from Microsoft tool from Microsoft tool from Microsoft Research. It's very new, it Research. It's very new, it Research. It's very new, it came out last came out last came out last week. Uh, it's week. Uh, it's week. Uh, it's called Assert. And called Assert. And called Assert. And Assert helps you Assert helps you Assert helps you write judges write judges write judges based on a specification. based on a specification. based on a specification. So you write a So you write a So you write a specification. You specification. You specification. You say, "Hey, this is an agent that say, "Hey, this is an agent that I'm building, and this is, you I'm building, and this is, you know, roughly what know, roughly what it should do, it should do, it should do, my vision of how my vision of how my vision of how it has to work, and, it has to work, and, it has to work, and, uh, what uh, what uh, what users it has to users it has to users it has to support." "And support." "And support." "And then it will generate then it will generate then it will generate a bunch of test a bunch of test a bunch of test cases." It will kind of cases." It will kind of cases." It will kind of expand your expand your expand your specification into, uh, specification into, uh, specification into, uh, actual actual actual evaluation rubrics and figure evaluation rubrics and figure evaluation rubrics and figure all of that out for you all of that out for you all of that out for you based on, you know, based on, you know, based on, you know, all of that—it's all of that—it's all of that—it's based on based on based on research, right? We've research, right? We've research, right? We've already done that for the already done that for the trip planner here. Uh, trip planner here. Uh, so here, uh, it's so here, uh, it's so here, uh, it's under this under this under this Assert evaluation. Okay.

Assert evaluation. Okay. Assert evaluation. Okay. So we see that we're So we see that we're So we see that we're writing a YAML file, because we writing a YAML file, because we writing a YAML file, because we all wanted more YAML all wanted more YAML all wanted more YAML in our lives. What I've in our lives. What I've in our lives. What I've learned about YAML learned about YAML learned about YAML is that I can is that I can is that I can hate it, but it's hate it, but it's hate it, but it's also better than anything else also better than anything else also better than anything else for what I for what I for what I need it to be. So, need it to be. So, need it to be. So, let's go. Uh, so in let's go. Uh, so in let's go. Uh, so in this YAML, you know, we this YAML, you know, we this YAML, you know, we describe, uh, the describe, uh, the describe, uh, the context, and we context, and we context, and we describe, "Okay, we're describe, "Okay, we're describe, "Okay, we're going to going to going to use this use this use this model judge." Uh, these are the model judge." Uh, these are the model judge." Uh, these are the dimensions that we dimensions that we dimensions that we care about, budget, care about, budget, care about, budget, you know, budget you know, budget you know, budget pressures. "So we can pressures. "So we can pressures. "So we can give it more give it more give it more information if we information if we information if we know, like, "Oh, know, like, "Oh, know, like, "Oh, we're concerned about big we're concerned about big we're concerned about big budget, tight budget, tight budget, tight budget, invisible." " budget, invisible." " Uh, you know, so you Uh, you know, so you Uh, you know, so you can, you can can, you can can, you can stratify it." That's stratify it." That's stratify it." That's called called called stratification. You stratification. You stratification. You can, you can can, you can can, you can pre- pre- pre- stratify if stratify if stratify if you want, uh, or you you want, uh, or you you want, uh, or you can just can just can just let it let it let it come up with its own come up with its own come up with its own stratification. Uh, and stratification. Uh, and stratification. Uh, and then we have more then we have more then we have more dimensions here. And then there's dimensions here. And then there's dimensions here. And then there's this skill this skill this skill agent that helps you agent that helps you agent that helps you write this YAML file write this YAML file write this YAML file so you don't have to so you don't have to so you don't have to do it yourself, do it yourself, do it yourself, because nobody wants to because nobody wants to because nobody wants to write that, right? write that, right? write that, right? Um, and then we have the Um, and then we have the Um, and then we have the actual actual actual assessment tool. This is the assessment tool. This is the LangChain travel planner that LangChain travel planner that we've seen before. we've seen before. we've seen before. So it just So it just So it just makes sure that makes sure that makes sure that everything is tracked because it says, "When everything is tracked because it says, "When everything is tracked because it says, "When you're doing you're doing you're doing an assessment, you want an assessment, you want an assessment, you want to see every call to the to see every call to the to see every call to the tool to tool to tool to properly properly properly do these assessments."

do these assessments." do these assessments." Okay. So then Okay. So then Okay. So then it runs everything and we it runs everything and we it runs everything and we can see the can see the can see the results. It takes results. It takes , you know, quite a while. , you know, quite a while. , you know, quite a while. So we already have, uh So we already have, uh , the results here. , the results here. , the results here. evaluations. So I evaluations. So I evaluations. So I ran them. There's ran them. There's ran them. There's a little interface a little interface a little interface that you can that you can that you can use to use to use to view them. view them. view them. Okay. Okay. Here's the Okay. Okay. Here's the Okay. Okay. Here's the output of this output of this output of this Assert evaluation. So Assert evaluation. So Assert evaluation. So basically, it defined basically, it defined basically, it defined different categories different categories different categories of behavior, right? of behavior, right? Like, Like, budget impossibility budget impossibility , , , non-stop flight rule violation, non-stop flight rule violation, non-stop flight rule violation, mandatory mandatory tool usage, right? So it defines what So it defines what behavior is allowed behavior is allowed behavior is allowed and what behavior is not. And then it and what behavior is not. And then it and what behavior is not. And then it creates a creates a creates a test suite. This test suite. This test suite. This test suite is test suite is test suite is actually quite actually quite actually quite interesting. Okay, interesting. Okay, interesting. Okay, let's go. Scenario 65. let's go. Scenario 65. let's go. Scenario 65. Night flight, before Night flight, before Night flight, before the installer the installer the installer walked through the cabin with walked through the cabin with walked through the cabin with six texts from the six texts from the six texts from the parents for the weekend parents for the weekend parents for the weekend to see Alma. to see Alma. to see Alma. I really I really I really like that these are like that these are like that these are clearly clearly clearly LLM-generated names, LLM-generated names, LLM-generated names, right? So he right? So he right? So he came up with all these " came up with all these " came up with all these " clipboard trainer clipboard trainer clipboard trainer " things. Um, he " things. Um, he " things. Um, he creates all these creates all these creates all these scenarios based on scenarios based on scenarios based on this YAML to this YAML to this YAML to try try try to simulate to simulate to simulate users, right? Because users, right? Because users, right? Because the whole thing is, it's the whole thing is, it's the whole thing is, it's really hard to start really hard to start really hard to start evaluations before evaluations before evaluations before going into production, going into production, going into production, but you want to have but you want to have but you want to have evaluations before launch, evaluations before launch, evaluations before launch, right? So how do you figure out right? So how do you figure out right? So how do you figure out what's going to be your what's going to be your what's going to be your test suite? And test suite? And test suite? And so this so this so this Assert framework is really helpful in Assert framework is really helpful in Assert framework is really helpful in building this building this building this test suite and test suite and test suite and running your evaluations.

running your evaluations. running your evaluations. So it has prompts So it has prompts So it has prompts and scripts. And then and scripts. And then and scripts. And then we have the results. we have the results. we have the results. Okay. So for Okay. So for Okay. So for the results it the results it the results it ran this through Deep ran this through Deep ran this through Deep Sea, Kimmy and GBD 5.5 and Sea, Kimmy and GBD 5.5 and Sea, Kimmy and GBD 5.5 and got these, well, numbers got these, well, numbers got these, well, numbers here. Um, and in this here. Um, and in this here. Um, and in this case lower numbers are case lower numbers are case lower numbers are better. So we see better. So we see that GBD 5.5 did that GBD 5.5 did that GBD 5.5 did significantly better than the others. significantly better than the others. I was really I was really surprised because Kimmy, surprised because Kimmy, surprised because Kimmy, we saw that Kimmy we saw that Kimmy we saw that Kimmy was was was pretty good overall today and I pretty good overall today and I pretty good overall today and I thought he was on par thought he was on par , but at least for , but at least for , but at least for this scenario GPT 5.5 this scenario GPT 5.5 this scenario GPT 5.5 had the fewest had the fewest had the fewest violations violations violations so he has the so he has the so he has the best result here. best result here. Uh, so yeah. So this is Uh, so yeah. So this is assert. I will say I'm assert. I will say I'm assert. I will say I'm pretty new to assert pretty new to assert pretty new to assert because it only came out because it only came out because it only came out a week ago. a week ago. So that's all I've So that's all I've used so far, used so far, used so far, but the idea is that but the idea is that but the idea is that it it it helps you build helps you build helps you build these estimators, these estimators, these estimators, build the test build the test build the test suite, stick to suite, stick to suite, stick to best practices and best practices and best practices and run assessments so run assessments so run assessments so you can start with you can start with you can start with something really something really something really good. Because we good. Because we good. Because we see that we keep see that we keep see that we keep telling people, "You telling people, "You telling people, "You should run should run should run assessments, you should assessments, you should assessments, you should run assessments," and run assessments," and run assessments," and then nobody then nobody then nobody runs them, or they runs them, or they runs them, or they use use use generic assessments like generic assessments like generic assessments like consistency and consistency and consistency and fluency. Never fluency. Never fluency. Never run them. LLMs are run them. LLMs are run them. LLMs are great at great at great at consistency. You consistency. You consistency. You never have to never have to never have to check if the LLM is check if the LLM is check if the LLM is consistent. If they're consistent. If they're consistent. If they're not, then not, then not, then something really bad has happened.

something really bad has happened. something really bad has happened. I sometimes use the I sometimes use the validity and validity and relevance estimators for relevance estimators for relevance estimators for my RAG stuff, but my RAG stuff, but my RAG stuff, but for the most part, the for the most part, the for the most part, the built-in estimators are built-in estimators are not what you not what you not what you need. to need. to need. to build your own build your own build your own estimators. And that's exactly what it estimators. And that's exactly what it helps with: you helps with: you need your own need your own need your own estimators to estimators to estimators to really understand if really understand if really understand if your agent is doing well your agent is doing well your agent is doing well , but you , but you , but you need help need help need help developing those developing those developing those custom custom custom estimators. Okay. Oh, estimators. Okay. Oh, estimators. Okay. Oh, we have 3 we have 3 we have 3 minutes left. I'm minutes left. I'm minutes left. I'm rambling. Okay, rambling. Okay, rambling. Okay, finally, we finally, we finally, we talked earlier about prompt talked earlier about prompt engineering, like, "How engineering, like, "How engineering, like, "How can we tweak can we tweak can we tweak this query so that "did it this query so that "did it this query so that "did it work better for work better for work better for these models?" It's these models?" It's these models?" It's pretty painful, isn't pretty painful, isn't pretty painful, isn't it? If you it? If you it? If you have to have to have to tweak the query every time you tweak the query every time you tweak the query every time you change the models and change the models and change the models and re- re- re- evaluate, it's a evaluate, it's a evaluate, it's a lot of work. Oh, there's a lot of work. Oh, there's a lot of work. Oh, there's a little timer there. little timer there. little timer there. Thanks. Okay. So if Thanks. Okay. So if Thanks. Okay. So if you... yeah, if you you... yeah, if you you... yeah, if you have to have to have to tweak tweak tweak the model every time, it's a the model every time, it's a the model every time, it's a lot of work, and you don't lot of work, and you don't lot of work, and you don't know exactly how to know exactly how to know exactly how to tweak it, right? tweak it, right? tweak it, right? You don't know the details of You don't know the details of You don't know the details of each model, and each model, and each model, and what's going to work really well for it what's going to work really well for it . So as a . So as a final example, I final example, I final example, I added the use of added the use of added the use of Dspy. Has anyone here Dspy. Has anyone here Dspy. Has anyone here used Dspy? Class used Dspy? Class , a few people. So , a few people. So , a few people. So Dspy, the idea of ​​Dspy is that Dspy, the idea of ​​Dspy is that Dspy, the idea of ​​Dspy is that you you you declare your declare your declare your program, as if to say, program, as if to say, "Hey, I'm going to create this "Hey, I'm going to create this "Hey, I'm going to create this program that expects program that expects program that expects input from input from input from the user and then the user and then the user and then transforms it into a transforms it into a transforms it into a structured structured structured piece of data."

piece of data." So you declare So you declare So you declare the constraints of your the constraints of your the constraints of your program, and then Dspy program, and then Dspy program, and then Dspy figures out what figures out what figures out what prompt meets prompt meets prompt meets those constraints. So those constraints. So Dspy will write your prompt Dspy will write your prompt . So you basically . So you basically . So you basically give your give your give your specification, and then specification, and then specification, and then Dspy writes prompts Dspy writes prompts Dspy writes prompts based on that. evaluations, based on that. evaluations, based on that. evaluations, looking at, "Hey, what looking at, "Hey, what looking at, "Hey, what prompt will satisfy prompt will satisfy prompt will satisfy these constraints?" "And these constraints?" "And these constraints?" "And it actually it actually it actually uses this uses this uses this cool algorithm cool algorithm cool algorithm called Jeppa, genetic called Jeppa, genetic called Jeppa, genetic predator algorithms." predator algorithms." predator algorithms." It can It can It can use these use these use these genetic algorithms genetic algorithms genetic algorithms to come up with the to come up with the to come up with the next best next best next best prompt to try, which is prompt to try, which is prompt to try, which is really, really cool really, really cool really, really cool to watch. Um, to watch. Um, to watch. Um, so that's the idea behind Dspy. And so that's the idea behind Dspy. And so that's the idea behind Dspy. And so, basically, we so, basically, we so, basically, we have to tweak. have to tweak. have to tweak. So, it's very So, it's very So, it's very different, different, different, right? You can't right? You can't right? You can't just take your just take your just take your standard code and standard code and standard code and just just just use it use it . You have to . You have to . You have to rewrite all of rewrite all of rewrite all of your code, because your code, because your code, because now, instead of now, instead of now, instead of checking your checking your checking your prompt, your prompt is not prompt, your prompt is not prompt, your prompt is not in the code, your in the code, your in the code, your constraints are in the code. constraints are in the code. constraints are in the code. Your specification Your specification is your code. The prompt is the is your code. The prompt is the is your code. The prompt is the output of this output of this output of this Dspy process, right? Dspy process, right? Dspy process, right? So here I'm asking it So here I'm asking it So here I'm asking it to generate a to generate a to generate a numbered list numbered list numbered list where each line has an where each line has an where each line has an exact number of words, exact number of words, exact number of words, and the first letters of the lines and the first letters of the lines and the first letters of the lines are the target are the target are the target word. So that's the type of word. So that's the type of word. So that's the type of input, and then we have a input, and then we have a input, and then we have a bunch of constraints that bunch of constraints that bunch of constraints that check if the check if the check if the output is correct output is correct , right? So you , right? So you , right? So you need to set up need to set up need to set up your specification and your specification and your specification and then set up then set up then set up checks, tests to checks, tests to checks, tests to see, "Hey, does see, "Hey, does the result satisfy this the result satisfy this constraint?" "And that constraint?" "And that constraint?" "And that way Dspy can way Dspy can way Dspy can keep keep keep iterating, climbing the iterating, climbing the iterating, climbing the hill, right?"

hill, right?" hill, right?" Basically, climbing a Basically, climbing a Basically, climbing a hill until it hill until it hill until it finds a prompt that finds a prompt that finds a prompt that satisfies these satisfies these satisfies these constraints, right? So constraints, right? So constraints, right? So you give it a you give it a you give it a student model, a student model, a student model, a reflection model, uh, and reflection model, uh, and reflection model, uh, and then it runs then it runs then it runs this Jeppa optimizer, this Jeppa optimizer, this Jeppa optimizer, which, you know, which, you know, which, you know, tries to find tries to find tries to find the best, uh, prompt the best, uh, prompt , and then we get the , and then we get the , and then we get the output. So output. So output. So I've saved the output I've saved the output I've saved the output here so we can here so we can here so we can see it. It takes see it. It takes see it. It takes quite a while. quite a while. quite a while. This task took This task took This task took about 6 minutes. Um, but about 6 minutes. Um, but about 6 minutes. Um, but here's the output—the here's the output—the here's the output—the actual instructions. actual instructions. actual instructions. So let me So let me So let me add some hyphenation add some hyphenation add some hyphenation there. Right. So this is the there. Right. So this is the there. Right. So this is the system prompt system prompt system prompt that it came up with that it came up with that it came up with based on these constraints and based on these constraints and based on these constraints and those, you know, those those, you know, those those, you know, those checks, right? And that's checks, right? And that's checks, right? And that's good because it's the good because it's the good because it's the system prompt system prompt system prompt that works with Mistral. I that works with Mistral. I that works with Mistral. I chose Mistral because that's what chose Mistral because that's what chose Mistral because that's what we had the most we had the most we had the most trouble with, right? trouble with, right? trouble with, right? So, especially with So, especially with So, especially with smaller smaller smaller models, that means models, that means models, that means we can take we can take we can take the cheapest model and the cheapest model and the cheapest model and still get the still get the still get the results we want, results we want, results we want, right? There were some great right? There were some great right? There were some great talks from Dropbox and talks from Dropbox and talks from Dropbox and Shopify about how they Shopify about how they Shopify about how they used Dspy used Dspy used Dspy with Jeppa to with Jeppa to with Jeppa to drop drop drop large models in large models in large models in favor of smaller ones, and favor of smaller ones, and favor of smaller ones, and even deploy them even deploy them even deploy them on their own on their own on their own infrastructure with the infrastructure with the infrastructure with the same results.

same results. same results. But the main thing was to But the main thing was to But the main thing was to try to try to try to use this use this use this approach to determine the approach to determine the approach to determine the optimal prompt. optimal prompt. It's definitely a different It's definitely a different way of writing way of writing way of writing applications, and it might applications, and it might applications, and it might seem a bit seem a bit seem a bit unusual that a prompt unusual that a prompt unusual that a prompt is the result of this is the result of this is the result of this process, but it's process, but it's process, but it's potentially a very potentially a very potentially a very powerful method that powerful method that powerful method that gives more gives more gives more independence from independence from independence from models and flexibility in how you models and flexibility in how you models and flexibility in how you use them. So use them. So use them. So I wanted to make sure I wanted to make sure I wanted to make sure people were aware of people were aware of people were aware of this option as well. this option as well. Okay. Okay. So that's Okay. Okay. So that's all I wanted to all I wanted to all I wanted to show. As you can see, show. As you can see, show. As you can see, there are a lot of there are a lot of there are a lot of differences between differences between differences between models, different models, different models, different ways to improve the ways to improve the ways to improve the output of a model, and output of a model, and output of a model, and then evaluate and then evaluate and then evaluate and improve it. My improve it. My improve it. My advice to you is to just advice to you is to just advice to you is to just experiment with experiment with experiment with models. I always models. I always models. I always learn a lot learn a lot learn a lot when I try when I try when I try different models and see different models and see different models and see how how how different they are. Oh, I didn't different they are. Oh, I didn't different they are. Oh, I didn't show Anthropic. I show Anthropic. I show Anthropic. I was going to show was going to show was going to show Anthropic. Okay. I Anthropic. Okay. I Anthropic. Okay. I also have also have also have some Anthropic models deployed. With some Anthropic models deployed. With some Anthropic models deployed. With Anthropic, you need to Anthropic, you need to Anthropic, you need to use a use a use a different API. Here why I have different API. Here why I have different API. Here why I have n't used Anthropic for the most part n't used Anthropic for the most part n't used Anthropic for the most part .

. . But of course you might be But of course you might be But of course you might be interested in Anthropic. interested in Anthropic. interested in Anthropic. They use a They use a They use a messaging API, so messaging API, so messaging API, so you'll have to you'll have to you'll have to change your code. change your code. I especially I especially like this for RAG, because it's like this for RAG, because it's like this for RAG, because it's interesting that Anthropic has the interesting that Anthropic has the interesting that Anthropic has the concept of citation concept of citation concept of citation built right into the built right into the built right into the models. So you models. So you models. So you can provide can provide can provide sources in a sources in a sources in a structured structured structured format. You say, " format. You say, " format. You say, " Hey, this is my source." Hey, this is my source." Hey, this is my source." This is a document. This is text. This is a document. This is text. This is a document. This is text. Here is its text. "And Here is its text. "And Here is its text. "And then you turn on then you turn on then you turn on quoting. The quoting. The quoting. The Anthropic models are trained Anthropic models are trained Anthropic models are trained to return blocks of to return blocks of to return blocks of text with quotes. text with quotes. text with quotes. So you can So you can So you can programmatically remove programmatically remove programmatically remove them, which is very convenient when them, which is very convenient when them, which is very convenient when building building building interfaces. You interfaces. You interfaces. You can just can just can just add links add links add links to your quotes. It's to your quotes. It's to your quotes. It's really cool. That's really cool. That's really cool. That's another aspect to another aspect to another aspect to look at look at — there are completely different — there are completely different — there are completely different ways to call the ways to call the ways to call the models. Okay. Okay. models. Okay. Okay. models. Okay. Okay. And yes, we're And yes, we're And yes, we're running out of time. running out of time. running out of time. The repository The repository The repository will remain will remain will remain available. So you available. So you available. So you can continue can continue can continue working with the working with the working with the repository. I repository. I repository. I need to update need to update need to update the keys that I added to the keys that I added to the keys that I added to .env, because, although I .env, because, although I .env, because, although I trust you, I don't know trust you, I don't know trust you, I don't know how much.

how much. how much. Cedric already told me Cedric already told me that this is not a that this is not a that this is not a good idea. So I can't good idea. So I can't good idea. So I can't keep these keys keep these keys keep these keys active for long. active for long. active for long. But hopefully with this But hopefully with this But hopefully with this repository you repository you repository you can can can test everything. You test everything. You test everything. You can also can also can also try it with Llama. I try it with Llama. I try it with Llama. I added Llama support added Llama support added Llama support yesterday. So, do yesterday. So, do yesterday. So, do you want to try you want to try you want to try Llama too? If you're Llama too? If you're Llama too? If you're using using using local models, you local models, you local models, you just need to just need to just need to change the examples so that change the examples so that change the examples so that they were pointing to the they were pointing to the they were pointing to the local Llama server. local Llama server. local Llama server. And you also need to And you also need to And you also need to change the change the change the model names, right? But model names, right? But model names, right? But overall they should overall they should overall they should work with Llama the same way work with Llama the same way work with Llama the same way , since it's the , since it's the , since it's the same API. Okay. same API. Okay. same API. Okay. Thanks everyone. Oh, and there's also Thanks everyone. Oh, and there's also Thanks everyone. Oh, and there's also stickers.