How to Actually Tokenmaxx and Save Millions — Alvin Pane, OutRival

Hello everyone, I'm Alvin, the Hello everyone, I'm Alvin, the head of the head of the head of the intelligent intelligent intelligent agents department at Outrival. agents department at Outrival. agents department at Outrival. I guess everyone has already heard I guess everyone has already heard I guess everyone has already heard of "token maxing". of "token maxing". of "token maxing". Companies are even Companies are even Companies are even creating rankings of creating rankings of creating rankings of those who use those who use those who use the most AI tokens. the most AI tokens. the most AI tokens. Meta even Meta even Meta even had a special had a special had a special position for this. position for this. position for this. Token legend. A badge of honor Token legend. A badge of honor Token legend. A badge of honor for the one who burns for the one who burns for the one who burns the most the most the most computing computing computing resources. It's a funny resources. It's a funny resources. It's a funny meme, but if you're meme, but if you're meme, but if you're building a real building a real building a real product, product, product, using tokens using tokens is a strange indicator of is a strange indicator of is a strange indicator of success. Tokens are not the success. Tokens are not the success. Tokens are not the work itself. This is what work itself. This is what work itself. This is what you pay for getting the you pay for getting the you pay for getting the job done. At job done. At job done. At Outrival, our digital Outrival, our digital Outrival, our digital workers complete workers complete workers complete complex tasks complex tasks complex tasks via phone, text, via phone, text, via phone, text, and email. and email. and email. At the application level, we At the application level, we At the application level, we process process process billions of tokens every day. And billions of tokens every day. And billions of tokens every day. And when you work at when you work at when you work at this scale, this scale, this scale, every extra query to the every extra query to the every extra query to the model model model translates into translates into translates into huge delays huge delays huge delays and millions of dollars in and millions of dollars in and millions of dollars in costs. Take a look at costs. Take a look at costs. Take a look at this request. I created a this request. I created a this request. I created a synthetic proxy synthetic proxy synthetic proxy that demonstrates what that demonstrates what that demonstrates what structural structural structural bloat looks like in the bloat looks like in the bloat looks like in the pipeline. Let's imagine pipeline. Let's imagine pipeline. Let's imagine we are classifying we are classifying we are classifying customer intent for the customer intent for the customer intent for the fictional company fictional company fictional company Harbor Byte. We've all seen something like this Harbor Byte. We've all seen something like this Harbor Byte. We've all seen something like this . There are 20 . There are 20 . There are 20 edge cases, edge cases, edge cases, overly pedantic overly pedantic overly pedantic definitions, and a paragraph definitions, and a paragraph definitions, and a paragraph that almost begs that almost begs that almost begs the model not the model not the model not to hallucinate. When you to hallucinate. When you to hallucinate. When you send that much send that much send that much text with each text with each text with each request, millions of request, millions of request, millions of times a day, it's times a day, it's times a day, it's no longer just a prompt, but a no longer just a prompt, but a no longer just a prompt, but a huge huge huge computational tax that computational tax that computational tax that you've imposed on yourself you've imposed on yourself . If you want to . If you want to . If you want to empower your agents with empower your agents with empower your agents with more capabilities, more capabilities, more capabilities, you need the you need the you need the architectural backend architectural backend architectural backend to do so. So to do so. So to do so. So today we are changing the today we are changing the today we are changing the concept of "token concept of "token maxing." It should maxing." It should maxing." It should n't be about n't be about n't be about burning the burning the burning the most

most most tokens, but about tokens, but about tokens, but about extracting the most extracting the most extracting the most intelligence from the intelligence from the intelligence from the least least least amount of tokens possible. So, amount of tokens possible. So, amount of tokens possible. So, here's how to really here's how to really here's how to really maximize maximize maximize the effectiveness of tokens the effectiveness of tokens . Let's move on to the . Let's move on to the . Let's move on to the action plan. First, action plan. First, action plan. First, observe. You can't observe. You can't observe. You can't fix what you fix what you fix what you can't can't can't measure. Configure a measure. Configure a measure. Configure a shared shared shared model client so that each model client so that each model client so that each attempt becomes attempt becomes attempt becomes part of the trace, part of the trace, part of the trace, combining combining combining model calls, tools, model calls, tools, model calls, tools, and retries and retries and retries into a single task. into a single task. into a single task. Record the Record the Record the prompt version, prompt version, prompt version, model settings, model settings, model settings, token usage token usage , latency, and , latency, and , latency, and cost. OpenTelemetry is a cost. OpenTelemetry is a great framework great framework great framework for tracking for tracking for tracking this. Now, with this this. Now, with this this. Now, with this telemetry data, you telemetry data, you telemetry data, you can create can create can create a benchmark of how your a benchmark of how your a benchmark of how your production is performing. But this is where production is performing. But this is where production is performing. But this is where most most most optimization efforts optimization efforts optimization efforts hit a hit a hit a wall. Let me wall. Let me wall. Let me tell you how this tell you how this tell you how this usually usually usually happens. You happens. You happens. You take your take your take your huge prompt, huge prompt, huge prompt, give it to a smaller, give it to a smaller, give it to a smaller, cheaper model, and cheaper model, and cheaper model, and run it on a run it on a run it on a few thousand few thousand few thousand historical examples historical examples historical examples from production. We from production. We from production. We did it. Our cheaper did it. Our cheaper did it. Our cheaper candidate model candidate model candidate model disagreed with the expensive disagreed with the expensive disagreed with the expensive production base production base production base about 18% of the about 18% of the time. Of course, we time. Of course, we time. Of course, we refused it.

refused it. refused it. She wasn't She wasn't She wasn't good enough. good enough. good enough. But what does But what does But what does that 18% actually mean? that 18% actually mean? Since language Since language models in the workplace models in the workplace deal with deal with ambiguous ambiguous ambiguous input, this 18% input, this 18% is just a summary is just a summary is just a summary figure. It figure. It figure. It hides a critically hides a critically hides a critically important variable: the important variable: the important variable: the underlying variability of the underlying variability of the underlying variability of the model itself. When model itself. When model itself. When the input the input the input is confusing, your is confusing, your is confusing, your working prompt working prompt will inevitably contradict your own will inevitably contradict your own previous decisions. previous decisions. To isolate this To isolate this hidden variable and hidden variable and hidden variable and understand what understand what your edit actually changed, we your edit actually changed, we asked a very asked a very asked a very simple question. Does the simple question. Does the simple question. Does the working model agree working model agree working model agree with itself? We with itself? We with itself? We conducted AA testing. conducted AA testing. conducted AA testing. We call this the " We call this the " ceiling test." We ceiling test." We ceiling test." We took our huge, took our huge, took our huge, expensive working model, expensive working model, expensive working model, ran it three times on the ran it three times on the ran it three times on the same input same input same input data, and compared data, and compared data, and compared the responses. We found the responses. We found that it contradicted that it contradicted that it contradicted itself about itself about itself about 7% of the time.

7% of the time. 7% of the time. The consistency of The consistency of The consistency of the results of the the results of the the results of the model itself was model itself was model itself was only about 93%. So, only about 93%. So, only about 93%. So, we demanded we demanded we demanded perfection from a perfection from a perfection from a cheap cheap candidate model that candidate model that candidate model that our our our existing expensive existing expensive existing expensive system could not achieve. Once you system could not achieve. Once you system could not achieve. Once you set this set this set this “ “ “ ceiling test” benchmark, you have a ceiling test” benchmark, you have a ceiling test” benchmark, you have a measurable baseline for measurable baseline for measurable baseline for comparison. Now comparison. Now comparison. Now you can start you can start you can start making work cheaper. making work cheaper. making work cheaper. There are three technical levers There are three technical levers that I usually that I usually that I usually use, and they use, and they use, and they work in sequence. work in sequence. work in sequence. The first is caching. The first is caching. In particular, In particular, structuring structuring structuring prompts to prompts to prompts to maximize the frequency of maximize the frequency of prefix cache usage is a very prefix cache usage is a very simple and effective simple and effective simple and effective way to reduce way to reduce way to reduce the cost of the cost of the cost of task execution. The second task execution. The second task execution. The second lever is to reduce lever is to reduce lever is to reduce the size of the the size of the the size of the prompt itself. So let's prompt itself. So let's prompt itself. So let's go back to the go back to the Harbor Bite prompt example. This prompt is Harbor Bite prompt example. This prompt is almost 5000 almost 5000 almost 5000 tokens long. And why is that? tokens long. And why is that? tokens long. And why is that? Because over time, Because over time, Because over time, the developers constantly the developers constantly the developers constantly added new paragraphs added new paragraphs added new paragraphs with rules.

with rules. For example, the rule " don't confuse don't confuse don't confuse enthusiasm with enthusiasm with enthusiasm with authority" is authority" is one of them. Plus, there were one of them. Plus, there were one of them. Plus, there were huge hard- huge hard- huge hard- coded coded coded transcripts that transcripts that transcripts that served as ready-made served as ready-made served as ready-made examples. No one examples. No one examples. No one wants to be the one who wants to be the one who wants to be the one who deletes a line and deletes a line and deletes a line and breaks a working breaks a working breaks a working system. So what to do? system. So what to do? system. So what to do? Automate it. Automate it. Automate it. Turn Turn Turn prompting into a prompting into a prompting into a search task. With a search task. With a search task. With a framework like DSPy, you framework like DSPy, you framework like DSPy, you can stop can stop can stop manually writing manually writing manually writing instructions. You instructions. You instructions. You define a clear define a clear define a clear interface contract, interface contract, interface contract, input data, input data, input data, output schema, and output schema, and output schema, and evaluation function. And then evaluation function. And then evaluation function. And then you pass it on to an you pass it on to an you pass it on to an optimizer like optimizer like optimizer like Jeppa. The optimizer Jeppa. The optimizer Jeppa. The optimizer treats your prompt treats your prompt treats your prompt as a search space as a search space . It works with your . It works with your . It works with your benchmark, benchmark, benchmark, trying trying trying to match to match to match historical marks, and historical marks, and historical marks, and uses uses uses error feedback to error feedback to error feedback to correct correct correct instructions. He instructions. He instructions. He removed almost all the removed almost all the removed almost all the long examples and long examples and long examples and replaced them with replaced them with replaced them with compact compact compact rules. The result is a rules. The result is a 51% reduction in the number of instruction tokens while 51% reduction in the number of instruction tokens while maintaining the same maintaining the same maintaining the same functionality.

functionality. functionality. Now you can Now you can Now you can check whether check whether check whether the quality is maintained. the quality is maintained. the quality is maintained. This is an instant This is an instant This is an instant increase in increase in increase in margin. The margin. The margin. The third lever is third lever is third lever is directly directly directly reducing the size of reducing the size of reducing the size of the model. With a concise the model. With a concise the model. With a concise prompt and a measured prompt and a measured prompt and a measured search cycle, you don't search cycle, you don't search cycle, you don't always need a always need a always need a huge, cutting-edge huge, cutting-edge huge, cutting-edge model. You can model. You can model. You can start aggressively start aggressively start aggressively reducing reducing reducing computational computational computational costs. There are two costs. There are two costs. There are two interesting strategies here. interesting strategies here. First, you can break a First, you can break a First, you can break a huge prompt into huge prompt into huge prompt into smaller parts. You smaller parts. You smaller parts. You use the use the use the cheaper model for cheaper model for cheaper model for initial initial initial routing or routing or routing or data extraction, and data extraction, and only engage the expensive frontier models for complex only engage the expensive frontier models for complex considerations. Second, considerations. Second, considerations. Second, you can use you can use you can use the same DSP search loop the same DSP search loop the same DSP search loop we just we just we just created, but created, but created, but connect a connect a connect a smaller model to it. You smaller model to it. You smaller model to it. You allow the smaller allow the smaller allow the smaller model to search model to search model to search its own prompt its own prompt space to space to space to match the match the match the reference answers reference answers generated by your generated by your generated by your larger model. larger model. larger model. Essentially, you Essentially, you Essentially, you use Jeppa use Jeppa use Jeppa to train a tiny to train a tiny to train a tiny model to achieve model to achieve model to achieve the performance of a the performance of a the performance of a huge model huge model huge model on your on your on your dataset. But the reality of dataset. But the reality of dataset. But the reality of optimization is not so optimization is not so optimization is not so simple when it simple when it simple when it comes to these comes to these comes to these small models. If small models. If small models. If you try to you try to you try to squeeze the behavior of a squeeze the behavior of a squeeze the behavior of a large model into a large model into a large model into a smaller one, you will find that smaller one, you will find that smaller one, you will find that you create new points of you create new points of you create new points of failure and failure and failure and retraining. And in retraining. And in retraining. And in many of our many of our downsizing experiments, downsizing experiments, we found that the we found that the we found that the prompts were not of sufficient prompts were not of sufficient prompts were not of sufficient quality for release.

quality for release. Which brings us to the Which brings us to the final hurdle. final hurdle. final hurdle. We've We've We've halved the Harbor Lights prompt, but halved the Harbor Lights prompt, but halved the Harbor Lights prompt, but now we have to prove now we have to prove now we have to prove that we didn't break the product that we didn't break the product . Remember, these . Remember, these . Remember, these models are probabilistic. The models are probabilistic. The production system is not production system is not always consistent always consistent always consistent with itself, and with itself, and with itself, and similarly our new similarly our new similarly our new cheaper candidate cheaper candidate cheaper candidate will have its own will have its own will have its own internal variations. internal variations. So you run a So you run a large-scale shadow large-scale shadow large-scale shadow deployment on the deployment on the deployment on the same input data and same input data and same input data and evaluate the system from evaluate the system from evaluate the system from three angles. three angles. Production vs. Production vs. production, candidate production, candidate production, candidate vs. candidate, and vs. candidate, and vs. candidate, and candidate vs. candidate vs. candidate vs. production. There is a production. There is a production. There is a useful useful useful mathematical aspect here. mathematical aspect here. mathematical aspect here. You take You take You take the difference between the the difference between the the difference between the two versions and two versions and two versions and subtract the average subtract the average subtract the average difference difference difference within each of within each of within each of them. What remains them. What remains gives a measure of gives a measure of gives a measure of how much how much how much their their their decision probabilities differ. This is where the decision probabilities differ. This is where the "ceiling" test comes into play. "ceiling" test comes into play. We don't guess. We We don't guess. We We don't guess. We have a mathematical basis have a mathematical basis have a mathematical basis for how much for how much for how much the responses change the responses change when we don't when we don't when we don't touch anything. And here's the secret touch anything. And here's the secret touch anything. And here's the secret to this drift.

to this drift. to this drift. Different doesn't Different doesn't Different doesn't necessarily mean necessarily mean necessarily mean wrong. If wrong. If wrong. If your optimizer your optimizer your optimizer has done its job, has done its job, has done its job, the distributions should be the distributions should be the distributions should be different because the different because the different because the cheaper prompt cheaper prompt cheaper prompt actually performs actually performs actually performs better. We are not better. We are not better. We are not necessarily necessarily necessarily trying to create a trying to create a trying to create a perfect clone of a perfect clone of a perfect clone of a production system, production system, production system, which may already have which may already have which may already have certain certain certain flaws built into it. So you flaws built into it. So you flaws built into it. So you stop stop stop panicking about panicking about panicking about gross differences, gross differences, gross differences, unload them and unload them and unload them and manually review manually review manually review some of them. Was the some of them. Was the some of them. Was the candidate better? candidate better? candidate better? Was the production Was the production Was the production system better? system better? system better? Were these simply two different Were these simply two different , equally acceptable , equally acceptable , equally acceptable ways of handling ways of handling ways of handling ambiguous input ambiguous input ? For Harbor Bite, let's say ? For Harbor Bite, let's say ? For Harbor Bite, let's say the inspection the inspection the inspection confirmed 443 confirmed 443 confirmed 443 regressions, but also regressions, but also regressions, but also found 2,463 significant found 2,463 significant found 2,463 significant improvements. That's almost improvements. That's almost improvements. That's almost six improvements for six improvements for six improvements for every regression at a every regression at a every regression at a prompt that's half as prompt that's half as prompt that's half as small. But you shouldn't small. But you shouldn't small. But you shouldn't release a product release a product release a product based solely on a based solely on a based solely on a point score. You point score. You point score. You have to take into account have to take into account have to take into account sampling noise. We have sampling noise. We have sampling noise. We have seen significant seen significant seen significant growth on this growth on this growth on this query set, but query set, but query set, but we need to know we need to know we need to know how confident we can how confident we can how confident we can be in these be in these be in these results. Therefore, we results. Therefore, we results. Therefore, we calculate a calculate a calculate a confidence interval confidence interval confidence interval for this net for this net for this net increase. You should increase. You should increase. You should strictly match strictly match strictly match pairs so that both pairs so that both pairs so that both versions answer the versions answer the versions answer the same queries and same queries and same queries and repeated repeated repeated conversation sessions remain conversation sessions remain conversation sessions remain grouped. To grouped. To grouped. To claim that a claim that a claim that a candidate is better, we candidate is better, we candidate is better, we want the lower want the lower want the lower bound of the confidence bound of the confidence bound of the confidence interval to be greater interval to be greater interval to be greater than zero. If than zero. If than zero. If the interval crosses the interval crosses the interval crosses zero, we need zero, we need zero, we need more evidence more evidence more evidence before making such statements before making such statements . And one final

. And one final . And one final check: you should check: you should check: you should perform these perform these perform these calculations separately for your calculations separately for your calculations separately for your critical critical critical categories. A great categories. A great categories. A great overall overall overall average may average may average may actually be actually be actually be hiding the fact hiding the fact hiding the fact that one of your vital that one of your vital customer requests is customer requests is failing. If failing. If failing. If your critical your critical your critical categories pass categories pass categories pass the test and the lower the test and the lower the test and the lower confidence limit is above confidence limit is above confidence limit is above zero—you have your zero—you have your zero—you have your answer. answer. answer. Release and Release and Release and continue to continue to continue to measure. So this measure. So this measure. So this process is a way process is a way process is a way to replace the guesswork in to replace the guesswork in to replace the guesswork in prompt engineering prompt engineering prompt engineering with a clear with a clear with a clear release rule. Cache the release rule. Cache the release rule. Cache the obvious things. obvious things. obvious things. Let Let Let the optimizer the optimizer the optimizer shorten your shorten your shorten your instructions. instructions. instructions. Choose the right Choose the right Choose the right model size and model size and model size and use the use the use the "ceiling" test to "ceiling" test to "ceiling" test to understand understand understand the variability. Then the variability. Then the variability. Then check if check if check if the quality is maintained. the quality is maintained. the quality is maintained. So, here's your So, here's your So, here's your homework for homework for homework for Monday morning. Monday morning. Monday morning. Choose your Choose your Choose your most expensive task. most expensive task. most expensive task. Measure your Measure your Measure your baseline variance. baseline variance. baseline variance. Run it Run it Run it through the search cycle and through the search cycle and through the search cycle and find out find out find out how much extra you're how much extra you're how much extra you're really paying for. really paying for. really paying for. We need to We need to We need to bring back bring back bring back the efficiency of the efficiency of the efficiency of token usage token usage . Let the rest of . Let the rest of . Let the rest of the industry compete the industry compete the industry compete for high scores in for high scores in for high scores in the ratings. The true the ratings. The true the ratings. The true token legends are token legends are token legends are those who make each those who make each those who make each individual token individual token individual token work for work for work for results.