Why AI Writing Still Reads Like Slop — Louis-François Bouchard, Towards AI

So, I'm here to So, I'm here to talk about all these talk about all these talk about all these words that we words that we words that we call "slop," and more call "slop," and more call "slop," and more importantly, I want to importantly, I want to importantly, I want to try to help try to help try to help fix what fix what fix what Karpati called the "slop Karpati called the "slop apocalypse," which apocalypse," which apocalypse," which he believed was supposed to he believed was supposed to he believed was supposed to start in 2026 start in 2026 , and I think we're already , and I think we're already , and I think we're already seeing it on seeing it on seeing it on social media. social media. So to try to So to try to fix this and fix this and fix this and stop the "slop", we stop the "slop", we stop the "slop", we first need to first need to first need to define it. define it. So, first of all, what So, first of all, what is a "slope"? There are is a "slope"? There are is a "slope"? There are many different many different many different definitions from different definitions from different definitions from different experts in the experts in the experts in the field, but I think field, but I think field, but I think we can all we can all we can all agree that reading agree that reading agree that reading something that looks like something that looks like something that looks like generated text is generated text is generated text is truly terrifying. truly terrifying. And it usually And it usually looks like this, with looks like this, with looks like this, with all these words all these words all these words that we know, like " that we know, like " delve" or "generative" and delve" or "generative" and delve" or "generative" and other other other things like that, and the main things like that, and the main things like that, and the main problem is that problem is that problem is that even as the models even as the models even as the models get better, with get better, with get better, with the advent of GPT-6.1 and other the advent of GPT-6.1 and other the advent of GPT-6.1 and other improved improved improved models, the "slop" is models, the "slop" is models, the "slop" is not going away. It not going away. It not going away. It just keeps just keeps just keeps scaling. So it scaling. So it scaling. So it looks like we need looks like we need looks like we need something more than something more than something more than just additional just additional just additional training of the models. And training of the models. And training of the models. And I really want to I really want to I really want to at least first at least first at least first define what " define what " slop" is so we can slop" is so we can slop" is so we can understand why it understand why it understand why it occurs. So, in my opinion occurs. So, in my opinion occurs. So, in my opinion , a "slope" , a "slope" , a "slope" consists of six consists of six consists of six main things.

main things. First, there are many words First, there are many words First, there are many words that are used too often by that are used too often by language models but language models but that we hardly ever that we hardly ever that we hardly ever use ourselves, such as “delve,” use ourselves, such as “delve,” use ourselves, such as “delve,” which we know well. which we know well. which we know well. Secondly, it is Secondly, it is Secondly, it is usually too usually too usually too general. He always general. He always general. He always talks about talks about talks about most things, most things, most things, most projects. most projects. most projects. He doesn't share anything He doesn't share anything He doesn't share anything specific. And I'm specific. And I'm specific. And I'm guilty of guilty of guilty of using this on using this on using this on LinkedIn, which is probably the " LinkedIn, which is probably the " purpose" of LinkedIn, so purpose" of LinkedIn, so purpose" of LinkedIn, so it's the norm there. And worse it's the norm there. And worse , sloppy "slop" - , sloppy "slop" - the text usually the text usually the text usually says nothing. These are says nothing. These are says nothing. These are just beautiful words just beautiful words just beautiful words put together that we put together that we put together that we don't even don't even don't even use in use in use in everyday life. And what's everyday life. And what's everyday life. And what's worse, not only does he worse, not only does he worse, not only does he say nothing, he also say nothing, he also say nothing, he also keeps keeps keeps repeating himself. So he repeating himself. So he repeating himself. So he repeats nothing, and he is repeats nothing, and he is repeats nothing, and he is too general. And, too general. And, too general. And, worse yet, he's worse yet, he's worse yet, he's hallucinating. As we hallucinating. As we hallucinating. As we know, models know, models know, models hallucinate, so we ca hallucinate, so we ca hallucinate, so we ca n't completely n't completely n't completely trust sloppy trust sloppy trust sloppy text. And lastly, the text. And lastly, the text. And lastly, the worst thing for me is that worst thing for me is that worst thing for me is that he usually he usually he usually has the wrong tone, has the wrong tone, has the wrong tone, or at least no or at least no or at least no tone at all, tone at all, tone at all, while trying while trying while trying to be everyone's tone to be everyone's tone to be everyone's tone at once. So he at once. So he at once. So he just doesn't talk like just doesn't talk like just doesn't talk like you. He doesn't you. He doesn't you. He doesn't share the things share the things share the things you would share.

you would share. So my goal today , along with the project , along with the project , along with the project I've been working on, I've been working on, I've been working on, is to first and foremost is to first and foremost is to first and foremost measure this measure this measure this sloppy sloppy sloppy writing style. Not just writing style. Not just writing style. Not just words. I don't want words. I don't want words. I don't want to create another " to create another " EQ-Bench slop score". It's EQ-Bench slop score". It's EQ-Bench slop score". It's about measuring about measuring about measuring how how how sloppy the text is. sloppy the text is. So to do this, So to do this, we are releasing a new benchmark today, right we are releasing a new benchmark today, right we are releasing a new benchmark today, right now, now, now, which which which we have called Tone-Bench. The we have called Tone-Bench. The main purpose of this main purpose of this benchmark is to measure benchmark is to measure benchmark is to measure how well the new models how well the new models how well the new models handle handle handle creative writing, which is creative writing, which is creative writing, which is also what I will try to also what I will try to also what I will try to determine. So, here's the determine. So, here's the determine. So, here's the plan for today. plan for today. plan for today. The plan is to The plan is to The plan is to justify whether justify whether justify whether we even need we even need we even need another benchmark? another benchmark? another benchmark? Then—how it works, Then—how it works, Then—how it works, what we measure, how we what we measure, how we what we measure, how we built it, and I built it, and I built it, and I want to share the want to share the want to share the development cost, development cost, development cost, our solutions, what our solutions, what we've already learned we've already learned we've already learned about AI writing about AI writing thanks to this thanks to this benchmark, and benchmark, and benchmark, and tell you a little more tell you a little more tell you a little more about it. So, let's about it. So, let's about it. So, let's dive into this.

dive into this. So, the first question is: So, the first question is: do we even do we even do we even need another need another need another benchmark? And, just benchmark? And, just benchmark? And, just looking at LinkedIn and looking at LinkedIn and looking at LinkedIn and the answers in X, my the answers in X, my the answers in X, my answers in X, I think I answers in X, I think I answers in X, I think I need it. It seems like need it. It seems like need it. It seems like people are spamming with people are spamming with people are spamming with more and more more and more more and more generated generated generated content. So, let content. So, let content. So, let it at least be of it at least be of it at least be of sufficient quality. And, sufficient quality. And, sufficient quality. And, more precisely, I really more precisely, I really more precisely, I really need it too. We are a need it too. We are a small team at small team at small team at Towards AI, my company, Towards AI, my company, Towards AI, my company, and we have a focus and we have a focus and we have a focus on on on education. We create education. We create education. We create a lot of educational a lot of educational a lot of educational content, content, content, mostly for mostly for mostly for AI engineering. And as a AI engineering. And as a AI engineering. And as a small team, we, small team, we, small team, we, of course, also of course, also of course, also use use use AI for this. And it's extremely AI for this. And it's extremely AI for this. And it's extremely annoying—constantly annoying—constantly annoying—constantly reviewing this reviewing this reviewing this junk and trying to junk and trying to junk and trying to fix it. We...I've fix it. We...I've fix it. We...I've posted hundreds of posted hundreds of posted hundreds of videos. We videos. We videos. We have published thousands of have published thousands of have published thousands of articles. We published a articles. We published a articles. We published a book. We are working book. We are working book. We are working on a second one, on a second one, on a second one, dedicated to dedicated to dedicated to AI engineering. So we AI engineering. So we AI engineering. So we really need the really need the really need the best models that best models that best models that will help us will help us will help us write better and not write better and not write better and not fill our texts with fill our texts with fill our texts with junk. And another junk. And another junk. And another nice addition to nice addition to nice addition to this—I this—I this—I shared information shared information shared information about this benchmark in X for quite some time about this benchmark in X for quite some time about this benchmark in X for quite some time while I was developing it.

while I was developing it. It received a very It received a very good response, and people good response, and people good response, and people seem to seem to seem to confirm that we confirm that we confirm that we really needed really needed really needed such a benchmark for such a benchmark for such a benchmark for creative writing. And creative writing. And creative writing. And finally, to prove finally, to prove finally, to prove that I consider us the that I consider us the that I consider us the team that should be team that should be team that should be working on this working on this working on this benchmark, I will say that benchmark, I will say that benchmark, I will say that I have been evaluating AI models I have been evaluating AI models I have been evaluating AI models since my master's since my master's since my master's and PhD days. And now at Towards AI, and PhD days. And now at Towards AI, and PhD days. And now at Towards AI, we are building we are building we are building assessment pipelines for our assessment pipelines for our assessment pipelines for our clients. We have 15 clients. We have 15 clients. We have 15 AI engineers, as well as AI engineers, as well as AI engineers, as well as creative writers and creative writers and creative writers and editors on the team. editors on the team. editors on the team. So we already have that for So we already have that for So we already have that for our own our own our own content. So I content. So I content. So I wanted to use this wanted to use this wanted to use this to create a to create a to create a benchmark and benchmark and benchmark and help the community even more. help the community even more. So, what exactly did we So, what exactly did we build? build? build? I hope I'm I hope I'm I hope I'm trying to share trying to share trying to share as much as much as much information as possible information as possible , and I'd be happy to , and I'd be happy to , and I'd be happy to chat if chat if chat if we can help we can help we can help you create something of your you create something of your you create something of your own or if I can own or if I can own or if I can just share just share just share our our our experiences. So what experiences. So what experiences. So what we've created is we've created is we've created is very subjective. This is a very subjective. This is a very subjective. This is a way to measure whether a way to measure whether a way to measure whether a model can write like model can write like model can write like me. And I define me. And I define me. And I define it this way: if I can it this way: if I can it this way: if I can read it out loud read it out loud read it out loud right now without right now without right now without wincing at words wincing at words wincing at words I wouldn't I wouldn't I wouldn't use myself. So, use myself. So, use myself. So, the process is very simple.

the process is very simple. the process is very simple. The benchmark, the The benchmark, the The benchmark, the benchmark process, the benchmark process, the benchmark process, the testing process is very testing process is very testing process is very simple. We have a simple. We have a simple. We have a prompt, which is a brief prompt, which is a brief prompt, which is a brief and research, and we and research, and we and research, and we send it to the send it to the send it to the model, hoping model, hoping model, hoping that it will generate an that it will generate an that it will generate an entire video script entire video script entire video script that I could write that I could write that I could write . Of course, this doesn't . Of course, this doesn't . Of course, this doesn't happen. It usually happen. It usually happen. It usually turns out pretty turns out pretty turns out pretty bad, but that's the bad, but that's the bad, but that's the main goal, and we main goal, and we main goal, and we evaluate this evaluate this evaluate this final script final script final script against our criteria against our criteria against our criteria and our entire and our entire and our entire evaluation pipeline evaluation pipeline . And the prompt we . And the prompt we . And the prompt we send to each send to each send to each model in the benchmark goes model in the benchmark goes model in the benchmark goes much deeper than much deeper than much deeper than that. It looks that. It looks that. It looks something like this: we something like this: we something like this: we have a lot of details have a lot of details have a lot of details about how I speak about how I speak , how I write, a bunch of , how I write, a bunch of , how I write, a bunch of examples from my examples from my examples from my previous videos. We previous videos. We previous videos. We have a lot of have a lot of have a lot of information about information about information about storytelling in general, storytelling in general, storytelling in general, active sentences, and active sentences, and active sentences, and how to write how to write how to write well. And we have well. And we have well. And we have instructions on instructions on instructions on what to generate and what what to generate and what what to generate and what to do for that to do for that to do for that task, after which task, after which task, after which the model should generate the model should generate the model should generate one scenario. And this is one scenario. And this is one scenario. And this is just one of them. In the just one of them. In the just one of them. In the current benchmark, current benchmark, current benchmark, we have 10 different we have 10 different we have 10 different scenarios that scenarios that scenarios that we have we have we have worked hard to worked hard to worked hard to make as good as possible make as good as possible make as good as possible , and we , and we , and we have created have created have created scoring criteria for them scoring criteria for them scoring criteria for them to help LLM to help LLM judges evaluate these judges evaluate these judges evaluate these generations. So, we generations. So, we generations. So, we have 10 different have 10 different have 10 different scenarios. Some of scenarios. Some of scenarios. Some of them are published, them are published, them are published, some are not. We are not some are not. We are not some are not. We are not saying what saying what saying what the benchmarks are exactly, to the benchmarks are exactly, to the benchmarks are exactly, to also check for also check for data contamination. So

data contamination. So what do we do with these what do we do with these what do we do with these 10 scenarios? We 10 scenarios? We 10 scenarios? We take all the models that take all the models that take all the models that exist and exist and exist and try to evaluate try to evaluate try to evaluate their results. This their results. This their results. This means that for means that for means that for each model, we each model, we each model, we repeat this repeat this repeat this process 10 times, process 10 times, process 10 times, generating an article at the generating an article at the generating an article at the prompt. prompt. prompt. We actually generate five of We actually generate five of We actually generate five of them. I'll them. I'll them. I'll tell you why soon. Then tell you why soon. Then tell you why soon. Then four different LLM judges four different LLM judges four different LLM judges analyze them. analyze them. analyze them. I recently added I recently added I recently added Jeff for Jeff for Jeff for testing. Not testing. Not testing. Not sure if I'll leave sure if I'll leave sure if I'll leave this in, but I added it this in, but I added it this in, but I added it just to just to just to see if see if see if it works well. And then we it works well. And then we it works well. And then we calculate the final calculate the final calculate the final score for the score for the score for the leaderboard. I'll go leaderboard. I'll go leaderboard. I'll go through all these steps through all these steps through all these steps from left to right. So, first of all from left to right. So, first of all from left to right. So, first of all , why are we , why are we , why are we generating generating generating five scenarios at all? This is to five scenarios at all? This is to check for check for intra-model intra-model intra-model variability, which I variability, which I variability, which I strongly recommend strongly recommend strongly recommend doing if you're doing if you're doing if you're creating your own creating your own creating your own estimates, because especially, I estimates, because especially, I estimates, because especially, I think, in creative think, in creative think, in creative tasks, as we tasks, as we tasks, as we see here in the see here in the see here in the Grok 4.7 example, some Grok 4.7 example, some Grok 4.7 example, some generations are outliers, generations are outliers, generations are outliers, where as we see from where as we see from where as we see from the benchmark, the model is the benchmark, the model is the benchmark, the model is as good as Opus 5.5 as good as Opus 5.5 , but only in one , but only in one , but only in one generation out of five. So generation out of five. So generation out of five. So we want to measure that we want to measure that , and the worse the model, , and the worse the model, , and the worse the model, the greater the the greater the the greater the variability. We variability. We variability. We have some conclusions have some conclusions have some conclusions on this, which I will on this, which I will on this, which I will share at the end.

share at the end. But yes, we found But yes, we found that five drafts that five drafts that five drafts seemed like a golden seemed like a golden seemed like a golden mean in terms of the mean in terms of the mean in terms of the cost for us to cost for us to cost for us to execute and execute and execute and measure that measure that measure that variability. So variability. So variability. So what exactly are we measuring what exactly are we measuring what exactly are we measuring and how can we and how can we and how can we quantify the quantify the quantify the quality of writing in general? It quality of writing in general? It quality of writing in general? It was a challenge and we was a challenge and we was a challenge and we worked on it for worked on it for worked on it for quite some time. It's quite some time. It's quite some time. It's all exposed on all exposed on all exposed on the benchmark now, and there the benchmark now, and there 's a lot more detail on the benchmark page as well 's a lot more detail on the benchmark page as well . But we . But we . But we measure nine measure nine measure nine different metrics that we've different metrics that we've different metrics that we've detailed, detailed, detailed, but I can boil but I can boil but I can boil them down to five main them down to five main them down to five main aspects. First, we aspects. First, we aspects. First, we measure the tone. I mean, does measure the tone. I mean, does measure the tone. I mean, does that sound like me? This that sound like me? This that sound like me? This means, can means, can means, can the model adapt the model adapt the model adapt to a specific tone? to a specific tone? Second, we Second, we measure the pure measure the pure measure the pure quality of the writing. Are quality of the writing. Are quality of the writing. Are there good hooks, there good hooks, there good hooks, good emotional good emotional good emotional transitions, and all the transitions, and all the transitions, and all the aspects we aspects we aspects we look for in a good look for in a good look for in a good story. Then we story. Then we story. Then we measure whether measure whether measure whether she follows she follows she follows the instructions. Does the instructions. Does the instructions. Does it meet the it meet the it meet the expected length expected length expected length and and and format? Did format? Did format? Did she place she place she place the right images the right images the right images and things in the right and things in the right and things in the right places? Fourth, places? Fourth, places? Fourth, we measure we measure we measure the accuracy of the research.

the accuracy of the research. We have a research We have a research brief for each brief for each brief for each scenario, and we just scenario, and we just scenario, and we just check whether there were check whether there were check whether there were hallucinations or whether the hallucinations or whether the hallucinations or whether the model used the model used the model used the research research research correctly. Finally, correctly. Finally, correctly. Finally, we measure "slap" using we measure "slap" using EQ-Bench's score as well as our EQ-Bench's score as well as our own measurements. And what own measurements. And what own measurements. And what it looks like from the it looks like from the it looks like from the judges' point of view judges' point of view judges' point of view looks something like this looks something like this looks something like this . We have about . We have about . We have about a thousand words of a thousand words of a thousand words of examples, examples, examples, instructions, and instructions, and instructions, and guidance on guidance on guidance on each metric, which each metric, which each metric, which we provide along with a we provide along with a we provide along with a script for script for script for analysis, my own analysis, my own analysis, my own script as a script as a script as a guide, and a guide, and a guide, and a brief and brief and brief and research. research. In total, it's a In total, it's a 14,000-word request that we 14,000-word request that we 14,000-word request that we send to each send to each send to each judge to then judge to then judge to then get feedback, get feedback, get feedback, which we'll delve into which we'll delve into which we'll delve into further. And therefore, the judge further. And therefore, the judge further. And therefore, the judge analyzes only the quality of analyzes only the quality of analyzes only the quality of the writing. He doesn't the writing. He doesn't the writing. He doesn't focus on focus on focus on length and doesn't know what length and doesn't know what length and doesn't know what model he's writing. We model he's writing. We model he's writing. We do this later using the do this later using the code itself. For one code itself. For one metric, like tone and metric, like tone and metric, like tone and voice, which is our voice, which is our voice, which is our primary criterion, primary criterion, primary criterion, it looks it looks it looks something like this: we something like this: we something like this: we have a lot of have a lot of have a lot of detailed descriptions detailed descriptions detailed descriptions of what my tone is, of what my tone is, of what my tone is, what what what to expect from it, and how to to expect from it, and how to to expect from it, and how to analyze it. Then analyze it. Then analyze it. Then the judge first the judge first the judge first divides divides divides the results into the results into the results into six categories, six categories, six categories, and after determining the and after determining the and after determining the category, gives category, gives category, gives a score. We found this to be the a score. We found this to be the a score. We found this to be the best way best way best way to grade for

to grade for to grade for such a qualitative such a qualitative such a qualitative task, as opposed task, as opposed task, as opposed to the binary system to the binary system to the binary system that is commonly advised that is commonly advised that is commonly advised , or the , or the , or the direct selection of direct selection of direct selection of categories or categories or categories or percentages. We percentages. We percentages. We found that it found that it found that it works really well. works really well. It contains many It contains many examples of what examples of what examples of what each score means in each score means in each score means in terms of terms of terms of writing for each writing for each writing for each task. And here's what task. And here's what the judge sends in response. It contains the judge sends in response. It contains scores for nine scores for nine scores for nine different criteria with different criteria with different criteria with justification for justification for justification for each score and an each score and an each score and an example such as: example such as: example such as: "I gave this "I gave this "I gave this score because of this score because of this score because of this specific example." specific example." But the main thing is that all But the main thing is that all this is automated. this is automated. this is automated. Of course, since we Of course, since we Of course, since we have 120 models, we have 120 models, we have 120 models, we generate five generate five generate five drafts for each drafts for each drafts for each of the 10 tasks. This quickly of the 10 tasks. This quickly of the 10 tasks. This quickly leads to thousands of leads to thousands of leads to thousands of scripts to scripts to scripts to review, so we can't review, so we can't review, so we can't review review review them ourselves. And them ourselves. And them ourselves. And the problem with LLM judges the problem with LLM judges the problem with LLM judges is that is that is that they have their own they have their own they have their own preferences. Some preferences. Some preferences. Some models models models prefer themselves.

prefer themselves. prefer themselves. Most models Most models Most models prefer themselves prefer themselves prefer themselves . And, in particular . And, in particular , OpenAI. And what I found , OpenAI. And what I found , OpenAI. And what I found quite funny quite funny quite funny while studying this while studying this while studying this is that Anthropic and is that Anthropic and is that Anthropic and OpenAI seem to be at war OpenAI seem to be at war , where they seem to , where they seem to have learned to hate have learned to hate each other after training, which is each other after training, which is each other after training, which is quite funny. But quite funny. But quite funny. But all this is to say that all this is to say that all this is to say that when you when you when you use LLM use LLM judges, such as GPT, judges, such as GPT, judges, such as GPT, and want to know and want to know and want to know which model is best which model is best which model is best for your task, for your task, for your task, be careful and be careful and be careful and use different use different use different families of models families of models families of models to make sure, to make sure, to make sure, because we see because we see because we see that OpenAI has a strong that OpenAI has a strong that OpenAI has a strong bias towards bias towards bias towards its own family of its own family of its own family of models, not just models, not just models, not just one model, but one model, but one model, but all OpenAI models. And all OpenAI models. And all OpenAI models. And to fix that, we to fix that, we to fix that, we listened to listened to listened to a few comments a few comments a few comments I got at X about I got at X about I got at X about using people using people using people for creative tasks, for creative tasks, for creative tasks, which would be ideal which would be ideal which would be ideal but pretty but pretty but pretty impractical. We still impractical. We still impractical. We still tried to do tried to do tried to do it in a scalable it in a scalable it in a scalable way, by way, by way, by manually comparing—me and three of manually comparing—me and three of manually comparing—me and three of my colleagues—1,000 pairs of AV my colleagues—1,000 pairs of AV my colleagues—1,000 pairs of AV generated generated generated scripts, not full scripts, not full scripts, not full texts, but texts, but texts, but significant parts of them that significant parts of them that significant parts of them that could be compared between could be compared between could be compared between two scripts, two scripts, two scripts, two generations.

two generations. Hopefully we will Hopefully we will do more and I do more and I do more and I want to make this public want to make this public want to make this public to get to get to get help from the help from the help from the community in the community in the community in the future, but not future, but not future, but not now. So, what did it now. So, what did it now. So, what did it cost us and how cost us and how cost us and how much work was much work was much work was involved? For me, for involved? For me, for involved? For me, for us, it was an us, it was an us, it was an initial investment of initial investment of initial investment of about 500 about 500 about 500 hours to create 10 hours to create 10 hours to create 10 scenarios, scenarios, scenarios, define all the define all the define all the criteria, find criteria, find criteria, find examples and try to examples and try to examples and try to assess the quality, which is assess the quality, which is assess the quality, which is very difficult. And then I very difficult. And then I very difficult. And then I just worked just worked just worked alone with four alone with four alone with four different 20X subscriptions different 20X subscriptions different 20X subscriptions for 4 months for 4 months for 4 months to build to build to build a benchmark, the rest of the stuff a benchmark, the rest of the stuff , experiment and , experiment and , experiment and change the change the metric ratios. So, we metric ratios. So, we spent spent spent thousands of dollars on this. And thousands of dollars on this. And thousands of dollars on this. And if you want to if you want to if you want to do this type of do this type of do this type of benchmark, you should benchmark, you should benchmark, you should know that it is know that it is know that it is quite expensive. Now, quite expensive. Now, quite expensive. Now, if I change just if I change just if I change just one thing in the criteria one thing in the criteria one thing in the criteria or in how the judge or in how the judge or in how the judge scores the generations, it will scores the generations, it will scores the generations, it will cost us cost us cost us over $2,000. And over $2,000. And over $2,000. And if I change just if I change just if I change just one line in the input one line in the input one line in the input prompt for the model prompt for the model prompt for the model during generation, it will during generation, it will during generation, it will cost us cost us cost us over $4,000. So over $4,000. So over $4,000. So experimenting and experimenting and experimenting and changing these things in changing these things in changing these things in prompts is very expensive.

prompts is very expensive. prompts is very expensive. Therefore, it is worth doing this Therefore, it is worth doing this Therefore, it is worth doing this carefully. So what carefully. So what carefully. So what can we get from can we get from can we get from this benchmark, from this benchmark, from this benchmark, from Tone Bench? You can Tone Bench? You can Tone Bench? You can find a lot of things find a lot of things find a lot of things on the website itself. There on the website itself. There on the website itself. There are a lot of details there. are a lot of details there. But the important thing is that I But the important thing is that I want to share what I want to share what I want to share what I 've learned over the past 4 've learned over the past 4 've learned over the past 4 months: for months: for months: for creative writing, creative writing, creative writing, you don't need you don't need you don't need to choose a model that to choose a model that to choose a model that will generate the perfect will generate the perfect will generate the perfect post on the first try. post on the first try. You just need to You just need to find the model that find the model that will annoy you the least on the way will annoy you the least on the way to the desired to the desired to the desired result. So, I result. So, I result. So, I think perfection wo think perfection wo think perfection wo n't happen anytime soon n't happen anytime soon n't happen anytime soon . This means that . This means that . This means that currently on the benchmark, the currently on the benchmark, the currently on the benchmark, the GLM 5.3 flash model is GLM 5.3 flash model is GLM 5.3 flash model is extremely good extremely good extremely good and very cheap. It is and very cheap. It is and very cheap. It is 50 times cheaper than the 50 times cheaper than the 50 times cheaper than the best model, Opus 5.5 best model, Opus 5.5 best model, Opus 5.5 extra high, which in extra high, which in extra high, which in turn is 2.5 times turn is 2.5 times turn is 2.5 times cheaper than Max thinking, cheaper than Max thinking, cheaper than Max thinking, but gives the same but gives the same but gives the same results. So you results. So you results. So you generally shouldn't generally shouldn't generally shouldn't use Max use Max use Max for writing. Additionally, the for writing. Additionally, the for writing. Additionally, the best way to best way to best way to write with models write with models write with models that we have found is to that we have found is to that we have found is to have an extremely have an extremely have an extremely long and detailed long and detailed long and detailed prompt. It can't prompt. It can't prompt. It can't be be be long enough, and the longer long enough, and the longer long enough, and the longer the better. You should the better. You should the better. You should add every example add every example add every example that comes to mind that comes to mind that comes to mind of what to do and how to do it.

of what to do and how to do it. of what to do and how to do it. The same goes for The same goes for The same goes for negative examples of things negative examples of things negative examples of things you don't you don't you don't like, things that like, things that like, things that sound wrong, sound wrong, sound wrong, words you don't want words you don't want words you don't want to see, and everything to see, and everything to see, and everything else. You can else. You can else. You can share each one, if share each one, if share each one, if you're writing newsletters or you're writing newsletters or you're writing newsletters or something like that, you something like that, you something like that, you can share them in their can share them in their can share them in their entirety, and it'll entirety, and it'll entirety, and it'll only be better. And what we've only be better. And what we've only be better. And what we've seen that really seen that really seen that really helps a lot is helps a lot is helps a lot is to use the to use the to use the smart judge only smart judge only smart judge only once, one once, one once, one to three times, to three times, to three times, otherwise it will start otherwise it will start otherwise it will start to repeat itself. So, to repeat itself. So, to repeat itself. So, just one review of another just one review of another just one review of another model with the same model with the same model with the same instructions that will instructions that will instructions that will give give give feedback to the author. feedback to the author. Actually, I do Actually, I do n't have any quantitative n't have any quantitative n't have any quantitative metrics on metrics on metrics on this, but it this, but it this, but it helps a lot. And for a helps a lot. And for a helps a lot. And for a company, if you company, if you company, if you are working with APIs and want to are working with APIs and want to are working with APIs and want to get the best get the best get the best results at results at results at scale, I suggest scale, I suggest scale, I suggest generating multiple generating multiple generating multiple drafts, say 10 drafts, say 10 drafts, say 10 variants of something variants of something variants of something using a using a using a cheaper model like cheaper model like GLM 5.3 flash. And then GLM 5.3 flash. And then GLM 5.3 flash. And then ask a ask a ask a smarter judge, like smarter judge, like Opus or Opus or Opus or whoever, to pick and whoever, to pick and whoever, to pick and combine the best combine the best combine the best parts of each of those parts of each of those parts of each of those drafts, which I drafts, which I drafts, which I think would still think would still think would still cost significantly cost significantly cost significantly less than one less than one less than one generation of Opus 5.5. And generation of Opus 5.5. And generation of Opus 5.5. And I think that will I think that will I think that will lead to better lead to better lead to better results, judging from what results, judging from what results, judging from what I've seen. And I've seen. And I've seen. And lastly: you need lastly: you need lastly: you need human verification.

human verification. human verification. This really cannot This really cannot This really cannot be avoided. But all these be avoided. But all these be avoided. But all these comparative tests comparative tests comparative tests measure, obviously, measure, obviously, measure, obviously, my voice, so it's my voice, so it's my voice, so it's very subjective, and it's very subjective, and it's very subjective, and it's just one point of view, just one point of view, just one point of view, so you shouldn't so you shouldn't so you shouldn't trust it blindly. This does trust it blindly. This does trust it blindly. This does not mean that the Opus 5.5 is the not mean that the Opus 5.5 is the best model. So, best model. So, best model. So, but ideally this is a but ideally this is a but ideally this is a strong signal to strong signal to strong signal to find which model find which model find which model writes well and which is less writes well and which is less writes well and which is less sloppy. Um, I advise sloppy. Um, I advise sloppy. Um, I advise you to build your own you to build your own you to build your own evaluation pipelines. evaluation pipelines. evaluation pipelines. If you want, we If you want, we If you want, we can help. You can help. You can help. You can write in the chat can write in the chat can write in the chat or contact us or contact us . As for the . As for the . As for the benchmark test, benchmark test, benchmark test, it's already working. We it's already working. We it's already working. We literally just literally just literally just published and published and published and released it. So, released it. So, released it. So, there are lots of graphs there are lots of graphs , lots of insights, , lots of insights, , lots of insights, lots of comparisons and lots of comparisons and implementation details, a full implementation details, a full methodology. So methodology. So methodology. So just check it out just check it out just check it out and hopefully and hopefully and hopefully it will be useful. it will be useful. it will be useful. Thank you.