I Made My AI Agents Faster Without a GPU Upgrade

This is one coding agent working through This is one coding agent working through a task on an RTX Pro 6000 turn by turn. a task on an RTX Pro 6000 turn by turn. a task on an RTX Pro 6000 turn by turn. On the first turn, it answers in about On the first turn, it answers in about On the first turn, it answers in about 0.12 of a second. And you can see it's 0.12 of a second. And you can see it's 0.12 of a second. And you can see it's getting a little bit slower, right? 23 getting a little bit slower, right? 23 getting a little bit slower, right? 23 24. And finally, turn 25, 19 seconds 24. And finally, turn 25, 19 seconds 24. And finally, turn 25, 19 seconds before it writes a single word. And this before it writes a single word. And this before it writes a single word. And this is because on every turn, it has more is because on every turn, it has more is because on every turn, it has more and more to read. [music] With agents, and more to read. [music] With agents, and more to read. [music] With agents, caching doesn't give you a flat speed caching doesn't give you a flat speed caching doesn't give you a flat speed up. The effect multiplies. The up. The effect multiplies. The up. The effect multiplies. The conversation gets longer at every turn. conversation gets longer at every turn. conversation gets longer at every turn. And every agent you add is rereading its And every agent you add is rereading its And every agent you add is rereading its own history on one GPU. So in this own history on one GPU. So in this own history on one GPU. So in this video, I'll go through what KV cache video, I'll go through what KV cache video, I'll go through what KV cache does for agents and where LM Cash fits does for agents and where LM Cash fits does for agents and where LM Cash fits in. Think about how many websites have in. Think about how many websites have in. Think about how many websites have your personal information. GitHub, your personal information. GitHub, your personal information. GitHub, developer tools, random services you developer tools, random services you developer tools, random services you signed up for years ago. Data brokers signed up for years ago. Data brokers signed up for years ago. Data brokers collect those details and sell them. collect those details and sell them. collect those details and sell them. Your personal information shouldn't be Your personal information shouldn't be Your personal information shouldn't be that easy to find. Privacy laws in that easy to find. Privacy laws in that easy to find. Privacy laws in dozens of countries give you the right dozens of countries give you the right dozens of countries give you the right to request that data brokers delete your to request that data brokers delete your to request that data brokers delete your information. But good luck hunting down information. But good luck hunting down information. But good luck hunting down hundreds of brokers, submitting hundreds of brokers, submitting hundreds of brokers, submitting requests, and checking whether your requests, and checking whether your requests, and checking whether your information shows up again. That's information shows up again. That's information shows up again. That's [music] where Incogn comes in. It [music] where Incogn comes in. It [music] where Incogn comes in. It automatically sends those removal automatically sends those removal automatically sends those removal requests and follows up with the brokers requests and follows up with the brokers requests and follows up with the brokers for me. My dashboard identified over 231 for me. My dashboard identified over 231 for me. My dashboard identified over 231 databases containing my details and most databases containing my details and most databases containing my details and most have already been cleared. And if I find have already been cleared. And if I find have already been cleared. And if I find my information on another website, I my information on another website, I my information on another website, I submit a link through custom removals submit a link through custom removals submit a link through custom removals and a privacy expert handles the removal and a privacy expert handles the removal and a privacy expert handles the removal for me. They find the data, request its for me. They find the data, request its for me. They find the data, request its removal, and keep following up. And for removal, and keep following up. And for removal, and keep following up. And for additional reassurance, Deoid has additional reassurance, Deoid has additional reassurance, Deoid has verified incogn's data removal process.

verified incogn's data removal process. verified incogn's data removal process. Now, it doesn't cover government records Now, it doesn't cover government records Now, it doesn't cover government records or random social posts, [music] but it or random social posts, [music] but it or random social posts, [music] but it can tackle data brokers and eligible can tackle data brokers and eligible can tackle data brokers and eligible websites. And there's a 30-day money websites. And there's a 30-day money websites. And there's a 30-day money back guarantee if you want to give it a back guarantee if you want to give it a back guarantee if you want to give it a try. Take your personal information back try. Take your personal information back try. Take your personal information back with incogn. Go to incogn.com/exist with incogn. Go to incogn.com/exist with incogn. Go to incogn.com/exist and use code lexiscin for 60% off an and use code lexiscin for 60% off an and use code lexiscin for 60% off an annual plan. Link is down below. Now, annual plan. Link is down below. Now, annual plan. Link is down below. Now, back to the video. back to the video. back to the video. So there's this project called LM cache. So there's this project called LM cache. So there's this project called LM cache. It's a cache layer for VLM. And VLM is It's a cache layer for VLM. And VLM is It's a cache layer for VLM. And VLM is that open source server that runs a that open source server that runs a that open source server that runs a large language model. But VLM already large language model. But VLM already large language model. But VLM already has a cache built in. So what does LM has a cache built in. So what does LM has a cache built in. So what does LM cache do that VLM doesn't? And how do cache do that VLM doesn't? And how do cache do that VLM doesn't? And how do both of them compare to no cache at all? both of them compare to no cache at all? both of them compare to no cache at all? First, let's take a look at what cache First, let's take a look at what cache First, let's take a look at what cache actually saves you. Before the model actually saves you. Before the model actually saves you. Before the model writes a single word, it reads your writes a single word, it reads your writes a single word, it reads your whole prompt and it takes notes while it whole prompt and it takes notes while it whole prompt and it takes notes while it reads. Let's just call it notes for reads. Let's just call it notes for reads. Let's just call it notes for simplicity. Technically, it's called KV simplicity. Technically, it's called KV simplicity. Technically, it's called KV cache. You've probably heard that term cache. You've probably heard that term cache. You've probably heard that term before. K&V stand for keys and values.

before. K&V stand for keys and values. before. K&V stand for keys and values. Just like studying for a test, if you Just like studying for a test, if you Just like studying for a test, if you keep your notes around, next time you keep your notes around, next time you keep your notes around, next time you have to reread your book, uh you don't have to reread your book, uh you don't have to reread your book, uh you don't have to reread the whole book. You just have to reread the whole book. You just have to reread the whole book. You just read whatever is new in the lesson that read whatever is new in the lesson that read whatever is new in the lesson that the teacher taught you. the teacher taught you. the teacher taught you. Pro tip, [music] I haven't been to Pro tip, [music] I haven't been to Pro tip, [music] I haven't been to college in a long time, by the way. I college in a long time, by the way. I college in a long time, by the way. I don't know. Is that how kids do it these don't know. Is that how kids do it these don't know. Is that how kids do it these days still? Here's the same 30 turn days still? Here's the same 30 turn days still? Here's the same 30 turn session. This is no cash. Look how much session. This is no cash. Look how much session. This is no cash. Look how much prompt processing has to happen with prompt processing has to happen with prompt processing has to happen with each turn. There's more and more and each turn. There's more and more and each turn. There's more and more and more. It rereads 115,000 tokens by the more. It rereads 115,000 tokens by the more. It rereads 115,000 tokens by the time it's done at the end. And that's time it's done at the end. And that's time it's done at the end. And that's over 2 million tokens over the whole over 2 million tokens over the whole over 2 million tokens over the whole session. With the cache though, this is session. With the cache though, this is session. With the cache though, this is VLM's prefix cache. By the way, with the VLM's prefix cache. By the way, with the VLM's prefix cache. By the way, with the cache, it never reads more than about cache, it never reads more than about cache, it never reads more than about 10,000 tokens every turn. That's 14 10,000 tokens every turn. That's 14 10,000 tokens every turn. That's 14 times less reading. But those notes times less reading. But those notes times less reading. But those notes aren't free. They live in the GPU memory aren't free. They live in the GPU memory aren't free. They live in the GPU memory right next to the model. The model right next to the model. The model right next to the model. The model itself already takes up a big chunk of itself already takes up a big chunk of itself already takes up a big chunk of the card. And whatever is left over, the card. And whatever is left over, the card. And whatever is left over, that's what your budget is for notes. that's what your budget is for notes. that's what your budget is for notes. That's usually why you don't want a 32 That's usually why you don't want a 32 That's usually why you don't want a 32 GB model to run on a 5090 with 32 GB of GB model to run on a 5090 with 32 GB of GB model to run on a 5090 with 32 GB of memory. Because hey, we need room for memory. Because hey, we need room for memory. Because hey, we need room for notes. When the memory fills up, the notes. When the memory fills up, the notes. When the memory fills up, the oldest notes get thrown out, and the oldest notes get thrown out, and the oldest notes get thrown out, and the next time you rerun them, you're back to next time you rerun them, you're back to next time you rerun them, you're back to rereading everything again. So, where do rereading everything again. So, where do rereading everything again. So, where do you keep the notes? Well, I compared you keep the notes? Well, I compared you keep the notes? Well, I compared four setups. One, no cache, where it four setups. One, no cache, where it four setups. One, no cache, where it rereads everything every time. We rereads everything every time. We rereads everything every time. We already seen what that does, right? Two, already seen what that does, right? Two, already seen what that does, right? Two, VLM's pre-built prefix cache. It helps.

VLM's pre-built prefix cache. It helps. VLM's pre-built prefix cache. It helps. It reuses the notes as long as the It reuses the notes as long as the It reuses the notes as long as the prompt starts the same way and keeps prompt starts the same way and keeps prompt starts the same way and keeps them in GPU memory inside the VLM them in GPU memory inside the VLM them in GPU memory inside the VLM process. And that one's on by default in process. And that one's on by default in process. And that one's on by default in the VLM I tested, by the way. So, when I the VLM I tested, by the way. So, when I the VLM I tested, by the way. So, when I did the no cache run, I actually had to did the no cache run, I actually had to did the no cache run, I actually had to turn it off. Three is VLM's own CPU turn it off. Three is VLM's own CPU turn it off. Three is VLM's own CPU offload, which lets those notes spill offload, which lets those notes spill offload, which lets those notes spill into regular RAM, system RAM, but it's into regular RAM, system RAM, but it's into regular RAM, system RAM, but it's still inside VLM. And four is our new still inside VLM. And four is our new still inside VLM. And four is our new friend, LM cache. It keeps the notes in friend, LM cache. It keeps the notes in friend, LM cache. It keeps the notes in a separate server outside of VLM in RAM a separate server outside of VLM in RAM a separate server outside of VLM in RAM and sometimes even on NVME drives. and sometimes even on NVME drives. and sometimes even on NVME drives. That's the SSDs. That's important to That's the SSDs. That's important to That's the SSDs. That's important to know because what happens when the VLM know because what happens when the VLM know because what happens when the VLM process dies or when you reboot the process dies or when you reboot the process dies or when you reboot the computer? [music] computer? [music] computer? [music] I started off on the DJX station GB300 I started off on the DJX station GB300 I started off on the DJX station GB300 running quen 3 235B. So a pretty decent running quen 3 235B. So a pretty decent running quen 3 235B. So a pretty decent size model. I fed it a 32,000 token size model. I fed it a 32,000 token size model. I fed it a 32,000 token document. Asked the question and then document. Asked the question and then document. Asked the question and then asked about it again with no cash about asked about it again with no cash about asked about it again with no cash about 1.35 seconds and any cash any one of 1.35 seconds and any cash any one of 1.35 seconds and any cash any one of those three I mentioned much better those three I mentioned much better those three I mentioned much better right they range from 112 to 122 right they range from 112 to 122 right they range from 112 to 122 seconds. Sorry milliseconds.

seconds. Sorry milliseconds. seconds. Sorry milliseconds. Milliseconds. Milliseconds. Milliseconds. Yeah much faster. About 11 or 12 times Yeah much faster. About 11 or 12 times Yeah much faster. About 11 or 12 times faster in fact. Now, 1.35 seconds faster in fact. Now, 1.35 seconds faster in fact. Now, 1.35 seconds doesn't sound like much, but with no doesn't sound like much, but with no doesn't sound like much, but with no cash, an agent pays that on every single cash, an agent pays that on every single cash, an agent pays that on every single turn. And we know what it's like waiting turn. And we know what it's like waiting turn. And we know what it's like waiting for an agent to do his thing, right? for an agent to do his thing, right? for an agent to do his thing, right? Yeah, we're spoiled. Next, I made it Yeah, we're spoiled. Next, I made it Yeah, we're spoiled. Next, I made it more difficult. What if the documents more difficult. What if the documents more difficult. What if the documents don't all fit in GPU memory? The DJX don't all fit in GPU memory? The DJX don't all fit in GPU memory? The DJX station has room for just under a station has room for just under a station has room for just under a million tokens of notes, so I sent it 48 million tokens of notes, so I sent it 48 million tokens of notes, so I sent it 48 documents. That's 1.5 million tokens. documents. That's 1.5 million tokens. documents. That's 1.5 million tokens. And I have a chart for that, too. VLM's And I have a chart for that, too. VLM's And I have a chart for that, too. VLM's GPU cache. That's that orange bar. 0% GPU cache. That's that orange bar. 0% GPU cache. That's that orange bar. 0% hits on 48 documents. That's bad. It hits on 48 documents. That's bad. It hits on 48 documents. That's bad. It filled up somewhere between 16 documents filled up somewhere between 16 documents filled up somewhere between 16 documents to 48 documents. And then it just kicked to 48 documents. And then it just kicked to 48 documents. And then it just kicked the old stuff out. And by the time I the old stuff out. And by the time I the old stuff out. And by the time I came back to a document, it was gone. So came back to a document, it was gone. So came back to a document, it was gone. So that ended up being about as slow as no that ended up being about as slow as no that ended up being about as slow as no cache at all. So then I let the cache cache at all. So then I let the cache cache at all. So then I let the cache spill into regular RAM. Ooh, the station spill into regular RAM. Ooh, the station spill into regular RAM. Ooh, the station has a huge amount of CPU memory. That's has a huge amount of CPU memory. That's has a huge amount of CPU memory. That's that LPDDR5X that LPDDR5X that LPDDR5X that's actually in the same pool. It's that's actually in the same pool. It's that's actually in the same pool. It's like unified memory with the HBM or the like unified memory with the HBM or the like unified memory with the HBM or the high bandwidth memory that we were using high bandwidth memory that we were using high bandwidth memory that we were using before that's next to the GPU. Anyway, I before that's next to the GPU. Anyway, I before that's next to the GPU. Anyway, I made a whole separate video on the DJX made a whole separate video on the DJX made a whole separate video on the DJX station and how the memory works and the station and how the memory works and the station and how the memory works and the breakdown. I'll link to it down below if breakdown. I'll link to it down below if breakdown. I'll link to it down below if you're interested. VLM's own offload 164 you're interested. VLM's own offload 164 you're interested. VLM's own offload 164 milliseconds. So, we got 99.9% milliseconds. So, we got 99.9% milliseconds. So, we got 99.9% hits there. And LMCH 140 milliseconds, hits there. And LMCH 140 milliseconds, hits there. And LMCH 140 milliseconds, almost the same. and they both fixed the almost the same. and they both fixed the almost the same. and they both fixed the problem. Well, that's great, right? So, problem. Well, that's great, right? So, problem. Well, that's great, right? So, then I hit it with 16 clients or about

then I hit it with 16 clients or about then I hit it with 16 clients or about 144 concurrent requests. No cache 187 144 concurrent requests. No cache 187 144 concurrent requests. No cache 187 seconds. That's a long time. VLM's GPU seconds. That's a long time. VLM's GPU seconds. That's a long time. VLM's GPU prefix cache 107 seconds with CPU system prefix cache 107 seconds with CPU system prefix cache 107 seconds with CPU system RAM offload from VLM 69 seconds. Some of RAM offload from VLM 69 seconds. Some of RAM offload from VLM 69 seconds. Some of you might comment about that down in the you might comment about that down in the you might comment about that down in the comments below. That's a 3minut job that comments below. That's a 3minut job that comments below. That's a 3minut job that now finishes in just over a minute. But now finishes in just over a minute. But now finishes in just over a minute. But LM cache and VLM's offload are pretty LM cache and VLM's offload are pretty LM cache and VLM's offload are pretty much tied. If LM's cache isn't faster, much tied. If LM's cache isn't faster, much tied. If LM's cache isn't faster, what's it for? what's it for? what's it for? So, I restarted VLM. VLM's prefix cache So, I restarted VLM. VLM's prefix cache So, I restarted VLM. VLM's prefix cache after the restart about 1.35 seconds. after the restart about 1.35 seconds. after the restart about 1.35 seconds. CPU offload about the same. They both CPU offload about the same. They both CPU offload about the same. They both lost everything because their notes, lost everything because their notes, lost everything because their notes, their cache lived inside the process their cache lived inside the process their cache lived inside the process that I just killed. LM cache, boom. 141 that I just killed. LM cache, boom. 141 that I just killed. LM cache, boom. 141 milliseconds. 99.9% hits. Now the milliseconds. 99.9% hits. Now the milliseconds. 99.9% hits. Now the difference shows up. See, its notes difference shows up. See, its notes difference shows up. See, its notes lived in a separate server that kept on lived in a separate server that kept on lived in a separate server that kept on running. And when VLM came back, it running. And when VLM came back, it running. And when VLM came back, it reconnected and picked up where it left reconnected and picked up where it left reconnected and picked up where it left off. First time I tried this, the new off. First time I tried this, the new off. First time I tried this, the new VLM crashed. It said there wasn't enough VLM crashed. It said there wasn't enough VLM crashed. It said there wasn't enough GPU memory. LM cache was still holding GPU memory. LM cache was still holding GPU memory. LM cache was still holding on to the GPU memory from the dead on to the GPU memory from the dead on to the GPU memory from the dead process, about 89 gigs of it for 2 process, about 89 gigs of it for 2 process, about 89 gigs of it for 2 minutes. Now, restart normally takes minutes. Now, restart normally takes minutes. Now, restart normally takes about 150 seconds, so it took 27 about 150 seconds, so it took 27 about 150 seconds, so it took 27 seconds this time. and 136 seconds of seconds this time. and 136 seconds of seconds this time. and 136 seconds of that was waiting for memory to be that was waiting for memory to be that was waiting for memory to be released. Now, there's a setting for released. Now, there's a setting for released. Now, there's a setting for that wait. I tried 6 seconds. It that wait. I tried 6 seconds. It that wait. I tried 6 seconds. It refused. The minimum is 30 seconds. And

refused. The minimum is 30 seconds. And refused. The minimum is 30 seconds. And at 30 seconds, the whole restart came at 30 seconds, the whole restart came at 30 seconds, the whole restart came down to 173 seconds. That's still about down to 173 seconds. That's still about down to 173 seconds. That's still about 20 to 30 seconds slower than just plain 20 to 30 seconds slower than just plain 20 to 30 seconds slower than just plain VLM, which already takes a while, but VLM, which already takes a while, but VLM, which already takes a while, but it's way better than 277. Next, I gave it's way better than 277. Next, I gave it's way better than 277. Next, I gave LMC only 40 gigs of RAM plus an NVME LMC only 40 gigs of RAM plus an NVME LMC only 40 gigs of RAM plus an NVME drive. with RAM only about 1.4 seconds. drive. with RAM only about 1.4 seconds. drive. with RAM only about 1.4 seconds. All misses with the disc about 265 All misses with the disc about 265 All misses with the disc about 265 milliseconds. Then I restarted it and milliseconds. Then I restarted it and milliseconds. Then I restarted it and it's still 265 milliseconds. It just it's still 265 milliseconds. It just it's still 265 milliseconds. It just read the cache the notes right off the read the cache the notes right off the read the cache the notes right off the disc and everything worked. disc and everything worked. disc and everything worked. First up, there's cash [music] blend. First up, there's cash [music] blend. First up, there's cash [music] blend. Reusing chunks of documents even when Reusing chunks of documents even when Reusing chunks of documents even when they show up in a different order. Why they show up in a different order. Why they show up in a different order. Why would you want that? Well, if you think would you want that? Well, if you think would you want that? Well, if you think of a rag app or an agent that pulls the of a rag app or an agent that pulls the of a rag app or an agent that pulls the same handful of files for lots of same handful of files for lots of same handful of files for lots of different questions, just in a different different questions, just in a different different questions, just in a different order each time, prefix caching only order each time, prefix caching only order each time, prefix caching only helps when the prompt starts exactly the helps when the prompt starts exactly the helps when the prompt starts exactly the same way. But that's not always the same way. But that's not always the same way. But that's not always the case. When I shuffle [music] the case. When I shuffle [music] the case. When I shuffle [music] the documents, less than 5% came back from documents, less than 5% came back from documents, less than 5% came back from cache. But unfortunately, it wouldn't cache. But unfortunately, it wouldn't cache. But unfortunately, it wouldn't run. One mode has no VLM support yet, so run. One mode has no VLM support yet, so run. One mode has no VLM support yet, so that one I couldn't test. Next, a that one I couldn't test. Next, a that one I couldn't test. Next, a different model, DeepS V4 Flash. VLM different model, DeepS V4 Flash. VLM different model, DeepS V4 Flash. VLM said that his GPU cache could hold about said that his GPU cache could hold about said that his GPU cache could hold about 1.4 million tokens. So, I threw 320 1.4 million tokens. So, I threw 320 1.4 million tokens. So, I threw 320 documents at it. That's about 10 million documents at it. That's about 10 million documents at it. That's about 10 million tokens cuz I wanted to overflow it. And tokens cuz I wanted to overflow it. And tokens cuz I wanted to overflow it. And it never ran out. I got 98% hits it never ran out. I got 98% hits it never ran out. I got 98% hits straight from GPU memory. Also, its straight from GPU memory. Also, its straight from GPU memory. Also, its notes are so small per token that about notes are so small per token that about notes are so small per token that about 17 million tokens would fit on the GPU.

17 million tokens would fit on the GPU. 17 million tokens would fit on the GPU. VLM's own estimate was off by more than VLM's own estimate was off by more than VLM's own estimate was off by more than 10 times, so my overflow test couldn't 10 times, so my overflow test couldn't 10 times, so my overflow test couldn't really overflow. LMC still won the really overflow. LMC still won the really overflow. LMC still won the restart test though. Right after VLM restart test though. Right after VLM restart test though. Right after VLM restart, 16 documents took 5.7 seconds restart, 16 documents took 5.7 seconds restart, 16 documents took 5.7 seconds with LM cache compared to 13.8 around with LM cache compared to 13.8 around with LM cache compared to 13.8 around there with just VLM's cache. Then I there with just VLM's cache. Then I there with just VLM's cache. Then I tried another pretty fresh model Quen tried another pretty fresh model Quen tried another pretty fresh model Quen [music] 3.8 flash next. It's a hybrid [music] 3.8 flash next. It's a hybrid [music] 3.8 flash next. It's a hybrid model. It mixes two kinds of layers. So model. It mixes two kinds of layers. So model. It mixes two kinds of layers. So its notes or KV cache aren't shaped like its notes or KV cache aren't shaped like its notes or KV cache aren't shaped like a normal models. VLM's own CPU offload a normal models. VLM's own CPU offload a normal models. VLM's own CPU offload wouldn't even start on it. So that by wouldn't even start on it. So that by wouldn't even start on it. So that by the time you're watching this video, the time you're watching this video, the time you're watching this video, some of these things might be outdated some of these things might be outdated some of these things might be outdated and some of them might already work. LMC and some of them might already work. LMC and some of them might already work. LMC cache only worked after I rebuilt it for cache only worked after I rebuilt it for cache only worked after I rebuilt it for a newer version. And once it ran, the 16 a newer version. And once it ran, the 16 a newer version. And once it ran, the 16 document round after restart took 12.3 document round after restart took 12.3 document round after restart took 12.3 seconds with LM cache VLMs 27.4. seconds with LM cache VLMs 27.4. seconds with LM cache VLMs 27.4. Finally, it was time to do some coding Finally, it was time to do some coding Finally, it was time to do some coding agents, which is the workload I care agents, which is the workload I care agents, which is the workload I care about most. I ran them on the DJX Spark about most. I ran them on the DJX Spark about most. I ran them on the DJX Spark with an older model, but still valid with an older model, but still valid with an older model, but still valid OSS120B scripted sessions. That means OSS120B scripted sessions. That means OSS120B scripted sessions. That means real files and real logs replayed real files and real logs replayed real files and real logs replayed exactly the same way for every setup.

exactly the same way for every setup. exactly the same way for every setup. Each session runs 30 turns [music] and Each session runs 30 turns [music] and Each session runs 30 turns [music] and conversation grows to almost 100,000 conversation grows to almost 100,000 conversation grows to almost 100,000 tokens. And I'm counting turns by tokens. And I'm counting turns by tokens. And I'm counting turns by minute. Basically, how many replies the minute. Basically, how many replies the minute. Basically, how many replies the agents get back. And the late turns with agents get back. And the late turns with agents get back. And the late turns with eight agents waited 104 seconds with no eight agents waited 104 seconds with no eight agents waited 104 seconds with no cash. That's almost 2 minutes per turn. cash. That's almost 2 minutes per turn. cash. That's almost 2 minutes per turn. And with cache, boom, 2.1 seconds in And with cache, boom, 2.1 seconds in And with cache, boom, 2.1 seconds in turns per minute. That's 5.6 times more turns per minute. That's 5.6 times more turns per minute. That's 5.6 times more work done in the same amount of time. work done in the same amount of time. work done in the same amount of time. Now, I wanted to show this side by side. Now, I wanted to show this side by side. Now, I wanted to show this side by side. This is on an RTX Pro 6000. Now, here's This is on an RTX Pro 6000. Now, here's This is on an RTX Pro 6000. Now, here's cash on one side, no cash on the other cash on one side, no cash on the other cash on one side, no cash on the other side at the same time on one GPU. Look side at the same time on one GPU. Look side at the same time on one GPU. Look at turns four through six. The cash side at turns four through six. The cash side at turns four through six. The cash side had up to 27,000 tokens cached, and it had up to 27,000 tokens cached, and it had up to 27,000 tokens cached, and it was still just as slow, 3 to 4 seconds. was still just as slow, 3 to 4 seconds. was still just as slow, 3 to 4 seconds. With that much cached, it should have With that much cached, it should have With that much cached, it should have taken about a second. So, what's up? taken about a second. So, what's up? taken about a second. So, what's up? Turns out VLM was batching them Turns out VLM was batching them Turns out VLM was batching them together. So, the fast one kept waiting together. So, the fast one kept waiting together. So, the fast one kept waiting on the slow one. So, if a cache request on the slow one. So, if a cache request on the slow one. So, if a cache request shares the GPU with a big uncashed one, shares the GPU with a big uncashed one, shares the GPU with a big uncashed one, it can lose a lot of its advantage.

it can lose a lot of its advantage. it can lose a lot of its advantage. So, then I changed the method. I So, then I changed the method. I So, then I changed the method. I recorded each setup alone with the GPU recorded each setup alone with the GPU recorded each setup alone with the GPU all to itself, then played them back all to itself, then played them back all to itself, then played them back together with real timings. [music] together with real timings. [music] together with real timings. [music] This is the RTX Pro 6000 running Quen This is the RTX Pro 6000 running Quen This is the RTX Pro 6000 running Quen 3.827B 3.827B 3.827B FB8. The agent session grows to 115,000 FB8. The agent session grows to 115,000 FB8. The agent session grows to 115,000 tokens here. One agent did 5.2 turns per tokens here. One agent did 5.2 turns per tokens here. One agent did 5.2 turns per minute. That's with no cash. 12.9 with minute. That's with no cash. 12.9 with minute. That's with no cash. 12.9 with VLM's prefix cache and 12.8 with LM VLM's prefix cache and 12.8 with LM VLM's prefix cache and 12.8 with LM cash. That's 2 and a half times. With cash. That's 2 and a half times. With cash. That's 2 and a half times. With eight agents, we have about eight turns eight agents, we have about eight turns eight agents, we have about eight turns per minute with no cash and up to almost per minute with no cash and up to almost per minute with no cash and up to almost 60 with cash. So split that across eight 60 with cash. So split that across eight 60 with cash. So split that across eight agents with no cash. Each one gets about agents with no cash. Each one gets about agents with no cash. Each one gets about one turn a minute. With cache, each one one turn a minute. With cache, each one one turn a minute. With cache, each one gets about seven. No cache barely scales gets about seven. No cache barely scales gets about seven. No cache barely scales because the GPU is busy reading. Every because the GPU is busy reading. Every because the GPU is busy reading. Every turn, it reads the whole prompt. With turn, it reads the whole prompt. With turn, it reads the whole prompt. With cache, it reads about 7% of it. And both cache, it reads about 7% of it. And both cache, it reads about 7% of it. And both caches keep scaling. In this case, LM caches keep scaling. In this case, LM caches keep scaling. In this case, LM cache is pretty close to VLM's cache. cache is pretty close to VLM's cache. cache is pretty close to VLM's cache. So, as the agent keeps going, the later So, as the agent keeps going, the later So, as the agent keeps going, the later turns, in this case, 21 through 30, they turns, in this case, 21 through 30, they turns, in this case, 21 through 30, they show it too. With no cache, the weight show it too. With no cache, the weight show it too. With no cache, the weight goes from about 14 seconds to 28 seconds goes from about 14 seconds to 28 seconds goes from about 14 seconds to 28 seconds to 42 as you go from one agent to eight to 42 as you go from one agent to eight to 42 as you go from one agent to eight agents. And then with either cash turned agents. And then with either cash turned agents. And then with either cash turned on, it's half a second to about two on, it's half a second to about two on, it's half a second to about two seconds. Now, back to that agent session seconds. Now, back to that agent session seconds. Now, back to that agent session from the beginning of the video. This from the beginning of the video. This from the beginning of the video. This time with both caches and the restart.

time with both caches and the restart. time with both caches and the restart. This is the fun one. At turn 15, both This is the fun one. At turn 15, both This is the fun one. At turn 15, both caches answer in about half a second. caches answer in about half a second. caches answer in about half a second. But then I restart VLM. And at the next But then I restart VLM. And at the next But then I restart VLM. And at the next turn, turn 16, VLM's cache takes 9 and turn, turn 16, VLM's cache takes 9 and turn, turn 16, VLM's cache takes 9 and 1/2 seconds. Nothing gets reused. The 1/2 seconds. Nothing gets reused. The 1/2 seconds. Nothing gets reused. The cash has to get rebuilt, but LM cache cash has to get rebuilt, but LM cache cash has to get rebuilt, but LM cache takes 1.1 seconds. It reused almost takes 1.1 seconds. It reused almost takes 1.1 seconds. It reused almost 60,000 tokens over the whole 30 turn 60,000 tokens over the whole 30 turn 60,000 tokens over the whole 30 turn session, not counting the restart session, not counting the restart session, not counting the restart itself. When you have no cash, it took itself. When you have no cash, it took itself. When you have no cash, it took about 350 seconds. Vom cache 65 seconds about 350 seconds. Vom cache 65 seconds about 350 seconds. Vom cache 65 seconds and LMC 55 seconds. That's almost 6 and LMC 55 seconds. That's almost 6 and LMC 55 seconds. That's almost 6 minutes versus about 1. Now, to be fair, minutes versus about 1. Now, to be fair, minutes versus about 1. Now, to be fair, if you do count the restart, LMC if you do count the restart, LMC if you do count the restart, LMC actually comes out about 20 seconds actually comes out about 20 seconds actually comes out about 20 seconds behind VLM's cache because that restart behind VLM's cache because that restart behind VLM's cache because that restart takes 88 seconds with LM cache versus 58 takes 88 seconds with LM cache versus 58 takes 88 seconds with LM cache versus 58 seconds with VLM cache. The cash didn't seconds with VLM cache. The cash didn't seconds with VLM cache. The cash didn't make the agent any smarter. It just make the agent any smarter. It just make the agent any smarter. It just stopped it from rereading the same stopped it from rereading the same stopped it from rereading the same stuff. LMC keeps those notes around. stuff. LMC keeps those notes around. stuff. LMC keeps those notes around. Those KV caches like I don't like saying Those KV caches like I don't like saying Those KV caches like I don't like saying cash [music] so much. So notes is a good cash [music] so much. So notes is a good cash [music] so much. So notes is a good way to call it. Um way to call it. Um way to call it. Um it's like moist. Some people don't like it's like moist. Some people don't like it's like moist. Some people don't like saying moist or hearing it. Well, cash saying moist or hearing it. Well, cash saying moist or hearing it. Well, cash it just I don't want to say it too much, it just I don't want to say it too much, it just I don't want to say it too much, you know? I don't know. Let's go with you know? I don't know. Let's go with you know? I don't know. Let's go with pink and purple this time. All right.

pink and purple this time. All right. pink and purple this time. All right. All right. I ran the agent test on two All right. I ran the agent test on two All right. I ran the agent test on two machines with two different models. And machines with two different models. And machines with two different models. And it's the same story there, too. With it's the same story there, too. With it's the same story there, too. With eight agents, it's 5.6 times more turns eight agents, it's 5.6 times more turns eight agents, it's 5.6 times more turns per minute on the Spark and 7.5 times on per minute on the Spark and 7.5 times on per minute on the Spark and 7.5 times on the RTX Pro 6000. The more agents you the RTX Pro 6000. The more agents you the RTX Pro 6000. The more agents you run, the bigger the payoff. The more you run, the bigger the payoff. The more you run, the bigger the payoff. The more you spend, the more you save. spend, the more you save. spend, the more you save. That's no, that's a different thing. That's no, that's a different thing. That's no, that's a different thing. Never mind. If you're running coding Never mind. If you're running coding Never mind. If you're running coding agents on your own hardware, use KV agents on your own hardware, use KV agents on your own hardware, use KV cache. And if you're using VLM, prefix cache. And if you're using VLM, prefix cache. And if you're using VLM, prefix caching is on by default, so leave it caching is on by default, so leave it caching is on by default, so leave it on. If everything fits inside GPU on. If everything fits inside GPU on. If everything fits inside GPU memory, that's all you need. If it memory, that's all you need. If it memory, that's all you need. If it doesn't, but fits in RAM, then VLM's own doesn't, but fits in RAM, then VLM's own doesn't, but fits in RAM, then VLM's own CPU offload keeps up with LM cache, but CPU offload keeps up with LM cache, but CPU offload keeps up with LM cache, but maybe your server restarts and you don't maybe your server restarts and you don't maybe your server restarts and you don't want every agent starting out cold or want every agent starting out cold or want every agent starting out cold or your cash outgrows even your RAM. That's your cash outgrows even your RAM. That's your cash outgrows even your RAM. That's when you turn LM cache on and that's when you turn LM cache on and that's when you turn LM cache on and that's when it's worth it. you're signing up when it's worth it. you're signing up when it's worth it. you're signing up for more moving parts and slower for more moving parts and slower for more moving parts and slower restarts and config pain on newer models restarts and config pain on newer models restarts and config pain on newer models because of all the [music] new stuff because of all the [music] new stuff because of all the [music] new stuff that you won't be able to try. Now, if that you won't be able to try. Now, if that you won't be able to try. Now, if you want to see that DJX station in you want to see that DJX station in you want to see that DJX station in action from beginning to the end right action from beginning to the end right action from beginning to the end right over here, watch that video and the RTX over here, watch that video and the RTX over here, watch that video and the RTX Pro 6000 build I did with four of them, Pro 6000 build I did with four of them, Pro 6000 build I did with four of them, watch this [music] video. Thanks for watch this [music] video. Thanks for watch this [music] video. Thanks for watching and I'll see you next time.