I Thought 2 TB of RAM Was Overkill… My AI Agents Disagree

AMD just announced a workstation that takes up to two terabytes of RAM. 2 terb. My first reaction was probably the same as yours. Who needs that? So, I decided to find out on this. This is a Thread Ripper build I did not long ago, and right now it's got two RTX Pro 6000s in there. It did have four, but I needed them for other projects. So, here's the plan. I'm going to baseline these, then get to work on system RAM and see how much that really matters for AI. >> For AI for AI. >> Now, the first time I ran agents on this thing, the thing that ran out first wasn't the GPUs. It was the memory. Later, I swapped both of these cards for one that cost just a fraction just to see if the GPU was even the part that mattered. And this is the real variable. 128 gigs of RAM in there right now. Later I double it to 256 and change nothing else. And the question is simple. Does the same work get done any faster? But I got the first version of this test completely wrong. So let me start there.

Now here's the thing we all do with local AI. We obsess over VRAM. And that's for a good reason. VRAM lives inside the GPU and it is much faster than system memory for this kind of work. This is why every spec sheet leads with it and why we all chase bigger and bigger cards. This machine already has 192 gigabytes of it, which is overkill for pretty much everybody, especially at today's prices. So, look at the AMD Halo Station once again coming in 2027. Ryzen Thread Ripper Pro, the world's best workstation processor. That's what I got in there on this ASUS motherboard. I'll leave the specs down below. So, look at this AMD Halo station again. two terabytes of system memory, 576 GB of HBM3 GPU memory. That's the fast high bandwidth memory. That is almost four times more of the slower memory than fast memory. I mean, they're all pretty fast. RAM is really fast, but that's done on purpose in their flagship AI machine. So, if VRAM is the fast stuff that everyone wants, why would they stack it with four times as much of the slow stuff? What do they know that uh I haven't covered yet on this channel? Ah, all right, enough theatrics. My first instinct was that RAM is just overflow.

It's a place where things spill when the GPU runs out. Well, that turned out to be not exactly right. Here's what actually happens. The model sits on the GPU, but the agent itself lives in system RAM. The orchestration, the tool calls, the growing context, the cache of everything it has seen already. We're not talking about chatting with a model or generating images. We're talking about agentic work. So the real question is how much agent work this machine can do at once. And that ceiling is set by RAM, system RAM. Now getting a clean answer was not smooth. The first benchmark I built leaned on the GPU and got muddied by the CPU. So I threw it out. Then I wondered whether the memory difference could even show up on a processor this fast. That's a Thread Ripper Pro 9975WX, by the way. Yeah, it's nice. So, how do you measure how much agent work a machine can do? It's not exactly tokens per second. I do plenty of those benchmarks on this channel. I run a growing number of real coding agents at the same time against one model. Between turns, each agent's conversation gets parked in system RAM instead of being rebuilt from scratch. So, how many of those can I stack before the system overflows? Eight agents was fine. 12 still climbing. Then at 16, the throughput just falls off the cliff. The weight for the first token also keeps growing. This is kind of expected though. Now 12 job agents all finished the job. At 16, not so much anymore. 10 of them failed. So 12 is kind of the ceiling at 128 GB. Here's the important

part. Through all of it, swap stayed at zero, which means everything wasn't RAM. And the CPU stayed around 6%. So that means it wasn't uh using the CPU that much. on a machine with 192 GB of VRAM. The thing that ran out first was the system memory. So, my new benchmark was working. I'll share that later, by the way. Oh, one more thing from this run. The biggest models were the fastest. A big model doesn't have to be a slow model. We'll come back to that. So, if the GPUs were not even the limit here on the most powerful setup I own, not that I tested, that I own cuz I tested some pretty powerful setups here recently. Oh my gosh. What would happen when I swap in instead of 192 GB, I swap in 32 GB of VRAM on this Radeon R9700 professional workstation GPU. This one's by ASUS. Feels so empty in here. This is going to get interesting. So, these days I'm always flipping between models. GPT for research, cloud for coding, nano banana for image generation, VO cling, and runway for video. Six tabs, six bills, and counting. Enter chat LLM teams. One dashboard houses every top LLM and route Olympics the right one.

GPT Mini for ultra fast answers. Cloud Sonnet for coding. Gemini Pro for massive context. They recently added Gemini 3 and GPT 5.1 the moment they dropped. Create professional presentations with graphs, charts, and deep research detailed content. Need human sounding copy? Human eyes rewrites text to defeat AI detectors. Need visuals? Pick frontier or open- source models. Nano Banana, Midjourney, Flux for images, Magnific for upscaling, plus VO, WAN, and Sora for video, all built in. You also get Abaca's AI deep agent to pretty much do anything. Build full stack apps, websites, reports with just text prompts and deploy them on the spot. They have Abaca's AI desktop, which is the brand new coding editor and assistant that lets you vibe code and build production ready apps. And the kicker, it's just $10 a month, less than one premium model. Head over to chatlm.abacus.a AI or click the link below to level up with Chat LLM teams.

After pulling the 192 GB of VRAM out and replacing it with just 32, which cost just a small fraction of the money by old logic, the VRAMm logic, this thing should just fall apart. And by this thing, I mean that thing. I had to move it. It was making too much noise. We cut the fast memory down to a sixth of what it was. Now, one thing you should know before I tell you the numbers, this is a different GPU at a completely different software stack. It's AMD, not the same stack as the RTX Nvidia cards. So, this is a completely fresh baseline, not a drop in swap of the numbers you saw earlier. So, the first question is not how fast this thing is, but whether it can run it at all. And the biggest model I threw at it is 124 billion parameters. That's Link 3.0 Flash. That's the same model I ran on the bigger card with the 131,000 token context. Now, look, let's be honest here. There's no way that a 124 billion parameter card will fit into a 32 gigs of VRAM. But hang on for a moment. Look at this. On the GPU, only 5 1/2 gigs. That's it. The rest of the model, about 90 gigs of it, is sitting in system RAM. And considering that this is a different stack, that's not even slow. About 38 tokens per second. And that's because it's a mixture of experts model that only fires a few experts per token. Remember, I mentioned earlier, a big model doesn't have to be a slow model. This is an example of that. Now, of course, compared to the RTX Pro 6000 on the exact same model, it's a lot faster. But the R9700 running 124 billion parameter model at 38 tokens per second is pretty usable. And it can do

this not because of the GPU, because of the system RAM. By the way, remember that benchmark that I talked about earlier? Well, I'm getting a little bit ahead of myself here, but it's ready in a 0.1 release right now. So, you can go check it out on GitHub. It's called Agent Headroom, and it's a suite of benchmarks for agents. There's rung, pace, crowd, tenants, and recall. I'm not going to go into the details of that right now. I'll save that for another video, perhaps a members video, but you can go and check it out. It's up on GitHub right now. You can check it out and run it for yourself and feel free to open up pull requests and contribute. Thanks. Now, capacity is one thing. The other question is, how much agent work can it do all at the same time? Well, you'd be surprised at eight agents, it's right there with the monster. Seven tokens per second against 8 1/2. That's close. But where the monster, I keep calling it the monster. I mean the RTX Pro 6000. It's not a monster. It just is what it is. Okay, let it live. It has more RAM. But where it kept climbing is 12 agents. Yeah, the R9700 kind of uh struggles there at 16 agents. Yeah. So, we dropped a little bit. The RTX sustained 12 agents and the cheap card sustains eight. And I know this is a bit of a tangent because we're not talking about the cards, the GPUs. Uh, I mean, the whole system kind of has to do with all of this, but I want to focus on the RAM. And here's the thing. This is the opposite of part one. On the monster, the limit was the RAM. Here with the

R9700, the limit is now the GPU, cuz now 32 gigs of VRAM only has room for so many live lanes at once. But system RAM still has plenty to spare. So, the GPU is the limit again, but only for how many, not for how big. one agent, the same model, both running fully on the card. The cheaper one is slower. 28 29 tokens per second there against 63. So, a bit under half the speed. That's the trade. You're paying in speed, not in capability here. So, putting it all together, RAM decides what this machine can run and VRAM decides how many of them can run at once. The expensive cards were buying parallelism, more agents in flight at the same time. the RAM was buying capability. In other words, models too big for any normal size GPU that is not $16,000 now. And even though the AMD Halo station is not going to be a cheap machine, I don't know what the price of it is, but that 2 TB that's the capability that you're buying there. So, if capability comes from RAM, the move is obvious. Give it more RAM.

And at 128 gigs, what's in that box right now, the next model up is just out of reach, just a little bit. Only 120 gigs. So, doubling the RAM to 256 should fit that model. Now, the moment of truth. We're going to replace the 128 with 256. Thank you to Kingston for kindly sending these over. Sorry. Each one of these DDR5 boards is ECC. That's error correcting memory. And each one is 32 GB. Bad cable management up here. You can blame Dan for that one. Dan is not here to defend himself, so it's not fair. I'll take the blame since it's my computer. Oh yeah, number one in. There's two good kinds of sweaty. One is putting in RAM, a lot of RAM, and the other one's working out. I know there's a third kind, but I'm not allowed to talk about that on this channel. It's a PG-rated channel. Yes. All right. All eight of them are seated. I really hope this turns on and trains them quickly. All right, here we go. All right, let's see what we got. Oh yeah, there they all are. 6,400 megat transfers per second, Kingston. 32 megabytes per stick. And we got eight of them. Good to go. It's only this loud when it's booting up. It's not normally this loud when it's just standing there working. Ah, there we go. I think it's

done. And yeah, it's pretty quiet now. I like that. >> All right, so the 256 gigs is in, 128 is out to be used on other projects. Don't worry, I'll put them to good use. And I've already run the tests. Nothing else was changed, just the RAM. And the question I really care about is, does more RAM get the same block of agent work done faster? And not that this is very scientific what I'm doing here, but I did have a hypothesis and it was going to be that 12 agents should pass. Now, 16 probably would too. So, let's take a look. 12 agents first. Last time on 128 gigs, 10 of those 12 failed. 30 times an agent came back and its conversation had been pushed out of RAM. So, what happens in that case? Well, the card has to rebuild it from scratch. And on 256 GB, zero. Not a single one. All 12 agents finished. So far so good. I went up to 16 and all 16 finished. Zero rebuilds again. RAM peaked at about 150 gigs, which by the way is more than the old setup even had. So check this out. The monster with 128 gigs did 12 agents. The cheap card with 128 did eight. And the cheap card with 256 did 16. And I think it had more room to grow. I just stopped at 16 at this point. And also, by the way, when I say cheap card, it's cheap

relative to the uh RTX Pro 6000. It's still a $2,000 card at this point. When they came out, they were $1,200. Everything is going up. So cheap relative, not cheap globally. Now, I don't want you walking away thinking that cheap card beat the monster card here. The monster card only had 128 gigs, too. I never gave it 256, so who knows what it would do at that point. We can make a hypothesis. So, is it faster? Well, each agent on its own, no. Throughput sits around seven tokens per second at 12 and 16 agents. And that's because the small card only has two slots for each agent on the GPU. More agents just means each one waits a little longer for its turn. But here, look at the whole job. Same 12 agents, same amount of work on 128 gigs. That run took 85 minutes and only two of the 12 agents even finished. On 256 though, all 12 agents were done in just 59 minutes. That's 26 minutes faster just because the card wasn't redoing the work it had already done. So why would anyone want 2 terb of RAM? Because that's where your agents live. The more of them you can run and the sooner the work gets done. But before this machine had one cheap card in it, it had four RTX Pro 6000s, 384 gigs of VRAM in one box. And we didn't even talk about that processor that's in there. You need to see what it took to build that. So watch

this video right over here. Thanks for watching and I'll see you next time.