The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI

Let's uh get started. Uh hi everyone. Let's uh get started. Uh hi everyone. Thank you for coming. Uh this is the Thank you for coming. Uh this is the Thank you for coming. Uh this is the late afternoon in the last day. Um so I late afternoon in the last day. Um so I late afternoon in the last day. Um so I really appre appreciate it. I'm gone, really appre appreciate it. I'm gone, really appre appreciate it. I'm gone, founder and CEO of friendly AI. Today I founder and CEO of friendly AI. Today I founder and CEO of friendly AI. Today I want to talk about agentic inference. So want to talk about agentic inference. So want to talk about agentic inference. So I'll first walk through what changed uh I'll first walk through what changed uh I'll first walk through what changed uh why it matters and how we rebuilt the why it matters and how we rebuilt the why it matters and how we rebuilt the inference cloud for agents. inference cloud for agents. inference cloud for agents. Before we go deeper let me briefly Before we go deeper let me briefly Before we go deeper let me briefly introduce friendly AI. Friendly AI is introduce friendly AI. Friendly AI is introduce friendly AI. Friendly AI is the frontier AI inference cloud for the frontier AI inference cloud for the frontier AI inference cloud for agents. So we run inference for agents agents. So we run inference for agents agents. So we run inference for agents at scale faster, cheaper and more at scale faster, cheaper and more at scale faster, cheaper and more reliably. We are born from a research reliably. We are born from a research reliably. We are born from a research team at S National University and those team at S National University and those team at S National University and those research roots still define us. We are research roots still define us. We are research roots still define us. We are the team that invented continuous the team that invented continuous the team that invented continuous batching. The inference optimization batching. The inference optimization batching. The inference optimization that is now standard across the industry that is now standard across the industry that is now standard across the industry and our orca work inspired 3LM a widely and our orca work inspired 3LM a widely and our orca work inspired 3LM a widely used open source framework.

used open source framework. used open source framework. Today we operate globally headquartered Today we operate globally headquartered Today we operate globally headquartered in San Francisco with a team in soul to in San Francisco with a team in soul to in San Francisco with a team in soul to scale frontier inference. scale frontier inference. scale frontier inference. As you know 2026 is a year agents go As you know 2026 is a year agents go As you know 2026 is a year agents go into massive production and it's driven into massive production and it's driven into massive production and it's driven by two trends coming together. First by two trends coming together. First by two trends coming together. First agents are going exponential. agents are going exponential. agents are going exponential. AI agents are driving explosive adoption AI agents are driving explosive adoption AI agents are driving explosive adoption across software operations and knowledge across software operations and knowledge across software operations and knowledge work. work. work. Second, open rate motors have reached Second, open rate motors have reached Second, open rate motors have reached the frontier and make agents economic. the frontier and make agents economic. the frontier and make agents economic. They now rival closed frontier motors in They now rival closed frontier motors in They now rival closed frontier motors in capability, which means you can run capability, which means you can run capability, which means you can run frontier quality agents on open motors frontier quality agents on open motors frontier quality agents on open motors with much lower token cost. with much lower token cost. with much lower token cost. Let me make the openweight motor part Let me make the openweight motor part Let me make the openweight motor part concrete. concrete. concrete. Open weight motors are now strong enough Open weight motors are now strong enough Open weight motors are now strong enough for these types of real agentic for these types of real agentic for these types of real agentic workflows. Here we gave the exact same workflows. Here we gave the exact same workflows. Here we gave the exact same task building a tower defense game with task building a tower defense game with task building a tower defense game with a coding agent to two models. On the a coding agent to two models. On the a coding agent to two models. On the left is GLM 5.2 an openw rate motor left is GLM 5.2 an openw rate motor left is GLM 5.2 an openw rate motor running on friendly AI. On the right is running on friendly AI. On the right is running on friendly AI. On the right is anthropics opus 4.8.

anthropics opus 4.8. anthropics opus 4.8. The important point is not that the The important point is not that the The important point is not that the outputs are identical. The point is that outputs are identical. The point is that outputs are identical. The point is that both complete the task at a level that both complete the task at a level that both complete the task at a level that is clearly usable for many agentic is clearly usable for many agentic is clearly usable for many agentic workflows. Open weight models have workflows. Open weight models have workflows. Open weight models have crossed the quality threshold crossed the quality threshold crossed the quality threshold but the economics are very different for but the economics are very different for but the economics are very different for the same task. Opus 4.8 cost about a the same task. Opus 4.8 cost about a the same task. Opus 4.8 cost about a $150 GLM 5.2 unfriendly AI cost 27 cents $150 GLM 5.2 unfriendly AI cost 27 cents $150 GLM 5.2 unfriendly AI cost 27 cents about 5.6 times cheaper. about 5.6 times cheaper. about 5.6 times cheaper. So this is the promise I mentioned So this is the promise I mentioned So this is the promise I mentioned earlier. Open rate models give you earlier. Open rate models give you earlier. Open rate models give you frontier quality agents at a fraction of frontier quality agents at a fraction of frontier quality agents at a fraction of cost. cost. cost. But motor cost is only one part of the But motor cost is only one part of the But motor cost is only one part of the story. To make agent actually fast and story. To make agent actually fast and story. To make agent actually fast and reliable, the inference stack itself has reliable, the inference stack itself has reliable, the inference stack itself has to change. So let's look at what to change. So let's look at what to change. So let's look at what actually happens inside an agentic actually happens inside an agentic actually happens inside an agentic workload. workload. workload. So first let's look at changes in the So first let's look at changes in the So first let's look at changes in the workload. In the past the dominant usage workload. In the past the dominant usage workload. In the past the dominant usage was chat. The basic unit was a request.

was chat. The basic unit was a request. was chat. The basic unit was a request. A person asks a question, the model A person asks a question, the model A person asks a question, the model answers and the person reads it. Latency answers and the person reads it. Latency answers and the person reads it. Latency meant how fast did I get one response. meant how fast did I get one response. meant how fast did I get one response. Agents are different. The basic unit is Agents are different. The basic unit is Agents are different. The basic unit is a task. A task may involve many model a task. A task may involve many model a task. A task may involve many model cores, many tool cores and it may run cores, many tool cores and it may run cores, many tool cores and it may run autonomously for a while. autonomously for a while. autonomously for a while. So the user does not really care about So the user does not really care about So the user does not really care about the latency of one individual request. the latency of one individual request. the latency of one individual request. The user cares about when the whole task The user cares about when the whole task The user cares about when the whole task is completed. is completed. is completed. That means we have to optimize for That means we have to optimize for That means we have to optimize for tasks, not just individual requests. tasks, not just individual requests. tasks, not just individual requests. Let's look at agent workflows more Let's look at agent workflows more Let's look at agent workflows more closely. An agent really runs a session closely. An agent really runs a session closely. An agent really runs a session made up of tasks. Each task typically made up of tasks. Each task typically made up of tasks. Each task typically runs in a loop. First, it plans, which runs in a loop. First, it plans, which runs in a loop. First, it plans, which usually means an LLM call. Then, it acts usually means an LLM call. Then, it acts usually means an LLM call. Then, it acts maybe by calling a tool. Then it maybe by calling a tool. Then it maybe by calling a tool. Then it observes the result and adds the that observes the result and adds the that observes the result and adds the that back into the context and it repeats back into the context and it repeats back into the context and it repeats this until the task is done.

this until the task is done. this until the task is done. So we are constantly alternating between So we are constantly alternating between So we are constantly alternating between LLM inference and one or more nonLM tool LLM inference and one or more nonLM tool LLM inference and one or more nonLM tool executions. So there is a gap between executions. So there is a gap between executions. So there is a gap between LLM calls. LLM calls. LLM calls. An agent can also create sub agents and An agent can also create sub agents and An agent can also create sub agents and run them in parallel. Agent inputs also look very different Agent inputs also look very different from chat. The graph here shows the from chat. The graph here shows the from chat. The graph here shows the prompt and completion length prompt and completion length prompt and completion length distributions of our internal coding distributions of our internal coding distributions of our internal coding agent runs with GLM 5.2 agent runs with GLM 5.2 agent runs with GLM 5.2 which we use day-to-day. which we use day-to-day. which we use day-to-day. They are much longer. They grow as the They are much longer. They grow as the They are much longer. They grow as the task progresses since every observation task progresses since every observation task progresses since every observation gets appended back into the context. gets appended back into the context. gets appended back into the context. There's an important pattern here. There's an important pattern here. There's an important pattern here. Consecutive agent steps usually share a Consecutive agent steps usually share a Consecutive agent steps usually share a huge prefix. If we recomp compute the huge prefix. If we recomp compute the huge prefix. If we recomp compute the same prefix every time, we are burning a same prefix every time, we are burning a same prefix every time, we are burning a lot of compute on work we already did. lot of compute on work we already did. lot of compute on work we already did. So this is one of the biggest So this is one of the biggest So this is one of the biggest opportunities in agentic inference.

opportunities in agentic inference. opportunities in agentic inference. So how token hungry are agents? Now So how token hungry are agents? Now So how token hungry are agents? Now let's look at a long horizon task let's look at a long horizon task let's look at a long horizon task example like deep research. example like deep research. example like deep research. We ran explained the spec decoding We ran explained the spec decoding We ran explained the spec decoding framework in VLM using code with GLM 5.2 framework in VLM using code with GLM 5.2 framework in VLM using code with GLM 5.2 unfriendly AI. unfriendly AI. unfriendly AI. There are multiple stages and each stage There are multiple stages and each stage There are multiple stages and each stage is composed of sub agents which run is composed of sub agents which run is composed of sub agents which run multiple inferences and tool cores. multiple inferences and tool cores. multiple inferences and tool cores. So it might run tens or even hundreds of So it might run tens or even hundreds of So it might run tens or even hundreds of inference steps sometimes over minutes inference steps sometimes over minutes inference steps sometimes over minutes or hours and the shared context keeps or hours and the shared context keeps or hours and the shared context keeps going the whole time. For the user what going the whole time. For the user what going the whole time. For the user what matters is not the latency of a single matters is not the latency of a single matters is not the latency of a single token or one core. What matters is when token or one core. What matters is when token or one core. What matters is when is my test completed. So agent inference is not just chat with So agent inference is not just chat with more requests. It's a different problem. more requests. It's a different problem. more requests. It's a different problem. The context grows over time. Tour work The context grows over time. Tour work The context grows over time. Tour work is interled between model cores. The is interled between model cores. The is interled between model cores. The number of model cores depends on the number of model cores depends on the number of model cores depends on the input. So you can't really plan around a input. So you can't really plan around a input. So you can't really plan around a fixed request rate plan. And the real fixed request rate plan. And the real fixed request rate plan. And the real metric is end to end test latency, not a metric is end to end test latency, not a metric is end to end test latency, not a single request latency.

This is where friendly AI comes in. We This is where friendly AI comes in. We rebuilt the frontier inference cloud rebuilt the frontier inference cloud rebuilt the frontier inference cloud specifically for agentic goal close specifically for agentic goal close specifically for agentic goal close around the challenges I just walked around the challenges I just walked around the challenges I just walked through through through and we set one goal optimize end to end and we set one goal optimize end to end and we set one goal optimize end to end test latency test latency test latency the task not just the request. So how do the task not just the request. So how do the task not just the request. So how do we do that? Let me show you the key we do that? Let me show you the key we do that? Let me show you the key engineering behind it. engineering behind it. engineering behind it. Here's the engineering map for how we Here's the engineering map for how we Here's the engineering map for how we think about it. We built the stack layer think about it. We built the stack layer think about it. We built the stack layer by layer around agent workflows. There by layer around agent workflows. There by layer around agent workflows. There are four big pillars I'm going to cover are four big pillars I'm going to cover are four big pillars I'm going to cover today. Prefix caching today. Prefix caching today. Prefix caching key value in short KB cache management, key value in short KB cache management, key value in short KB cache management, cash aware routing, agent aware cash aware routing, agent aware cash aware routing, agent aware optimization optimization optimization and of course underneath we need model and of course underneath we need model and of course underneath we need model layer optimization like sparse attention layer optimization like sparse attention layer optimization like sparse attention for long context techniques to reduce for long context techniques to reduce for long context techniques to reduce errors, fast corners, resilience serving errors, fast corners, resilience serving errors, fast corners, resilience serving and more. In this talk, I'm going to and more. In this talk, I'm going to and more. In this talk, I'm going to focus on the four pillars.

focus on the four pillars. focus on the four pillars. Let's start with prefix caching. Since Let's start with prefix caching. Since Let's start with prefix caching. Since Asian steps share a large prefix, we Asian steps share a large prefix, we Asian steps share a large prefix, we compute key value for that prefix once compute key value for that prefix once compute key value for that prefix once and cache it. Then on later steps, we and cache it. Then on later steps, we and cache it. Then on later steps, we reuse the cache key value and only reuse the cache key value and only reuse the cache key value and only process the new suffix. process the new suffix. process the new suffix. Reading from cache is much cheaper than Reading from cache is much cheaper than Reading from cache is much cheaper than recmp computing prefill. So this recmp computing prefill. So this recmp computing prefill. So this improves time to first token and reduces improves time to first token and reduces improves time to first token and reduces compute on every step. And the longer compute on every step. And the longer compute on every step. And the longer the task runs in agents, the more the task runs in agents, the more the task runs in agents, the more valuable this becomes. But caching only works if the KV cache But caching only works if the KV cache actually fits and can move around actually fits and can move around actually fits and can move around efficiently. So we need strong KV cache efficiently. So we need strong KV cache efficiently. So we need strong KV cache management. We use frugal memory management. We use frugal memory management. We use frugal memory management to pack more active context management to pack more active context management to pack more active context onto each GPU memory. We use KB onto each GPU memory. We use KB onto each GPU memory. We use KB contigation to reduce the memory contigation to reduce the memory contigation to reduce the memory footprint. footprint. footprint. We you we use hierarchical caching We you we use hierarchical caching We you we use hierarchical caching across GPU memory, host memory and across GPU memory, host memory and across GPU memory, host memory and disks. So we can go beyond GPU limits.

disks. So we can go beyond GPU limits. disks. So we can go beyond GPU limits. And we also use distributed caching. So And we also use distributed caching. So And we also use distributed caching. So one prefix can be served across one prefix can be served across one prefix can be served across replicas, not just inside one instance. At global cluster scale, routing becomes At global cluster scale, routing becomes really important. A knife load balancer really important. A knife load balancer really important. A knife load balancer may spread requests evenly across GPU may spread requests evenly across GPU may spread requests evenly across GPU clusters, but it can destroy cache clusters, but it can destroy cache clusters, but it can destroy cache locality. locality. locality. A cache aware router at a global scale A cache aware router at a global scale A cache aware router at a global scale does something smarter. It sends a does something smarter. It sends a does something smarter. It sends a request to part that already has the request to part that already has the request to part that already has the right prefix cached turning a cord right prefix cached turning a cord right prefix cached turning a cord prefill into one cache ship. At the same prefill into one cache ship. At the same prefill into one cache ship. At the same time, it still has to balance load. So time, it still has to balance load. So time, it still has to balance load. So one part doesn't become a hot spot. one part doesn't become a hot spot. one part doesn't become a hot spot. In this example, the two requests of In this example, the two requests of In this example, the two requests of task A go to the same part one for cache task A go to the same part one for cache task A go to the same part one for cache locality. The next piece is agent aware The next piece is agent aware optimization. And this is the next optimization. And this is the next optimization. And this is the next fronture of agentic inference. Today, fronture of agentic inference. Today, fronture of agentic inference. Today, most systems schedule each LLM core as most systems schedule each LLM core as most systems schedule each LLM core as if it were independent.

if it were independent. if it were independent. They don't really understand that this They don't really understand that this They don't really understand that this core is part of a longer agent program. core is part of a longer agent program. core is part of a longer agent program. But if the optimizer knows the agent But if the optimizer knows the agent But if the optimizer knows the agent level context, we can make better level context, we can make better level context, we can make better decisions. For example, preempting the decisions. For example, preempting the decisions. For example, preempting the right work, right work, right work, speculatively speculatively speculatively prefilling context for an unlikely next prefilling context for an unlikely next prefilling context for an unlikely next step or making a better cache eviction step or making a better cache eviction step or making a better cache eviction decision based on agent level context. decision based on agent level context. decision based on agent level context. So the goal is to reduce ant latency not So the goal is to reduce ant latency not So the goal is to reduce ant latency not just make one call look fast. When we put all of this together, this When we put all of this together, this is the payoff. We are using the same is the payoff. We are using the same is the payoff. We are using the same model GLM 5.2 with kilo code to create a model GLM 5.2 with kilo code to create a model GLM 5.2 with kilo code to create a simple mobile game. We ran the same task simple mobile game. We ran the same task simple mobile game. We ran the same task with model APIs of friendly AI and with model APIs of friendly AI and with model APIs of friendly AI and another well-known inference provider. another well-known inference provider. another well-known inference provider. As you can see, friendly completes the As you can see, friendly completes the As you can see, friendly completes the same task end to end tox same task end to end tox same task end to end tox thanks to our Asenticentric cloud thanks to our Asenticentric cloud thanks to our Asenticentric cloud design.

design. design. So what does this unlock in practice? a So what does this unlock in practice? a So what does this unlock in practice? a stronger production agent stack. stronger production agent stack. stronger production agent stack. Take an agent you already like. Now plug Take an agent you already like. Now plug Take an agent you already like. Now plug in openweight frontier models like GLM in openweight frontier models like GLM in openweight frontier models like GLM 5.2, Minimax and Kimi served on friendly 5.2, Minimax and Kimi served on friendly 5.2, Minimax and Kimi served on friendly AI. The motor gives you frontier quality AI. The motor gives you frontier quality AI. The motor gives you frontier quality capability and better economics. capability and better economics. capability and better economics. Friendly AI gives you the speed, Friendly AI gives you the speed, Friendly AI gives you the speed, reliability and endtoend test reliability and endtoend test reliability and endtoend test performance needed in production. That performance needed in production. That performance needed in production. That combination, quality, speed, combination, quality, speed, combination, quality, speed, reliability, and cost is what makes reliability, and cost is what makes reliability, and cost is what makes agents actually useful and economical in agents actually useful and economical in agents actually useful and economical in production. Friendly AI is currently powering teams Friendly AI is currently powering teams in production from AI native startups to in production from AI native startups to in production from AI native startups to global enterprises. global enterprises. global enterprises. I'd like to highlight a couple here. I'd like to highlight a couple here. I'd like to highlight a couple here. Hilo is a hugely popular Asantic AI Hilo is a hugely popular Asantic AI Hilo is a hugely popular Asantic AI coding tool serving millions of users.

coding tool serving millions of users. coding tool serving millions of users. LG is a global enterprise whose LG is a global enterprise whose LG is a global enterprise whose businesses range from electronics to businesses range from electronics to businesses range from electronics to healthcare to energy. Very different healthcare to energy. Very different healthcare to energy. Very different companies, but they all need the same companies, but they all need the same companies, but they all need the same thing. Fast, reliable, cost effective thing. Fast, reliable, cost effective thing. Fast, reliable, cost effective agentic inference. This testimonial from our client Kilo This testimonial from our client Kilo says it all. Over the past year, Kilo says it all. Over the past year, Kilo says it all. Over the past year, Kilo Code has tested several inference Code has tested several inference Code has tested several inference providers hosting both open and closed providers hosting both open and closed providers hosting both open and closed models. In a split test of GLM5 usage models. In a split test of GLM5 usage models. In a split test of GLM5 usage compared against other thirdparty compared against other thirdparty compared against other thirdparty providers and direct usage from the providers and direct usage from the providers and direct usage from the model lab G.A.I., Friendly AI was model lab G.A.I., Friendly AI was model lab G.A.I., Friendly AI was consistently seven times faster with a consistently seven times faster with a consistently seven times faster with a significantly lower error rate. Today, significantly lower error rate. Today, significantly lower error rate. Today, friendly AI is a core component of the friendly AI is a core component of the friendly AI is a core component of the killer stack. killer stack. killer stack. And you can consume this however fits And you can consume this however fits And you can consume this however fits your stack. Model API is the fastest way your stack. Model API is the fastest way your stack. Model API is the fastest way to start. Core frontier open weight to start. Core frontier open weight to start. Core frontier open weight models through our serless API.

models through our serless API. models through our serless API. Dedicated endpoints give you your own Dedicated endpoints give you your own Dedicated endpoints give you your own isolated deployment with guaranteed isolated deployment with guaranteed isolated deployment with guaranteed SLAs's for production workloads. And SLAs's for production workloads. And SLAs's for production workloads. And BYOG bring your own GPU lets you run BYOG bring your own GPU lets you run BYOG bring your own GPU lets you run friendly inference on your own friendly inference on your own friendly inference on your own infrastructure. Same stack, three ways infrastructure. Same stack, three ways infrastructure. Same stack, three ways to deploy. To wrap up, there are three things to To wrap up, there are three things to remember. First, frontier open weight remember. First, frontier open weight remember. First, frontier open weight models make production agents models make production agents models make production agents economically scalable. economically scalable. economically scalable. Second, agents are not just chat with Second, agents are not just chat with Second, agents are not just chat with more cores. Agentic inference requires more cores. Agentic inference requires more cores. Agentic inference requires optimizing end to end task latency with optimizing end to end task latency with optimizing end to end task latency with the challenges I mentioned. the challenges I mentioned. the challenges I mentioned. Third, friendly AI is built as an Third, friendly AI is built as an Third, friendly AI is built as an inference cloud for that word. Fast, inference cloud for that word. Fast, inference cloud for that word. Fast, reliable, cost effective agentic reliable, cost effective agentic reliable, cost effective agentic inference. Thank you for attending my session. If Thank you for attending my session. If you're building agents, give uh frontier you're building agents, give uh frontier you're building agents, give uh frontier openweight motors a try on friendly openweight motors a try on friendly openweight motors a try on friendly today. You can get started at today. You can get started at today. You can get started at friendly.ai in minutes. And uh thank friendly.ai in minutes. And uh thank friendly.ai in minutes. And uh thank you. I'll be around after the session.

you. I'll be around after the session. you. I'll be around after the session. Um thank you. Um thank you. Um thank you. >> [applause]