One Day, $1,000 in Tokens: An AI War Room for Legacy Code — Ford Prior, GEICO

For the past few years, it For the past few years, it seems like all we've been seems like all we've been seems like all we've been talking about is talking about is talking about is how to speed up our how to speed up our how to speed up our SDLC. You all know what SDLC. You all know what SDLC. You all know what I'm talking about. Namely, about this I'm talking about. Namely, about this I'm talking about. Namely, about this big big big development stage in the middle, development stage in the middle, development stage in the middle, you know? Where we you know? Where we you know? Where we did a lot of did a lot of did a lot of coding and coding and coding and intellectual intellectual intellectual work. And of course, work. And of course, work. And of course, at least where I at least where I at least where I work, that step has been work, that step has been work, that step has been effectively erased, right effectively erased, right ? And we got ? And we got ? And we got a lot of time back. But I think a lot of time back. But I think , and I'm not a prophet, , and I'm not a prophet, , and I'm not a prophet, imagining a future where imagining a future where imagining a future where agents do agents do agents do the work and humans the work and humans the work and humans stay in stay in stay in the middle as the middle as the middle as managers managers managers signing off on every signing off on every signing off on every step. But I work at step. But I work at step. But I work at Geico, and we're not a Geico, and we're not a Geico, and we're not a technology technology technology company. We are a company. We are a risk management company, and risk management company, and risk management is risk management is risk management is embedded in everything we embedded in everything we embedded in everything we do, including do, including do, including how we develop and how we develop and how we develop and deliver deliver deliver software. So software. So software. So while we saw this while we saw this while we saw this inner loop inner loop inner loop accelerate accelerate accelerate using using using Gene Kim's concept, Gene Kim's concept, Gene Kim's concept, we were kind of we were kind of we were kind of trapped in the outer trapped in the outer trapped in the outer loop, which is everything loop, which is everything loop, which is everything else that else that else that happens outside of happens outside of happens outside of coding and coding and coding and development. And we development. And we development. And we ask ourselves what ask ourselves what ask ourselves what life would be like life would be like life would be like outside these walls, in a outside these walls, in a outside these walls, in a world where the external world where the external world where the external cycle keeps pace with the cycle keeps pace with the cycle keeps pace with the internal. So this is the internal. So this is the internal. So this is the story of how we story of how we story of how we broke free, and my broke free, and my broke free, and my goal with this talk is goal with this talk is to inspire you to to inspire you to to inspire you to do the same. When do the same. When do the same. When Geico manages America's fleet of vehicles Geico manages America's fleet of vehicles Geico manages America's fleet of vehicles or the risks or the risks or the risks associated with it, and associated with it, and something

something like an accident happens on the road, we like an accident happens on the road, we like an accident happens on the road, we incur what we incur what we incur what we call a loss, call a loss, which is usually which is usually reported by one reported by one reported by one of our of our of our insureds and insureds and insureds and becomes an insurance becomes an insurance becomes an insurance claim. And claim. And claim. And claims settlement technology is claims settlement technology is where the policy where the policy where the policy comes to life and gets comes to life and gets comes to life and gets the customer back on the road the customer back on the road the customer back on the road as quickly as possible with as quickly as possible with as quickly as possible with minimal minimal minimal inconvenience, which we inconvenience, which we inconvenience, which we call resolution call resolution . My name is Ford . My name is Ford . My name is Ford Pryor. I am a senior Pryor. I am a senior Pryor. I am a senior engineer and technical engineer and technical engineer and technical evangelist evangelist evangelist working in the claims technology group working in the claims technology group at GEICO. I seem to be the at GEICO. I seem to be the only technical only technical only technical evangelist at Berkshire evangelist at Berkshire evangelist at Berkshire Hathaway, but that means Hathaway, but that means Hathaway, but that means that in our group of 300 that in our group of 300 that in our group of 300 engineers, we have engineers, we have engineers, we have about 32 teams, as about 32 teams, as about 32 teams, as we called them, with a we called them, with a we called them, with a technical lead and a technical lead and a technical lead and a business partner for business partner for business partner for each of them. And my each of them. And my each of them. And my job is to job is to job is to find and find and find and implement implement implement best practices in best practices in best practices in our SDLC. I run the our SDLC. I run the Claims Resolution Innovation Lab, Claims Resolution Innovation Lab, and I share and I share and I share stories like this to stories like this to stories like this to engage the engage the engage the world's best engineers in world's best engineers in world's best engineers in our problem area, our problem area, our problem area, which we find very which we find very which we find very interesting, not to mention interesting, not to mention interesting, not to mention important important important to the American to the American to the American economy. So, economy. So, economy. So, a little more a little more a little more context on how context on how context on how we develop we develop software. Like software. Like many organizations, many organizations, many organizations, we view our we view our we view our work through the lens of work through the lens of work through the lens of semantic semantic semantic versioning.

versioning. versioning. About 6 months About 6 months About 6 months ago, we ago, we ago, we analyzed our analyzed our analyzed our task queues and task queues and task queues and discovered that discovered that discovered that after after after completing the next completing the next completing the next patch or update, our engineers patch or update, our engineers patch or update, our engineers had only 25% of the time, or had only 25% of the time, or had only 25% of the time, or even less, to work even less, to work even less, to work on large-scale on large-scale on large-scale changes that changes that changes that keep us keep us keep us competitive competitive competitive and ahead of our and ahead of our and ahead of our competitors. It's like competitors. It's like competitors. It's like going from V1 to V2 and going from V1 to V2 and going from V1 to V2 and V3. So, we began V3. So, we began V3. So, we began strategically strategically strategically investing in investing in investing in managed GEICO AI agents across managed GEICO AI agents across managed GEICO AI agents across platforms, platforms, platforms, using different using different using different models to models to models to delegate work delegate work delegate work from specification to from specification to from specification to patch deployment patch deployment patch deployment and small tasks. and small tasks. Our goal is to redirect these redirect these redirect these efforts so that our efforts so that our efforts so that our brilliant engineers brilliant engineers brilliant engineers can focus can focus can focus on what really on what really on what really matters and brings matters and brings matters and brings them joy: them joy: them joy: work at scale. work at scale. work at scale. Which, as I mentioned, Which, as I mentioned, Which, as I mentioned, is the key to our is the key to our is the key to our competitive competitive competitive advantage. So, this is our advantage. So, this is our advantage. So, this is our current state. But when current state. But when current state. But when we look at this we look at this we look at this large-scale work, we large-scale work, we large-scale work, we see that we are limited in the see that we are limited in the see that we are limited in the possibilities of possibilities of possibilities of collaboration in rooms collaboration in rooms collaboration in rooms where the where the where the right specialists must gather at the right specialists must gather at the right specialists must gather at the right time to right time to right time to make the right make the right make the right decisions at each decisions at each decisions at each stage of the life stage of the life stage of the life cycle necessary cycle necessary cycle necessary to bring the to bring the to bring the product to the working product to the working product to the working environment. We environment. We environment. We lack not only lack not only lack not only collaboration collaboration , but also intelligence: , but also intelligence: , but also intelligence: the ability to connect the ability to connect the ability to connect all the variables all the variables all the variables surrounding a problem, surrounding a problem, surrounding a problem, deepening our deepening our deepening our understanding, including understanding, including understanding, including fact-checking, fact-checking, fact-checking, code analysis—in short, code analysis—in short, code analysis—in short, everything needed to everything needed to everything needed to deliver insights as deliver insights as deliver insights as quickly as quickly as quickly as solutions are needed. And that is what

solutions are needed. And that is what solutions are needed. And that is what ultimately ultimately ultimately gives us gives us gives us the confidence to the confidence to the confidence to bring these bring these bring these large-scale changes to the large-scale changes to the large-scale changes to the market. So, it's market. So, it's market. So, it's collaboration and collaboration and collaboration and intelligence. We lack intelligence. We lack intelligence. We lack both of these components both of these components both of these components when scaling when scaling when scaling big changes. So big changes. So big changes. So to solve this to solve this to solve this issue, we created a issue, we created a issue, we created a multiplayer multiplayer multiplayer AI-driven " AI-driven " war room." war room." war room." It consists of a It consists of a It consists of a list of key list of key list of key decision-makers decision-makers decision-makers and subject matter and subject matter and subject matter experts, a driver to experts, a driver to experts, a driver to manage these manage these manage these complex AI complex AI tools, and an AI- tools, and an AI- centric centric centric productivity suite that productivity suite that productivity suite that includes our includes our discussion channels and discussion channels and interactive design interactive design studio. I can't studio. I can't studio. I can't name name name the supplier, but you the supplier, but you the supplier, but you can easily can easily can easily guess. This is guess. This is guess. This is required for all required for all required for all our shared our shared our shared elements, and of course elements, and of course elements, and of course we have our we have our we have our code repositories. All of code repositories. All of code repositories. All of this is combined into a powerful this is combined into a powerful this is combined into a powerful combination of combination of combination of workflows, workflows, workflows, agent skills, MCPs, agent skills, MCPs, agent skills, MCPs, enterprise MCPs, and enterprise MCPs, and enterprise MCPs, and agent fleets to agent fleets to agent fleets to do the hardest do the hardest do the hardest work, validation, work, validation, work, validation, data synthesis, and more. data synthesis, and more. data synthesis, and more. So my question So my question So my question was: how was: how was: how can we speed up the can we speed up the can we speed up the provisioning of provisioning of agent-callable APIs for our agent-callable APIs for our legacy core legacy core legacy core functions? For functions? For functions? For context, our context, our context, our legacy legacy legacy claims processing systems claims processing systems are millions of lines of are millions of lines of are millions of lines of code, right? So, the code, right? So, the code, right? So, the deep context deep context deep context that we're trying to that we're trying to that we're trying to create here is create here is create here is built once and built once and built once and then can be then can be reused. So, that's just a little bit of additional additional context. But I context. But I context. But I bet that bet that bet that with a thousand with a thousand with a thousand dollars in AI tokens, dollars in AI tokens, dollars in AI tokens, in one day we in one day we in one day we could completely could completely could completely map one map one map one domain, including domain, including domain, including graphical graphical graphical representations of the business

representations of the business topology, all the topology, all the topology, all the complex code in our complex code in our complex code in our legacy systems, legacy systems, legacy systems, and everything in between; and and everything in between; and and everything in between; and we can get the we can get the we can get the agent to find and agent to find and agent to find and suggest one suggest one suggest one major change, which in major change, which in major change, which in our case we our case we our case we limited to an API limited to an API limited to an API called by the agent called by the agent called by the agent in our legacy in our legacy in our legacy claims systems, and claims systems, and claims systems, and deploy it all to deploy it all to deploy it all to our our our agent platform to agent platform to agent platform to demonstrate to the demonstrate to the demonstrate to the boss at 4pm. I put boss at 4pm. I put boss at 4pm. I put pressure on myself, pressure on myself, pressure on myself, but we ran this but we ran this but we ran this experiment three times experiment three times experiment three times across three different across three different code modernization tasks. This code modernization tasks. This little taxonomy— little taxonomy— little taxonomy— update, update, update, transform, transform, transform, rethink— rethink— is taken from the Anthropic is taken from the Anthropic is taken from the Anthropic code modernization plugin code modernization plugin code modernization plugin , which I’ll , which I’ll , which I’ll talk about in a minute. talk about in a minute. talk about in a minute. But here's the thing: But here's the thing: But here's the thing: first we first we first we looked at our looked at our looked at our rental domain, which is rental domain, which is rental domain, which is relatively simple, it's relatively simple, it's relatively simple, it's all the technology that all the technology that all the technology that ensures ensures ensures you get a rental you get a rental you get a rental car when your car when your car when your primary car primary car primary car gets into an accident. gets into an accident. gets into an accident. We implemented a newer We implemented a newer We implemented a newer version of the same version of the same version of the same technology, which technology, which technology, which is actually the definition of is actually the definition of is actually the definition of an update, and an update, and an update, and extended the existing API extended the existing API . It was an interesting . It was an interesting . It was an interesting story because the agent story because the agent story because the agent used a pretty used a pretty used a pretty interesting process for interesting process for data mining and analysis to figure out data mining and analysis to figure out that we could combine that we could combine that we could combine several facts and several facts and several facts and save eight and a save eight and a save eight and a half thousand half thousand half thousand queries per month by queries per month by queries per month by providing the client with providing the client with providing the client with up-to-date up-to-date up-to-date rental information. This was the rental information. This was the rental information. This was the first experiment first experiment first experiment we conducted, and the we conducted, and the we conducted, and the second was the second was the second was the transformation of our transformation of our transformation of our loss domain. And loss domain. And loss domain. And losses are probably losses are probably our most difficult... No, sorry. I

No, sorry. I No, sorry. I mean losses, I'm looking mean losses, I'm looking mean losses, I'm looking at the notes. The at the notes. The at the notes. The loss domain is how we loss domain is how we loss domain is how we estimate the dollar amount of the estimate the dollar amount of the estimate the dollar amount of the claimed loss. claimed loss. claimed loss. So for this So for this So for this case, we created a case, we created a case, we created a set of new APIs set of new APIs set of new APIs called by the agent called by the agent to unlock the to unlock the to unlock the broader broader broader workflow that workflow that workflow that required the required the required the damage data. And finally, we damage data. And finally, we damage data. And finally, we rethought our rethought our rethought our recycling domain, recycling domain, recycling domain, which is perhaps the which is perhaps the which is perhaps the most complex of the most complex of the most complex of the three. This is essentially what three. This is essentially what happens happens happens when your car is when your car is when your car is beyond repair. beyond repair. beyond repair. In this case, we In this case, we In this case, we prepared a fully prepared a fully prepared a fully tested tested tested migration plan with migration plan with migration plan with API specifications and API specifications and API specifications and design mockups. design mockups. design mockups. Approved, and this is the Approved, and this is the Approved, and this is the most important part most important part , approved by the , approved by the , approved by the product and product and product and technology departments at technology departments at technology departments at every level. So, every level. So, every level. So, this is a fully this is a fully this is a fully tested tested tested project plan, ready to be project plan, ready to be project plan, ready to be implemented, which, as implemented, which, as implemented, which, as you know, you know, you know, is a major milestone in itself is a major milestone in itself when everyone has the when everyone has the when everyone has the same vision. same vision. same vision. Including technical Including technical Including technical details. And, to be honest details. And, to be honest details. And, to be honest , in this , in this , in this case, he case, he case, he understood the understood the understood the subject areas very well. And subject areas very well. And subject areas very well. And the story with the recovery work the story with the recovery work was that was that he found a bug in the he found a bug in the he found a bug in the working environment. working environment. working environment. I can't say I can't say I can't say more, but it was. more, but it was. more, but it was. So, So, So, anyway, you might be anyway, you might be anyway, you might be interested in knowing interested in knowing interested in knowing a little more detail a little more detail a little more detail about how it all about how it all about how it all worked. Actually, worked. Actually, worked. Actually, our friends at Anthropic our friends at Anthropic our friends at Anthropic provided a bunch of stuff, and provided a bunch of stuff, and provided a bunch of stuff, and we also provided a we also provided a we also provided a lot of stuff ourselves. So on lot of stuff ourselves. So on lot of stuff ourselves. So on the left we have the left we have the harness, the model, this the harness, the model, this thing called Fleet thing called Fleet thing called Fleet Kit, which I'll explain in a Kit, which I'll explain in a Kit, which I'll explain in a moment, and this plugin that moment, and this plugin that moment, and this plugin that did the did the did the code modernization task. This is code modernization task. This is code modernization task. This is on the left, provided by on the left, provided by on the left, provided by the supplier. By the supplier. By the supplier. By the way, we could the way, we could the way, we could replace any of replace any of replace any of this with another this with another this with another vendor, another vendor, another vendor, another model, another model, another model, another framework, etc. And on framework, etc. And on framework, etc. And on the right was something the right was something the right was something we can't replace we can't replace . This is all GEICO, and we . This is all GEICO, and we . This is all GEICO, and we need it a need it a need it a lot. We had lot. We had lot. We had context in our

context in our context in our knowledge base, including knowledge base, including software standards and all this software standards and all this archaeology of archaeology of archaeology of past solutions that past solutions that past solutions that we had to conduct, we had to conduct, we had to conduct, a list of tools that an a list of tools that an a list of tools that an agent could agent could agent could access on access on access on my machine, and my machine, and my machine, and instructions for instructions for instructions for agents on how to agents on how to agents on how to use them. We had use them. We had use them. We had methods. I'm not going to talk methods. I'm not going to talk methods. I'm not going to talk about this in order, about this in order, about this in order, but we had methods but we had methods but we had methods that are essentially all of that are essentially all of that are essentially all of our skills and our skills and our skills and all of those all of those domain-specific, domain-specific, highly optimized highly optimized highly optimized ways of developing ways of developing software that are software that are probably probably probably different from different from different from yours. And the last one is yours. And the last one is judgment, that is, who judgment, that is, who judgment, that is, who can decide what and can decide what and can decide what and at what stage, right? at what stage, right? at what stage, right? So, both from the perspective of a So, both from the perspective of a subject matter expert and subject matter expert and from the perspective of from the perspective of from the perspective of the management the management the management decision maker. And we decision maker. And we decision maker. And we grouped them into two grouped them into two grouped them into two repositories. repositories. I'll explain why in a minute. I'll explain why in a minute. I'll explain why in a minute. But it allowed us to But it allowed us to But it allowed us to make it so that make it so that make it so that agents could agents could agents could add to them and add to them and add to them and correct them, and correct them, and correct them, and also that people could also that people could also that people could do the same. But do the same. But do the same. But this AI, and this is one of the this AI, and this is one of the this AI, and this is one of the most interesting parts, most interesting parts, most interesting parts, is expensive to operate is expensive to operate is expensive to operate at scale. at scale. at scale. And so we needed a And so we needed a And so we needed a park manager, a sort of park manager, a sort of PhD-level park manager. This was a PhD-level park manager. This was a small collaboration with small collaboration with small collaboration with Anthropic, and it's not Anthropic, and it's not Anthropic, and it's not open source yet, but I open source yet, but I open source yet, but I can tell you how can tell you how can tell you how it works today, and it works today, and it works today, and I hope you I hope you I hope you can build your can build your can build your own version, because it's own version, because it's own version, because it's not high math.

not high math. not high math. So, FleetKit is a clearly So, FleetKit is a clearly So, FleetKit is a clearly structured collection of structured collection of structured collection of rules, protocols, and rules, protocols, and rules, protocols, and instructions instructions instructions written in written in written in markdown format. Optimized markdown format. Optimized markdown format. Optimized for the Claude environment, for the Claude environment, for the Claude environment, in particular Fable, and in particular Fable, and in particular Fable, and designed around the designed around the designed around the main task of main task of transforming transforming transforming architecture. And as you architecture. And as you architecture. And as you might guess, might guess, might guess, these kinds of things don't these kinds of things don't these kinds of things don't work very well with work very well with work very well with some of the earlier some of the earlier some of the earlier iterations of the models. iterations of the models. iterations of the models. I think Opus 48 and I think Opus 48 and I think Opus 48 and newer versions are what I think newer versions are what I think newer versions are what I think could could could handle this handle this handle this type of control. This is like a type of control. This is like a type of control. This is like a level above the level above the level above the actual system actual system actual system prompts, right? These are prompts, right? These are prompts, right? These are the rules that an the rules that an the rules that an agent must follow. agent must follow. And it's quite simple. And it's quite simple. He takes a goal, He takes a goal, He takes a goal, breaks it down into breaks it down into breaks it down into parts, sends parts, sends parts, sends leaders to leaders to leaders to conduct conduct conduct research and research and research and determine an approach. determine an approach. determine an approach. Then all of this Then all of this Then all of this falls into a queue with falls into a queue with falls into a queue with human participation. You can human participation. You can human participation. You can use GitHub use GitHub use GitHub issues, for example, something issues, for example, something issues, for example, something that people can that people can that people can comment on and comment on and comment on and improve, or where improve, or where improve, or where other agents can other agents can other agents can add more add more add more context. And when context. And when everything looks good from an approval perspective, everything looks good from an approval perspective, he sends he sends he sends developers to do developers to do developers to do the rest. In this the rest. In this the rest. In this case we case we case we use beads.

use beads. use beads. Not sure if you Not sure if you Not sure if you know about beads, but know about beads, but know about beads, but they help they help they help agents track agents track agents track everything within their own everything within their own everything within their own space. Ultimately, space. Ultimately, space. Ultimately, the goal is to the goal is to the goal is to create what we create what we create what we call a significant call a significant call a significant change, like a change, like a change, like a change request in our change request in our change request in our systems or a systems or a systems or a code change in our code change in our code change in our repositories. The repositories. The repositories. The main idea is main idea is main idea is that that software development becomes software development becomes less like less like less like painting on a canvas painting on a canvas painting on a canvas and more like working in a and more like working in a and more like working in a factory. Okay, I factory. Okay, I factory. Okay, I need to hurry up. need to hurry up. need to hurry up. So, on the left you So, on the left you So, on the left you see these three see these three see these three repositories. They repositories. They repositories. They contain everything you contain everything you contain everything you need to run need to run need to run Fleet Kit, including its Fleet Kit, including its Fleet Kit, including its source code. By the way, source code. By the way, source code. By the way, all of these slides all of these slides all of these slides are available in a are available in a are available in a blog post, which I'll blog post, which I'll blog post, which I'll link to at the very link to at the very link to at the very end. Because of these end. Because of these end. Because of these spotlights, I can't see spotlights, I can't see spotlights, I can't see who's taking the photos, but I who's taking the photos, but I who's taking the photos, but I just want to say: just want to say: just want to say: if you want if you want if you want to see to see to see the renderings I'm the renderings I'm the renderings I'm showing, they're all in the showing, they're all in the showing, they're all in the blog post blog post blog post published published published today. And I'll leave a today. And I'll leave a today. And I'll leave a link to it at link to it at link to it at the end of the slides. So, the end of the slides. So, the end of the slides. So, all of this is started, all of this is started, all of this is started, pulled into a pulled into a pulled into a Cloud Code session with the plugin, Cloud Code session with the plugin, Cloud Code session with the plugin, activated, and you activated, and you activated, and you give it a task. give it a task. give it a task. He accepts or He accepts or He accepts or creates a channel in Slack or creates a channel in Slack or creates a channel in Slack or on our on our on our communication communication communication platform. And actually, platform. And actually, platform. And actually, this work channel this work channel this work channel becomes a place where everyone becomes a place where everyone becomes a place where everyone communicates.

communicates. communicates. All All All participants in the "war participants in the "war participants in the "war room", people room", people room", people important for important for important for decision-making in the decision-making in the decision-making in the project, are invited. And each time an project, are invited. And each time an project, are invited. And each time an agent reaches agent reaches agent reaches a stage while following a stage while following a stage while following its its its nine-step nine-step nine-step process, it creates process, it creates process, it creates one of two one of two one of two artifacts. By the way, artifacts. By the way, artifacts. By the way, on the right, where it says on the right, where it says on the right, where it says "military room," "military room," "military room," he can tag he can tag he can tag individual people individual people individual people based on a list. based on a list. based on a list. For example, if there are For example, if there are For example, if there are any missing any missing any missing details about a product, details about a product, details about a product, it will find out who is responsible for it it will find out who is responsible for it it will find out who is responsible for it and mention and mention and mention that specific person. that specific person. that specific person. In addition, it In addition, it In addition, it creates creates creates worksheets that worksheets that worksheets that are actually very high-quality HTML are actually very high-quality HTML are actually very high-quality HTML using Cloud using Cloud using Cloud Artifacts. People can Artifacts. People can Artifacts. People can fill them out. They are fill them out. They are fill them out. They are automatically automatically automatically saved. This is saved. This is saved. This is needed to needed to needed to fill in the gaps. fill in the gaps. fill in the gaps. In addition, it In addition, it In addition, it creates design creates design creates design documentation, documentation, documentation, including complex including complex including complex visualizations of our visualizations of our visualizations of our architecture that did architecture that did architecture that did not exist before. The not exist before. The not exist before. The teams were happy to teams were happy to teams were happy to receive them because they could receive them because they could receive them because they could keep them, and keep them, and keep them, and then we would edit then we would edit then we would edit them, and the agent could them, and the agent could them, and the agent could read our modified read our modified read our modified document again, for example, document again, for example, document again, for example, in draw.io or Figma. After in draw.io or Figma. After in draw.io or Figma. After which all this is which all this is which all this is pulled back into the pulled back into the pulled back into the session again. OK. If you session again. OK. If you session again. OK. If you looked looked looked into our “war into our “war into our “war room” in Richmond, room” in Richmond, room” in Richmond, Virginia, Virginia, Virginia, where I’m from, this summer, you would where I’m from, this summer, you would where I’m from, this summer, you would see either a big see either a big see either a big Cloud Code screen with a bunch of Cloud Code screen with a bunch of Cloud Code screen with a bunch of people around it, or people around it, or people around it, or one of these two one of these two one of these two screens. On the left is the screens. On the left is the product worksheet we had to product worksheet we had to fill out, and on the right is one fill out, and on the right is one of our of our charting tools. The main point charting tools. The main point and what and what and what impressed me the most impressed me the most was that these artifacts was that these artifacts was that these artifacts were of really high were of really high were of really high quality. So quality. So quality. So the question never arose: " the question never arose: " Why is the agent asking

Why is the agent asking Why is the agent asking me stupid questions?" me stupid questions?" » We thought, "Wow, that's a » We thought, "Wow, that's a » We thought, "Wow, that's a really great really great really great question that we question that we question that we never never never thought about when thought about when thought about when identifying an edge identifying an edge identifying an edge case in business or case in business or case in business or technology." And the technology." And the technology." And the design artifacts, as design artifacts, as design artifacts, as I said, the teams I said, the teams I said, the teams could keep to themselves, could keep to themselves, could keep to themselves, and often they and often they and often they described things that no described things that no described things that no one had time for, one had time for, one had time for, because, as you know, not because, as you know, not because, as you know, not every engineer likes to every engineer likes to every engineer likes to draw diagrams. draw diagrams. draw diagrams. Okay, I'm Okay, I'm Okay, I'm missing something here, but missing something here, but missing something here, but we're moving on because we're moving on because we're moving on because I have I have I have about 3 minutes left. Um, about 3 minutes left. Um, about 3 minutes left. Um, these " these " war room" tools, war room" tools, war room" tools, well, to put it well, to put it well, to put it simply—there are two simply—there are two simply—there are two types of agents, " types of agents, " code divers" as we code divers" as we code divers" as we called them, who called them, who called them, who were doing were doing were doing intensive analysis intensive analysis intensive analysis and and and graphing of our graphing of our graphing of our outdated object- outdated object- oriented languages. This oriented languages. This oriented languages. This was happening in the cloud, was happening in the cloud, was happening in the cloud, and we also had and we also had and we also had local local local agents who had agents who had agents who had access to my entire access to my entire access to my entire laptop, all the MCPs laptop, all the MCPs laptop, all the MCPs on it, many of on it, many of on it, many of which were not yet in which were not yet in which were not yet in production, were production, were enterprise-specific, enterprise-specific, highly optimized highly optimized highly optimized for our for our for our problems, and could problems, and could problems, and could produce very valuable produce very valuable produce very valuable data, including data, including data, including fragmented versions of the fragmented versions of the fragmented versions of the codebases that codebases that codebases that were being analyzed. were being analyzed. were being analyzed. So, there is a bindings file So, there is a bindings file that I provided to the agent that I provided to the agent that I provided to the agent to describe what to describe what to describe what is on my is on my is on my computer, how it computer, how it computer, how it works, and how to works, and how to works, and how to map it to work map it to work map it to work with the with the with the fleet kit. And fleet kit. And fleet kit. And you can see the you can see the you can see the built-in MCP servers.

built-in MCP servers. Very quickly: the Very quickly: the code modernization plugin code modernization plugin is the heart of the whole is the heart of the whole is the heart of the whole system, and what's system, and what's system, and what's inside it, I inside it, I inside it, I think you'll find think you'll find think you'll find quite interesting, so quite interesting, so quite interesting, so I recommend you explore it I recommend you explore it I recommend you explore it after finishing after finishing after finishing this presentation to this presentation to this presentation to get more get more get more information. And I information. And I information. And I will move on. will move on. will move on. After all, the After all, the After all, the operator's laptop, meaning my operator's laptop, meaning my operator's laptop, meaning my laptop, was the laptop, was the execution environment, okay? execution environment, okay? I had to I had to I had to complete nine complete nine complete nine steps steps steps starting with the starting with the starting with the loading stage, so loading stage, so loading stage, so eight steps after eight steps after eight steps after that. And what I want you to that. And what I want you to that. And what I want you to take away from this very take away from this very take away from this very rich slide is rich slide is rich slide is this: on the left are this: on the left are this: on the left are all the steps all the steps all the steps branching off from the branching off from the code modification plugin. code modification plugin. Okay, we adapted Okay, we adapted Okay, we adapted it to our needs it to our needs , as I advise , as I advise , as I advise everyone to do. And on the right— everyone to do. And on the right— these orange elements— these orange elements— are the created are the created worksheets. Therefore, it worksheets. Therefore, it worksheets. Therefore, it required required required human approval. And during human approval. And during human approval. And during this journey to this journey to this journey to modernization, you modernization, you modernization, you can see that there can see that there was a lot of human was a lot of human involvement at the beginning, and less at the end involvement at the beginning, and less at the end , and the foundation of it all , and the foundation of it all , and the foundation of it all was this was this code modernization plugin. And another interesting code modernization plugin. And another interesting detail: the detail: the detail: the code modernization plugin code modernization plugin code modernization plugin performs performs performs deterministic deterministic deterministic analysis, checking analysis, checking analysis, checking whether the whether the whether the previous step created the previous step created the previous step created the necessary artifacts, necessary artifacts, necessary artifacts, including the file.

including the file. So, some interesting So, some interesting lessons. Of course, I lessons. Of course, I lessons. Of course, I burned a lot of burned a lot of burned a lot of tokens. And as a tokens. And as a tokens. And as a small conclusion: small conclusion: small conclusion: do not let Claude, who do not let Claude, who do not let Claude, who runs Fable, runs Fable, runs Fable, create cron create cron jobs to jobs to jobs to check the work, check the work, check the work, as it as it as it will start a Fable session every time will start a Fable session every time will start a Fable session every time and instantly exhaust and instantly exhaust and instantly exhaust your budget. Um, but, um your budget. Um, but, um , some other interesting , some other interesting , some other interesting findings findings findings involved involved involved hint caching, hint caching, hint caching, which turned out to be much more which turned out to be much more which turned out to be much more efficient than I efficient than I efficient than I expected. As well as expected. As well as expected. As well as some atypical some atypical some atypical features of features of features of subagent behavior subagent behavior . But one interesting . But one interesting . But one interesting observation observation observation was that was that was that after creating the after creating the after creating the context and context and context and distributing tasks distributing tasks distributing tasks between agents to between agents to between agents to write our write our write our complex complex complex legacy code, we legacy code, we legacy code, we ran tests ran tests ran tests against newly against newly against newly written written written reference code and reference code and reference code and found that Fable or found that Fable or found that Fable or Sonnet 5.5 handled it just as well Sonnet 5.5 handled it just as well Sonnet 5.5 handled it just as well , and , and , and sometimes better, than Fable. sometimes better, than Fable. sometimes better, than Fable. So, um, choosing a model So, um, choosing a model is the key to success. is the key to success. is the key to success. Um, and looking to the Um, and looking to the Um, and looking to the future, um, I think future, um, I think future, um, I think that enhanced that enhanced that enhanced access control access control access control will be our will be our will be our greatest greatest greatest achievement, achievement, achievement, at least in my at least in my at least in my opinion. Um, we also opinion. Um, we also opinion. Um, we also have some have some have some interesting updates to make to the interesting updates to make to the decision models in our decision models in our framework and the boundaries of framework and the boundaries of framework and the boundaries of our model selection system our model selection system our model selection system . And also, um, our . And also, um, our . And also, um, our context consisted context consisted context consisted simply of markdown files, simply of markdown files, simply of markdown files, friends. We essentially didn't friends. We essentially didn't friends. We essentially didn't have sophisticated have sophisticated have sophisticated tools for tools for tools for visualizing visualizing visualizing connections. We didn't connections. We didn't connections. We didn't use use use anything fancy anything fancy anything fancy like, um, Graphify or like, um, Graphify or like, um, Graphify or Obsidian. I mean, Obsidian. I mean, Obsidian. I mean, if you find me if you find me if you find me after the talk, I'd after the talk, I'd after the talk, I'd love to hear your love to hear your love to hear your advice on more advice on more advice on more optimized optimized optimized search and

search and search and contextual analysis that we contextual analysis that we contextual analysis that we use to use to use to work in the "war work in the "war work in the "war room." Um, and room." Um, and room." Um, and lastly—audio as a lastly—audio as a lastly—audio as a data source. I would have data source. I would have data source. I would have liked to have turned on liked to have turned on liked to have turned on recording so that our recording so that our recording so that our conversations would be automatically conversations would be automatically conversations would be automatically transcribed, transcribed, transcribed, but instead but instead but instead I had to I had to I had to sit and frantically sit and frantically sit and frantically write down everything that write down everything that write down everything that was discussed in the was discussed in the was discussed in the meetings. Um, and it was meetings. Um, and it was meetings. Um, and it was quite a quite a quite a distraction, and I'm distraction, and I'm distraction, and I'm sure I sure I sure I missed a lot. Um, so we missed a lot. Um, so we missed a lot. Um, so we need a way to need a way to need a way to incorporate information incorporate information incorporate information from the "war room" with from the "war room" with from the "war room" with less less less data loss. And finally, how data loss. And finally, how data loss. And finally, how can we scale can we scale can we scale this across the entire this across the entire this across the entire organization? Um, organization? Um, organization? Um, find find find a group in your company that a group in your company that a group in your company that can't make can't make can't make decisions, implement decisions, implement decisions, implement it, develop it, it, develop it, it, develop it, control costs, control costs, control costs, and make it your own. At and make it your own. At and make it your own. At GEICO, we're trying GEICO, we're trying GEICO, we're trying to figure out how, while these to figure out how, while these to figure out how, while these agents agents agents are improving and are improving and are improving and looking for better ways, looking for better ways, looking for better ways, can we engage can we engage can we engage people in that people in that people in that process? And how do we process? And how do we process? And how do we engage our engage our engage our best, best, best, brightest brightest brightest engineers, who seem to engineers, who seem to engineers, who seem to receive receive receive divine divine divine inspiration to inspiration to inspiration to choose the next choose the next choose the next target while the agents target while the agents target while the agents conquer the first? conquer the first? conquer the first? Okay, that's all. Okay, that's all. Okay, that's all. Um, we broke out of Um, we broke out of Um, we broke out of our outer our outer our outer loop. Now it's your loop. Now it's your loop. Now it's your turn. Um, and here's a blog turn. Um, and here's a blog turn. Um, and here's a blog you can you can you can take a picture of if you take a picture of if you take a picture of if you want. Okay, thank you want. Okay, thank you want. Okay, thank you for your time.