Burn Your Flags: Interactive CLIs for Agents — Mark Lummus & Navinkumar Patil, PayPal

Hello everyone. Hello everyone. Welcome. My name is Welcome. My name is Welcome. My name is Mark Llamas. I'm the Mark Llamas. I'm the Mark Llamas. I'm the head of head of developer tools at PayPal. With developer tools at PayPal. With me is Naveen Kumar. He is me is Naveen Kumar. He is me is Naveen Kumar. He is our technical lead for the our technical lead for the developer tools team. Today developer tools team. Today we'll talk a little we'll talk a little we'll talk a little about CLI. The title of about CLI. The title of about CLI. The title of our presentation is our presentation is our presentation is somewhat provocative. And somewhat provocative. And somewhat provocative. And you will understand why at you will understand why at you will understand why at the end of the end of the end of the presentation. So, the presentation. So, the presentation. So, today we're today we're today we're going to talk about a going to talk about a going to talk about a specific type of CLI, the specific type of CLI, the specific type of CLI, the interactive CLI. We interactive CLI. We interactive CLI. We have prepared an exercise for you have prepared an exercise for you have prepared an exercise for you . We will provide you with . We will provide you with . We will provide you with the opportunity, for those the opportunity, for those the opportunity, for those who want to participate who want to participate , because this is a , because this is a , because this is a workshop after all. We will give you a workshop after all. We will give you a workshop after all. We will give you a CLI that you can CLI that you can CLI that you can enhance to work enhance to work enhance to work in an agent in an agent in an agent environment. This is the environment. This is the environment. This is the goal of our goal of our workshop today. We will give you workshop today. We will give you enough time. We'll enough time. We'll enough time. We'll talk for a few talk for a few talk for a few minutes, minutes, minutes, try to try to try to talk as talk as talk as little as possible, and then you little as possible, and then you little as possible, and then you start. It's quite start. It's quite start. It's quite interesting, I think interesting, I think interesting, I think someone said that everything someone said that everything someone said that everything new is well forgotten new is well forgotten new is well forgotten old. When I started old. When I started old. When I started my career many my career many my career many years ago, I already years ago, I already years ago, I already had gray hair.

had gray hair. Everything was through the CLI. Everything Everything was through the CLI. Everything was a was a was a command line interface. Do command line interface. Do command line interface. Do any of you remember any of you remember any of you remember working with MS-DOS before working with MS-DOS before working with MS-DOS before Microsoft Windows? That's right. Microsoft Windows? That's right. Microsoft Windows? That's right. At my first job At my first job At my first job out of college, I out of college, I out of college, I designed embedded designed embedded designed embedded systems. You systems. You systems. You connected to your connected to your connected to your computer via a computer via a computer via a serial port, serial port, serial port, opened a opened a opened a VT100-compatible terminal, and VT100-compatible terminal, and VT100-compatible terminal, and started typing— started typing— that was your CLI. It was the that was your CLI. It was the that was your CLI. It was the same way of same way of same way of interacting with a interacting with a interacting with a computer. Today, computer. Today, computer. Today, CLIs are experiencing CLIs are experiencing CLIs are experiencing a kind of a kind of a kind of renaissance. I think renaissance. I think renaissance. I think maybe it's the cloud maybe it's the cloud maybe it's the cloud code that made them so code that made them so code that made them so popular, and now popular, and now popular, and now we have codex. Many we have codex. Many we have codex. Many add CLI tools add CLI tools add CLI tools to make it easier to to make it easier to to make it easier to access the access the access the capabilities they capabilities they capabilities they want to provide to the agent. want to provide to the agent.

Now about the Now about the interactive CLI, interactive CLI, interactive CLI, which is why we are here which is why we are here which is why we are here today. Agents today. Agents today. Agents are constantly turning are constantly turning are constantly turning to bash, and that's a good thing. But to bash, and that's a good thing. But to bash, and that's a good thing. But if they run an if they run an if they run an interactive CLI, this interactive CLI, this interactive CLI, this can be a problem, can be a problem, can be a problem, because not all shells because not all shells because not all shells handle handle handle interactivity well. And interactivity well. And interactivity well. And if they can't if they can't if they can't cope, they cope, they cope, they just silently just silently just silently finish the job. You finish the job. You finish the job. You won't even know. won't even know. won't even know. No errors, No errors, No errors, nothing. But why is this nothing. But why is this nothing. But why is this important? important? Interactive CLIs are Interactive CLIs are great because the great because the great because the next question next question next question can arise can arise can arise depending on how depending on how depending on how you answered the you answered the you answered the previous one. The number of previous one. The number of previous one. The number of flags increases flags increases flags increases if you try to if you try to if you try to create them for all create them for all create them for all possible possible possible flow options. The choice is flow options. The choice is flow options. The choice is contextual and contextual and contextual and may depend on what is may depend on what is may depend on what is happening happening happening in the environment. So, in the environment. So, in the environment. So, if your CLI if your CLI if your CLI calls an API, calls an API, calls an API, gets a response, and gets a response, and gets a response, and then then then a question arises based on a question arises based on a question arises based on that response, that response, that response, handling that handling that handling that with flags is with flags is with flags is very, very difficult. And very, very difficult. And very, very difficult. And fuses can fuses can fuses can be part of an be part of an be part of an interactive interactive interactive process. Here is an example.

process. Here is an example. Here on the left we Here on the left we see a sample CLI. It is see a sample CLI. It is see a sample CLI. It is interactive. CLI interactive. CLI interactive. CLI opened an opened an opened an interactive prompt, interactive prompt, interactive prompt, arrow control. arrow control. The person sees it. A person can A person can use use use arrows, but the agent arrows, but the agent arrows, but the agent cannot see it. cannot see it. The agent sees a The agent sees a redraw cycle, redraw cycle, redraw cycle, not a question. And not a question. And not a question. And the environment loses the environment loses the environment loses interface state. So interface state. So , here is the failure scenario. , here is the failure scenario. , here is the failure scenario. This happens quietly This happens quietly This happens quietly and imperceptibly. The usual and imperceptibly. The usual and imperceptibly. The usual solution is to simply solution is to simply solution is to simply add checkboxes. add checkboxes. Flags are suitable Flags are suitable for simple cases. for simple cases. You know in advance You know in advance what the what the what the scenarios will be. This is a scenarios will be. This is a scenarios will be. This is a one-time pass. one-time pass. one-time pass. All options are independent. All options are independent. A great option for a A great option for a simple flag.

simple flag. But when responses But when responses change based change based change based on context or on context or on context or reactions to reactions to reactions to real-time events, it real-time events, it real-time events, it becomes a problem. You becomes a problem. You becomes a problem. You really can't really can't really can't use use use checkboxes in checkboxes in checkboxes in situations like this. An agent is situations like this. An agent is situations like this. An agent is not an not an not an automation script. The automation script. The automation script automation script knows nothing. All knows nothing. All knows nothing. All decisions are laid out decisions are laid out decisions are laid out in advance. in advance. The agent can reason, The agent can reason, react, react, react, ask, ask, ask, retry, and retry, and retry, and go back. And go back. And go back. And most importantly, the agent most importantly, the agent most importantly, the agent can traverse a can traverse a can traverse a decision tree if you decision tree if you decision tree if you provide one. provide one. provide one. That won't work with flags. So, That won't work with flags. So, That won't work with flags. So, we already have a model for what's we already have a model for what's happening here. It's happening here. It's called " called " accessibility". So accessibility". So accessibility". So if you compare what if you compare what if you compare what we do for we do for we do for people with what people with what people with what needs to be done for needs to be done for needs to be done for agents, the solutions agents, the solutions agents, the solutions become obvious.

become obvious. It's the same discipline , but with a new reader. , but with a new reader. So, here's the So, here's the difficulty. How do difficulty. How do difficulty. How do we preserve safeguards we preserve safeguards we preserve safeguards and guidelines and guidelines and guidelines while ceasing to pretend while ceasing to pretend while ceasing to pretend that agents and people that agents and people that agents and people have the same have the same have the same capabilities? I'm not capabilities? I'm not capabilities? I'm not going to answer going to answer going to answer that question. This is the that question. This is the that question. This is the goal of our goal of our goal of our seminar. So, seminar. So, seminar. So, the task is this. We have a the task is this. We have a the task is this. We have a CLI. We call it CLI. We call it CLI. We call it Barista 9000. It Barista 9000. It Barista 9000. It will take will take will take coffee orders. And coffee orders. And coffee orders. And it won't work it won't work it won't work if you try to if you try to if you try to run it through an run it through an run it through an agent because it's agent because it's agent because it's interactive. And if interactive. And if interactive. And if you decide to take you decide to take you decide to take part in the challenge, part in the challenge, part in the challenge, the winner will receive a Mac the winner will receive a Mac the winner will receive a Mac Mini. Take it with Mini. Take it with Mini. Take it with you today. It's in you today. It's in you today. It's in that bag right there.

that bag right there. That was a challenge in That was a challenge in itself—getting the Mac Mini itself—getting the Mac Mini . I'll be honest. Good. . I'll be honest. Good. Now the simple rules. We will give you the CLI. You will We will give you the CLI. You will make changes to make changes to make changes to this CLI. Run this this CLI. Run this this CLI. Run this through an agent. And it through an agent. And it through an agent. And it will send some will send some will send some information to our information to our information to our server. And then it server. And then it server. And then it will update the will update the will update the leaderboard. We'll leaderboard. We'll leaderboard. We'll show the show the show the leaderboard in a moment. We leaderboard in a moment. We leaderboard in a moment. We don't have two entrances, don't have two entrances, don't have two entrances, so we'll replace the so we'll replace the so we'll replace the slides with a slides with a slides with a leaderboard so you can leaderboard so you can leaderboard so you can see it. And here is see it. And here is see it. And here is Naveen. He will tell Naveen. He will tell Naveen. He will tell you more about this you more about this . . . Thank you, Mark. OK. Thank you, Mark. OK. Class. Yes. I think this is an Class. Yes. I think this is an exciting task exciting task exciting task for everyone. And this topic is for everyone. And this topic is for everyone. And this topic is very close to me. very close to me. But, all But, all else aside, Mark, are else aside, Mark, are else aside, Mark, are we going to have coffee after we we going to have coffee after we we going to have coffee after we build this?

build this? build this? I hope so. I hope so. I hope so. If we If we If we succeed. succeed. succeed. Okay. Class. Mhm. So, Okay. Class. Mhm. So, Okay. Class. Mhm. So, yes. Let me, before yes. Let me, before yes. Let me, before we start we start we start the task, I want the task, I want the task, I want to invite you on a to invite you on a to invite you on a journey, okay? When I journey, okay? When I journey, okay? When I started my started my started my engineering career, I started engineering career, I started engineering career, I started working with IDEs. I working with IDEs. I working with IDEs. I started started started using Eclipse using Eclipse using Eclipse because I mostly because I mostly because I mostly wrote in Java. Over wrote in Java. Over wrote in Java. Over time, I switched to time, I switched to time, I switched to IntelliJ from JetBrains. It IntelliJ from JetBrains. It IntelliJ from JetBrains. It was amazing. was amazing. Looking back five or six years, back five or six years, back five or six years, everything started everything started everything started to evolve. to evolve. to evolve. New IDEs have appeared on the market New IDEs have appeared on the market New IDEs have appeared on the market , most notably Microsoft VS Code , most notably Microsoft VS Code , which has spawned many , which has spawned many , which has spawned many other IDEs as well. So, other IDEs as well. So, other IDEs as well. So, if we look if we look if we look now, Microsoft had its now, Microsoft had its now, Microsoft had its own editor, VS Code own editor, VS Code . But later, . But later, . But later, new AI IDEs started to appear, new AI IDEs started to appear, new AI IDEs started to appear, such as Cursor, Windsurf, and these are such as Cursor, Windsurf, and these are such as Cursor, Windsurf, and these are just a few of them. In just a few of them. In just a few of them. In all these journeys, CLI all these journeys, CLI all these journeys, CLI has remained constant, has remained constant, has remained constant, hasn't it? At some hasn't it? At some hasn't it? At some point I thought, " point I thought, " Okay, you know, CLI is Okay, you know, CLI is Okay, you know, CLI is slowly dying."

slowly dying." slowly dying." Maybe they won't Maybe they won't Maybe they won't come back. But, come back. But, come back. But, you know, I you know, I you know, I actually actually actually like the dialogue like the dialogue like the dialogue from the movie "The from the movie "The from the movie "The Hangover" where all Hangover" where all Hangover" where all four friends in the four friends in the four friends in the car say, "Hey, car say, "Hey, car say, "Hey, we're back." we're back." Imagine CLIs saying Imagine CLIs saying this to each other. And in this to each other. And in this to each other. And in the world of AI, they are indeed the world of AI, they are indeed the world of AI, they are indeed back. Isn't that right back. Isn't that right back. Isn't that right ? Whoever bought it, ? Whoever bought it, ? Whoever bought it, I don't know. But the whole I don't know. But the whole I don't know. But the whole story is that story is that story is that now CLIs have changed. And now CLIs have changed. And now CLIs have changed. And who is responsible for this who is responsible for this who is responsible for this ? ? ? Agents. Isn't that right? So Agents. Isn't that right? So Agents. Isn't that right? So this workshop this workshop this workshop focuses focuses focuses mostly on agents mostly on agents mostly on agents and your CLIs, and it will be a and your CLIs, and it will be a and your CLIs, and it will be a hands-on class. There hands-on class. There are rules here. There are are rules here. There are no rules here. no rules here. no rules here. So feel free So feel free So feel free to experiment to experiment to experiment as much as you can. And you as much as you can. And you as much as you can. And you know about the prizes here. know about the prizes here. So yes, we have... maybe Vinod, maybe Vinod, maybe Vinod, can I get a can I get a can I get a laptop? So, we have a laptop? So, we have a laptop? So, we have a few things. So, few things. So, few things. So, we have. Yes. So, we we have. Yes. So, we we have. Yes. So, we have a repository, this is PayPal have a repository, this is PayPal have a repository, this is PayPal Cafe. There is a CLI there that you Cafe. There is a CLI there that you Cafe. There is a CLI there that you can download.

can download. can download. This is a basic CLI. It wo This is a basic CLI. It wo This is a basic CLI. It wo n't be as easy for the Mac n't be as easy for the Mac n't be as easy for the Mac Mini. Yes. So you Mini. Yes. So you Mini. Yes. So you will get some will get some will get some initial commands. initial commands. initial commands. Try Try Try to work with him. to work with him. to work with him. Register. Register. Register. Enter your Enter your desired username. Your desired username. Your nicknames, whatever. And nicknames, whatever. And nicknames, whatever. And try try try making coffee with making coffee with making coffee with it. it. it. OK? This is the goal. And OK? This is the goal. And OK? This is the goal. And throughout throughout throughout this task, we this task, we this task, we want you to be want you to be want you to be innovative in innovative in innovative in your approaches. It will your approaches. It will your approaches. It will get more interesting in the future get more interesting in the future get more interesting in the future when agents when agents when agents start start start using CLI. using CLI. using CLI. So think about it So think about it So think about it from that perspective. from that perspective. I will change that. I just I will change that. I just changed my laptops changed my laptops changed my laptops here. During the session, we here. During the session, we here. During the session, we will not give you will not give you will not give you any tips at the any tips at the any tips at the start. When we start. When we start. When we feel like feel like feel like enough people enough people enough people have installed have installed have installed the repository and can the repository and can the repository and can run the CLI, then run the CLI, then run the CLI, then maybe every 15-20 maybe every 15-20 maybe every 15-20 minutes we'll minutes we'll minutes we'll share a good share a good share a good idea. I wouldn't idea. I wouldn't idea. I wouldn't call it a call it a call it a best practice.

best practice. best practice. I'll just call it a I'll just call it a I'll just call it a good idea. Because I good idea. Because I good idea. Because I feel that we are still at the very feel that we are still at the very feel that we are still at the very beginning of the journey beginning of the journey beginning of the journey and we don't yet know what and we don't yet know what and we don't yet know what practices are practices are practices are best. We seem to be best. We seem to be best. We seem to be looking for them. So these are looking for them. So these are looking for them. So these are good ideas. I also good ideas. I also good ideas. I also hope that hope that hope that at the end we at the end we at the end we hear some great ideas hear some great ideas hear some great ideas from you, the audience, from you, the audience, from you, the audience, so we can so we can so we can share them. So share them. So share them. So just keep that in just keep that in just keep that in mind. We are all learning. mind. We are all learning. mind. We are all learning. This is a journey. We This is a journey. We This is a journey. We will come back to this will come back to this will come back to this near the end. It near the end. It near the end. It doesn't work. Give us doesn't work. Give us doesn't work. Give us a couple of minutes. I think we a couple of minutes. I think we a couple of minutes. I think we will solve the technical will solve the technical will solve the technical problems. It seems like people problems. It seems like people problems. It seems like people are already using it.

are already using it. are already using it. So, on the right you So, on the right you So, on the right you will see "get CLI". And if will see "get CLI". And if will see "get CLI". And if you see PayPal cafe {slash} you see PayPal cafe {slash} you see PayPal cafe {slash} rules, that's where rules, that's where rules, that's where all the rules will be laid out all the rules will be laid out all the rules will be laid out . You can . You can . You can start working with the CLI start working with the CLI start working with the CLI while we while we while we figure it out here. OK. figure it out here. OK. Do we have a URL? Oh, Do we have a URL? Oh, is that her? No. Thank you. Yes is that her? No. Thank you. Yes . . . Hey, that's it. Hey, that's it. Hey, that's it. Perfectly. Class.

Perfectly. Class. Perfectly. Class. Thank you. Thank you. Thank you. Thank you very much. Thank you very much. Thank you very much. Gay. Gay. Class. So, you see, Class. So, you see, it's Oh, we're seeing some it's Oh, we're seeing some it's Oh, we're seeing some registrations. Okay, registrations. Okay, registrations. Okay, on the right you will see on the right you will see on the right you will see "get CLI". You can "get CLI". You can "get CLI". You can download it, and download it, and download it, and I hope you I hope you I hope you see it. It's in the see it. It's in the see it. It's in the lower right corner. And there lower right corner. And there lower right corner. And there are certain rules here. You are certain rules here. You are certain rules here. You can click on can click on can click on these rules and a these rules and a these rules and a list of rules will appear list of rules will appear list of rules will appear . So, we'll . So, we'll . So, we'll probably spend probably spend probably spend about 25–30 minutes, and about 25–30 minutes, and about 25–30 minutes, and in between we'll in between we'll in between we'll insert slides to insert slides to insert slides to make things more make things more make things more interesting. interesting. Where is the laptop for the other Where is the laptop for the other slides? At Benno. slides? At Benno. slides? At Benno. Good. This is appropriate. And Good. This is appropriate. And Good. This is appropriate. And we are here to help.

we are here to help. we are here to help. If you have any If you have any If you have any problems, please problems, please problems, please raise your hand. We raise your hand. We raise your hand. We will try to help will try to help will try to help you as best we can. you as best we can. you as best we can. We had a little We had a little We had a little internal debate now. internal debate now. internal debate now. There are great There are great There are great models. An interesting models. An interesting models. An interesting feature of this feature of this feature of this problem is that if you problem is that if you problem is that if you use use use the highest mode of the highest mode of the highest mode of thinking in thinking in thinking in the most modern LLMs, you the most modern LLMs, you the most modern LLMs, you can probably can probably can probably solve it by brute solve it by brute solve it by brute force. But I feel force. But I feel force. But I feel that in the current environment, that in the current environment, teams are starting to teams are starting to think about their think about their think about their token budget. So token budget. So token budget. So you don't necessarily you don't necessarily you don't necessarily want to spend want to spend want to spend your expensive "smart your expensive "smart " tokens to run a " tokens to run a " tokens to run a CLI tool. So the

CLI tool. So the CLI tool. So the real challenge is— real challenge is— real challenge is— can you get Haiku, can you get Haiku, can you get Haiku, Sonnet, or something on Sonnet, or something on Sonnet, or something on that level that level that level to run on your CLI? We to run on your CLI? We to run on your CLI? We know you can know you can know you can do it with the do it with the do it with the most powerful. most powerful. most powerful. Can you do it Can you do it Can you do it with something in between that is with something in between that is with something in between that is faster and cheaper? It's an faster and cheaper? It's an faster and cheaper? It's an interesting challenge if interesting challenge if interesting challenge if you want you want you want to take it. Sounds to take it. Sounds to take it. Sounds fun, right? Perfectly. fun, right? Perfectly.

Speaking Speaking of coffee, Mark, did of coffee, Mark, did of coffee, Mark, did you know that San you know that San Francisco has the best Francisco has the best Francisco has the best bakeries in the entire bakeries in the entire bakeries in the entire United States? Did United States? Did United States? Did you know about this fact? you know about this fact? you know about this fact? I didn't know about this. I didn't know about this. I didn't know about this. You should You should You should definitely definitely definitely continue. Tell continue. Tell continue. Tell me more. me more. me more. Um, I've been doing Um, I've been doing Um, I've been doing a few a few a few bakery tours around town, and it's bakery tours around town, and it's bakery tours around town, and it's so much fun. You so much fun. You so much fun. You should definitely check out all should definitely check out all should definitely check out all these amazing bakeries these amazing bakeries these amazing bakeries here. here. Good. Good. Yes. Do Yes. Do Yes. Do you have any favorites? you have any favorites? you have any favorites? Butter and Crumble. My Butter and Crumble. My Butter and Crumble. My favorite. Although I favorite. Although I favorite. Although I couldn't get there. couldn't get there. Perfectly. So, currently, Perfectly. So, currently, we have J Gray collects in first place we have J Gray collects in first place we have J Gray collects in first place . So guys, . So guys, . So guys, please please please catch up with him or catch up with him or catch up with him or her. her.

You can You can register your register your register your names using the init command. I hope names using the init command. I hope names using the init command. I hope some of you have already some of you have already some of you have already figured this out. figured this out.

I will start. I will start. Come on. Come on. Come on. Yes. Yes. Yes. Steeply. Um, let's Steeply. Um, let's Steeply. Um, let's move on to move on to move on to the presentations. In the the presentations. In the the presentations. In the meantime, you can meantime, you can meantime, you can start working. I start working. I start working. I mean, you'll mean, you'll mean, you'll continue working with the continue working with the continue working with the CLI that you have. But, CLI that you have. But, CLI that you have. But, as a first tip, as a first tip, as a first tip, you know, a good idea is you know, a good idea is to start to start to start using tmux. using tmux. using tmux. How many of you here How many of you here How many of you here know about tmux? Good. There are know about tmux? Good. There are know about tmux? Good. There are quite a few people quite a few people quite a few people here. Um, so, yes. Um, here. Um, so, yes. Um, here. Um, so, yes. Um, in short, in short, in short, start by launching tmux start by launching tmux start by launching tmux and type the help command. and type the help command. Help is very important, Help is very important, because every AI agent because every AI agent because every AI agent today today today relies on the relies on the relies on the help command before it can help command before it can use any other command. other command. So, Yes. Sending So, Yes. Sending keys and capturing keys and capturing keys and capturing the panel will help the panel will help the panel will help you in many you in many you in many ways. So, tmux, for ways. So, tmux, for ways. So, tmux, for those who don't know about it those who don't know about it those who don't know about it , is a tool for , is a tool for , is a tool for working with multiple working with multiple working with multiple terminal terminal terminal sessions at the same time, sessions at the same time, sessions at the same time, which allows you to which allows you to which allows you to use the use the panel splitting feature. In this panel splitting feature. In this example, as you can see— example, as you can see— I mean, I mean, I mean, the screen is too small for me— the screen is too small for me— the screen is too small for me— let let let me zoom in a little bit.

me zoom in a little bit. me zoom in a little bit. Yes. tmux {hyphen} d. This Yes. tmux {hyphen} d. This Yes. tmux {hyphen} d. This flag detaches the flag detaches the flag detaches the entire CLI and moves entire CLI and moves entire CLI and moves it into a completely it into a completely it into a completely different tmux instance different tmux instance different tmux instance for you. So, you for you. So, you for you. So, you see the size of the see the size of the see the size of the captured tmux panel; captured tmux panel; captured tmux panel; As I said, you As I said, you As I said, you can split can split can split panels to work panels to work panels to work in the CLI, and that's one of those in the CLI, and that's one of those in the CLI, and that's one of those cool features that cool features that cool features that helps agents helps agents helps agents do a bunch of do a bunch of do a bunch of tasks. Imagine that tasks. Imagine that tasks. Imagine that when you need to when you need to when you need to run multiple run multiple run multiple agents, you can agents, you can agents, you can open different sessions open different sessions open different sessions and tmux keeps a log of and tmux keeps a log of and tmux keeps a log of what you what you what you are working on and restores are working on and restores are working on and restores those sessions when those sessions when those sessions when needed. This is useful needed. This is useful needed. This is useful for agents to for agents to for agents to interact when interact when interact when multiple sessions are open multiple sessions are open multiple sessions are open , and it's a great , and it's a great , and it's a great way for way for way for long-running long-running long-running tasks to be tasks to be tasks to be run inside run inside run inside tmux. The second good idea tmux. The second good idea is IPC side channels. As is IPC side channels. As is IPC side channels. As I mentioned earlier, tmux I mentioned earlier, tmux I mentioned earlier, tmux was a nice was a nice was a nice upgrade. So,

upgrade. So, upgrade. So, instead of instead of instead of screen scraping, tmux screen scraping, tmux screen scraping, tmux uses uses uses visual output visual output . You can start . You can start . You can start using using using side channels. The side side channels. The side side channels. The side channels will help channels will help channels will help you identify the you identify the you identify the arrow keys arrow keys arrow keys and understand how they and understand how they and understand how they work—I'll talk about that work—I'll talk about that work—I'll talk about that on the on the on the next slide. next slide. But the agent needs to know But the agent needs to know what the output JSON will be what the output JSON will be . This becomes a . This becomes a . This becomes a critical point in the critical point in the critical point in the entire report: the entire report: the entire report: the source JSON files are what source JSON files are what the agents need. Since we are the agents need. Since we are dealing with dealing with dealing with different different different personalities; personalities; personalities; Previously, we Previously, we Previously, we focused only focused only focused only on people. Now we have on people. Now we have on people. Now we have agents. So agents. So agents. So consider both of these consider both of these consider both of these roles so that your CLI can roles so that your CLI can roles so that your CLI can effectively effectively effectively leverage leverage leverage both both both uses. So, uses. So, uses. So, about side channels for about side channels for about side channels for those who don't know: those who don't know: those who don't know: let me explain in let me explain in let me explain in simple terms. simple terms. Think of it as Think of it as adding a REST API to adding a REST API to adding a REST API to your web application.

your web application. A website provides people with a A website provides people with a visual interface visual interface . And an API provides a machine- . And an API provides a machine- structured structured structured interface. So interface. So interface. So think of the side think of the side think of the side channel as something channel as something channel as something identical. This is one identical. This is one identical. This is one analogy I analogy I analogy I can refer to. can refer to. Read state, Read state, send input send input send input as data. So, as data. So, as data. So, we have a barista brew {hyphen}{hyphen} we have a barista brew {hyphen}{hyphen} we have a barista brew {hyphen}{hyphen} agent. Now, here's this agent. Now, here's this agent. Now, here's this command that will help command that will help command that will help you run your CLI with you run your CLI with you run your CLI with background agents, background agents, background agents, right? Um, you want right? Um, you want right? Um, you want your agents to your agents to your agents to run in the background run in the background run in the background . And you add . And you add . And you add all these commands, they all these commands, they all these commands, they will help you will help you will help you try to try to try to use this in use this in use this in your current CLIs. your current CLIs. See if See if you can make you can make you can make some progress on the some progress on the some progress on the barista brew app that barista brew app that barista brew app that is on your laptops. is on your laptops.

Now there are a few things Now there are a few things you you you could actually provide as an could actually provide as an could actually provide as an example here. You example here. You example here. You see Barista Brew ask, and see Barista Brew ask, and see Barista Brew ask, and then there's a bunch of then there's a bunch of then there's a bunch of information there. Oh, what is this? information there. Oh, what is this? information there. Oh, what is this? Okay, fields, then Okay, fields, then Okay, fields, then small, large, small, large, small, large, medium, maybe medium, maybe medium, maybe coffee sizes, and yes, the coffee sizes, and yes, the coffee sizes, and yes, the default is default is default is large. What does this large. What does this large. What does this even mean? This is even mean? This is even mean? This is input data. So, all input data. So, all you need to you need to you need to do is provide this do is provide this do is provide this input so that the input so that the input so that the agent knows about it, and agent knows about it, and agent knows about it, and you should do it you should do it you should do it in machine-readable in machine-readable in machine-readable formats. So, this is where formats. So, this is where formats. So, this is where this thing comes this thing comes this thing comes into play. So don't just into play. So don't just into play. So don't just tell the agent what the tell the agent what the tell the agent what the person sees, okay? person sees, okay? Think outside the box . What are . What are . What are the agents going to think? They have the agents going to think? They have the agents going to think? They have different skills and you different skills and you different skills and you should start should start should start using those using those using those skills. The same skills. The same skills. The same request, but now with a request, but now with a request, but now with a different intelligence. different intelligence. So, on the left you see, So, on the left you see, I was talking about different I was talking about different I was talking about different personalities. One is personalities. One is personalities. One is people, and the other is agents.

people, and the other is agents. The CLI is the same, but when The CLI is the same, but when you look at how you look at how you look at how it's structured, it's structured, it's structured, people need a set of people need a set of people need a set of options. But on the other options. But on the other options. But on the other hand, you have agents to hand, you have agents to hand, you have agents to whom you can provide whom you can provide whom you can provide all this information in all this information in all this information in one input one input one input file. Good? So, this is where file. Good? So, this is where file. Good? So, this is where we call we call we call it it it context injection. You provide context injection. You provide context injection. You provide context in a way context in a way context in a way that allows me that allows me that allows me to give an example. to give an example. to give an example. You're actually You're actually You're actually running out of oat running out of oat running out of oat milk, okay? And you milk, okay? And you milk, okay? And you only have two only have two only have two gallons of oat gallons of oat gallons of oat milk left. Tell milk left. Tell milk left. Tell your agent what your agent what your agent what information they information they information they need to ask the need to ask the need to ask the user for. user for. Confirm if Confirm if two ounces is enough. two ounces is enough. two ounces is enough. Do you want me to Do you want me to Do you want me to choose a replacement for you? choose a replacement for you? This information should be This information should be provided provided provided using using using context injection context injection context injection as a technique. And you as a technique. And you as a technique. And you can can can use this in the use this in the use this in the CLI that you actually CLI that you actually CLI that you actually build. Try build. Try build. Try using this.

using this. using this. Maybe you will be at the Maybe you will be at the Maybe you will be at the highest position on the highest position on the highest position on the leaderboard. So leaderboard. So , we started the journey from T , we started the journey from T , we started the journey from T marks, which is basically marks, which is basically marks, which is basically screen scraping, to screen scraping, to screen scraping, to implementing skills implementing skills . So, T marks are . So, T marks are . So, T marks are minimal effort. I minimal effort. I minimal effort. I think many of think many of think many of you may have you may have you may have tried this. But this is where tried this. But this is where tried this. But this is where auxiliary channels auxiliary channels auxiliary channels help you help you help you provide agents with provide agents with provide agents with useful useful useful feedback, and feedback, and feedback, and introducing introducing introducing context offers context offers context offers different scenarios to think about different scenarios to think about that aren’t possible that aren’t possible with humans. So, with humans. So, with humans. So, yes. Continue to yes. Continue to yes. Continue to improve your improve your improve your results. results. Do you want to show it Do you want to show it now? now? now? Yes. Good. This will take Yes. Good. This will take Yes. Good. This will take some time.

some time. some time. Here, please. Here, please. Here, please. OK. Steeply. We OK. Steeply. We OK. Steeply. We see a lot of see a lot of see a lot of activity here. A. Someone is activity here. A. Someone is activity here. A. Someone is really close to really close to really close to 5000. Is it Bank? Oh, Monk. Okay 5000. Is it Bank? Oh, Monk. Okay . Are there any questions at this . Are there any questions at this . Are there any questions at this point? Does anyone want point? Does anyone want point? Does anyone want answers, have answers, have answers, have any questions? any questions? Sorry? Sorry? What happens if several What happens if several What happens if several people get 5,000? people get 5,000? Okay, let me Okay, let me repeat that question repeat that question repeat that question for you. What happens for you. What happens for you. What happens if several people if several people if several people reach a score of reach a score of reach a score of 5,000 points? Mark, 5,000 points? Mark, 5,000 points? Mark, do you want to answer do you want to answer do you want to answer this?

this? The idea was that The idea was that we look at we look at we look at the score, and in case of a the score, and in case of a the score, and in case of a tie, who tie, who tie, who took the least amount of took the least amount of took the least amount of time. Now, looking time. Now, looking time. Now, looking at the results, I see a at the results, I see a at the results, I see a few who completed few who completed few who completed three out of three in 0 three out of three in 0 three out of three in 0 seconds. I see many who seconds. I see many who seconds. I see many who managed in 0 managed in 0 managed in 0 seconds. So I seconds. So I seconds. So I need to consult with someone need to consult with someone need to consult with someone , because this , because this , because this shouldn't happen. But shouldn't happen. But shouldn't happen. But the idea is that we the idea is that we the idea is that we look at look at look at the result first and then at the the result first and then at the the result first and then at the response time, so that an response time, so that an response time, so that an accurate response accurate response accurate response given quickly has an given quickly has an given quickly has an advantage over an accurate advantage over an accurate advantage over an accurate response given response given response given slowly. slowly.

So far we have one So far we have one leader. leader. leader. Okay, let me Okay, let me Okay, let me try. try. This is the end of the This is the end of the leaderboard. Nicholas, keep leaderboard. Nicholas, keep leaderboard. Nicholas, keep playing. Currently 0 playing. Currently 0 playing. Currently 0 seconds. seconds. Yes. Let's Yes. Let's wait another 2 wait another 2 wait another 2 minutes. Okay, then minutes. Okay, then minutes. Okay, then we can move on to we can move on to we can move on to discussing ideas.

discussing ideas. discussing ideas. Maybe we'll spend 10 or Maybe we'll spend 10 or Maybe we'll spend 10 or 20 minutes on ideas, 20 minutes on ideas, 20 minutes on ideas, depending on depending on depending on how how how engaged they are. OK. engaged they are. OK. In a few minutes, we In a few minutes, we will open a space will open a space will open a space for the exchange of ideas. for the exchange of ideas. I hope we I hope we hear great ideas hear great ideas hear great ideas that surpass that surpass that surpass ours, because I know they ours, because I know they ours, because I know they exist. exist.

Mark, did you expect Mark, did you expect so many people so many people so many people to score over 4,000? to score over 4,000? to score over 4,000? Yes. Certainly. At this Yes. Certainly. At this Yes. Certainly. At this conference? conference? conference? Certainly. Certainly. Certainly. Perfectly. Perfectly. You weren't expecting it? You weren't expecting it? No, I expected it. Do No, I expected it. Do No, I expected it. Do you think anyone can you think anyone can you think anyone can reach 5,000? I do reach 5,000? I do n't know. We'll leave n't know. We'll leave that open. Why do that open. Why do that open. Why do n't we open n't we open n't we open the discussion now?

the discussion now? the discussion now? Yes? Yes? Yes? Of course. Of course. Of course. We are happy to We are happy to We are happy to hear ideas. So hear ideas. So , did anyone in the room , did anyone in the room , did anyone in the room come up with other ideas come up with other ideas come up with other ideas that helped them that helped them that helped them achieve such a achieve such a achieve such a high result high result high result so quickly? so quickly? Nobody wants to Nobody wants to leave their leave their leave their workplace. Oh, workplace. Oh, workplace. Oh, we have Yes. Let me, we have Yes. Let me, we have Yes. Let me, let me come let me come let me come to you. Because I can barely to you. Because I can barely to you. Because I can barely hear you. Good.

hear you. Good. This is a very easy This is a very easy benchmark to manipulate benchmark to manipulate , because the only thing that , because the only thing that , because the only thing that matters is matters is matters is executing a single executing a single executing a single order with very order with very order with very low latency. low latency. So if you make a So if you make a deterministic deterministic deterministic parser, which completely parser, which completely parser, which completely defeats the spirit of this defeats the spirit of this defeats the spirit of this competition, but you competition, but you competition, but you make such a parser make such a parser make such a parser and put something and put something and put something very close to very close to very close to the server, you can the server, you can the server, you can quickly reduce quickly reduce quickly reduce the latency if you just the latency if you just the latency if you just use "brute use "brute use "brute force". force". force". Of course. Of course. Of course. What is this that What is this that What is this that you did? you did? you did? This is what I'm doing This is what I'm doing This is what I'm doing now. now. now. Good. OK. You Good. OK. You Good. OK. You said something interesting. A said something interesting. A deterministic deterministic parser that runs in parser that runs in parser that runs in your CLI. Of course. your CLI. Of course. I actually I actually like the idea of like the idea of like the idea of determinism in CLI. determinism in CLI. The reason is that The reason is that if the CLI is a if the CLI is a if the CLI is a tool for tool for tool for using LLM, then you using LLM, then you using LLM, then you definitely want definitely want definitely want it to be it to be it to be deterministic. LLM deterministic. LLM is an unpredictable is an unpredictable is an unpredictable player in this player in this player in this situation. We want situation. We want situation. We want the CLI to be, as someone the CLI to be, as someone the CLI to be, as someone described, a small, described, a small, described, a small, sharp tool sharp tool . He has to be very, . He has to be very, . He has to be very, very focused on very focused on very focused on his work. And his work. And his work. And every time you give every time you give every time you give it a command, you it a command, you it a command, you want to get the same want to get the same response. So, you don't response. So, you don't want to embed an want to embed an want to embed an agent in a CLI and then agent in a CLI and then agent in a CLI and then pass that CLI to pass that CLI to pass that CLI to another agent because you

another agent because you another agent because you don't know what you're going don't know what you're going don't know what you're going to get. Are there any other to get. Are there any other to get. Are there any other interesting ideas? Of course, interesting ideas? Of course, interesting ideas? Of course, we...Many people we...Many people we...Many people have figured out how to circumvent have figured out how to circumvent have figured out how to circumvent the system. Where there's a Mac mini, the system. Where there's a Mac mini, the system. Where there's a Mac mini, there's a way, I there's a way, I there's a way, I guess. guess. guess. Yes. Yes. Good. Do you understand? And Good. Do you understand? And this one will wait 5 minutes. this one will wait 5 minutes. this one will wait 5 minutes. 5 more minutes. This 5 more minutes. This 5 more minutes. This result has not result has not result has not changed for changed for changed for some time. Before some time. Before some time. Before the site goes down. Did you the site goes down. Did you the site goes down. Did you refresh the page?

refresh the page? refresh the page? Yes. Yes. I'm just saying we're I'm just saying we're connecting. Oh, there connecting. Oh, there connecting. Oh, there it is. OK.

it is. OK. I think the network is I think the network is slowing down a bit. The network here is slowing down a bit. The network here is slowing down a bit. The network here is a bit slow. a bit slow. Many, many Many, many computers. That computers. That sounds good. sounds good. Yes, he didn't fall. And Yes, he didn't fall. And Yes, he didn't fall. And now everything is frozen. That's now everything is frozen. That's all. There all. There will be no changes in the results will be no changes in the results . Of course. Do . Of course. Do we want to wait a we want to wait a few more minutes or are we few more minutes or are we few more minutes or are we done?

done? done? Yes, about 5 minutes. Yes, about 5 minutes. Yes, about 5 minutes. 5 minutes, okay. Do 5 minutes, okay. Do you want to make you want to make a presentation for a presentation for a presentation for closing remarks? closing remarks? We'll wait We'll wait about 5 more minutes and about 5 more minutes and about 5 more minutes and then we'll finish. Where, then we'll finish. Where, then we'll finish. Where, where is Bonk? Raise your where is Bonk? Raise your where is Bonk? Raise your hand. hand. Who is Bonk? Okay, Who is Bonk? Okay, we have a few people. A we have a few people. A we have a few people. A wide smile, a wide smile, a wide smile, a wider smile. And we wider smile. And we wider smile. And we really want to hear really want to hear really want to hear how you did it. So how you did it. So , as soon as we , as soon as we , as soon as we announce it, we'll give announce it, we'll give announce it, we'll give you a chance to tell us you a chance to tell us you a chance to tell us what you've done. I would be what you've done. I would be what you've done. I would be happy to happy to happy to listen. I need to listen. I need to listen. I need to move here.

move here. move here. Yes. Yes. Yes. Wait a minute. Wait a minute. Sorry, I should have Sorry, I should have asked for another asked for another asked for another one, I should have one, I should have one, I should have asked for another asked for another asked for another microphone in the microphone in the microphone in the question room. question room. Uh, yeah, Bonk is just an Uh, yeah, Bonk is just an agent thing agent thing agent thing built on the built on the built on the Cloud Agent SDK. Uh, he Cloud Agent SDK. Uh, he Cloud Agent SDK. Uh, he built all of this. Before built all of this. Before built all of this. Before that, I already had a kind of that, I already had a kind of evaluation skill that evaluation skill that creates an environment and creates an environment and creates an environment and optimizes it. So, optimizes it. So, optimizes it. So, essentially, he created a essentially, he created a essentially, he created a deterministic CLI deterministic CLI solver, and now solver, and now solver, and now it just runs in it just runs in it just runs in Tmax and "bombs" your APIs Tmax and "bombs" your APIs . . Of course. Of course. Yes. Yes. Yes. OK. OK. OK. Also, I think I was Also, I think I was Also, I think I was lucky because at lucky because at lucky because at one point I reached a one point I reached a one point I reached a speed of about 29 speed of about 29 speed of about 29 milliseconds. Probably milliseconds. Probably milliseconds. Probably because we are in the same because we are in the same because we are in the same region. region. region. Of course. Of course. Of course. Network delay. Network delay. Network delay. Good. And this is all Good. And this is all Good. And this is all done on Claude? done on Claude? done on Claude? Uh, on Claude agent SDK. Uh, on Claude agent SDK. Uh, on Claude agent SDK. Of course. I Of course. I Of course. I just manage it just manage it just manage it through Slack.

through Slack. through Slack. And what model? And what model? And what model? Opus 4.8. Opus 4.8. Opus 4.8. Of course. Of course. Of course. Yes. Have Yes. Have Yes. Have you tried any you tried any other other other models? models? models? Uh, to create a CLI Uh, to create a CLI Uh, to create a CLI or... or... or... For your agent, For your agent, For your agent, for your agent. How would for your agent. How would for your agent. How would we do it? we do it? we do it? CLI is fully CLI is fully CLI is fully deterministic. So I deterministic. So I ...Agent created a CLI and ...Agent created a CLI and ...Agent created a CLI and now just now just now just runs it in Tmax. runs it in Tmax. runs it in Tmax. Of course. Of course. Of course. Yes. Yes. Yes. Perfectly. Perfectly. Perfectly. Yes. Yes. Yes. Congratulations. Congratulations. Congratulations. Thank you. Thank you. Let's move on to the Let's move on to the next one, Danny. Where is next one, Danny. Where is next one, Danny. Where is Danny? Danny? You see me, he You see me, he put it together. put it together. Last. What do we want to know? What do we want to know? Yes. How did you Yes. How did you Yes. How did you build it? build it?

Yes, I know nothing about coding at all . I...I use...I'm . I...I use...I'm basically a "vibe coder", basically a "vibe coder", basically a "vibe coder", I have a couple of agents, I have a couple of agents, I have a couple of agents, and I just said, " and I just said, " Hey, here's the problem." " Hey, here's the problem." " Read it, then Read it, then Read it, then correct and correct." " correct and correct." " So I just have So I just have So I just have Claude code in VSC." Have Claude code in VSC." Have you ever you ever built a CLI before? Uh, built a CLI before? Uh, built a CLI before? Uh, no. I saw no. I saw no. I saw a few hours ago that it a few hours ago that it a few hours ago that it was apparently possible to was apparently possible to was apparently possible to win a prize. So I win a prize. So I win a prize. So I took it upon myself and said, took it upon myself and said, took it upon myself and said, "Hey, deal with this. "Hey, deal with this. " "Explain it to me as " "Explain it to me as " "Explain it to me as if I, you know, don't if I, you know, don't if I, you know, don't know anything." "And then know anything." "And then know anything." "And then explain it as if I were explain it as if I were explain it as if I were five years old." five years old." Yeah, I...I don't know. So . . . And then we got... And then we got... Yeah, we got some Yeah, we got some Yeah, we got some good results, and I good results, and I good results, and I said, "Hey, we said, "Hey, we said, "Hey, we need to do this need to do this need to do this faster." "Give me faster." "Give me faster." "Give me some suggestions," and some suggestions," and some suggestions," and yes, seriously. So, I yes, seriously. So, I yes, seriously. So, I think we have think we have think we have two scripts running, the first one two scripts running, the first one two scripts running, the first one was doing 49.98, and I said, " was doing 49.98, and I said, " Hey, make a copy and give Hey, make a copy and give Hey, make a copy and give me some tips on how me some tips on how me some tips on how to get 49.99." to get 49.99." to get 49.99." Yes. Did you get any Yes. Did you get any Yes. Did you get any advice?

advice? advice? The agent is pretty The agent is pretty The agent is pretty automated, automated, automated, so I just say, " so I just say, " Hey, here's the beginning, here's Hey, here's the beginning, here's Hey, here's the beginning, here's the end. Give me the end. Give me the end. Give me a few questions at a few questions at a few questions at the beginning if I need to, and the beginning if I need to, and the beginning if I need to, and then fix the then fix the then fix the problems. So I don't problems. So I don't problems. So I don't have to deal with it have to deal with it have to deal with it ." Here it is ." Here it is ." Here it is . Sorry, that's not . Sorry, that's not . Sorry, that's not very good... very good... very good... No, no, that's good. I'm No, no, that's good. I'm No, no, that's good. I'm sure many of you are sure many of you are sure many of you are wondering: is wondering: is wondering: is writing a CLI something writing a CLI something writing a CLI something you'll ever you'll ever you'll ever do yourself? Will you do yourself? Will you do yourself? Will you ask an agent to ask an agent to ask an agent to do it for you? do it for you? Obviously, we have Obviously, we have a lot of people in the room who will a lot of people in the room who will a lot of people in the room who will just point Claude just point Claude just point Claude or Codex at this and or Codex at this and or Codex at this and say, "Make a CLI. say, "Make a CLI. Make it fast. Make it fast. Make it Make it Make it deterministic." And those are deterministic." And those are deterministic." And those are three ideas, right? three ideas, right? OK. You want to change You want to change the slide. the slide.

Here he is. So, we discussed all our proposals. We heard some ideas. The We heard some ideas. The main idea was to main idea was to main idea was to let the let the let the agent handle agent handle agent handle everything for you. I everything for you. I everything for you. I like this idea. like this idea. like this idea. Anything else before we Anything else before we Anything else before we finish? Good. That's finish? Good. That's finish? Good. That's it. The conclusion for us it. The conclusion for us it. The conclusion for us based on our based on our based on our instructions and instructions and instructions and recommendations: give recommendations: give recommendations: give it good context, a it good context, a it good context, a side channel, and side channel, and side channel, and tell it tell it tell it to use tmux. to use tmux. These are three good ideas These are three good ideas to get you started. Thank you to get you started. Thank you to get you started. Thank you all. Monk, do you want all. Monk, do you want all. Monk, do you want to come over?

to come over? to come over? Monk is still Monk is still Monk is still the winner for me, the winner for me, the winner for me, right? right? right? Yes. Yes. Yes. OK. OK. OK. No, no, we've already No, no, we've already No, no, we've already decided. We decided decided. We decided decided. We decided this 5 minutes ago. this 5 minutes ago. this 5 minutes ago. Good. Good. Good. Hey, I'm not going to Hey, I'm not going to Hey, I'm not going to change that. We have already change that. We have already change that. We have already decided. decided. Hooray! Perfectly. Do you want to take... Do you want to take... Where is it? Where is it? We'll just take a We'll just take a photo. photo. photo. your... your... your... Well, social media. Well, social media. Well, social media. That's right. That's right. This is a real magnet. This is a real magnet. This is a real magnet. This is a real magnet. This is a real magnet. I didn't know those I didn't know those I didn't know those existed. existed. Good. Good. Perfectly. Where did this Perfectly. Where did this Perfectly. Where did this come from?