Does Your Computer Belong To Codex? I Went To OpenAI To Ask.

Is the computer my computer or is the Is the computer my computer or is the computer really Codex's computer and I computer really Codex's computer and I computer really Codex's computer and I just have a corner and talk to it? just have a corner and talk to it? just have a corner and talk to it? Because I find a lot of the time I'm Because I find a lot of the time I'm Because I find a lot of the time I'm using Whisper Flow and I'm talking to using Whisper Flow and I'm talking to using Whisper Flow and I'm talking to the agent and then there's this like the agent and then there's this like the agent and then there's this like cursor running around on these windows cursor running around on these windows cursor running around on these windows and it's doing UX user acceptance and it's doing UX user acceptance and it's doing UX user acceptance testing on something I'm building and I testing on something I'm building and I testing on something I'm building and I don't even know what's happening in all don't even know what's happening in all don't even know what's happening in all of my windows anymore. when the context of my windows anymore. when the context of my windows anymore. when the context relevant to that role became generally relevant to that role became generally relevant to that role became generally available to the mind. available to the mind. available to the mind. >> Voice is by far the most efficient way >> Voice is by far the most efficient way >> Voice is by far the most efficient way to get context in and to communicate to get context in and to communicate to get context in and to communicate information. information. information. >> But reading is much faster than >> But reading is much faster than >> But reading is much faster than listening. listening. listening. >> And then actually like last week it >> And then actually like last week it >> And then actually like last week it found a pretty significant error in my found a pretty significant error in my found a pretty significant error in my taxes taxes taxes >> in your favor. >> in your favor. >> in your favor. >> In my favor. >> In my favor. >> In my favor. >> Ooh. that ended up saving, you know, >> Ooh. that ended up saving, you know, >> Ooh. that ended up saving, you know, thousands of dollars and I was like, thousands of dollars and I was like, thousands of dollars and I was like, this is something that I would have this is something that I would have this is something that I would have never have figured out never have figured out never have figured out >> and I underestimated the scaling law >> and I underestimated the scaling law >> and I underestimated the scaling law potential of computer use. I get to be potential of computer use. I get to be potential of computer use. I get to be here at OpenAI today. I am so excited. I here at OpenAI today. I am so excited. I here at OpenAI today. I am so excited. I know so many of you have asked more know so many of you have asked more know so many of you have asked more questions than I can count about what I questions than I can count about what I questions than I can count about what I can ask these guys at OpenAI. The the can ask these guys at OpenAI. The the can ask these guys at OpenAI. The the conversation I'm about to have is one conversation I'm about to have is one conversation I'm about to have is one I've been so excited to share with you I've been so excited to share with you I've been so excited to share with you because it gets at the heart of what because it gets at the heart of what because it gets at the heart of what makes AI go from just the chatbot in makes AI go from just the chatbot in makes AI go from just the chatbot in your pocket to something that does your pocket to something that does your pocket to something that does meaningful work. So I'm going to be meaningful work. So I'm going to be meaningful work. So I'm going to be sitting down with Ashe and Andrew and sitting down with Ashe and Andrew and sitting down with Ashe and Andrew and they're both deep into that transition they're both deep into that transition they're both deep into that transition building the future of how we all work building the future of how we all work building the future of how we all work together. And really we're going to go together. And really we're going to go together. And really we're going to go beyond the interface. What we're going beyond the interface. What we're going beyond the interface. What we're going to talk about is what does it feel like to talk about is what does it feel like to talk about is what does it feel like here at Open AI to do work differently

here at Open AI to do work differently here at Open AI to do work differently in the age of AI. No, they don't have in the age of AI. No, they don't have in the age of AI. No, they don't have all the answers figured out, but we're all the answers figured out, but we're all the answers figured out, but we're going to talk together about what that going to talk together about what that going to talk together about what that looks like in practice. We're going to looks like in practice. We're going to looks like in practice. We're going to talk together about lessons we can all talk together about lessons we can all talk together about lessons we can all take for how we work differently. And take for how we work differently. And take for how we work differently. And we're ultimately going to get to, I we're ultimately going to get to, I we're ultimately going to get to, I think, a point in the conversation where think, a point in the conversation where think, a point in the conversation where we can all understand some of the we can all understand some of the we can all understand some of the long-term trends that you can see from long-term trends that you can see from long-term trends that you can see from inside this building that are going to inside this building that are going to inside this building that are going to shape where we should go with our shape where we should go with our shape where we should go with our careers, where we should go with our careers, where we should go with our careers, where we should go with our skill sets, where we should go with our skill sets, where we should go with our skill sets, where we should go with our fluency and AI and what we focus on fluency and AI and what we focus on fluency and AI and what we focus on learning over the next 6 months to a learning over the next 6 months to a learning over the next 6 months to a year. So, I've been super excited for year. So, I've been super excited for year. So, I've been super excited for this. I can't wait for you to dive in. this. I can't wait for you to dive in. this. I can't wait for you to dive in. Andrew Ash, I'm so excited we get to Andrew Ash, I'm so excited we get to Andrew Ash, I'm so excited we get to chat. Uh maybe first could you tell us chat. Uh maybe first could you tell us chat. Uh maybe first could you tell us about who you are and how you came to be about who you are and how you came to be about who you are and how you came to be at OpenAI? at OpenAI? at OpenAI? >> Sure. Want to start? >> Sure. Want to start? >> Sure. Want to start? >> Yeah. Um I'm Andrew. I started uh work >> Yeah. Um I'm Andrew. I started uh work >> Yeah. Um I'm Andrew. I started uh work on the Codex app which is now our whole on the Codex app which is now our whole on the Codex app which is now our whole desktop app and work on a variety of desktop app and work on a variety of desktop app and work on a variety of things there. Um I've been here about a things there. Um I've been here about a things there. Um I've been here about a year now, a little over a year. year now, a little over a year. year now, a little over a year. >> Cool. >> Cool. >> Cool. >> And where were you before that?

>> And where were you before that? >> And where were you before that? >> Um I was always in a startup world. So I >> Um I was always in a startup world. So I >> Um I was always in a startup world. So I was a founder for was a founder for was a founder for >> I don't know something like eight years. >> I don't know something like eight years. >> I don't know something like eight years. um doing little things and then I you um doing little things and then I you um doing little things and then I you know got a chance to connect with some know got a chance to connect with some know got a chance to connect with some people here and uh things have just people here and uh things have just people here and uh things have just really taken off. really taken off. really taken off. >> Yeah. Yeah, that makes sense. Yeah. What >> Yeah. Yeah, that makes sense. Yeah. What >> Yeah. Yeah, that makes sense. Yeah. What about you? about you? about you? >> Cool. I'm Ash. I've been here for >> Cool. I'm Ash. I've been here for >> Cool. I'm Ash. I've been here for threeish years. I joined OpenAI to lead threeish years. I joined OpenAI to lead threeish years. I joined OpenAI to lead the tragedy enterprise team actually and the tragedy enterprise team actually and the tragedy enterprise team actually and then I bounced around a bunch doing a then I bounced around a bunch doing a then I bounced around a bunch doing a bunch of consumer stuff and then bunch of consumer stuff and then bunch of consumer stuff and then recently I've been focused on leading recently I've been focused on leading recently I've been focused on leading our productivity and engineering pillar our productivity and engineering pillar our productivity and engineering pillar um which is our goal is to bring useful um which is our goal is to bring useful um which is our goal is to bring useful agents to everyone starting by focusing agents to everyone starting by focusing agents to everyone starting by focusing on knowledge work and developers. on knowledge work and developers. on knowledge work and developers. And I guess maybe that's a great place And I guess maybe that's a great place And I guess maybe that's a great place to jump in. I am curious with the launch to jump in. I am curious with the launch to jump in. I am curious with the launch of Astra, Astra's now not just inside of Astra, Astra's now not just inside of Astra, Astra's now not just inside the walls, outside the walls, everyone the walls, outside the walls, everyone the walls, outside the walls, everyone has it. What do you guys see as the most has it. What do you guys see as the most has it. What do you guys see as the most surprising use case that you've seen for surprising use case that you've seen for surprising use case that you've seen for Astra out in the wild since it's come Astra out in the wild since it's come Astra out in the wild since it's come out? >> The most surp I mean the fruitfly has >> The most surp I mean the fruitfly has been uh a little wild. been uh a little wild. been uh a little wild. >> That has been a little crazy, hasn't it?

>> That has been a little crazy, hasn't it? >> That has been a little crazy, hasn't it? >> Yes. But I I think the the one I didn't >> Yes. But I I think the the one I didn't >> Yes. But I I think the the one I didn't realize would take off, and part of it's realize would take off, and part of it's realize would take off, and part of it's just because it's fun, is is the just because it's fun, is is the just because it's fun, is is the Blender. Um people just simulating these Blender. Um people just simulating these Blender. Um people just simulating these 3D environments. It's not the most 3D environments. It's not the most 3D environments. It's not the most useful thing that we use it for useful thing that we use it for useful thing that we use it for internally, but it's been pretty wild to internally, but it's been pretty wild to internally, but it's been pretty wild to watch people create these games and watch people create these games and watch people create these games and environments that they walk around in environments that they walk around in environments that they walk around in that, you know, photorealistic. Every that, you know, photorealistic. Every that, you know, photorealistic. Every model that comes out has these spikes of model that comes out has these spikes of model that comes out has these spikes of things that can just do really well. And things that can just do really well. And things that can just do really well. And uh one of Astra's spikes has been 3D and uh one of Astra's spikes has been 3D and uh one of Astra's spikes has been 3D and it's been it's been it's been >> And did you guys know that 3D was a >> And did you guys know that 3D was a >> And did you guys know that 3D was a spike internally when you released it? spike internally when you released it? spike internally when you released it? >> I think we had people who were playing >> I think we had people who were playing >> I think we had people who were playing with those capabilities. I think you with those capabilities. I think you with those capabilities. I think you know kind of alluding to what Andrew is know kind of alluding to what Andrew is know kind of alluding to what Andrew is saying like I think we're surprised by saying like I think we're surprised by saying like I think we're surprised by how much it's resonated. Like I had a how much it's resonated. Like I had a how much it's resonated. Like I had a family member who's like doing a home family member who's like doing a home family member who's like doing a home renovation like just paste a PDF of renovation like just paste a PDF of renovation like just paste a PDF of their blueprint into Astra and made an their blueprint into Astra and made an their blueprint into Astra and made an entire like 3D model of the renovation entire like 3D model of the renovation entire like 3D model of the renovation that you can like walk through and like that you can like walk through and like that you can like walk through and like fully accurate and they actually made fully accurate and they actually made fully accurate and they actually made some like change decisions based on that some like change decisions based on that some like change decisions based on that which is like pretty mind-blowing like which is like pretty mind-blowing like which is like pretty mind-blowing like how that's like cross the chasm from how that's like cross the chasm from how that's like cross the chasm from just like a power user tool to something just like a power user tool to something just like a power user tool to something that anyone can use to to leverage their that anyone can use to to leverage their that anyone can use to to leverage their life. What are some of the design and life. What are some of the design and life. What are some of the design and interface differences the way you guys interface differences the way you guys interface differences the way you guys have had to think differently when have had to think differently when have had to think differently when you're starting to build for we'll call you're starting to build for we'll call you're starting to build for we'll call it non-verifiable domains in the it non-verifiable domains in the it non-verifiable domains in the knowledge work space? I think with knowledge work space? I think with knowledge work space? I think with knowledge work we've really seen that knowledge work we've really seen that knowledge work we've really seen that like less is more like the the more you like less is more like the the more you like less is more like the the more you can sort of abstract away exactly what can sort of abstract away exactly what can sort of abstract away exactly what the agent is doing but still provide the agent is doing but still provide the agent is doing but still provide enough that people can trust that it's enough that people can trust that it's enough that people can trust that it's providing the right outputs. Um that's

providing the right outputs. Um that's providing the right outputs. Um that's made it much more accessible to people made it much more accessible to people made it much more accessible to people and and and >> yeah I mean if you take the computer use >> yeah I mean if you take the computer use >> yeah I mean if you take the computer use capability which is really capability which is really capability which is really state-of-the-art as an example we've state-of-the-art as an example we've state-of-the-art as an example we've always been able to do some amount of always been able to do some amount of always been able to do some amount of computer use. computer use. computer use. >> Mhm. when it was just a developer tool, >> Mhm. when it was just a developer tool, >> Mhm. when it was just a developer tool, we could get away with showing the we could get away with showing the we could get away with showing the scripts that were running to scripts that were running to scripts that were running to [clears throat] do this computer use, [clears throat] do this computer use, [clears throat] do this computer use, right? If it's going to click around right? If it's going to click around right? If it's going to click around this app, we can show the scripts, you this app, we can show the scripts, you this app, we can show the scripts, you know, it's running that the selectors know, it's running that the selectors know, it's running that the selectors it's clicking on the things like that. it's clicking on the things like that. it's clicking on the things like that. >> Um, >> Um, >> Um, >> as we've shifted this to to hit a larger >> as we've shifted this to to hit a larger >> as we've shifted this to to hit a larger audience, we have to build out like audience, we have to build out like audience, we have to build out like >> you can't show that script. You got to >> you can't show that script. You got to >> you can't show that script. You got to know that that's now using notes or know that that's now using notes or know that that's now using notes or using the browser and put a little, you using the browser and put a little, you using the browser and put a little, you know, picturein picture display of it know, picturein picture display of it know, picturein picture display of it clicking around and add the pointer and clicking around and add the pointer and clicking around and add the pointer and right. So they're all coding agents right. So they're all coding agents right. So they're all coding agents under the hood. So we added, you know, under the hood. So we added, you know, under the hood. So we added, you know, friendly pointer that clicks around and friendly pointer that clicks around and friendly pointer that clicks around and and things like that. But fundamentally, and things like that. But fundamentally, and things like that. But fundamentally, it's it's very similar in what the model it's it's very similar in what the model it's it's very similar in what the model is good at. is good at. is good at. >> You know, computer use has been one of >> You know, computer use has been one of >> You know, computer use has been one of those vocally self-critical moments for those vocally self-critical moments for those vocally self-critical moments for me. In 2425, me. In 2425, me. In 2425, looking at where computer use was, looking at where computer use was, looking at where computer use was, nobody had it right. like it was janky.

nobody had it right. like it was janky. nobody had it right. like it was janky. Yeah. Yeah. Yeah. >> Any implementation I looked at. >> Any implementation I looked at. >> Any implementation I looked at. >> And my thesis at the time was that if >> And my thesis at the time was that if >> And my thesis at the time was that if you use a computer as a human, you're you use a computer as a human, you're you use a computer as a human, you're going to be good at using it as a human. going to be good at using it as a human. going to be good at using it as a human. And if you use it as a machine, you're And if you use it as a machine, you're And if you use it as a machine, you're going to want a machine readable going to want a machine readable going to want a machine readable interface. You're going to want an MCP. interface. You're going to want an MCP. interface. You're going to want an MCP. You're going to want an API. You're You're going to want an API. You're You're going to want an API. You're going to want something that you can going to want something that you can going to want something that you can interact with. And I underestimated the interact with. And I underestimated the interact with. And I underestimated the scaling law potential of computer use. scaling law potential of computer use. scaling law potential of computer use. And when it got good enough, there's And when it got good enough, there's And when it got good enough, there's just this massive tipping point into just this massive tipping point into just this massive tipping point into latent compute that doesn't require MCP latent compute that doesn't require MCP latent compute that doesn't require MCP and doesn't require API. And now I find and doesn't require API. And now I find and doesn't require API. And now I find myself using all of these use cases that myself using all of these use cases that myself using all of these use cases that are suddenly fast and they're suddenly are suddenly fast and they're suddenly are suddenly fast and they're suddenly fluent. Like I can use it to look fluent. Like I can use it to look fluent. Like I can use it to look through a bunch of annoying like through a bunch of annoying like through a bunch of annoying like paperwork that I never would have had paperwork that I never would have had paperwork that I never would have had access to via MCP or API because like if access to via MCP or API because like if access to via MCP or API because like if it's government paperwork, they're not it's government paperwork, they're not it's government paperwork, they're not building that. Like that's not building that. Like that's not building that. Like that's not happening. happening. happening. >> And now it's just done. Like I can just >> And now it's just done. Like I can just >> And now it's just done. Like I can just take care of it. And I think that's been take care of it. And I think that's been take care of it. And I think that's been >> an aha moment for me, but also for a lot >> an aha moment for me, but also for a lot >> an aha moment for me, but also for a lot of folks who are watching is how good of folks who are watching is how good of folks who are watching is how good that's become as quickly as it has. that's become as quickly as it has. that's become as quickly as it has. >> Yeah. It's really the the the universal >> Yeah. It's really the the the universal >> Yeah. It's really the the the universal connector.

connector. connector. >> Yeah. >> Yeah. >> Yeah. >> Right. There are many cases where MCP is >> Right. There are many cases where MCP is >> Right. There are many cases where MCP is faster, more token efficient. faster, more token efficient. faster, more token efficient. >> Yeah. >> Yeah. >> Yeah. >> But the world in which you get MCP for >> But the world in which you get MCP for >> But the world in which you get MCP for everything that you do is just that's everything that you do is just that's everything that you do is just that's not not not >> it's a non-existent world. Yeah. >> it's a non-existent world. Yeah. >> it's a non-existent world. Yeah. >> So you're right. I mean, computer use >> So you're right. I mean, computer use >> So you're right. I mean, computer use for a while for us was it was an for a while for us was it was an for a while for us was it was an impressive party trick impressive party trick impressive party trick >> where it's it's very visual. It it >> where it's it's very visual. It it >> where it's it's very visual. It it seemed impressive, but to actually work seemed impressive, but to actually work seemed impressive, but to actually work with it with it with it >> is kind of slow, right? And now it's so >> is kind of slow, right? And now it's so >> is kind of slow, right? And now it's so fast that fast that fast that >> we miss sometimes if an agent in the >> we miss sometimes if an agent in the >> we miss sometimes if an agent in the background is using it where it's like, background is using it where it's like, background is using it where it's like, oh, we don't we didn't realize that it oh, we don't we didn't realize that it oh, we don't we didn't realize that it actually switched from MCP to computer actually switched from MCP to computer actually switched from MCP to computer use for that one thing that isn't in the use for that one thing that isn't in the use for that one thing that isn't in the MCP. It just it just happened, right? MCP. It just it just happened, right? MCP. It just it just happened, right? And you look back and you're like, oh, And you look back and you're like, oh, And you look back and you're like, oh, interesting. I was clicking around. I interesting. I was clicking around. I interesting. I was clicking around. I didn't even notice. Right. Um, it's didn't even notice. Right. Um, it's didn't even notice. Right. Um, it's crazy. crazy. crazy. >> We are running into this situation where >> We are running into this situation where >> We are running into this situation where we are asking ourselves, is the computer we are asking ourselves, is the computer we are asking ourselves, is the computer my computer or is the computer really my computer or is the computer really my computer or is the computer really Codex's computer and I just have a Codex's computer and I just have a Codex's computer and I just have a corner and talk to it? Because I find a corner and talk to it? Because I find a corner and talk to it? Because I find a lot of the time I'm using Whisper Flow lot of the time I'm using Whisper Flow lot of the time I'm using Whisper Flow and I'm talking to the agent and then and I'm talking to the agent and then and I'm talking to the agent and then there's this like cursor running around there's this like cursor running around there's this like cursor running around on these windows and it's doing UX user on these windows and it's doing UX user on these windows and it's doing UX user acceptance testing on something I'm acceptance testing on something I'm acceptance testing on something I'm building and I don't even know what's building and I don't even know what's building and I don't even know what's happening in all of my windows anymore.

happening in all of my windows anymore. happening in all of my windows anymore. And so I'm curious when you think about And so I'm curious when you think about And so I'm curious when you think about knowledge work and where it's going, how knowledge work and where it's going, how knowledge work and where it's going, how do you think about compute availability? do you think about compute availability? do you think about compute availability? How do you think about how we will be How do you think about how we will be How do you think about how we will be using computers as humans versus how we using computers as humans versus how we using computers as humans versus how we will be directing agents in the computer will be directing agents in the computer will be directing agents in the computer experience? experience? experience? >> I mean in many ways we we we spent >> I mean in many ways we we we spent >> I mean in many ways we we we spent decades building up this computer UI decades building up this computer UI decades building up this computer UI apps websites like thing, right? That is apps websites like thing, right? That is apps websites like thing, right? That is far more complex than I think anyone far more complex than I think anyone far more complex than I think anyone really wanted. really wanted. really wanted. >> Yes. >> Yes. >> Yes. >> To do everything, right? And you feel >> To do everything, right? And you feel >> To do everything, right? And you feel that in your life, you know, I I have that in your life, you know, I I have that in your life, you know, I I have two kids and filling out the forms on I two kids and filling out the forms on I two kids and filling out the forms on I have three. I feel you, have three. I feel you, have three. I feel you, >> right? You know, [laughter] life has >> right? You know, [laughter] life has >> right? You know, [laughter] life has become so worked work workshaped with become so worked work workshaped with become so worked work workshaped with this stuff and it's it's actually it's this stuff and it's it's actually it's this stuff and it's it's actually it's worse because these tools are often like worse because these tools are often like worse because these tools are often like PDFs, right? Or you're faxing something. PDFs, right? Or you're faxing something. PDFs, right? Or you're faxing something. Pediatricians still use fax. It's Pediatricians still use fax. It's Pediatricians still use fax. It's >> all it's all like that, right? It's far >> all it's all like that, right? It's far >> all it's all like that, right? It's far more complex than we ever wanted it to more complex than we ever wanted it to more complex than we ever wanted it to be. And I there are some ways that I see be. And I there are some ways that I see be. And I there are some ways that I see the codeex moment, the chachi team the codeex moment, the chachi team the codeex moment, the chachi team moment with computer use as some some moment with computer use as some some moment with computer use as some some sort of like let's get back to what sort of like let's get back to what sort of like let's get back to what technology was supposed to help us with, technology was supposed to help us with, technology was supposed to help us with, right? Describe your problem.

right? Describe your problem. right? Describe your problem. >> Um, describe what you're what you need >> Um, describe what you're what you need >> Um, describe what you're what you need help with and it's going to click around help with and it's going to click around help with and it's going to click around that. It's going to find the website. that. It's going to find the website. that. It's going to find the website. It's going to scan the PDF. It's going It's going to scan the PDF. It's going It's going to scan the PDF. It's going to, you know, send the facts. It's going to, you know, send the facts. It's going to, you know, send the facts. It's going to file this away so that on my phone to file this away so that on my phone to file this away so that on my phone later I can be like, "Hey, when was the later I can be like, "Hey, when was the later I can be like, "Hey, when was the tetanus vaccine that we got?" like tetanus vaccine that we got?" like tetanus vaccine that we got?" like >> can you pull up this >> can you pull up this >> can you pull up this >> form and send it to the sports camp, >> form and send it to the sports camp, >> form and send it to the sports camp, right? Like all of this stuff is is I right? Like all of this stuff is is I right? Like all of this stuff is is I think think think >> helping navigate the complexity that's >> helping navigate the complexity that's >> helping navigate the complexity that's just been built over decades and kind of just been built over decades and kind of just been built over decades and kind of never really dealt with, right? never really dealt with, right? never really dealt with, right? >> Never dealt with. In fact, it's on >> Never dealt with. In fact, it's on >> Never dealt with. In fact, it's on average gotten worse every year. I find average gotten worse every year. I find average gotten worse every year. I find >> I don't even think twice about asking >> I don't even think twice about asking >> I don't even think twice about asking codeex to do that anymore. And it feels codeex to do that anymore. And it feels codeex to do that anymore. And it feels like there's that's not just an like there's that's not just an like there's that's not just an intelligence problem, that's also an intelligence problem, that's also an intelligence problem, that's also an integrations and data availability integrations and data availability integrations and data availability problem. And I'm curious how you guys problem. And I'm curious how you guys problem. And I'm curious how you guys thought about walking over that bridge, thought about walking over that bridge, thought about walking over that bridge, how you thought about sort of opening up how you thought about sort of opening up how you thought about sort of opening up the computer beyond computer use with the computer beyond computer use with the computer beyond computer use with like, you know, the email integrations like, you know, the email integrations like, you know, the email integrations you guys have done, the Slack you guys have done, the Slack you guys have done, the Slack integrations, etc. Um, and as you look integrations, etc. Um, and as you look integrations, etc. Um, and as you look ahead, to what extent are you looking at ahead, to what extent are you looking at ahead, to what extent are you looking at the integrations piece as it's mostly the integrations piece as it's mostly the integrations piece as it's mostly done because now computer use is so good done because now computer use is so good done because now computer use is so good versus no, we still want to invest very versus no, we still want to invest very versus no, we still want to invest very heavily here because there's more heavily here because there's more heavily here because there's more capability when we invest in these deep capability when we invest in these deep capability when we invest in these deep integrations. I think for a long time, integrations. I think for a long time, integrations. I think for a long time, you know, chatbt has had connectors, you know, chatbt has had connectors, you know, chatbt has had connectors, plugins, whatever you want to call them plugins, whatever you want to call them plugins, whatever you want to call them for a while. Um, the model was not for a while. Um, the model was not for a while. Um, the model was not always the best at using them. I think always the best at using them. I think always the best at using them. I think in the past 6 months or so, suddenly, in the past 6 months or so, suddenly, in the past 6 months or so, suddenly, much like with computer use, there was much like with computer use, there was much like with computer use, there was like a flipping point. Suddenly, it was like a flipping point. Suddenly, it was like a flipping point. Suddenly, it was like, oh my god, we need to give make like, oh my god, we need to give make like, oh my god, we need to give make sure it has access to anything that sure it has access to anything that sure it has access to anything that someone would want to give it access to

someone would want to give it access to someone would want to give it access to because that just each incremental because that just each incremental because that just each incremental connector or plugin that we add like connector or plugin that we add like connector or plugin that we add like multiplies the power of this product. is multiplies the power of this product. is multiplies the power of this product. is we focus a ton on latency, reliability, we focus a ton on latency, reliability, we focus a ton on latency, reliability, etc. to make sure that like when the etc. to make sure that like when the etc. to make sure that like when the model uses this connector, a it has model uses this connector, a it has model uses this connector, a it has access to as many possible tools as access to as many possible tools as access to as many possible tools as third party service provides. Like we're third party service provides. Like we're third party service provides. Like we're reflecting all of those, b like they reflecting all of those, b like they reflecting all of those, b like they work and they work reliably and c work and they work reliably and c work and they work reliably and c they're fast and token efficient. Um, they're fast and token efficient. Um, they're fast and token efficient. Um, and I think we'll continue doing that, and I think we'll continue doing that, and I think we'll continue doing that, but in the cases where we can make a but in the cases where we can make a but in the cases where we can make a better connector and for for those better connector and for for those better connector and for for those specific products, like why not, right? specific products, like why not, right? specific products, like why not, right? Like we're just giving an additional Like we're just giving an additional Like we're just giving an additional power to the model. And I'm curious when power to the model. And I'm curious when power to the model. And I'm curious when you look at dog fooding internally like you look at dog fooding internally like you look at dog fooding internally like you've been describing, how have you you've been describing, how have you you've been describing, how have you seen uh the shape of work start to seen uh the shape of work start to seen uh the shape of work start to change, what is the thing that you sort change, what is the thing that you sort change, what is the thing that you sort of know internally as a fluency habit or of know internally as a fluency habit or of know internally as a fluency habit or as a learning habit or something that as a learning habit or something that as a learning habit or something that has been really helpful inside this has been really helpful inside this has been really helpful inside this company that you would want people to company that you would want people to company that you would want people to know? But I think one thing that we at know? But I think one thing that we at know? But I think one thing that we at least for me this is the my personal least for me this is the my personal least for me this is the my personal trick is try it like you know you you trick is try it like you know you you trick is try it like you know you you come to work or or you're home wherever come to work or or you're home wherever come to work or or you're home wherever you're like looking at your computer and you're like looking at your computer and you're like looking at your computer and you have something to do you're like oh you have something to do you're like oh you have something to do you're like oh I need to update a presentation or I I need to update a presentation or I I need to update a presentation or I need to send someone an email or I need need to send someone an email or I need need to send someone an email or I need to do whatever and you're like natural to do whatever and you're like natural to do whatever and you're like natural instinct is going to be to go start instinct is going to be to go start instinct is going to be to go start doing that doing that doing that >> and I think something that was very >> and I think something that was very >> and I think something that was very >> important in my journey was like to just >> important in my journey was like to just >> important in my journey was like to just be like no I'm not going to do that I'm be like no I'm not going to do that I'm be like no I'm not going to do that I'm going to try [laughter] it going to try [laughter] it going to try [laughter] it >> in codeex or chatbt Yeah, I think that >> in codeex or chatbt Yeah, I think that >> in codeex or chatbt Yeah, I think that first step of going from like doing first step of going from like doing first step of going from like doing everything on your own to starting with everything on your own to starting with everything on your own to starting with automation I think is is a big gap.

automation I think is is a big gap. automation I think is is a big gap. >> We've set up our environment to be very >> We've set up our environment to be very >> We've set up our environment to be very codeex and chatb native, right? So the codeex and chatb native, right? So the codeex and chatb native, right? So the whole thing kind of works and we use whole thing kind of works and we use whole thing kind of works and we use normal third party tools for plenty of normal third party tools for plenty of normal third party tools for plenty of our work, right? It's not like we've our work, right? It's not like we've our work, right? It's not like we've made everything. We just we've made sure made everything. We just we've made sure made everything. We just we've made sure that they're connected well. that they're connected well. that they're connected well. >> And so, you know, people will be asking >> And so, you know, people will be asking >> And so, you know, people will be asking questions like, hey, what why why do we questions like, hey, what why why do we questions like, hey, what why why do we do a certain thing or like how do I do a certain thing or like how do I do a certain thing or like how do I connect to this thing? And the answer is connect to this thing? And the answer is connect to this thing? And the answer is usually like just talk talk to it like usually like just talk talk to it like usually like just talk talk to it like press the voice button ask. And it's press the voice button ask. And it's press the voice button ask. And it's it's um it's really interesting how you it's um it's really interesting how you it's um it's really interesting how you can kind of get through like cut through can kind of get through like cut through can kind of get through like cut through the noise, Slack, Google documents, like the noise, Slack, Google documents, like the noise, Slack, Google documents, like all of the stuff that people have worked all of the stuff that people have worked all of the stuff that people have worked on and kind of get the full picture. I on and kind of get the full picture. I on and kind of get the full picture. I think it's interesting to watch or you think it's interesting to watch or you think it's interesting to watch or you know everybody here uses codeex or chat know everybody here uses codeex or chat know everybody here uses codeex or chat work now for most of their job and for a work now for most of their job and for a work now for most of their job and for a while that wasn't yet true, right? Um while that wasn't yet true, right? Um while that wasn't yet true, right? Um there was a period between I don't know there was a period between I don't know there was a period between I don't know the first half of this year that like the first half of this year that like the first half of this year that like function by function was making that function by function was making that function by function was making that kind of grand transition to how they kind of grand transition to how they kind of grand transition to how they worked and for each one it was like a worked and for each one it was like a worked and for each one it was like a slightly different tipping point. Um, slightly different tipping point. Um, slightly different tipping point. Um, coding was obviously the first one, coding was obviously the first one, coding was obviously the first one, right? Um, and then who was who was right? Um, and then who was who was right? Um, and then who was who was second?

second? second? >> I heard legal was >> I heard legal was >> I heard legal was >> early because there's so much material, >> early because there's so much material, >> early because there's so much material, right? right? right? >> Um, and I think it's like a combination >> Um, and I think it's like a combination >> Um, and I think it's like a combination of what types of work the models have of what types of work the models have of what types of work the models have been good at at what time, but also then been good at at what time, but also then been good at at what time, but also then what kind of connectors and file access, what kind of connectors and file access, what kind of connectors and file access, like what does your context look like? like what does your context look like? like what does your context look like? The coding one was the easiest because The coding one was the easiest because The coding one was the easiest because everything was just like a local CLI, everything was just like a local CLI, everything was just like a local CLI, >> right? >> right? >> right? >> Yeah. >> Yeah. >> Yeah. >> Like you don't need a Git connector. >> Like you don't need a Git connector. >> Like you don't need a Git connector. You'd need the Git CLI. You need the GH You'd need the Git CLI. You need the GH You'd need the Git CLI. You need the GH CLI, right? CLI, right? CLI, right? >> And so, so we didn't have a great >> And so, so we didn't have a great >> And so, so we didn't have a great connector story in I don't know, connector story in I don't know, connector story in I don't know, January, December, right? But you would January, December, right? But you would January, December, right? But you would code with it. And then it's like legal code with it. And then it's like legal code with it. And then it's like legal legal's got local Microsoft documents legal's got local Microsoft documents legal's got local Microsoft documents all over the place which the model could all over the place which the model could all over the place which the model could read. And then it was it was kind of read. And then it was it was kind of read. And then it was it was kind of during this first half of the year that during this first half of the year that during this first half of the year that like connectors got really good and then like connectors got really good and then like connectors got really good and then computer use got really good. And so computer use got really good. And so computer use got really good. And so depending on the context that was depending on the context that was depending on the context that was required for any given role, it kind of required for any given role, it kind of required for any given role, it kind of like tipped tipped that over the edge like tipped tipped that over the edge like tipped tipped that over the edge when you know this critical mass right when you know this critical mass right when you know this critical mass right >> when the context relevant to that role >> when the context relevant to that role >> when the context relevant to that role became generally available to the became generally available to the became generally available to the >> generally available in performance and >> generally available in performance and >> generally available in performance and yeah so it was it was actually more that yeah so it was it was actually more that yeah so it was it was actually more that than like I guess the normal stuff that than like I guess the normal stuff that than like I guess the normal stuff that you would expect about how technical a you would expect about how technical a you would expect about how technical a role is or something like that.

role is or something like that. role is or something like that. >> That's a really good point. In fact, one >> That's a really good point. In fact, one >> That's a really good point. In fact, one one of the things I've been taking away one of the things I've been taking away one of the things I've been taking away is it feels like the question of whether is it feels like the question of whether is it feels like the question of whether a role is technical is becoming less and a role is technical is becoming less and a role is technical is becoming less and less of a question I find useful. less of a question I find useful. less of a question I find useful. >> Mhm. >> Mhm. >> Mhm. >> Yeah. It's been one of the things that >> Yeah. It's been one of the things that >> Yeah. It's been one of the things that has been super cool to see in the past has been super cool to see in the past has been super cool to see in the past few weeks. like I've been talking to or few weeks. like I've been talking to or few weeks. like I've been talking to or spending some time with people in the spending some time with people in the spending some time with people in the finance team and business teams and um finance team and business teams and um finance team and business teams and um they've been moving away from Excel in they've been moving away from Excel in they've been moving away from Excel in some places to like create these using some places to like create these using some places to like create these using chatb sites like effectively apps but chatb sites like effectively apps but chatb sites like effectively apps but then they're sharing across their teams then they're sharing across their teams then they're sharing across their teams and like iterating on over time and like and like iterating on over time and like and like iterating on over time and like making new versions of and updating for making new versions of and updating for making new versions of and updating for like new use cases and things like that like new use cases and things like that like new use cases and things like that and so for all intents and purposes they and so for all intents and purposes they and so for all intents and purposes they are you know developers in that in that are you know developers in that in that are you know developers in that in that realm and like they don't have a lot of realm and like they don't have a lot of realm and like they don't have a lot of the tools it was like kind kind of the tools it was like kind kind of the tools it was like kind kind of anxietyinducing for me because it's like anxietyinducing for me because it's like anxietyinducing for me because it's like okay they they don't have any of the okay they they don't have any of the okay they they don't have any of the tools that we have in order to make our tools that we have in order to make our tools that we have in order to make our jobs easier like git and version control jobs easier like git and version control jobs easier like git and version control and stuff they're kind of just yoling it and stuff they're kind of just yoling it and stuff they're kind of just yoling it with you know the what we've provided with you know the what we've provided with you know the what we've provided them and so there's so much more to them and so there's so much more to them and so there's so much more to build in order to like make their lives build in order to like make their lives build in order to like make their lives easier but like the fact that they're easier but like the fact that they're easier but like the fact that they're able to do it now itself is I think able to do it now itself is I think able to do it now itself is I think super cool super cool super cool >> I think touching on sites is interesting >> I think touching on sites is interesting >> I think touching on sites is interesting I know there's been a ton of investment I know there's been a ton of investment I know there's been a ton of investment in that space for you guys in the past in that space for you guys in the past in that space for you guys in the past call it month or two do you see a like call it month or two do you see a like call it month or two do you see a like if you look at knowledge work and kind if you look at knowledge work and kind if you look at knowledge work and kind of where knowledge [clears throat] work of where knowledge [clears throat] work of where knowledge [clears throat] work happens. Does sites feel like it's a big happens. Does sites feel like it's a big happens. Does sites feel like it's a big new general purpose lever around new general purpose lever around new general purpose lever around knowledge work?

knowledge work? knowledge work? >> I think there's >> I think there's >> I think there's this idea of like artifacts that are this idea of like artifacts that are this idea of like artifacts that are created. I think historically, you know, created. I think historically, you know, created. I think historically, you know, docs, slides, sheets, docs, slides, sheets, docs, slides, sheets, >> um, PDFs, etc. With AI, like maybe >> um, PDFs, etc. With AI, like maybe >> um, PDFs, etc. With AI, like maybe there's like a new like AI native there's like a new like AI native there's like a new like AI native artifact in the future. Um, I think the artifact in the future. Um, I think the artifact in the future. Um, I think the thing that sites does a really good job thing that sites does a really good job thing that sites does a really good job of is like sometimes the thing that of is like sometimes the thing that of is like sometimes the thing that you're trying to express doesn't fit you're trying to express doesn't fit you're trying to express doesn't fit cleanly in any of these buckets. like cleanly in any of these buckets. like cleanly in any of these buckets. like maybe it needs to be interactive, maybe maybe it needs to be interactive, maybe maybe it needs to be interactive, maybe it needs to be persistent in some way, it needs to be persistent in some way, it needs to be persistent in some way, etc. And maybe you because you're quote etc. And maybe you because you're quote etc. And maybe you because you're quote unquote nontechnical might not even know unquote nontechnical might not even know unquote nontechnical might not even know that like you know the requirements but that like you know the requirements but that like you know the requirements but you don't know how to articulate them in you don't know how to articulate them in you don't know how to articulate them in a way that like an engineer might but a way that like an engineer might but a way that like an engineer might but you can just describe your problem to you can just describe your problem to you can just describe your problem to Andrew's point to the model and it can Andrew's point to the model and it can Andrew's point to the model and it can create this application for you. Um and create this application for you. Um and create this application for you. Um and it fills that gap of like okay you could it fills that gap of like okay you could it fills that gap of like okay you could never do this in a sheet or a doc or never do this in a sheet or a doc or never do this in a sheet or a doc or slide deck um but now you're able to do slide deck um but now you're able to do slide deck um but now you're able to do this with a slide. So, so I do think this with a slide. So, so I do think this with a slide. So, so I do think like for knowledge work it enables like for knowledge work it enables like for knowledge work it enables people to be much more expressive than people to be much more expressive than people to be much more expressive than they otherwise would be. So I think they otherwise would be. So I think they otherwise would be. So I think that's going to be a huge unlock. that's going to be a huge unlock. that's going to be a huge unlock. >> It feels like in most situations most >> It feels like in most situations most >> It feels like in most situations most job roles there's some degree to which job roles there's some degree to which job roles there's some degree to which it's iterative. I would be curious to it's iterative. I would be curious to it's iterative. I would be curious to what extent you have seen differences in what extent you have seen differences in what extent you have seen differences in user behavior when people are say in user behavior when people are say in user behavior when people are say in work versus people in codecs and how work versus people in codecs and how work versus people in codecs and how they are starting to use they are starting to use they are starting to use perhaps a more rhythmic or iterative perhaps a more rhythmic or iterative perhaps a more rhythmic or iterative workflow versus maybe a goal oriented workflow versus maybe a goal oriented workflow versus maybe a goal oriented development workflow where you just have development workflow where you just have development workflow where you just have to get the code to production and it to get the code to production and it to get the code to production and it needs to be shipped.

needs to be shipped. needs to be shipped. >> Definitely more similar than they have >> Definitely more similar than they have >> Definitely more similar than they have been. I think you know over the past been. I think you know over the past been. I think you know over the past number of years software development has number of years software development has number of years software development has become more iter iterative incorporated become more iter iterative incorporated become more iter iterative incorporated that's a fair yeah that's a fair yeah that's a fair yeah >> um so I think like the shape of if you >> um so I think like the shape of if you >> um so I think like the shape of if you were to call it the product development were to call it the product development were to call it the product development life cycle of like an internal tool that life cycle of like an internal tool that life cycle of like an internal tool that is being built as a chat site looks a is being built as a chat site looks a is being built as a chat site looks a lot like your best practice on how to lot like your best practice on how to lot like your best practice on how to give something to customers right so give something to customers right so give something to customers right so everybody's kind of now surfacing everybody's kind of now surfacing everybody's kind of now surfacing something to some sort of customer right something to some sort of customer right something to some sort of customer right you know their HR might make a site to you know their HR might make a site to you know their HR might make a site to help us identify talent and like that we help us identify talent and like that we help us identify talent and like that we are the customer of that product that are the customer of that product that are the customer of that product that they are maintaining. So everyone's sort they are maintaining. So everyone's sort they are maintaining. So everyone's sort of starting to maintain a product of you of starting to maintain a product of you of starting to maintain a product of you know that captures their type of know that captures their type of know that captures their type of expertise for another audience. Um I expertise for another audience. Um I expertise for another audience. Um I think in many cases their cycle is think in many cases their cycle is think in many cases their cycle is faster because it's a you know smaller faster because it's a you know smaller faster because it's a you know smaller internal audience right they don't worry internal audience right they don't worry internal audience right they don't worry about about about >> backwards compatibility of a billion >> backwards compatibility of a billion >> backwards compatibility of a billion chat GBT users and so um sometimes they chat GBT users and so um sometimes they chat GBT users and so um sometimes they teach software developers here a lesson teach software developers here a lesson teach software developers here a lesson on [laughter] on [laughter] on [laughter] >> you can just iterate and >> you can just iterate and >> you can just iterate and >> of course you don't have a billion users >> of course you don't have a billion users >> of course you don't have a billion users but but but >> yeah that's an interesting point as >> yeah that's an interesting point as >> yeah that's an interesting point as well. What is it that we should be well. What is it that we should be well. What is it that we should be betting on in terms of skills? what is betting on in terms of skills? what is betting on in terms of skills? what is it that we should be betting on in terms it that we should be betting on in terms it that we should be betting on in terms of uh trend lines that we can keep an of uh trend lines that we can keep an of uh trend lines that we can keep an eye on and continue to expect to improve eye on and continue to expect to improve eye on and continue to expect to improve that are predictable enough that we can that are predictable enough that we can that are predictable enough that we can say in 6 months X or Y is going to be a say in 6 months X or Y is going to be a say in 6 months X or Y is going to be a lot better. I think that

lot better. I think that lot better. I think that creation of these work products for lack creation of these work products for lack creation of these work products for lack of a better word like sites or the tools of a better word like sites or the tools of a better word like sites or the tools or documents or whatever is going to or documents or whatever is going to or documents or whatever is going to continue to improve the model's continue to improve the model's continue to improve the model's capability to do it and that's on a mult capability to do it and that's on a mult capability to do it and that's on a mult multiple dimensions. So like one is just multiple dimensions. So like one is just multiple dimensions. So like one is just like the quality of the outputs right like the quality of the outputs right like the quality of the outputs right two is like the ability for it to two is like the ability for it to two is like the ability for it to understand your true intent like what understand your true intent like what understand your true intent like what you mean the ability for it to pull you mean the ability for it to pull you mean the ability for it to pull context in to site sources accurately context in to site sources accurately context in to site sources accurately whatever all those things are going to whatever all those things are going to whatever all those things are going to get a lot better. Um, I think what that get a lot better. Um, I think what that get a lot better. Um, I think what that means is then what you're bottlenecked means is then what you're bottlenecked means is then what you're bottlenecked by is like ideas. by is like ideas. by is like ideas. >> The ability to like think about what to >> The ability to like think about what to >> The ability to like think about what to do next to have the agent come up with do next to have the agent come up with do next to have the agent come up with something to to Andrew's point, validate something to to Andrew's point, validate something to to Andrew's point, validate whether or not that's useful with either whether or not that's useful with either whether or not that's useful with either yourself or like the people you work yourself or like the people you work yourself or like the people you work with and then go back to the agent and with and then go back to the agent and with and then go back to the agent and ask for like iterate. So I think the ask for like iterate. So I think the ask for like iterate. So I think the skill of like working with the agent is skill of like working with the agent is skill of like working with the agent is the thing that is going to stay constant the thing that is going to stay constant the thing that is going to stay constant and like can you continue to get better and like can you continue to get better and like can you continue to get better at that and then you can assume that at that and then you can assume that at that and then you can assume that each iteration the quality will keep each iteration the quality will keep each iteration the quality will keep getting better and you know maybe the getting better and you know maybe the getting better and you know maybe the iteration cycle gets shorter and then iteration cycle gets shorter and then iteration cycle gets shorter and then you can do 10 things in parallel where you can do 10 things in parallel where you can do 10 things in parallel where previously you were only able to do five previously you were only able to do five previously you were only able to do five or so on and so forth. I think the the or so on and so forth. I think the the or so on and so forth. I think the the other side of that is that it has become other side of that is that it has become other side of that is that it has become so easy to make things that so easy to make things that so easy to make things that taste on convergence is another taste on convergence is another taste on convergence is another bottleneck that we sometimes have which bottleneck that we sometimes have which bottleneck that we sometimes have which is like there's there's a lot being made is like there's there's a lot being made is like there's there's a lot being made at any given time.

at any given time. at any given time. >> Yeah. >> Yeah. >> Yeah. >> Right. We want to ship as focused as >> Right. We want to ship as focused as >> Right. We want to ship as focused as cohesive and coherent a product as cohesive and coherent a product as cohesive and coherent a product as possible to help people possible to help people possible to help people >> solve their stuff. So, how do we take >> solve their stuff. So, how do we take >> solve their stuff. So, how do we take that thing over there that looks cool that thing over there that looks cool that thing over there that looks cool and those three things and that thing and those three things and that thing and those three things and that thing and that thing and put that together, and that thing and put that together, and that thing and put that together, mold that into something that makes mold that into something that makes mold that into something that makes sense, is coherent to people, is sense, is coherent to people, is sense, is coherent to people, is tasteful, right? Cuz now it's like you tasteful, right? Cuz now it's like you tasteful, right? Cuz now it's like you can ship anything. We could ship every can ship anything. We could ship every can ship anything. We could ship every single product that's ever been single product that's ever been single product that's ever been >> You can you should Well, that's another >> You can you should Well, that's another >> You can you should Well, that's another question. [laughter] question. [laughter] question. [laughter] Um, I think that's that's one is really Um, I think that's that's one is really Um, I think that's that's one is really having clarity of thought of like having clarity of thought of like having clarity of thought of like >> what what do we really need to do here? >> what what do we really need to do here? >> what what do we really need to do here? >> And that like both of you called out >> And that like both of you called out >> And that like both of you called out mental skills like you talked about the mental skills like you talked about the mental skills like you talked about the fluency of working with agents. Um, and fluency of working with agents. Um, and fluency of working with agents. Um, and you talked about decision making and you talked about decision making and you talked about decision making and kind of how you make good decisions with kind of how you make good decisions with kind of how you make good decisions with good taste. Um, good taste. Um, good taste. Um, I think to me one of the things that I think to me one of the things that I think to me one of the things that I've been reflecting on is when I was I've been reflecting on is when I was I've been reflecting on is when I was earlier in my career, there was this earlier in my career, there was this earlier in my career, there was this obviously there was work that was obviously there was work that was obviously there was work that was intellectually rigorous that I had to intellectually rigorous that I had to intellectually rigorous that I had to do, but there was also lots of busy work do, but there was also lots of busy work do, but there was also lots of busy work that would just fill the dates as well.

that would just fill the dates as well. that would just fill the dates as well. And now all of that is gone. And now all of that is gone. And now all of that is gone. >> Like largely the busy work is something >> Like largely the busy work is something >> Like largely the busy work is something I can give to an agent and it just gets I can give to an agent and it just gets I can give to an agent and it just gets done. done. done. But that means that the question then But that means that the question then But that means that the question then bounces back to me, what am I doing with bounces back to me, what am I doing with bounces back to me, what am I doing with my time? How am I maximizing my agency? my time? How am I maximizing my agency? my time? How am I maximizing my agency? Do I have the ideas to actually give AI Do I have the ideas to actually give AI Do I have the ideas to actually give AI what it wants? Am I making decisions what it wants? Am I making decisions what it wants? Am I making decisions that are tasteful or am I pushing slop? that are tasteful or am I pushing slop? that are tasteful or am I pushing slop? And it feels like that's a question that And it feels like that's a question that And it feels like that's a question that we've all been posed. There's a level of we've all been posed. There's a level of we've all been posed. There's a level of slight discomfort on having to, slight discomfort on having to, slight discomfort on having to, >> you know, reinvent yourself over and >> you know, reinvent yourself over and >> you know, reinvent yourself over and over again. Um, at the same time, you over again. Um, at the same time, you over again. Um, at the same time, you know, I know, I know, I I grew up like a software engineer, but I grew up like a software engineer, but I grew up like a software engineer, but also a designer, and there were always also a designer, and there were always also a designer, and there were always things that I saw in my head, wanted to things that I saw in my head, wanted to things that I saw in my head, wanted to make that were beyond make that were beyond make that were beyond where I was on the engineering side. where I was on the engineering side. where I was on the engineering side. >> Mhm. >> Mhm. >> Mhm. >> And I feel somewhat liberated to be able >> And I feel somewhat liberated to be able >> And I feel somewhat liberated to be able to deliver things without the busy work. to deliver things without the busy work. to deliver things without the busy work. Right. Right. Right. >> There's no inherent meaning in that. >> There's no inherent meaning in that. >> There's no inherent meaning in that. It's the meaning is in this the end that It's the meaning is in this the end that It's the meaning is in this the end that the thing that you created. One of the the thing that you created. One of the the thing that you created. One of the things I'm curious about is how you guys things I'm curious about is how you guys things I'm curious about is how you guys think about uh token efficiency. It think about uh token efficiency. It think about uh token efficiency. It feels like we like, you know, maybe feels like we like, you know, maybe feels like we like, you know, maybe April or so token maxing was the thing April or so token maxing was the thing April or so token maxing was the thing and everyone was talking about token and everyone was talking about token and everyone was talking about token maxing and token leaderboards and now maxing and token leaderboards and now maxing and token leaderboards and now the pendulum is swung back. I've been the pendulum is swung back. I've been the pendulum is swung back. I've been having a ton of fun switching the usual having a ton of fun switching the usual having a ton of fun switching the usual assumptions. So, a lot of the usual assumptions. So, a lot of the usual assumptions. So, a lot of the usual assumptions are you pick the smart model assumptions are you pick the smart model assumptions are you pick the smart model as the orchestrator and the dumb model as the orchestrator and the dumb model as the orchestrator and the dumb model is the executor. I've been playing with is the executor. I've been playing with is the executor. I've been playing with switching it and having Luna be the switching it and having Luna be the switching it and having Luna be the orchestrator on extra high and having

orchestrator on extra high and having orchestrator on extra high and having Astra be the executor with sub agents Astra be the executor with sub agents Astra be the executor with sub agents and then I just poke Luna on agency and and then I just poke Luna on agency and and then I just poke Luna on agency and not uh under scoping what I'm building not uh under scoping what I'm building not uh under scoping what I'm building with another Astra model every now and with another Astra model every now and with another Astra model every now and then but it doesn't get the full context then but it doesn't get the full context then but it doesn't get the full context window so it's not chewing up all those window so it's not chewing up all those window so it's not chewing up all those tokens and then the super long running tokens and then the super long running tokens and then the super long running orchestration thread lives in Luna and I orchestration thread lives in Luna and I orchestration thread lives in Luna and I found that that is tremendously token found that that is tremendously token found that that is tremendously token efficient and it still gets me a lot a efficient and it still gets me a lot a efficient and it still gets me a lot a lot of the way to where I want to go. lot of the way to where I want to go. lot of the way to where I want to go. And I'm I'm curious like, have you heard And I'm I'm curious like, have you heard And I'm I'm curious like, have you heard of that? Is that a surprise to you? Do of that? Is that a surprise to you? Do of that? Is that a surprise to you? Do you have other tips or tricks for how you have other tips or tricks for how you have other tips or tricks for how you're token efficient? Um, yeah, you're token efficient? Um, yeah, you're token efficient? Um, yeah, >> I have heard of this. I Yeah, I've heard >> I have heard of this. I Yeah, I've heard >> I have heard of this. I Yeah, I've heard I've heard more and more of it. Um, I've heard more and more of it. Um, I've heard more and more of it. Um, >> as this pressure comes in, >> as this pressure comes in, >> as this pressure comes in, >> not just because >> not just because >> not just because people didn't realize they should be people didn't realize they should be people didn't realize they should be token efficient, but because as the token efficient, but because as the token efficient, but because as the models get larger, it kind of pushes you models get larger, it kind of pushes you models get larger, it kind of pushes you to be more token efficient, right? Um, to be more token efficient, right? Um, to be more token efficient, right? Um, per unit of delivery. Um I mean I think per unit of delivery. Um I mean I think per unit of delivery. Um I mean I think we've seen some real leaps in efficiency we've seen some real leaps in efficiency we've seen some real leaps in efficiency on Astra, right? There are cases where on Astra, right? There are cases where on Astra, right? There are cases where Astra is just an order of magnitude more Astra is just an order of magnitude more Astra is just an order of magnitude more efficient than our previous models and efficient than our previous models and efficient than our previous models and that's phenomenal. Um and the code mode that's phenomenal. Um and the code mode that's phenomenal. Um and the code mode uh stuff for computer use makes it even uh stuff for computer use makes it even uh stuff for computer use makes it even better. Um, I think we're also we're better. Um, I think we're also we're better. Um, I think we're also we're also trying to take some of the things also trying to take some of the things also trying to take some of the things you described in, you know, I've got you described in, you know, I've got you described in, you know, I've got this multi- aent setup where I'm picking this multi- aent setup where I'm picking this multi- aent setup where I'm picking this model for this and this model for this model for this and this model for this model for this and this model for this and this model for this and baking this and this model for this and baking this and this model for this and baking a little bit more of that into how a little bit more of that into how a little bit more of that into how models work by default.

models work by default. models work by default. >> No, I got it. >> No, I got it. >> No, I got it. >> And [laughter] and you know, we want to >> And [laughter] and you know, we want to >> And [laughter] and you know, we want to be able to bring that to everybody, be able to bring that to everybody, be able to bring that to everybody, >> right? >> right? >> right? >> Yeah. Yeah. And then like Andrew >> Yeah. Yeah. And then like Andrew >> Yeah. Yeah. And then like Andrew mentioned like we've been doing a lot of mentioned like we've been doing a lot of mentioned like we've been doing a lot of work on kind of the the details like work on kind of the the details like work on kind of the the details like making sure that you know the skills making sure that you know the skills making sure that you know the skills that we're publishing are the most token that we're publishing are the most token that we're publishing are the most token efficient for the model. Making sure efficient for the model. Making sure efficient for the model. Making sure that the harness and how it's using that the harness and how it's using that the harness and how it's using computer use with with things like code computer use with with things like code computer use with with things like code mode is is is working in a way where mode is is is working in a way where mode is is is working in a way where like we're using as few tokens as like we're using as few tokens as like we're using as few tokens as possible. So it's something that we want possible. So it's something that we want possible. So it's something that we want to continue pushing over time. Like I to continue pushing over time. Like I to continue pushing over time. Like I think the what you were mentioning about think the what you were mentioning about think the what you were mentioning about Astra doing the execution isn't Astra doing the execution isn't Astra doing the execution isn't surprising to me because one thing that surprising to me because one thing that surprising to me because one thing that we've seen is like when we're when we've seen is like when we're when we've seen is like when we're when you're purely measuring like tokens per you're purely measuring like tokens per you're purely measuring like tokens per per you know unit of time it's not the per you know unit of time it's not the per you know unit of time it's not the exact measurement that you should be exact measurement that you should be exact measurement that you should be thinking of even as a costconscious thinking of even as a costconscious thinking of even as a costconscious consumer because really what you want to consumer because really what you want to consumer because really what you want to measure is like how much did you spend measure is like how much did you spend measure is like how much did you spend to do the unit of work that you wanted to do the unit of work that you wanted to do the unit of work that you wanted to accomplish. Exactly. And so like you to accomplish. Exactly. And so like you to accomplish. Exactly. And so like you know while while Astra may you know be know while while Astra may you know be know while while Astra may you know be more expensive on a per token more expensive on a per token more expensive on a per token measurement it's like much more measurement it's like much more measurement it's like much more efficient. And so like what we want to efficient. And so like what we want to efficient. And so like what we want to do is like figure out how people can do is like figure out how people can do is like figure out how people can validate that like the ROI not validate that like the ROI not validate that like the ROI not necessarily just measuring like how many necessarily just measuring like how many necessarily just measuring like how many tokens you're spending and multiplying tokens you're spending and multiplying tokens you're spending and multiplying by price.

by price. by price. >> And I get people who say well my >> And I get people who say well my >> And I get people who say well my employees use it for email uh they use employees use it for email uh they use employees use it for email uh they use it as an answer engine uh and they use it as an answer engine uh and they use it as an answer engine uh and they use it for coding. And then there's just it for coding. And then there's just it for coding. And then there's just sort of this long pause and then you're sort of this long pause and then you're sort of this long pause and then you're you're sort of supposed to kind of fill you're sort of supposed to kind of fill you're sort of supposed to kind of fill in and say and then the next thing is in and say and then the next thing is in and say and then the next thing is how do you think about building an how do you think about building an how do you think about building an outcome focused product in a world where outcome focused product in a world where outcome focused product in a world where knowledge workers may not have the knowledge workers may not have the knowledge workers may not have the ability to cleanly articulate outcomes ability to cleanly articulate outcomes ability to cleanly articulate outcomes at the start of the process but may get at the start of the process but may get at the start of the process but may get there over time. like I I often use it there over time. like I I often use it there over time. like I I often use it to go through data, find to go through data, find to go through data, find groups that are not quite retaining on groups that are not quite retaining on groups that are not quite retaining on the product or you know getting the product or you know getting the product or you know getting onboarded fast enough and just like onboarded fast enough and just like onboarded fast enough and just like giving me clarity around that giving me clarity around that giving me clarity around that >> you know jobs that would take days >> you know jobs that would take days >> you know jobs that would take days beforehand to like you know find the beforehand to like you know find the beforehand to like you know find the data tables to write the SQL queries data tables to write the SQL queries data tables to write the SQL queries >> to build the dashboards and say oh no >> to build the dashboards and say oh no >> to build the dashboards and say oh no that was the wrong query it doesn't tell that was the wrong query it doesn't tell that was the wrong query it doesn't tell me anything so now I got to go back and me anything so now I got to go back and me anything so now I got to go back and do it again and nowadays I can you know do it again and nowadays I can you know do it again and nowadays I can you know get a report that's kind of like hey get a report that's kind of like hey get a report that's kind of like hey like this is an area of opportunity for like this is an area of opportunity for like this is an area of opportunity for us and I share that with my team and us and I share that with my team and us and I share that with my team and >> I think that you know we play a lot with >> I think that you know we play a lot with >> I think that you know we play a lot with how can we push it in different areas how can we push it in different areas how can we push it in different areas and it's not always you know ready here and it's not always you know ready here and it's not always you know ready here and we're doing it a lot for our and we're doing it a lot for our and we're doing it a lot for our research of what can we productize and research of what can we productize and research of what can we productize and so I I realized that you know people so I I realized that you know people so I I realized that you know people don't have the time to play with it the don't have the time to play with it the don't have the time to play with it the way that we do and we've got to bake way that we do and we've got to bake way that we do and we've got to bake some of this in so the other day we some of this in so the other day we some of this in so the other day we released a a data plugin for example released a a data plugin for example released a a data plugin for example that helps with a lot of these these um that helps with a lot of these these um that helps with a lot of these these um data questions and just really data questions and just really data questions and just really streamlining like how reports are made streamlining like how reports are made streamlining like how reports are made and things of that nature. Um but I

and things of that nature. Um but I and things of that nature. Um but I think this is something that we want to think this is something that we want to think this is something that we want to to play with, right? Is you shouldn't to play with, right? Is you shouldn't to play with, right? Is you shouldn't have like if we if if the model knows have like if we if if the model knows have like if we if if the model knows how to answer any question you type into how to answer any question you type into how to answer any question you type into it, then you shouldn't have to ask every it, then you shouldn't have to ask every it, then you shouldn't have to ask every one of those questions. Like we should one of those questions. Like we should one of those questions. Like we should be able to provide you a 9:00 a.m. be able to provide you a 9:00 a.m. be able to provide you a 9:00 a.m. report on, hey, here's the things that report on, hey, here's the things that report on, hey, here's the things that you're not asking, right? Yeah, you're not asking, right? Yeah, you're not asking, right? Yeah, >> your job is retention and you're not >> your job is retention and you're not >> your job is retention and you're not asking about these groups, but they are asking about these groups, but they are asking about these groups, but they are actually a huge blind spot for you, actually a huge blind spot for you, actually a huge blind spot for you, right? Like right? Like right? Like >> they're turning, right? Exactly. Like >> they're turning, right? Exactly. Like >> they're turning, right? Exactly. Like that's the type of thing I think we need that's the type of thing I think we need that's the type of thing I think we need to get right next. to get right next. to get right next. >> Yeah. Yeah. And and proactivity is sort >> Yeah. Yeah. And and proactivity is sort >> Yeah. Yeah. And and proactivity is sort of taking the context problem and of taking the context problem and of taking the context problem and pumping it up on steroids because then pumping it up on steroids because then pumping it up on steroids because then you have to not only be confident of the you have to not only be confident of the you have to not only be confident of the context, but also confident of the context, but also confident of the context, but also confident of the insight and confident of the insight and confident of the insight and confident of the directionality of the insight and directionality of the insight and directionality of the insight and confident of your degree of confidence confident of your degree of confidence confident of your degree of confidence so that you know when you're coming out so that you know when you're coming out so that you know when you're coming out and saying something, you're able to and saying something, you're able to and saying something, you're able to say, "I know this is true because X, Y, say, "I know this is true because X, Y, say, "I know this is true because X, Y, and Z, and therefore I recommend C and Z, and therefore I recommend C and Z, and therefore I recommend C action." action." action." >> Yeah, >> Yeah, >> Yeah, >> it's super interesting. It's an area >> it's super interesting. It's an area >> it's super interesting. It's an area where you want to be careful, where you want to be careful, where you want to be careful, >> especially with efficiency, >> especially with efficiency, >> especially with efficiency, >> right? You can't run this thing and just >> right? You can't run this thing and just >> right? You can't run this thing and just do inference all night, super expensive do inference all night, super expensive do inference all night, super expensive in inference for 12 hours while you're in inference for 12 hours while you're in inference for 12 hours while you're sleeping and eat up somebody's entire, sleeping and eat up somebody's entire, sleeping and eat up somebody's entire, you know, usage bill you know, usage bill you know, usage bill >> for, you know, three lousy insights that >> for, you know, three lousy insights that >> for, you know, three lousy insights that are kind of spammy, right?

are kind of spammy, right? are kind of spammy, right? >> So, this is something that we that we >> So, this is something that we that we >> So, this is something that we that we are thinking about a lot and getting are thinking about a lot and getting are thinking about a lot and getting right. right. right. >> Yeah. The token efficiency piece comes >> Yeah. The token efficiency piece comes >> Yeah. The token efficiency piece comes in there. The model choice I'm sure in there. The model choice I'm sure in there. The model choice I'm sure comes in there. comes in there. comes in there. >> Um >> Um >> Um >> the bar is generally higher for how good >> the bar is generally higher for how good >> the bar is generally higher for how good we have to be when the user hasn't we have to be when the user hasn't we have to be when the user hasn't explicitly asked question. explicitly asked question. explicitly asked question. >> I think of voice as an interface of >> I think of voice as an interface of >> I think of voice as an interface of course. When you guys think about model course. When you guys think about model course. When you guys think about model capability, computer use where these capability, computer use where these capability, computer use where these things are going. things are going. things are going. >> What are the trends in interface design >> What are the trends in interface design >> What are the trends in interface design that you guys are paying attention to as that you guys are paying attention to as that you guys are paying attention to as you think about where say codecs or work you think about where say codecs or work you think about where say codecs or work could be expressed in a year or so that could be expressed in a year or so that could be expressed in a year or so that like we wouldn't even think about now? I like we wouldn't even think about now? I like we wouldn't even think about now? I think I mean you touched on voice so I think I mean you touched on voice so I think I mean you touched on voice so I think the one to to call out like think the one to to call out like think the one to to call out like >> I think the this is one where like you >> I think the this is one where like you >> I think the this is one where like you see early adopters see early adopters see early adopters so far ahead of the curve of everyone so far ahead of the curve of everyone so far ahead of the curve of everyone else like when I walk around the office else like when I walk around the office else like when I walk around the office I see some people with like six I see some people with like six I see some people with like six different mics on their desk and that's different mics on their desk and that's different mics on their desk and that's what they use to communicate with you what they use to communicate with you what they use to communicate with you know codeex or tragedy media work all know codeex or tragedy media work all know codeex or tragedy media work all day but that's clearly not made it um day but that's clearly not made it um day but that's clearly not made it um out to everyone and I think part of that out to everyone and I think part of that out to everyone and I think part of that is you know the product needs to do a is you know the product needs to do a is you know the product needs to do a much better job people discovering that much better job people discovering that much better job people discovering that this capability even exists. Part of it this capability even exists. Part of it this capability even exists. Part of it is like we need to bring people along on is like we need to bring people along on is like we need to bring people along on like changing from you know typing or like changing from you know typing or like changing from you know typing or tapping to like speaking and like how tapping to like speaking and like how tapping to like speaking and like how much more fluid and and frictionless much more fluid and and frictionless much more fluid and and frictionless that is as an interface. Um, so I think that is as an interface. Um, so I think that is as an interface. Um, so I think we have a lot of work to do there in we have a lot of work to do there in we have a lot of work to do there in terms of like making it feel like this terms of like making it feel like this terms of like making it feel like this is the next thing like codeex or chatbt is the next thing like codeex or chatbt is the next thing like codeex or chatbt work is or chbt is with you everywhere work is or chbt is with you everywhere work is or chbt is with you everywhere um because you have access to this voice

um because you have access to this voice um because you have access to this voice interface for people who get it they interface for people who get it they interface for people who get it they truly get it but we haven't done enough truly get it but we haven't done enough truly get it but we haven't done enough to like show people that that's to like show people that that's to like show people that that's possible. possible. possible. >> So even beyond like building a net new >> So even beyond like building a net new >> So even beyond like building a net new interface I think voice is one where interface I think voice is one where interface I think voice is one where like we have it already but like it's like we have it already but like it's like we have it already but like it's about how do we bring it to everyone. about how do we bring it to everyone. about how do we bring it to everyone. >> Yeah. >> Yeah. >> Yeah. >> I think the the interesting other side >> I think the the interesting other side >> I think the the interesting other side of voice is that voice is of voice is that voice is of voice is that voice is by far the most efficient way to get by far the most efficient way to get by far the most efficient way to get context in and to communicate context in and to communicate context in and to communicate information, information, information, >> but reading is much faster than >> but reading is much faster than >> but reading is much faster than listening. listening. listening. >> That's right. >> That's right. >> That's right. >> And so I I think like playing with this >> And so I I think like playing with this >> And so I I think like playing with this combination of voice in and like combination of voice in and like combination of voice in and like generative visuals out is very generative visuals out is very generative visuals out is very interesting to me. And we've done a interesting to me. And we've done a interesting to me. And we've done a little bit with our visualized skill and little bit with our visualized skill and little bit with our visualized skill and our generative UI, but I I don't think our generative UI, but I I don't think our generative UI, but I I don't think we've we've we've >> done anything big here yet. >> done anything big here yet. >> done anything big here yet. >> Yeah. Um, there are many times where I >> Yeah. Um, there are many times where I >> Yeah. Um, there are many times where I want to be speaking into this this want to be speaking into this this want to be speaking into this this product, into this model, providing product, into this model, providing product, into this model, providing context at a speed I can't type, context at a speed I can't type, context at a speed I can't type, >> but not having it speak back to me, >> but not having it speak back to me, >> but not having it speak back to me, having it show me things and, you know, having it show me things and, you know, having it show me things and, you know, ask me questions. I'm a very visual ask me questions. I'm a very visual ask me questions. I'm a very visual person, right? Um, so we're we're person, right? Um, so we're we're person, right? Um, so we're we're looking at a bunch of that, too.

looking at a bunch of that, too. looking at a bunch of that, too. >> Yeah. >> Yeah. >> Yeah. >> Yeah. I I I think that there's more to >> Yeah. I I I think that there's more to >> Yeah. I I I think that there's more to do here, too. This the the size of do here, too. This the the size of do here, too. This the the size of interaction needed is not always the interaction needed is not always the interaction needed is not always the same. Mhm. same. Mhm. same. Mhm. >> And so as we think about things like >> And so as we think about things like >> And so as we think about things like token efficiency, even token efficiency, even token efficiency, even >> sometimes a single question can unblock >> sometimes a single question can unblock >> sometimes a single question can unblock like a ton of token usage and a much like a ton of token usage and a much like a ton of token usage and a much better outcome. Just some clarifying better outcome. Just some clarifying better outcome. Just some clarifying questions. You'll see Astra asking more questions. You'll see Astra asking more questions. You'll see Astra asking more questions than previous models. But as questions than previous models. But as questions than previous models. But as tasks get long, the ability to direct tasks get long, the ability to direct tasks get long, the ability to direct questions to different devices you might questions to different devices you might questions to different devices you might have, have, have, >> you know, like I don't, you know, people >> you know, like I don't, you know, people >> you know, like I don't, you know, people talk about coding at the playground talk about coding at the playground talk about coding at the playground while their kids are at the playground. while their kids are at the playground. while their kids are at the playground. I'm like, well, I don't I don't love I'm like, well, I don't I don't love I'm like, well, I don't I don't love that idea. Like, I want to spend time that idea. Like, I want to spend time that idea. Like, I want to spend time with them. with them. with them. >> You want to be spending time with the >> You want to be spending time with the >> You want to be spending time with the kids. Yeah. kids. Yeah. kids. Yeah. >> But it's also very frustrating to come >> But it's also very frustrating to come >> But it's also very frustrating to come back to my desk and see something done back to my desk and see something done back to my desk and see something done in a way that in a way that in a way that >> isn't wrong, but like one question could >> isn't wrong, but like one question could >> isn't wrong, but like one question could have made this better. have made this better. have made this better. >> Yeah. >> Yeah. >> Yeah. >> Like just send me the yes or no out >> Like just send me the yes or no out >> Like just send me the yes or no out while I'm out. I'll hit, you know, I'll while I'm out. I'll hit, you know, I'll while I'm out. I'll hit, you know, I'll give answer and like unblock you. Right. give answer and like unblock you. Right. give answer and like unblock you. Right. >> The same way that sometimes somebody, >> The same way that sometimes somebody, >> The same way that sometimes somebody, you know, sends you a Slack message to you know, sends you a Slack message to you know, sends you a Slack message to just get unblocked, right?

just get unblocked, right? just get unblocked, right? >> Yeah. >> Yeah. >> Yeah. >> Um, so we'll do more there. Yeah, I >> Um, so we'll do more there. Yeah, I >> Um, so we'll do more there. Yeah, I think and that's another example of like think and that's another example of like think and that's another example of like smart proactivity it seems like where smart proactivity it seems like where smart proactivity it seems like where it's like learning to ask the right it's like learning to ask the right it's like learning to ask the right question. Another workflow pattern I've question. Another workflow pattern I've question. Another workflow pattern I've noticed in myself is I am increasingly noticed in myself is I am increasingly noticed in myself is I am increasingly reading the intermediate messages that reading the intermediate messages that reading the intermediate messages that are printed and I am aggressively are printed and I am aggressively are printed and I am aggressively shaping the run of the agent as I see shaping the run of the agent as I see shaping the run of the agent as I see those me messages shape out. Sometimes those me messages shape out. Sometimes those me messages shape out. Sometimes it's like I'm directly responding and it's like I'm directly responding and it's like I'm directly responding and saying no no no no that's wrong do this saying no no no no that's wrong do this saying no no no no that's wrong do this over here differently. And sometimes over here differently. And sometimes over here differently. And sometimes it's actually taking a print out and in it's actually taking a print out and in it's actually taking a print out and in the middle of the run handing it to the middle of the run handing it to the middle of the run handing it to another agent saying, "This doesn't make another agent saying, "This doesn't make another agent saying, "This doesn't make a lot of sense to me. Can you give me a lot of sense to me. Can you give me a lot of sense to me. Can you give me fresh eyes on this?" And then fresh eyes on this?" And then fresh eyes on this?" And then >> the runs are long enough I can cycle >> the runs are long enough I can cycle >> the runs are long enough I can cycle that back around and say, "Okay, this is that back around and say, "Okay, this is that back around and say, "Okay, this is how I think we should proceed." how I think we should proceed." how I think we should proceed." >> Do you guys see sort of mid agent work >> Do you guys see sort of mid agent work >> Do you guys see sort of mid agent work activity as something that's an emerging activity as something that's an emerging activity as something that's an emerging category across a wider user base? Is category across a wider user base? Is category across a wider user base? Is that just me being weird as like a a that just me being weird as like a a that just me being weird as like a a power user or what's been your power user or what's been your power user or what's been your experience? experience? experience? >> I do think that as the scope of what the >> I do think that as the scope of what the >> I do think that as the scope of what the agent can take on gets larger, so like agent can take on gets larger, so like agent can take on gets larger, so like both like the value of it and then also both like the value of it and then also both like the value of it and then also the time it takes to do it, the time it takes to do it, the time it takes to do it, >> like we really need to nail the >> like we really need to nail the >> like we really need to nail the experience of like what does it feel experience of like what does it feel experience of like what does it feel like to interrupt or interject in the like to interrupt or interject in the like to interrupt or interject in the middle of a roll out. Not only for the middle of a roll out. Not only for the middle of a roll out. Not only for the things that you're describing where like things that you're describing where like things that you're describing where like maybe the agent is doing the wrong maybe the agent is doing the wrong maybe the agent is doing the wrong thing, but maybe the user is wrong. Like thing, but maybe the user is wrong. Like thing, but maybe the user is wrong. Like I've had many situations where like I I've had many situations where like I I've had many situations where like I ask it for something and then 20 minutes ask it for something and then 20 minutes ask it for something and then 20 minutes later I'm like wait a second that's not later I'm like wait a second that's not later I'm like wait a second that's not >> that was wrong.

>> that was wrong. >> that was wrong. >> Yeah that's [laughter] not at all the >> Yeah that's [laughter] not at all the >> Yeah that's [laughter] not at all the thing that I want. I need to go back to thing that I want. I need to go back to thing that I want. I need to go back to it and make that and that needs to feel it and make that and that needs to feel it and make that and that needs to feel like really fluid. So I think that's like really fluid. So I think that's like really fluid. So I think that's somewhere we need to invest in more but somewhere we need to invest in more but somewhere we need to invest in more but definitely something that we're seeing definitely something that we're seeing definitely something that we're seeing where like people are like starting to where like people are like starting to where like people are like starting to learn that this is birectional the learn that this is birectional the learn that this is birectional the entire time not just like entire time not just like entire time not just like >> question response question response. Um >> question response question response. Um >> question response question response. Um I think there's probably much more we I think there's probably much more we I think there's probably much more we can do in the interface to make that can do in the interface to make that can do in the interface to make that clear. clear. clear. >> Yeah. No, that makes a lot of sense. >> Yeah. No, that makes a lot of sense. >> Yeah. No, that makes a lot of sense. Another pattern that I've observed in Another pattern that I've observed in Another pattern that I've observed in myself, we talked about iterative work myself, we talked about iterative work myself, we talked about iterative work earlier. One of the ways that I have earlier. One of the ways that I have earlier. One of the ways that I have found and that others I've talked to found and that others I've talked to found and that others I've talked to have found that they actually work with have found that they actually work with have found that they actually work with AI is they use AI almost as like that uh AI is they use AI almost as like that uh AI is they use AI almost as like that uh bouncing ball idea generation machine bouncing ball idea generation machine bouncing ball idea generation machine and you're kind of bouncing. You're and you're kind of bouncing. You're and you're kind of bouncing. You're like, "No, that's not it, but I wouldn't like, "No, that's not it, but I wouldn't like, "No, that's not it, but I wouldn't have thought of that. No, that's not it, have thought of that. No, that's not it, have thought of that. No, that's not it, but I Oh, that's it." And like you have but I Oh, that's it." And like you have but I Oh, that's it." And like you have to go through a series of nos and the to go through a series of nos and the to go through a series of nos and the nos help clarify your intent. Um, I nos help clarify your intent. Um, I nos help clarify your intent. Um, I don't I don't know there's an interface don't I don't know there's an interface don't I don't know there's an interface for that yet, but it feels like that's a for that yet, but it feels like that's a for that yet, but it feels like that's a very different mode for me than let's very different mode for me than let's very different mode for me than let's get work done because I have clear get work done because I have clear get work done because I have clear intent and I need both in certain intent and I need both in certain intent and I need both in certain situations. Like in one case it's do I situations. Like in one case it's do I situations. Like in one case it's do I understand what I want and in the other understand what I want and in the other understand what I want and in the other case it is okay now let's get it done.

case it is okay now let's get it done. case it is okay now let's get it done. There was a very similar conversation I There was a very similar conversation I There was a very similar conversation I was having I think it was yesterday was having I think it was yesterday was having I think it was yesterday about the idea of like randomness or about the idea of like randomness or about the idea of like randomness or noise. noise. noise. >> Mhm. that like this is a parameter that >> Mhm. that like this is a parameter that >> Mhm. that like this is a parameter that you can if you're you know using every you can if you're you know using every you can if you're you know using every knob on like the API you can set knob on like the API you can set knob on like the API you can set parameters like this but if if you zoom parameters like this but if if you zoom parameters like this but if if you zoom out and you're like hey I've got this out and you're like hey I've got this out and you're like hey I've got this plan that I want to be achieved whether plan that I want to be achieved whether plan that I want to be achieved whether whether that's with a model or even with whether that's with a model or even with whether that's with a model or even with a company of people right there's a a company of people right there's a a company of people right there's a certain amount of like noise and jitter certain amount of like noise and jitter certain amount of like noise and jitter and experimentation that you accept that and experimentation that you accept that and experimentation that you accept that is somewhere between zero and like chaos is somewhere between zero and like chaos is somewhere between zero and like chaos >> because you know that It actually helps >> because you know that It actually helps >> because you know that It actually helps maximize. maximize. maximize. >> Yeah. >> Yeah. >> Yeah. >> Like global maximize, right? Like you're >> Like global maximize, right? Like you're >> Like global maximize, right? Like you're not going to get stuck in like a local not going to get stuck in like a local not going to get stuck in like a local maximum situation because you've got a maximum situation because you've got a maximum situation because you've got a little bit of play where it's like, oh little bit of play where it's like, oh little bit of play where it's like, oh yeah, these people over here are trying yeah, these people over here are trying yeah, these people over here are trying something new that maybe might be a something new that maybe might be a something new that maybe might be a waste. And I I feel I feel very similar waste. And I I feel I feel very similar waste. And I I feel I feel very similar about these models where I'm like about these models where I'm like about these models where I'm like >> I want you to follow the instructions, >> I want you to follow the instructions, >> I want you to follow the instructions, but I want you to have like a little bit but I want you to have like a little bit but I want you to have like a little bit of a little bit of playroom so that when of a little bit of playroom so that when of a little bit of playroom so that when we're talking about this stuff, like we're talking about this stuff, like we're talking about this stuff, like yes, it might take a little bit longer, yes, it might take a little bit longer, yes, it might take a little bit longer, might spend a little bit more, but might spend a little bit more, but might spend a little bit more, but you're kind of outside of the narrow you're kind of outside of the narrow you're kind of outside of the narrow band. I don't know know exactly what band. I don't know know exactly what band. I don't know know exactly what that looks like, but I'm with you on that looks like, but I'm with you on that looks like, but I'm with you on this.

this. this. >> Yeah, >> Yeah, >> Yeah, >> that resonates for me. >> that resonates for me. >> that resonates for me. >> I would be remiss if I did not ask cuz I >> I would be remiss if I did not ask cuz I >> I would be remiss if I did not ask cuz I get asked it a ton by leadership. if get asked it a ton by leadership. if get asked it a ton by leadership. if they're rolling out AI in an they're rolling out AI in an they're rolling out AI in an organization almost always the initial organization almost always the initial organization almost always the initial few months is it's very much like okay few months is it's very much like okay few months is it's very much like okay we've got some people who are adopting we've got some people who are adopting we've got some people who are adopting this and they're super passionate we've this and they're super passionate we've this and they're super passionate we've got group B and they're adopting us kind got group B and they're adopting us kind got group B and they're adopting us kind of figuring it out and group C is like I of figuring it out and group C is like I of figuring it out and group C is like I don't know what is this and when you don't know what is this and when you don't know what is this and when you talk about group A which is the talk about group A which is the talk about group A which is the passionate adopters almost always the passionate adopters almost always the passionate adopters almost always the story is one of individual productivity story is one of individual productivity story is one of individual productivity and the next question I get asked is and the next question I get asked is and the next question I get asked is what about teams and so my question to what about teams and so my question to what about teams and so my question to you is when you think about how AI has you is when you think about how AI has you is when you think about how AI has kind of tipped over these departments kind of tipped over these departments kind of tipped over these departments inside OpenAI over the last 6 months. inside OpenAI over the last 6 months. inside OpenAI over the last 6 months. How has that translated into team level How has that translated into team level How has that translated into team level productivity or team level uh ability to productivity or team level uh ability to productivity or team level uh ability to get more done? I think in all cases it's get more done? I think in all cases it's get more done? I think in all cases it's identifying a process or a problem that identifying a process or a problem that identifying a process or a problem that your team faces maybe on some repeated your team faces maybe on some repeated your team faces maybe on some repeated basis and then that individual the power basis and then that individual the power basis and then that individual the power user the early adopter whatever them user the early adopter whatever them user the early adopter whatever them going after and figuring out how can we going after and figuring out how can we going after and figuring out how can we make this better with AI. I think like make this better with AI. I think like make this better with AI. I think like the biggest hero cases we've seen is the biggest hero cases we've seen is the biggest hero cases we've seen is like someone's like, "Okay, we have to like someone's like, "Okay, we have to like someone's like, "Okay, we have to update this financial model every two update this financial model every two update this financial model every two weeks. I'm going to make that into a weeks. I'm going to make that into a weeks. I'm going to make that into a chatbd site. Now other people can do chatbd site. Now other people can do chatbd site. Now other people can do that too. Now it's more expressive, that too. Now it's more expressive, that too. Now it's more expressive, etc." Or I'm going to create an etc." Or I'm going to create an etc." Or I'm going to create an automation that put publishes all of our automation that put publishes all of our automation that put publishes all of our bugs for our non-retentive cohorts into bugs for our non-retentive cohorts into bugs for our non-retentive cohorts into Slack and now everyone on my team can Slack and now everyone on my team can Slack and now everyone on my team can start pulling from those things. And so start pulling from those things. And so start pulling from those things. And so it's like getting beyond like you it's like getting beyond like you it's like getting beyond like you creating an output for yourself and creating an output for yourself and creating an output for yourself and creating something that your team can creating something that your team can creating something that your team can see and get value from because I think see and get value from because I think see and get value from because I think for like you know leaders in

for like you know leaders in for like you know leaders in organizations who are like excited about organizations who are like excited about organizations who are like excited about deploying AI. I think when the early deploying AI. I think when the early deploying AI. I think when the early adopters are encouraged to do that then adopters are encouraged to do that then adopters are encouraged to do that then people also learn cuz they learn by people also learn cuz they learn by people also learn cuz they learn by seeing like oh that's something that I seeing like oh that's something that I seeing like oh that's something that I could have done and they also get value could have done and they also get value could have done and they also get value immediately versus like trying to run immediately versus like trying to run immediately versus like trying to run you know a training or something where you know a training or something where you know a training or something where it's like much more abstract. it's like much more abstract. it's like much more abstract. >> Yeah. Yeah, that makes a ton of sense. >> Yeah. Yeah, that makes a ton of sense. >> Yeah. Yeah, that makes a ton of sense. Okay, last question. I'm really curious. Okay, last question. I'm really curious. Okay, last question. I'm really curious. When you guys are using AI daytoday, When you guys are using AI daytoday, When you guys are using AI daytoday, what is I most people I know have sort what is I most people I know have sort what is I most people I know have sort of an NP hard problem, a problem where of an NP hard problem, a problem where of an NP hard problem, a problem where they just keep like running AI at it and they just keep like running AI at it and they just keep like running AI at it and eventually it cracks. Eventually the eventually it cracks. Eventually the eventually it cracks. Eventually the scaling laws come through and there's scaling laws come through and there's scaling laws come through and there's something that works. Um, for me, I had something that works. Um, for me, I had something that works. Um, for me, I had one of those moments when a I could one of those moments when a I could one of those moments when a I could finally read my six calendars and finally read my six calendars and finally read my six calendars and actually tell me what was available in actually tell me what was available in actually tell me what was available in my day and not just read two of them and my day and not just read two of them and my day and not just read two of them and give me false windows of failability. give me false windows of failability. give me false windows of failability. Uh, what is that for you guys? What is Uh, what is that for you guys? What is Uh, what is that for you guys? What is that moment that was really difficult? that moment that was really difficult? that moment that was really difficult? And has AI crossed it yet? Have you had And has AI crossed it yet? Have you had And has AI crossed it yet? Have you had that moment where it's like, oh yeah, that moment where it's like, oh yeah, that moment where it's like, oh yeah, okay, this was really hard. I never okay, this was really hard. I never okay, this was really hard. I never thought I would get there. I thought it thought I would get there. I thought it thought I would get there. I thought it would be a while and now it hit it and would be a while and now it hit it and would be a while and now it hit it and I'm surprised. There's like a pet one I'm surprised. There's like a pet one I'm surprised. There's like a pet one that I do every time there's a new model that I do every time there's a new model that I do every time there's a new model and it's it's hard because it's it's not and it's it's hard because it's it's not and it's it's hard because it's it's not just technically challenging. It's a just technically challenging. It's a just technically challenging. It's a it's a coordination problem which is it's a coordination problem which is it's a coordination problem which is >> you know we we have this crossplatform >> you know we we have this crossplatform >> you know we we have this crossplatform app app app >> and I have this dream of someday like >> and I have this dream of someday like >> and I have this dream of someday like someday with any given powerful model someday with any given powerful model someday with any given powerful model you should be able to maintain the like you should be able to maintain the like you should be able to maintain the like absolute platform native on all platform

absolute platform native on all platform absolute platform native on all platform right you know you're doing like old right you know you're doing like old right you know you're doing like old style objective C UI kit on or style objective C UI kit on or style objective C UI kit on or [laughter] app kit on Mac right you're [laughter] app kit on Mac right you're [laughter] app kit on Mac right you're like yep like we're win UI3 on Windows like yep like we're win UI3 on Windows like yep like we're win UI3 on Windows 11 right and this is a coordination 11 right and this is a coordination 11 right and this is a coordination problem because we're moving so fast problem because we're moving so fast problem because we're moving so fast that it just it has been impossible to that it just it has been impossible to that it just it has been impossible to move all the features in lock step. move all the features in lock step. move all the features in lock step. >> Like if if we were moving a tenth of the >> Like if if we were moving a tenth of the >> Like if if we were moving a tenth of the speed, this wouldn't be as much of an speed, this wouldn't be as much of an speed, this wouldn't be as much of an issue. But we use crossplatform because issue. But we use crossplatform because issue. But we use crossplatform because of the speed, right? And someday like of the speed, right? And someday like of the speed, right? And someday like we'll get back to it. We'll get back to we'll get back to it. We'll get back to we'll get back to it. We'll get back to it. We'll be able to fully rewrite all it. We'll be able to fully rewrite all it. We'll be able to fully rewrite all of these apps at exactly the perfect of these apps at exactly the perfect of these apps at exactly the perfect thing for them and keep them all in lock thing for them and keep them all in lock thing for them and keep them all in lock step and make a Rust version. I don't step and make a Rust version. I don't step and make a Rust version. I don't know. Um [laughter] we're not there yet, know. Um [laughter] we're not there yet, know. Um [laughter] we're not there yet, but um we're getting a we're getting a but um we're getting a we're getting a but um we're getting a we're getting a lot closer. I can tell you that. I mean, lot closer. I can tell you that. I mean, lot closer. I can tell you that. I mean, I I have the Codex app rewritten like I I have the Codex app rewritten like I I have the Codex app rewritten like every time we have a new model, I'm every time we have a new model, I'm every time we have a new model, I'm like, like, like, >> "Hey, try it try it in GPUi with Rust, >> "Hey, try it try it in GPUi with Rust, >> "Hey, try it try it in GPUi with Rust, you know, like you know, like you know, like >> and just see what happens." >> and just see what happens." >> and just see what happens." >> And just see what happens. And uh >> And just see what happens. And uh >> And just see what happens. And uh >> Astra got Astra got a pretty good >> Astra got Astra got a pretty good >> Astra got Astra got a pretty good version out on like several at a time. version out on like several at a time. version out on like several at a time. So, So, So, >> okay, >> okay, >> okay, >> we'll see. >> we'll see. >> we'll see. >> Yeah, fingers crossed. Okay, >> Yeah, fingers crossed. Okay, >> Yeah, fingers crossed. Okay, >> yeah, I mean, I guess taking the other >> yeah, I mean, I guess taking the other >> yeah, I mean, I guess taking the other side of this is something that just got side of this is something that just got side of this is something that just got crossed like for a long time. So, I was crossed like for a long time. So, I was crossed like for a long time. So, I was a very early adopter of like connecting a very early adopter of like connecting a very early adopter of like connecting AI to like everything in my life either AI to like everything in my life either AI to like everything in my life either via the browser or connectors. And I via the browser or connectors. And I via the browser or connectors. And I have an automation that's like um you have an automation that's like um you have an automation that's like um you know, how can I save money in my life?

know, how can I save money in my life? know, how can I save money in my life? I'd say like probably 2 months and it I'd say like probably 2 months and it I'd say like probably 2 months and it never found anything useful. Um and it's never found anything useful. Um and it's never found anything useful. Um and it's actually quite complicated problem cuz actually quite complicated problem cuz actually quite complicated problem cuz it goes through my email. It is logged it goes through my email. It is logged it goes through my email. It is logged into a bunch of stuff via the browser. into a bunch of stuff via the browser. into a bunch of stuff via the browser. It has some connectors, etc. It has some connectors, etc. It has some connectors, etc. >> Um you know, it's like maybe it asks me >> Um you know, it's like maybe it asks me >> Um you know, it's like maybe it asks me to cancel subscription or whatever, but to cancel subscription or whatever, but to cancel subscription or whatever, but that's it. And then actually like last that's it. And then actually like last that's it. And then actually like last week it found a pretty significant error week it found a pretty significant error week it found a pretty significant error in my taxes in my taxes in my taxes >> in your favor. >> in your favor. >> in your favor. >> In my favor. >> In my favor. >> In my favor. >> Oo. >> Oo. >> Oo. >> That ended up saving you know thousands >> That ended up saving you know thousands >> That ended up saving you know thousands of dollars and I was like this is of dollars and I was like this is of dollars and I was like this is something that I would have never have something that I would have never have something that I would have never have figured out. Like this just would never figured out. Like this just would never figured out. Like this just would never have happened because the amount of have happened because the amount of have happened because the amount of coordination required to figure this coordination required to figure this coordination required to figure this out. out. out. >> It was huge. Like I had to remember >> It was huge. Like I had to remember >> It was huge. Like I had to remember something that happened multiple years something that happened multiple years something that happened multiple years ago and pulled that forward and all of ago and pulled that forward and all of ago and pulled that forward and all of that. And so for me that was like pretty that. And so for me that was like pretty that. And so for me that was like pretty eye opening again of just like eye opening again of just like eye opening again of just like >> the more context you give it the models >> the more context you give it the models >> the more context you give it the models have gotten to a point where like have gotten to a point where like have gotten to a point where like >> they very much exceed what I'd be able >> they very much exceed what I'd be able >> they very much exceed what I'd be able to do on my own and so we I need to to do on my own and so we I need to to do on my own and so we I need to continue pushing in my personal life in continue pushing in my personal life in continue pushing in my personal life in that direction. that direction. that direction. >> Yeah. Yeah. One of the ones that Astra >> Yeah. Yeah. One of the ones that Astra >> Yeah. Yeah. One of the ones that Astra crossed for me that made me chuckle. Uh crossed for me that made me chuckle. Uh crossed for me that made me chuckle. Uh so a few months ago I was in Seattle.

so a few months ago I was in Seattle. so a few months ago I was in Seattle. Tibo had flown up. I had a brief chat Tibo had flown up. I had a brief chat Tibo had flown up. I had a brief chat with him and at the time I said, "Tibo, with him and at the time I said, "Tibo, with him and at the time I said, "Tibo, make me a model that can make Legos cuz make me a model that can make Legos cuz make me a model that can make Legos cuz I love Legos. I want a model and and I love Legos. I want a model and and I love Legos. I want a model and and like it's saturating out at like 50 like it's saturating out at like 50 like it's saturating out at like 50 pieces, 10 pieces at the time." pieces, 10 pieces at the time." pieces, 10 pieces at the time." >> And now, yeah, I can hit a thousand >> And now, yeah, I can hit a thousand >> And now, yeah, I can hit a thousand piece Lego set. It's not a problem. piece Lego set. It's not a problem. piece Lego set. It's not a problem. >> It's It's wild to cross the physical. >> It's It's wild to cross the physical. >> It's It's wild to cross the physical. Yeah. Yeah. Yeah. >> Right. I remember a few models ago when >> Right. I remember a few models ago when >> Right. I remember a few models ago when I did like the first um book for my I did like the first um book for my I did like the first um book for my kids. kids. kids. >> Yeah. >> Yeah. >> Yeah. >> And I was stringing them together. I was >> And I was stringing them together. I was >> And I was stringing them together. I was like, "All right, you're going to write like, "All right, you're going to write like, "All right, you're going to write a script and then you're going to take a script and then you're going to take a script and then you're going to take the script and you're going to write an the script and you're going to write an the script and you're going to write an image genen prompt for every page and image genen prompt for every page and image genen prompt for every page and you're going to pass that into the long you're going to pass that into the long you're going to pass that into the long ago. ago. ago. >> It was not." And now you can kind of >> It was not." And now you can kind of >> It was not." And now you can kind of just say like, "Hey, make my kids a book just say like, "Hey, make my kids a book just say like, "Hey, make my kids a book about themselves, like a kids book. Go about themselves, like a kids book. Go about themselves, like a kids book. Go get it printed and it will." And it's get it printed and it will." And it's get it printed and it will." And it's just like once things enter real life, just like once things enter real life, just like once things enter real life, you're holding them. you're holding them. you're holding them. >> It feels different. >> It feels different. >> It feels different. >> It feels different. It does. It does. >> It feels different. It does. It does. >> It feels different. It does. It does. Well, thank you for the time today, Well, thank you for the time today, Well, thank you for the time today, guys. I really appreciate it. It's been guys. I really appreciate it. It's been guys. I really appreciate it. It's been a lot of fun. a lot of fun. a lot of fun. >> Yeah. Thanks. A lot of fun. Uh, as a