Build Your Own Finance Agent Inside ChatGPT - Stefano Fabbri & Oliver Steffmann, OpenAI

[music] So hello everyone. Uh glad that you know there's quite a crowd here even if it's holiday. It's Monday morning. It's 9:00 a.m. So I'm Stephan Fabri. I'm part of the open AAI the applied AI engineering team alongside Oliver and also we have in a crowd another part of uh our team Dom. So feel free during the session you know if you don't want to ask ask question we also have a third man that you can leverage. So today's session is going to be a two-hour workshop so it's going to be interactive you are going to use chib work and codeex more into the detail later and we really want to kind of show you the capabilities um of like our product within the financial services space. How can you merge, you know, financial data, you know, LLM, AI, intelligence, and the right output to put everything together to help like your day-to-day business. So, today we're going to focus more on a wealth management case given like you know, we know in the crowd there are a lot of people coming from asset manager, investment banks. So, we thought that um this could be like an interesting case.

So to give you a little bit of a background like uh our team focus mostly on interacting with clients. So we spend all our days working with like some of you guys helping really to understand you know how can I transform like put this model into production at my company and extract the maximum possible value. So for each dollar that you invest with us, our task is really to help you maximize the value extract from it. So no matter if it's operation, if it's an agent, if it's like a code coding productivity, that's more or less our day-to-day and our team covers mostly like the financial institution here in New York. So that's about me, Oliver. Yeah, I'm working in the same team also long history in financial services, asset managers and also investor in banking but I'm a builder at heart. So today we want to talk about also how to build with the platform um and show you the use case here but yeah happy to help um I work with big customers um in the room probably a couple of people I have seen already. So if you approach us want to solve a problem with the API or with codeex yeah we are the right people.

So before continuing I can I ask you to scan this QR code and put your uh information is mostly the email address. The reason is because we want to give you access to a specific works uh workspace um within OpenAI. So you're going to get access to the financial data and you're going to have unlimited token for the session. Plus we also enable the ultraast mode. So is a specific like mode for like our latest model. So 6.1 soul and six Astra that reaches like a 300 token per second on average. So I feel that like for a workshop makes it easier to like you know create something in a faster way. So I'm going to show the QR code later but please uh it's really important otherwise you're not going to get access. >> So just maybe our colleague in the room also if you need it now you come later maybe just people sitting there. So we need that to get access to the sh GBT work website basically. So that's what we need.

So um just kind of a preview of the session today. This is kind of like a tentative schedule. So probably we're going to like go over and mix and match a little bit. But we're going to give you a quick introduction uh on like chibi for work uh work for financial services and like kind of like more the ecosystem. Then we're going to spend 10 minutes focusing a little bit more on the setup making sure that all of you guys can access um judg work for financial services and then we're going to have kind of like two main session that I'm going to be led by Oliver like the first one where we're going to focus more on work and then we're going to move to codeex and finally um we kept like 20 minutes mostly like for question and kind of like general feedback and like about uh you know what's going on in the space and you know whatever question that you might have like both on the coding and also like more on the industry wise. So let's now go to a quick introduction on our services. So as you probably know we have three main interfaces three main surfaces that we use. Chajibbt that is like you know the original one we have chibbity work that is basically something that it's more focused for knowledge workers so it's like you know it can run agent it can work it works with plug-in but it's not

really like codeex that is more meant for like engineering type of task. So these are the three surfaces that right now we are offering and as already mentioned we're going to focus on chargibility work first and then codeex. So what we did is basically within the charge work and charge financial services we tried to create a product that is enterprise focused that basically it's fine-tuned like the model is the same but the product itself is fine-tuned for the specific like financial services type of user. So there are like three main component to this product. So first of all the data. So it's we have some like built-in data that I think it's they're high quite high quality but also we you can access to other third party data vendor directly within uh chachbt so the agent can interact directly with the data sources without any um overhead. Then second we really like pushed [clears throat] our model and trained them on specific financial um type of work. So like you know we can think about like you know M&A equity research and so on so forth. And third component is the are the artifacts. So basically we really pushed in the in the recent months the capability of our model to

actually build presentation build spreadsheet and also being able to leverage like reference template that from like you know historical example you know and reproduce them like with the for the current analysis. So yeah maybe just to add to it. So in the in the labs what we do now the first part will really focus on this plug-in share GBT for financial services. So it sits in the web surface the website basically and then we show all the feature because how to use a financial data is one thing that's like a one benefit here but also there's certain ways or actually in instructs the prompt to present it the information in a chart or use also some verification and also have some annotations where the data comes from. So all that if you hang out for the lab will be very important to see and that's sort of the domain specific solution we to showcase. So let's start a little bit like with um the building data and that's why we want to on board you to this specific version because charge for financial services is only available like on the enterprise side. So these are some of the vendor that are built in within the product. So we like had partnership with like the Lupa, Elsag, Pitchbook, Crunchbase and other vendor. And basically to give you an idea like of

like what's available out of the box, we have like SEC feelings, we have market data. Market data of course are delayed. So uh I think there is a 15 minutes delayed but still you have like real-time market data. you have news and like we also have like you know private company profiles and earning calls transcript and the main difference here is that these data sources are like really piped into the product. So it's not just like the row data that is entering but like we did like a specific fine-tuning of of the databases to make sure that like this these data are easy to to be consumed by the model. So what we've seen is like like a nice improvement on like how the model is actually able to ingest this data because we kind of like wrap it in in a format that is like not not just like you know the row data set but it's more um fitted for the LLM.

>> Yeah. So maybe just one thing to add. I mean I have done something similar myself actually inside the company and it's quite hard if you pull down all the SEC files and you have it scale the data you have to make sure you grab the new information I mean I have done it in a in an asset management case but I think what's really here unique is basically you you have those agents running collecting this data we have the financial data pro providers that work with us but also you have to way have a context layer behind it that actually is efficient enough to serve it and that's often a technical challenge. So that's what we can show you also later. That's a sort of what the product has inside. Now it's there's some technical solution for providing that at speed and then like moving a little bit more on like you know how to leverage this data to create analysis like what our team our product team did is like build several plugins specifically for financial tasks. So specifically we can talk about the uh the investment banking one and autary research one. So within these plugins there that is like what we define uh at open eye as a plug-in is kind of like a library of skills plus MCPS and what we built is like we we partner with like top investment banking, top asset management, top equity researcher to build the kind of like most flexible but

also most like realistic plugins like skills that you can actually leverage as a baseline for your analysis. So what we've seen not only in the area of like financial services but what we when is that when we provide specific skills these skills present like a good baseline. So let's say that you are an equity research it wants to streamline uh his [clears throat] them their job what you can do is literally like take the the skill out of the box. there's been a lot of work behind it and then based on using that skill to create like you know a schedule job to create like basically a pipeline that try to reproduce what you're aim to and you use that as a baseline and on top of that you try to improve you try to customize to your taste and adding all the beats that are currently missing but what we've seen is that like these type of plugins [snorts] provide like a strong blaze line so you don't have to start from zero you start from like something that is built out of industry knowledge and then let's say that the 80% is there and then you add your 20% customization based on like your specific needs, your specific like view and so on so forth.

So finally again these are like some of the use cases that are already out of the box. You know these are a little bit more skewed on the investment bank. We're going to work today more on the asset management. If you think about you know pitchbook prep uh preparation, earning analysis, valuation, LBO modeling, uh everything comes out of the box and it's uh quite interesting because even myself like I I come from like a background in in capital markets and I had like to like interact with few people from investment banking talking about valu valuation and just like leveraging this plug-in like running few example locally was able like to bring me a little bit up to speed to understand um you know the product the the the problem behind um what the clients were talking about. So let's this is kind of like a little bit like the initial presentation about like you know what's the effort on uh the financial services. We are now going to move to the more like building part. And as a final reminder please add your details here. It's really really important and I'm gonna stay here for a minute making sure that everyone has it.

>> Yeah, maybe I think it would be good point um when we switch I will show you a little bit what we're doing hands on now that you can just confirm you can access the site and then also be able to log in into shb work. >> Yeah. Anyone has having problem putting the data in like the email address? I think it is a good sign. So we can now switch to the fun part. Um so this is actually the website we're going to use for the building part. So please um um feel free to scan it and if the it ask for a code the code is there but it should be in theory that you write QR. >> Yeah maybe we wait a little bit before we switch because we have a lot of time today. Um maybe just make sure everyone can access it and then I can talk a little bit also about the content and I show you also some um technical details.

So what we'll do we we saw some of the nice out ofthe-box features in um shb work for financial service. So you have data but also sort of customized use cases. So the first part we thought what would be the benefit for a wealth advisor business with shbd for financial service. So I think what we want to do and show here how can you actually make this process more efficient because at the end of the day you want to talk talk to your advisor more. Now, you don't want to uh have a team that's working, preparing, analyzing portfolios, figuring out um what happened, explaining it, preparing the meeting, um preparing um maybe also for the customer a deck that explains why were those changes happening in the portfolio, what needs to be done to bring it back in balance. So, the first part is really the use case. You bring this product shed GBT for financial service to your wealth advisor teams and they can basically accelerate your workflows. So it's an enduser tool in the team. I mean often in my experience also have very different work workflows or different way of working. So it's customizable and we walk you through this use case. a customer there's a portfolio change happening and you want to go all the way

preparing um the P&L or the portfolio attribution and then showing the customer what to do to bring it back into balance. So that's the use case number one and um we provide a couple of prompts there and then I don't know who in the room is oh is a question So yeah, we will fix that. Um, that's normal. Yeah. >> Let's go. >> So, uh, I'm going to work on it. You can keep going. So, like is the website like the the latest slide, right? >> The shbt work you I think. Yeah. So, my colleague work on it and I will do a little bit more to explain it. So, and maybe for the people in the room here, I don't know. We have a couple people. What's what's your background? I mean, you're like a developer, builder, engineer, an AI exp who is like more in a technical role, developing, building something in in the room. So, we have quite I can't see it fully, but yeah, it's a half maybe. So I think for those people that are more hands-on and technical now and I want to explain also the second part of the workshop once we get your access is actually in our service in the codec side the desktop solution the codeex would be also it can be used by end users business users knowledge workers

but this in this case we want to do a little bit more customization of it because let's assume you you deploy CHB financial service in your company successfully and it meets the business users need. But then the question is how can I extend it? How can I make my data coming with it? How can I customize it? So the second part of the workshop we thought there are two there basically two ways of doing it. You could say let's build a custom application. I get an open AI API key. Let's do some web UI. It's easy to do also with codex and web UI. I deployed in my company and then yeah I have the typical software development life cycle or is this path A or path B is you basically think about it oh my users have already in the enterprise shed GBT for work they already have codeex most of them why don't you keep sort of this need that I have for my business inside this ecosystem So like we explore in the second part basically what would you do as an internal developer using codeex to create a skill a plug in and a plug-in extension. I explain what it is in a bit to actually implement a full application

that could sit inside the surface. So basically with that with those things I will walk you through what those things are. you can customize basically the workflows. I mean, everyone I think probably knows now what a skill is. Um, it's a markup that provides a little bit more customization for you and reusable. But then once you actually want to connect your data, you need an MCP, you need some tools, you need to define all that. And then that's where you have a plugin. Now the plugin is sort of a little bit heavier and you can customize it around it. So we create a plugin there and then with a plugin it can lifts on a desktop it can live also in the website and then what also we just recently announced is actually on dev day plug-in extensions. So what we do at the end we also actually customize the UI a little bit so you can customize the surface to fit your use case. So we have an advisor that would use shbdt for financial service in the web but the workflow you implement can actually also be sitting in the in the desktop application and then so we exploring that >> one second >> no >> okay >> so we get there [laughter] so so I have to talk more and probably I can talk you through it so maybe can you do it um on a um I can show maybe the content also

Yeah. >> Yeah. Yeah. So, but just I think the interesting part is we did do in this workshop once we get access to it to you and don't worry if you can't get it out we the website is available later also. It's really what would I do as an developer with a company an enterprise that already has this platforms inside the company and can I avoid a little bit my work or reduce my work to deploy applications? No, that's a neat idea I think that becomes possible. >> So, just a question uh about the uh built-in tools you mentioned like uh transcripts and filings and all that sort of uh other data that you have there and that you I guess the harness has some awareness. Is it just that the the harness and skills are there or is there something that you did to the model? Just any kind of curious what what is makes it so much better. The party mic is on as well.

>> Okay. So, just for the people that have not the full link, um that's a workshop access code. Can you get to it? So, someone just confirm it works. >> Okay. You see, fix it. [laughter] So maybe also explaining um what we use under the hood for the workshop here. So that's actually um the sites feature the capability to host a website and it's quite u easy and in this case it's a public veite actually. So what's very neat um to create nice looking um functional UIs actually. So just I just wait a little bit that everyone is um can get to the workshop setup.

>> We are also sending the emails for um accessing the the workspace as mentioned before it should come up. >> Yeah. So, while we do it, um I want to also I think we have little um like bottles or something for everyone here and um not for everyone, for the people that participate today. And to make it a little bit fun um we actually once you're able to be um signed up in the room and I see we already have quite a few people there. Um so this is also created with um the sites application. So, everyone who was able to get into the um website um got 100,000 virtual money already. So, to make it a little fun and of course we all know the market always moves up and down and it goes up uh maybe a little bit down now. But if you complete the steps um you get additional funding and then um at the end of the day we want at the end of the workshop we want to see um the top three people.

Yeah. And um they get some bottles for sure. So to make a little fun, it's a little bit uh to keep it interesting. But um so for for the website now because I think we see people are in there now. So what I want to um basically point out so the first thing we really need to have access to shed GBT for financial services and um that's basically the website here and you can see you have financial data and you can see the plug-in um shed GBD for financial services. So that's sort of the thing my colleague works on onboards you and when you have access to this website which sht for financial services and in the financial data um that we showed in the presentation um you are free to go and follow the instructions under core capabilities. No. So you can say setup once and um we have a little bit a zip file to have some data there for the customer example and etc. So it's really the wealth management use case. How can I help my advisor to be faster and then I definitely would recommend um each of the subsections you can do on your own.

So if you're interested more portfolio questions, uh research or charting, um feel free to start one of those sections yourself and then um when you do it and have any question, please raise your hand also in the room. We want to help you there. Um and it shows really the highlights of the the main features of the product here. And then for the people that are more interested in in sort of building um I think it is a question you have to have codecs on your desktop. No. So so for the people that aren't you can install it. Um I think that's the second part and I I think it feel free. I mean if you rather want to envision that product more what is the product doing? What are the out of the box capabilities? stay in the core capabilities once you have access the website but with codeex um I think that's the idea what I was saying earlier how can we develop maybe with an AI native tool in an enterprise applications now workflows etc so and then here um we walks you through the skill plugin um we're doing also an MCP use case here to augment this workflow we do on the first side and then um I definitely want to highlight try out the plug-in extension and I think what's a plug-in extension I just want to explain

it a little bit um I think we I think people know now there are ways of extending um AI harnesses with skills and the plug-in is sort of little bit heavier and you see the structure it has skills and has an MCP server It has a tool configuration and then you also have an UI probably to surface it in a way how you want it. That's cool. That's the one thing I mentioned but it's sort of what I want to say is also possible. Now it's basically you have the shed GBT or the desktop the CEX application and you see this highlighted areas here. You could actually customize it depending on the conversation that maybe your wealth management business has with codeex or chat GBT. No. So you could say if I type in a certain thing and it's a certain file type suddenly it pops out a UI an extension that you control and can deploy and make it more custom. No. And there are some examples actually. So yeah, I want to highlight those are the things um we could um if feel free to proceed there. So the people that have already codex start with setup once and the people that still get access to the website um please start with um the setup once there. Yeah, there was a

question actually which is important. So, so shedgbt for financial services lives exclusively in our shedgbt work web interface. No, so that might could change but right now it's the only way to access that. No. So you basically I mean you have sort of I mean I have seen enterprises that have the webs GBT work, they have the codec, they have both and even knowledge who users have both. Um so you have to patch it together a little bit. It might change in the future but right now it sits there but it doesn't it doesn't matter. You can actually extend plugins on a desktop but also on the share GBT work site. Now the plugins live there too. So you have the options there and currently plug-in extensions the customization of the UI actually lives only on the desktop side now so you cannot do it yet in the web but it will be probably it's actually on our website explained that is coming up um and so just be aware of it um it's all pretty new here and um the surfaces provide different capabilities just be aware of it and right now if you like a real builder and say hey I want to get the financial data with a CLI now

because Codex has also like CLI you want to put it in a pipeline you're not able to actually access the data now yeah or it's basically not provided yet so as mentioned before here whilst you're in charge you have you can select the model we suggest you to go with either Astra or 6.1 Soul and if you can click on the on the speed you can pick ultra fast that is going to make like your life much better so and yeah it's just the promps you follow the promps and I mean you're learning by reading a little bit what the product actually provides for you so I mean the shibbity for financial services I think when you go on this market the the historical market data and also research you see some of the key features now that sort of as Stefano said um we focus on to get that in the product more tuned not fine-tuned but tuned to your use cases now so everyone has the email has logged into the correct workspace and can use like ch work fantastic went almost as as mostly as I would have.

>> Yeah. So I mean I I maybe for the people that just want to listen I can do it in parallel also a couple things. Um but any question I think we rather want to help you with questions so feel free. Yeah >> you mentioned the specific setup for financial aid. uh you had mentioned uh with the specific setup for financial services has does it have another ontology layer like what else have you built? I think there was a question asked earlier I didn't quite get the get the answer what specific things you had done beyond the model itself. So I think like what I mentioned before is really like the data ontology. So it's not just like connecting the data sources as if you know it would be like you know an API call. We actually the data ontology has been changed in order to make it like extremely easy for the model to get extract the right information and also there is like kind of a context of like for each type of data source what's relevant what's not.

So if you think about 10ks or 10 QS like is not just like providing the file but is also explaining you know what is what within the file. I think that's kind of the on the data point that's the kind of like key differentiator and also when when like not just from the integrated data sources but also from like the plug-in that we partner with like uh data provider that you can login into if you have access that's the same concept. So it's not just getting the data is actually getting data with the right contents in the right format. >> Yeah. And also when we talk about plugins um these plugins is not of course like at the end like if you look at the skills are kind of like some markdown files some templates but there's been like there's actual expert knowledge we've been like partnering with financial institution for quite a long time and when we built this product we had like a selected number of partner like top investment banking and so on so forth that we worked with in order to validate the quality of the product. So it's is literally like the the the kind of knowledge that we build to put to transform like like the model into a product that can be used by like financial experts. So >> yeah. Yeah. So I I think um there I mean if I've just talked about it before if you do it yourself I mean you say I want to do the context layer the ontology myself at scale and it's quite large

data now we have to ingest now so you can do it in an scalable product with agents like doing public data but also have the data providers and there's a lot of ways of making sure it finds the right data. So I think um we actually have a couple um blog post how our data architecture is behind. Now there's something interesting what services we use open source services but I think it shows you there's a lot of know how we build actually for billion users basically like a context efficiently no and so that's the same thing that's under the hood powering this product no so I think it's maybe the engineering side now and the context engineering um right now it's it's an early product and I think you will see how we involve it now but also lives on GBT work for now not via CLI and yeah let's see how the world goes here >> is there any way to access some of these more primitive data structures that you guys have developed like if you have ontologies or like a graph >> so like no what's under the hood it's not really accessible uh because this is like a product meant a little bit more for the knowledge user not the pure engineer but uh I think it's without enter too many the details is literally like each data set was like on boarded with the kind of like

provider to really understand like what's the best way to serve it and also it's it's mostly like the understanding of the data set. So it's not just like it's literally like you know what's the information like what's the relevant information. So how you know like a simple example like you know when you on board market prices versus return like explaining all these detail on like you know if you want to do this operation if you want to extract this information don't like how to avoid like basic uh misconception and so on so forth. So >> did you did you guys when you were talking about 10K and 10 Q sort of embedding did you guys have like custom embeddings and stuff is that is that at the back end of this? >> No no no no it's it's literally like we create the data set and like pro pre-process the data in a way that the model can access easily like think about like a kind of a tool call but like a really customized tool call for that data set. So, no worries.

>> Yeah. I mean, I'm just showing you. I mean, I'm doing now the poms. Now, if you're on the website, you just copy the pom, put it in the product and um and then also when you when you see now the icons here. So, you see a little bit um is actually picked up the plugin. So, think about it. The plugin is a way that's hinted in the prompt. Um I hinted in the prompt to use shbt for financial services. So it follows a plug-in concept currently and then um you can see actually the financial data that's under the hood used and the internal tool calls basically in this plug-in happening. So it's really um like a plug-in concept right now and then um yeah under the hood we provides a back end basically that's scalable. So I would say like to give you like the really high level view of this product. I think it's really cool because you know this is basically the interface that like every like knowledge user like the equity research analyst is going to use. But what we are also going to do in the second part is like thinking about how can you add plugins like uh we all know that you know these data sources are not enough. They're like you're probably gonna have like internal tools.

You're gonna have internal data set. You can build your own plug-in with your own skill like especially as we mentioning before like you want to do the last layer of customization. So as an engineer as like you can build this tool that then like you can use as a plug-in in charge for work. So your end user your knowledge user is going to have this as a base. So let's say this is like a really strong baseline on how to interact with data, how to build like specific financial analysis, how to produce like output like Excel, PowerPoint, but you can add a customization layer that your company or like with like the specific tools with specific data set with specific logic is going to make it unique. So >> yeah, so just point out I mean you you do it yourself also um but I think it shows you the underlying tool calls happening in the sharp financial service you don't have the full access to the sort of the context there but the second part is um this customizable UI so so there I mean we had just recently intelligent UI basically yeah now when you get in response you basically have a more um tuned UI answer basically through the response now and this is one example here and that is something that the plug-in provides too and so it has this context when I ask questions about revenue that you want to see it in a

certain way and maybe as a table and the chart so those are the things that are addition so if you use a the Astra model out of the box it would not do it necessarily automatically like that now you have to basically do it yourself so I just want to point All those little things that make a difference um because it's already a little bit um not it's not fine-tuned model but it's sort of has context how to present data. Yeah. [snorts] We check the pogus here. Okay. I see Mike, Zack, and Cool Deep [laughter] doing well. Oh, and this one is also accessible by mobile in case uh you know you want to do use a different interface. This is not just like desktop. It's is also from like the mobile app works.

>> Yeah. So, I think that's like the one thing I want to point out. market prices would be awesome because [laughter] I know I'm I'm basically uh right now you have to be in in the surface here. So if I ask the question now give me the historical market prices and um yeah I think that would be really neat to have an API but as is always like with data providers you have to be careful uh how the licensing works here but yeah in this case um it's basically um like a very important thing you want to know the market prices in an easy way and there would be also an opportunity if I built it in-house um to take that on because you all have those internal systems and then have an extension a plug-in sorry a plugin that sits maybe in chat GBT work and then also on the desktop side and does exactly that now because you have already databases access to internal data and then yeah maybe use the same um visualizations and then um provide an extension besides a product that's out of the Yeah, it's um it takes a little bit. So even the little pumps here a couple minutes. So if you impatient um and you want to go a little faster uh you know how to change the model or you basically

the actually you know what you can show like if you click on this uh little thunder on the left >> you can go oh doesn't oh doesn't >> no oh only on 61 soul if you go on 61 soul I thought like ultra fast was allowed here. >> Okay. Fantastic. Use that. It's really beautiful. >> Yeah. Yeah. I I mean the the benefit of working at Open you can use ultra fast. [laughter] >> No, it should be working also on the web. >> Yeah. >> Oh yeah. I mean I I did the right model now. So yeah. So yeah, do ultra. Um and also I think I'm just what I said the benefit of working in openi you can actually use ultra for your normal work a lot um and it's actually I think it's a day and night I think um if if this becomes sort of more accessible for everyone um it's just I mean I mean how many people probably waited sometimes for some agent work and then you do some couple things in parallel but now now I think yeah I I really appre appreciate it and um and I think um want to point out it's it's it's very helpful doing your work. Yeah, >> I would say like on this topic uh bringing a lot of like discussion that

we had with clients I think like token efficiency and token cost is like I think for the last three months has been like probably the main priority. So of course right now you know we are talking about Astra so that is like our top performing model but what you observe that is that of course if you're building something completely new a new codebase you're doing research using the most powerful model gives you a hedge but when you're building something like more like a repeating uh repeatable task you can get top level top model performance if you build the right context. So if you build the right skill that explains more in details you know what you expect to the model uh then even leveraging model like Luna or even like 61 6.1 sold is actually is extremely good can really give you the top of of the performance. So you don't have to go for like the most expensive model always the state-of-the-art model.

What we've seen as a common pattern is really like you can think about like building the skills in a proper way that is going to maximize [clears throat] the performance and this is like a common or if you're working in more like multi- aent setup what we've seen is that the orchestrator needs to be like probably like the smartest of the agent but the single sub agent don't need to be like the Astra you can switch to something like Terra something like Luna and still get like basically 95% of the overall uh result. So of course to do all this measurement it's about the use cases about like kind of like building evals is about actually measuring the performance of what you're doing but you know Astra is more like for what is like completely new research new codebase but also the other models especially even Luna that is our cheapest like I don't remember like out of my like by heart like the the price but is is extremely cheap We've seen it like that with the right skills can be leveraged and produce extremely high quality content from like you know reports to analysis. It's just that like you need to use the right skill and make sure that with the right skill the performance is actually matching your expectation. But that's a great strategy that we've seen working across the board you know for actually optimizing the cost. Especially if you

think that like let's say that you are running like this like wealth management conser concurge on like your kind of like entire universe of client that scales up pretty quickly both in terms of like cost and also in time time needed like using models like Luna that with the right set of skills can really like help you uh mitigate this kind of problem. Sorry. >> Yeah. No, I think yeah, I think the point about I mean the models are changing quickly and there are more models coming but yeah it's often I mean every time we we have a model also I think you have to figure out what's your effort level sort of now because the capabilities sometimes sit differently now the price point versus um the capabilities so I think I'm I'm basically in my day right now I'm using a lot 6.1 soul and then I can have the luxury to use ultra of And then but I think ever sometimes when I created the workshop actually the website um so this all was created of course with Astra and um even the more advanced live room view I think. Yeah, those are the things I handed off to Astra basically. Um, I let it run. I I was on Friday. I let it run and it came back after that and then I just fine-tune it a little bit more the website with um soul because it was quicker and more efficient and yeah, I think it's very important every time a new model comes figure out the

capabilities, the effort level. Um, and yeah, it's it's always changing. I cannot say we can't >> I would say like let's while you know while you go through um the various exercises like the question for you would be like any initial thoughts on like you know how are you using this model right now this type of product and models within your like day-to-day and what do you think are the limitation or the potential anyone wants to like hot takes on um using this type of product. No one. >> Yeah. Okay. We'll have later a little bit more questions. But so, so I think when I when I saw the first time the product, I mean, we have it seen internally earlier. Um, I said, "Okay, great. [laughter] But I need a little bit more." So maybe we can pivot. Um, this product is there and it's I think definitely geared to the knowledge worker. But let's think about the builders. So I have a little passion for it because my experience is definitely in the building side also and um um I I want to go a little bit um on on the plugins again. So so I think the plug-in architecture sort of the folder structure what's inside and plug in um is nicely described here. So you see the

skills MCP server which connects to something outside also via tools and then you have some UI components that could be rendered doing usage of the plug-in and JGBT work for financial service is a plug-in. So, and then um the the thing that you can do then with an architecture like that and that's maybe a segue now to go more on the builder side is come up with an architecture that leverage all that in a in a in a way that sort of brings all those different pieces together. So, this is sort of an architecture diagram um because you have to ask the question what's your surface. So when I say service, we have shed GBT work. We have which sits in the website. We have codeex which sits locally or we have also the shed locally in the desktop. Um and then as a builder I mean we all have now access usually with codecs but I see this actually coming to the knowledge workers more and then there will be probably more similarity in the UI for both um use cases in the future. So you have it on your desktop and you can do a lot more in codeex because now you can install a local MCP actually. Now you can create with code very with promps an MCP server that maybe goes to your internal security portfolio database or client database maybe or Salesforce could be also there used etc. But then you build

it there. You have a pro you have a prototype usually as a developer there. Now you basically connect to your local website sort of and you have a codex running and it connects to the MCP but um you you also want to use a plug-in infrastructure. So the plug-in infrastructure sits in sht managed. So I also maybe just point it out. So if you want to get now your own plugins in shed GBT work um you can install a plug-in creator. Yeah. So you can install this plugin and when you install this plugin you can also install plugins in the JGBT workite. So that's another way of customization it and the same thing you can do on the desktop side. I I don't know if I confuse you. It's but it's basically think about the surface and you have different avenues to customize it. So just be aware the website plug-in creator is one option and then in your desktop of course you can install and plug in and if you have users you could figure out an enterprise mechanism to deploy those plugins with your codeex deployment no so we have um like a management business everyone gets codex and also access to shbt work you can

figure out what's the folder structure with the plugins that's managed by the enterprise and then deploy through your existing mechanism and that that's a huge benefit. I mean if I would build now an application myself I have to worry about how to log in how to make sure my access control is there. So if you build an application on the side in enterprise it takes quite a time to get it out and approved. But if you build it in an in surface that's already approved and running and managed you can hook in. No. So that's sort of the plug here. So yeah, we would then use some of the OpenAI plugin um plug-in infrastructure. In this case, the SHBT financial services access to our context data where we have financial data etc. It still sits in the web service only. You can't use the CLI. And then the next thing is you would connect in your plug-in with an tool called your databases. and that would be another MCP or you do it um like in a different way and you can use authentication etc and making sure it's sort of an enterprise managed way of connecting it to the OpenAI hosted environment and all that can be done. So this would be sort of one way of doing a little bit more customization but you stay in this thing

and I I think I I can just I'm very excited about all those options now. I mean I I love to build agents myself and host it, but to get it out approved in an enterprise is often very hard and if you can leverage something and also um if you have it in codex there are some um guard rails policies that can actually prevent that a plugin can go outside of the sandbox. Now the sandbox is sits in the codeex you have sort of controls there and then if I deploy code I just have to figure out an enterprise approved way to get my plug-in deployed. No, so that's sort of the I think it's a neat idea to explore and it can only get better in the future. >> Yeah. And I and I think that like when we talk about enterprise control like staying like within [snorts] you know this surface of chibbity work and like which plug-in can be deployed to which employee is like really common request.

So all the enterprise control here available so you can really control the access control who access the data who access the plug-in and so on so forth. So this I feel I feel that like what Oliver was mentioning is really like here especially if charg work is your final surface for your users the like transforming like an idea a prototype into a kind of like deployed plug-in that can actually be used by your end your end product um end user it's like much faster than what like we're used to. We have a question. >> So you have to have server. [sighs] >> So depends on the plug-in. So it depends on the plug-in depends like what you need to do like a plugins could even be like a a set of skills that you know. >> Yeah. Then then you will have to create like a plugin like an MCPlay.

>> Exactly. >> And also to manage the authentic authentication. >> Yeah. Yeah. So so so maybe because now think about in this world where this is a platform which I think it is already what to do when in those terms. So so first of all a skill. So we have like some markup files. So I would create in natural language um like maybe a couple markup files what to do and I think also it's like quite easy now to create a skill with some help from codeex also and then it's sort of it's a reusable u method of reusing prompts I would say or knowledge no and and I think that's available currently on the codec side so if you have a developer we're using skills usually to develop faster It's not actually available in the web side right now. But then the interesting part is a plugin is what you said is with a skill I can't do really an enterprise integration. I have some reusability um in the workflow that I can scale and then can share it across the company. But the plugin is where oh now I have to authenticate systemto system or maybe I want to do on behalf of the user. No, you have to sort of authenticate um for the user that's actually logged in into your codeex side. So, so that's why the plugins will

be a package where you have the capabilities maybe also some skills that you want to reuse. But then to your point, I need some code, I need some logic, I need to have some way of getting my credentials for authentication, etc. So, that's a plug in here. That's a plugin. And you saw the folder structure before. And then it uses MCP connections and there's also uh a UI component part of it. There's an app UI standard like sort of a way of doing all UI components with a plug-in and then all those things can be zipped up basically in a in a package and that's a plug-in. So it's really I would say the plug-in are most powerful because they contain the super set of everything. And then what I mentioned before is um once you actually in the surface so if I go back to the plug-in extensions and I think I want to give you I think this is I think a good example to understand it because I had also a hard time to understand what is an extension but here it's maybe more visually so I think this an example so I mean you see that's sort of the surface you're familiar with GBT but on the bottom left. You see there's a way of adding your own icon, your own app hook basically. And then once you click it, now this is all now managed by a plug-in that you can control. So, and the beauty

is it's just not just this little icon there you can customize. You can customize uh different parts of the surface. So we can customize the settings, we can customize the navigation and then also think about it. I mean it's I if I go now in in in an enterprise and I have to go to five different websites to my job. I don't want to do that actually. I want to stay in my favorite tool which is AI powered has a very good frontier model hopefully um and I can have my conversation for my workflow and I don't have to leave it. So what I would like to do is I have my I'm doing my work as a end user but then it pops up it knows a context. Oh I have now something specialty around getting um maybe an ad u like a client meeting like a client meeting and have to prepare. So it knows in the conversation and it will pop up my customized UI for that. Now maybe the that your W business asked you about a certain thing you want to do quicker or you want to see a certain metric from an internal system for this customer profile. That's what you can do now and you don't have to leave the surface. So that's a I think I want to give you sort of an idea where

those concepts sit and um if you take that away from the session that would be already awesome. Yeah. So any questions? Yeah. So maybe I switch in in my visualization I set up the codec side. So would be a question also in the room. Um, who has uh Codex running locally already? Okay, I see. Yeah, like a fair amount. >> Yeah, codex of CHP app right now where it they're the same surface. So, yeah. So I will just play along a little bit because now it's more um building a plugin for me. Yeah. And everything becomes just a problem nowadays. So, Yeah, I think it will take some time. So

even with Ultra because I think even if you use Ultra, you have to build code and compile or not compile but run it on locally. Yeah. So maybe that's also one thing um because I'm doing a lot like this of course for like workshops like that. It's basically um sort of I'm on working on my desktop now. So my I can have a couple more shed conversations open and do another project with maybe create another plugin or something like that. But yeah, I think you really want to scale um basically where where basically the uh the agent runs or basically you want to have a cloud environment. So that's another thing that's very useful. Um I'm doing also for financial service a lot of um this appsac security scanning right now. So it's a very common use case and yeah it takes even on a desktop you have occupied a large repo and you do like sort of a appseac inspection it clocks your compute locally now so you want to move that sort of to a so remote site now so this case also if if you have a lot of tasks running um I think yeah we have a couple capabilities out of the box there but also you can customize it where actually the compute or the codeex runs now in this case the codex runs locally which takes even if the model as fast. It takes time or it's it takes as much time as my laptop has.

No. So, but yeah, it would be good to to have those infrastructure components also inside an enterprise where you basically can easily spin and spawn compute. No, and then um you hand over task and um it's basically um you come back when the work is done. Yeah. >> Yeah. >> Yeah. So the there are a couple of couple of angles here. Yes, the idea is really especially with plugins and like you know and the skills underneath is really like to create not not to have everyone with different skill but to actually build like kind of a marketplace and share them across teams and that works also with what we were mentioning before that is more like the templates. So artifact templates could be like a website, could be a PowerPoint presentation, it could

be an Excel. What we've seen is that like you want to have like we even within the plugins you can build some like add some kind of historical example as files and then like the the plug-in will use that as a reference. So let's say that we are in the context of like you know piece of research that needs to be created for this case like the your you know private you know private wealth clients what you can do is actually maybe provide like few historical example and then like ask like within the skill to use that as a reference then of course you can modify them but like the idea is really like to be as as reusable as possible. So because like we we know that like format matters a lot and that's why like uh even if doesn't really like connects to the financial aspect of it like we we put a lot of effort like in artifact creation. So if you probably seen like over the last six months the capabilities of our model to produce not just correct you know a spreadsheet but to actually format spreadsheet in like highest possible level of like formatting that like is matching like investment banking standard really went up. So we have like several benchmark we can show you but I think that like the best example is like try it yourself. If you have like some template or spreadsheet presentation PDF that you want that you have and just try

to say like okay with this new piece of analysis like build it in in this output you're going to see that probably uh the output is going to be quite strong and then of course that's the last mile where you probably going to have like to customize the skill a little bit to say like hey these are couple of details that are not correct but like if you mention it in the skill then at the next iteration are going to be correct. So yes, the best part is like having sharable skills because you also want to kind of keep fine-tuning them. So another like interesting point is that are the skill reusable between different type of model different families of model like the answer is that like if we move within charge families yes. So the idea is that probably you know if you move from like you know five to six so like a year a year uh kind of like jump you need some adjustment but in theory like between incremental updates like the [snorts] what we've seen is that usually you need less the specification because the model are getting smarter and smarter but these skills tends to be quite reusable and like the cost of maintenance is is kind of reasonable. not to elevate it.

So um >> yeah I think yeah maybe the skill can be a rabbit hole also actually so because yeah I think the skill reusability and then also maintain it um I have seen some customers go a little wild a lot of skills and then the teams I mean it's difficult to share and you have to standardize it sort of so this is more like an organizational problem but I think if you want to like say you like an app builder I would rather lean to plug in because you control also as a deployment and can package it a little bit more. If I I mean we're sitting on a font office or like some quant I mean they have everyone has skills on their on the on their codeex basically. So I think the question is then if two people in your team do similar things, they're creating a skill, how can you make sure it's like reusable, you know? So I would say um if it's a sort of when you have a couple team members and they're figuring out actually do a similar task maybe with a little bit optionality, I think it's highly recommended to actually remove some skills in your team because it's difficult to maintain. And I I actually advised a customer recently that I think you have too many skills. I mean maybe just try to come up with a good like like like a definition of the core capabilities. If you do like an portfolio analyszis or you do some trading analyszis thing um keep that as a skill but not for everything in your

workflow because it becomes kind of hard to maintain and also the point what you had is um you still have to retest with the models now because um what we what is one benefit maybe if you work with us is also you can access to the early preview models now so when you have a relationship for us because then you can see also what's impacting your business when you have certain promps and and also and you can test it early now and make sure when it's live then you actually make sure that it's the same expected behavior now but yeah my advice is if you want to control it stick to skills you have a builder that actually thought about it what should be like encapsulated encapsulated no and then if you see too many people have too many skills it can also slow down actually the processing so Um it's not as like a straight answer there but you have to manage it. It's my recommendation. Yeah. >> I think as a as a general trend that we've seen is that uh um so our team mostly focuses on API use cases. So uh use cases that like where you know you're going to use response API. So um eating like a different an like a fourth surface. But I've been an open eye for like few months now. And what you've seen already in like this kind of a short period is that lot of use cases that before were only able we were only able

to package if you were able if you were like actually writing code using API and API keys. Now they can be perfectly recreated like um within uh with the plug-in. So we to give an example guys is a little bit outside of the world management. We built few months ago like an internal use case of like how to to create an advanced piece of financial research. So what you can think about like as a deep research but specifically for finan for financial type of problems using uh the API. So it was like kind of an orchestration of like multiple agents. So the like the schematics was pretty simple like on a high level. So you have what we call a planner. So basically an agent that ingest the question and split the question between like sub sub question and for each sub question create basically a sub agent that is going to investigate in an independent manner and then there's going to be kind of like what we call a critique agent a feedback agent that is basically taking all the output from these different independent uh research putting them putting them together validating that there is no not conflict if there is conflict between different findings. It uh creates automatically new sub agent to investigate this like this like kind of contradiction you know till a moment where like we the final report reaches like the quality and

answers all the question that you want to cover. So really think about like an orchestration of sub agent and like kind of like a general orchestration that is really dynamic. Right now you can do we like actually Dom recreated that same use case using a plug-in talking about the deep research and that is way more usable for clients because first of all you know the end user doesn't have to leave the the interface to actually like you know enter like a different type of product that it can just stay in codeex it can actually visualize like what are the sub agents doing, what are investigating, you can create even a specific uh interface for the user to like understand what's going on. And with the current like multi- aent capabilities like you can really see everything in Codex, do everything in CEX and deploy you know instead of having like creating a new codebase just create a plug-in and ship it into Codex.

So >> why would why would you want to So I think we can ask the author. >> Yeah. >> Yeah. And also there's a Matthew and Zack and Cohen. I think you guys are leading here. I mean it's a fun thing. I mean let's see at the end. So but it's a tough race here. >> Uh yeah. So when it comes to where and why and when we would use plug-in capabilities typically the the original example of that the fundamental deep deep research for equities uh was an application that I built about how to do deep parallel validated work in terms of that space out of professional quality. It ended up leveraging a significant amount of orchestration. It meant like a lot of deterministic nondeterministic workloads within that process and it was a standalone application. What started to make a lot of sense is that a lot of the people doing that work weren't going to standalone individual one-off applications to do that work. They started consolidating where that work happened and that happened to be in codeex or jetp work. So it makes a lot of sense to bring that surface area and then the choice of how that how we want to like rebuild that came down to the fact of can I leverage the capabilities of codeex including its sub aent capabilities including its ability to direct dynamic cap uh agent structures to allow plugins and connected to different data sources instead of building these bespoke connections and it started to make a lot of sense to say hey this works as a plug-in specifically focused on utilizing and reusing the LM capabilities to handle things like dynamic tool calling specifically that

in line with how models do that. So we call that ind in distribution tool calling. It's one of the things that makes things faster and more accurate in terms of how those tools happen. So all these things happen to line up to like make sense of like instead of building the standalone agent, build this functionality into a set of a of a workflow, a hard or soft workflow depending on the gates you want to build as and then use codecs as the main surface area, the product area for that. So I was basically taking an old school style really manually tuned application that was using large language models in some nodes and traditional terministic workloads others and consolidating them into a codeex native process which specifically for the use and ease of the user which is you don't have to go to this one hosted site. It becomes locally generated. It becomes customizable based on their needs. They can basically add hey these are the data sources I want to use instead. These are the types of gates so I have like quality bar gates. So it started to become less of a black box in this predefined application and more this evolving process that made sense to live alongside the consumer.

>> Thanks D. >> Yeah. So so I just have to maybe add to the builders here also a little bit. So I think under the hood I think we have the codex harness which is open source here basically. So that's also a big competitive advantage I think because you have basically the full access to the source code which running basically what I showed today here and then you run it on the desktop that's one option with a manage product and that's what you see here it's basically but then also you can take that and move it with an SDK to the server side or an appunner there's a way of doing that programmatically And then of course you have the same builder capabilities. You have you see you have the plugins that it's basically um becomes a way where you can plug hook into an existing system that definitely has a lot of traction now and um gives you multiple choices when you have this research workflow what my colleague just mentioned maybe created very quickly.

Maybe a business tech or quant or a researcher has be able to do it. But then you want to scale it to an application on the server side. What do you do? How can you take the work you did already and scale it to the team? So you can take that and with the codex SDK you can hook it up in a real interface also if you need to. No, but you don't it's just a way that you think about it. It becomes very fluid where you can reuse your work and where you host it in the future. Yeah. So, let's quick check time. We have 40 minutes. Yeah. Maybe 50 minutes at the end question or >> Yeah, we're going to have like 50 minutes Q&A at the end, but feel free to interrupt at us any time. So I would you know keep talking a little bit on Codex just to give a little bit more context on like why we think it's a great product and as as we mentioned like the backbone of Codex so the uprunner and uh is a completely open source so even for more like to deploy this product within enterprise it gives like a lot of transparency to see exactly you know the first party app of of course is proprietary but like the engine that is run running behind is fully open source. So yeah it's not just

transparent but it's also fully customizable. So uh I think that's kind of like really interesting um angle. So within enterprise you can actually choose to customize the uprunner that then is going to leverage the first party interface to your needs. If you need any level of a customization it's there. Additionally like what we think compared to you know other type of interfaces is literally that um our model are trained within with this harness. So our model you know you know there's a lot of pre-training and post- trainining but when we think about post- trainining the postraining is is leveraging like our own harness. So the our model basically are trained to work you know within like multiple harnesses but specifically on this one they're going to squeeze more performance for this kind of like trivial reason because they like within the training phase they're already exposed to to this codebase. So especially like recently if we think about all the multi- aent capabilities so what Dom was mentioning so more lots of like workflows the most advanced one are leveraging like sub agents and the basically we the agent itself is taking care of like deciding when a sub agent is needed and when it's not. Of course, if you move like uh within what we call the ultra, not ultra fast, the ultra

mode, basically there like we are pushing the agent to leverage a little bit more these sub agent uh spudging capabilities. But especially when working with sub agents working sub aent codeex provides like a a much better performance and on top of that what you have on on the background is a compaction mode. So compaction mode is basically nothing but like a secondary model that is going to take the context of the current conversation and keep it compacting in order like to remove like the redundancies keep that the um the size you know within the the limits and it's a really high quality model. So what we've seen and this is a little bit more in what we've seen kind of like in the research type of task where like an extremely long context is needed. Compaction mode really allowed us to solve problem where like much longer context uh were required. So this basically allowed us to perform extremely well in what we call the long running task. So um like we as a team we will frequently work with hedge funds and prop trading firms. So as an example I that I run myself is that I took like one of the papers from like quant finance like you know brownie emotion and like option pricing and I literally asked the model and we are not even talking about Astro it was like few

months ago so it was like 5.5 to within codeex just replicate the the codebase of the mo of the paper so basically replicate the codebase of the paper generate the numerical result and try to see if they match what this what the paper mentioned. So there is like a task that you know it I think it ran for a couple of hours and where like it would have been impossible to run without compaction mode and then for example after two hours I was kind of kind of impressed that the model was able to fully reproduce this like you know really quant heavy codebase with like you know stoastic barium motion simulation calibration of models on top of that and that was like basically a task that without compaction mode is not achievable and compaction mode is already available first also via API but you know if you use the codeex arness is available in the arness and you don't even have to think about it yeah so I mean I'm running in par I think the promps some of people have done it um yeah it's running um creating the basically the plug-in extension right now um yeah it's basically um you see on the right side the way of in this example, my customized UI could be hooked up into the surface. So, it's like a small example only, but yeah, it's it's pretty easy, of course, just to prompt. Um, but yeah, it shows you

some UI options there. And then, um, if you look at our website, there are a lot more examples actually how our partners like Figma or Adobe have created a little bit more an app now where you can do a more complex thing. But I I really think I mean I have said it many times is I think that's where probably the industry is moving to now. You think about this more you have an AI platform and I want to plug in in that and then it's basically a marketplace and enterprises have their own inside and I think that's my personal view of course um but I see a strong resemblance here and um yeah it's I think it's a builder you have to just take as an opportunity you know to take advantage of. Yeah. >> Yeah. And I I would say that uh when when I joined OpenAI, a lot of people were still using Codex CLI as like you know mo most of people like you know coming from an engineering background like we're all used to CLI and I've seen that the percentage of like extremely technical people moving to the first party app and almost never leaving codeex as an app is getting like bigger and bigger. So uh as a former [clears throat] quant I still do around some experiment and right now like I used to have like my Jupyter notebook always open right now like is also already like within codex. So what we really think and I think one of the message that we want to pass is really like even as a technical person codeex

is an amazing surface and it's getting like all the you know kind of surface within codeex that you need and then you can as a technical person create plugins and ship those plugins to your like knowledge but non-technical users and the the way that you can actually put put this plug-in in production is can be quite stream streamlined thanks to enterprise control and you can even you can give like really powerful tool to your end user in like weeks instead of month and I think that especially for financial services where like lot of data ingestion is needed every time there's maybe like a new data source something that needs to be changed like a new analysis needs to be done this this new approach this new like AI based approach is going to like really help maximize the productivity in a way that we've seen you know starting in 2025 for coding so that the famous like we all all heard like 10x 100x productivity on the engineering side our objective here is like to bring that booster also to the knowledge workers so that's kind of I think the the vision behind the product and financial services thanks to this like mix of like artifacts, data sources like and technical analysis feels like um kind of like an immediate

match. But while sorry >> any question on all this topic that we covered so far >> I know we are here to discuss financial services but do you see that happening in other industries? Is it the financial services because the data is quite structured? You build the ontology layer. Is open AI doing that for healthcare? Is something else happening? So, >> so the answer Okay, the answer is yes. So, um we are working on multiple uh verticals and healthcare is definitely um uh one of the most priority the biggest priorities also in terms of like research. Um I would say that the other one that we get exposed a lot is cyber security. cyber security is change is dra drastically changing and in the space for example aside of like the more the more like cyber security specific model that we offer like our offering is called daybreak um is also we also like basically build again a plug-in and what and again it's the same concept as as before the idea is that like your cyber security team doesn't need to be an AI expert team or at least at the beginning so we bas Basically, you know, we provide you the model, we provide the the ARS, the open source ARS uh to work with, but we also provide you like a

plug-in for like, for example, for doing like assets uh static asset scanning that is going to is tuned on our model. So, it's going to help you create that baseline in order that like you know out of the box you can like models, harness, plugins to run everything. Actually we have like a library of skills that helps you to do everything from the scanning to the the final like review and patching. But you know this is not [laughter] we're focusing more financial services but our focus is really like across industries. Of course there are industry where like the benefit is like easy to measure finance cyber security healthcare research in general and others where like use cases like are not as broad. So >> so I think it's for the extending platform builder. It's more I think our digital native businesses or like you would say sort of software providers that you usually look for new avenues to sort of distribute also their solutions now. So I think our team works sort of with them a lot more.

I think what we are here because we in New York also met financial services. I think it's what you say it's a structured data is it it lends to itself to use actually AI. Now AI is sort of um with their data and I think it's kind of always hard to sort of have those data silos in those companies and then how can you actually leverage all this data in a context which is not there. It's not ontologies etc. is not there. So sometimes I think what I see as a strategy is sort of you have a it's it will take so much time to proper context layer across all the data stores but why don't we start from the bottom up too now so bottom up would be you have sort of the shb work of codex and then you have teams that basically connecting bottom up to all the data sources and build sort of this context over time while you modernize your data architecture over time. So it's sort of the AI would be super efficient and I think we have the luxury internally. We don't have those legacy things. We have this context layer already. It's it's amazing what you can do then as a workforce if it's actually unlocked. But you have to bottom up and then top down will take more time for you actually to do it at scale. Yeah.

So let's see. um co Matthew and Zakia I think we have this my screen saver I want to talk one more thing I I think um it's probably to to mention also um so when you install like sort of any extension inside a codeex no I think there's also you have to maybe you actually have some code running then on the desktop of someone else no so or you have it in the website no so in our security teams we have a lot of discussion with them and um how can you make sure they don't do any harm I mean there must be some sandbox and I think um maybe just sandboxing with codeex is a very important concept to understand because it's another benefit to go actually go this way because let's say you have a plugin that has to do a tool call so you want to limit where the tool can go from an URL you You always have to have multiple controls. You want to start with your enterprise controls, networking security, endpoint inspections, but once you have a an application on a platform, you want to have also controls inside there. So the codeex provides actually uh policies um the config files toml that are managed on an administrator site as an enterprise but also push down when you do a deployment and in those sides you can decide okay

I I don't want to get that this plugin can go to a certain website no and I want to make sure certain things can be only done also on my workspace let's say a team develops inside an enterprise something but it could not go like to maybe like an sensitive project folder. Now, so you can limit actually um with policies inside codeex an enterprise policy. No, what is can be accessed locally via the codeex client and also what network what URLs um maybe what even plugins certain people can do. And I think that's important to understand. There's a sophisticated way to do that and and that's basically the configuration that you can see also here. It's the custom config toml. So this is something your enterprise administrator would define once on an enterprise level for different teams. It pushes it down and it has all those things I just mentioned. the security sandboxing because it will run whatever it's hosted inside the codeex client will run inside that you still have to protect tools etc yourself but I think that's already a benefit um for this architecture and then the next thing is we all know that it's sort of difficult to I mean the security team goes all the way left we want to we want to we we

want to do um um like any sensitive operations but then the business user says hey it's so slow I always have to click please go to this plugin go to this website and and I think um what also it's unique here we have a uh like the way of doing a guardian like we can actually um automate a little bit the decision- making when it's critical or not critical and this is sort of a way also to make sure when you use a codex setup it's sort of balancing what security needs as a policy but also what's convenient or comfortable for the end user and this helps to make sure when it's sensitive and when it's not sensitive do I need to go to this workspace folder or not so I think that's very important also to highlight because I think this is the thing you have to also explain your users now when you build plugins >> no yeah like and basically every every big institution we work with they will customize uh the configuration for approval permission. So that's like a really like usually when we talk about like deploying codex across the enterprise that's the first like point that we discuss about and like again there is a big flexibility there. Any other question?

>> So, so maybe maybe almost the end. I think should we >> get our invite and then maybe ask some question if you uh co Matthew and Zack I think uh the leading as every good tech company we brought a little bit of merch. [laughter] So like should we say that Co, Matthew, and Zach are the one getting an amazing water bottle branded OpenAI [laughter] picked this morning from the office. So, [laughter] if you if you're still here, let's see. Otherwise, so let's start like uh Coin. I don't know how to pronounce it correctly. >> Yeah. Come up. >> Come up here. Yeah. >> Yeah. Thanks for answering the question.

It was not too hard. So >> yeah. So maybe >> tell me about you. >> Yeah. So uh my name is Um I'm working as a data engineer in the Netherlands. Um so it's pretty interesting to see also the crowd over here. I've been to AI engineer in London. Um and looking forward also to meet a lot of people over here and to see what people are doing here especially in the financial industry. Um that's about me. Yeah. >> And you are from company >> data. So we are consultant company uh yeah working worldwide and I'm working for the for the Dutch uh section. >> Thank you. >> Then we have Matthew. Yeah. Awesome. Yeah. You have to also say something now. [laughter] >> I hope it was fun. >> Yeah.

>> Yeah. Oh, thank you. This was this was awesome. Uh my name is Matt. Uh company's x1wealth.com and we're building an AI native family office. So, I'm doing a ton with codeex and this is very helpful. >> Appreciate it. >> I think an AI native sounds very good. I think I [laughter] would invest in that. >> Yeah. Cool. Yeah. Thank you. Then last one Zach. Okay. >> Yeah. Come up. >> Hi. So, I'm Zach. I'm the founder of the Clockworks and I build apps that use AI to explore new things that we haven't really been able to do before. I focus mostly on development topics, but you know, I'm always looking for new opportunities. So, >> thank you. >> I have a Yeah, good. Yeah, I have an idea. So I think the homework is for you by [laughter] to build a plug-in extension and put it on the marketplace or plug in. Yeah. >> Do you know anyone that can help us push it through?

>> Oh, you already there. So >> yeah, he's he's he gets it. >> Yeah. So yeah, this is also another thing. So I mean with all the um marketplaces now so there's a little bit you have to submit it because it can go also to external customers but I think if if you see it like this this is I think a signal for people I mean this is maybe B to B2B outside marketplace now but also don't forget the internal enterprise thing is the same concept yeah but I think it's really a signal where you have to leverage the distribution channels that are sort of emerging. So the emerging distribution channels and and we have seen it all with the app store and all how viral it is now. So basically think it is another way and of course the SM review if it's public now so and we just have to make sure it's smooth and safe and secure and it fits the right category now but yeah it's a I think yeah just it's a signal that's [laughter] what I want to say.

Yeah. >> Yeah. I know that's I think is that's also the beauty of an open source code base and I would say that like again today we cover more like you know our product surface so codeex charg work but we do also a lot of interaction with clients when working on bespoke use cases and specifically like on uh API use cases like of course that's a little bit more complicated and there's a lot of topics that we can discuss but like a lot of the engagement that we do is actually understanding uh the use cases helping like you know bring best breast practices because we are the first user internal user of our product we do speak with a lot of people in the industry. So for example as we like mentioned before this magic word evals I think is like the most important one. So literally like helping clients you know understanding you know how to design the right eval for the right problem and the question is always like you know we talk about you know boosting productivity but especially when you want to go to production you need to have a way to measure like actually the performance of your product and it's you know again when we're talking about like specific you know chat thread like there you don't need any valves but when you build your agents then it's needed And for example you know best practices there on like how to build evals understanding like out of an evals data set the

performance why the performance is not meeting the requirement you know what the kind of like you know probably like it's not about the model it's about how to improve the arness the prompting the connection with the tools and sometimes it's also like a model limitation so we also connect back to research to understand like where our model are hitting the limits where they're not behaving even if in the perfect environment as expected and ideally you know thanks to your feedback at the next model iteration that is is kind of fixed or improved. We have a question sorry. >> Yes. So on the topic of evals, I was quite curious to see where what your perspective is in terms of the market's maturity in terms of adoption of evals and understanding of the fundamentals and even going into advanced topics or more advanced topics like auto research type of workflows. >> Okay, so this is a pretty big question. Let's start with the evals. Uh I would say that evils are the most usually the blocker because building evils and this is my personal perspective. So happy to uh hear like different thoughts. The complexity comes from the fact that evals usually incorporates a lot of like business business knowledge. So coming from uh SMMES so subject matter experts and actually they are built by engineers. So uh it's having you know the intersection of like a team that has like subject matter experts and

engineers to build evals is not trivial. So usually I always and again here there is a bias because in my previous job I was like what is called a quantitative researcher like for me whenever you build an agent evals is the most important piece because you cannot do optimization if you don't do not measure the performance and it's not just you know like the one line of like oh I'm you know I have my evils and it's like 70% there but it's also understanding the underlying metrics of like you you know first of all like you know where are we performing well where are we are not performing well and so on so forth so I think like evas is like a really hard part but it's not because it's a difficult technology is because like you need to basically put like expert like expert expertise within like kind of like an engineering created data set so then best practices there like to keep it short I think like always like start simple Don't don't start with the LLM as a judge. Start simple and then add complexity because otherwise like you'd never understand if like the quality of your evals is not good enough or the quality of like the agent is not good enough. So that's for me like the like kind of like best practice and one kind of cheat code that we always suggest when you use LLM as an agent don't ask to rank one to five to score

one to five always ask yes or no it's much better try to decompose that one to five into a true or false and you know like right now like doing some promotion we also release like the decision API so you can have literally a model that you know it's quite good is based on out of Luna and it's going to tell you like you know you know quickly super quickly yes or no and the probability and the kind of the confidence around that answer so I see that like a lot of possible application for a decision API uh could be actually within the evils so that's the first part second part is auto research it is like a really big topic it is um something that uh I think really started significant this here and uh what I've seen is that like and there are benchmarks so like we reach really good coding capabilities already last year but like our model specifically really like improved on the mathematical capabilities more recently especially with like Astra and 5.6 six. So I think like especially from a kind of quant perspective like the intersection of coding and math research really allows you for this like more auto research task. So I would suggest like for you for example like one of the auto research task was what I mentioned before replicating a paper. So of course like there are different level of like complexity there but what we have

especially if you think about one specific metric that you want to improve to hill climb as we say in AI terms like we have what is called the goal mode in codeex. So basically a goal mode is basic you have to specify you know what's the target and it's better to have like kind of a target that is like a number a score. Uh one example to keep it simple is you know like try you have this agent you have this EVAS data set this benchmark try to calibrate the prompt in order to maximize the performance of the of the agent and what happens there is that the agent looks at the target and then keeps iterating on like new ideas and new hypothesis. Of course, this is kind of like the the kind of like most s simplistic kind of like auto research mode, but like some of the result that we published also like in terms of mathematics early this year were you were created using goal mode in uh in codeex actually for one of those we also published the prompt. So I think like if you're interested in the space reading actually the public prompt that we gave to Codex in order to prove that theorem it was a conjecture uh there is a different sorry for the mathematician in the room. Um I think that's the most interesting like really interesting piece of content because

like shows you a little bit more the logic of like this because like multi- aent and sub aent approaches. So um like for example like especially when you do research task you're you're probably going to use ultra uh mode then you're going to really see that the agent is going to leverage like the sub agent automatically to actually like run some independent piece of research in parallel then like merge them and iterate. So I would say that you know these are the two really good entry point for evils and auto research and I don't want to understate like even if it's simple like we actually use goal mode to achieve like significant results. So um it's quite powerful. >> Yeah. Plus one on goal mode. Yeah. I think I think yeah just half a year ago you had to worry about the prompt and you had to worry about how to orchestrate the agents and all this stuff. So now I you move back now you you really have to think about what should the agent do as a goal and I think that's very powerful because I mean the models will be more capable to actually do all that than automatically what we used to do what we do still partially manually and I think I highly recommend plus one on the goal mode and also I think today we had chativity financial services plugin basically shown but I think our team really I mean also my colleague Dom

we're focusing with financial service customers or any fintech etc to actually implement it at scale and also those difficult problems and there's a lot of nuances in the evaluations that our team brings so we're not going away we just want to make sure you leverage our models and the platform at the scale I mean this was more simpler today but approach us if you have any use cases happy to work with you >> yeah know I really want to stress the fact that it's kind of like a twod directional conversation. So, you know, if you already have a relationship with OpenAI, please, you know, hit your account director like feedbacks are more than welcome and also it's part of our job really making sure that, you know, it's not just nice presentation, but actually it's going to work at your own companies. So, um that's really like, you know, we are here. But any other question? Okay.

>> One of the questions I have. >> Thank you. Uh, one of the questions I have is uh during the presentation there was a lot of um uh readonly skills and plugin curious for uh the clients or any skills that you've seen that actually writes back into the system or like system of records that uh the team has been building with. >> That's a good question. I think Dom has a has a even better answer. >> Yeah. Hey, I'll come up here. Uh, yeah. So, when it comes to plug-in structures, uh, and how they can and should be interacting with larger systems, there's a couple ways to do that. Uh, and it depends on like what you're trying to build. So if I'm building a fullblown like plug-in extension, I'm really just orchestr I'm using the codeex as the orchestration system, but those tools and processes can absolutely store to remote locations and to interact with those systems. The easiest way to do that is to interact with the local file system. So if you tell it, hey, within this project where I'm scoping this out, it can it can write back to these processes or you give it instructions for the model to actually like the output of the tool is this and the output of the or the instruction of a skill set is to say, hey, persist this in this location. Uh you'll see a lot of persistent structuring uh within codeex.

It's gotten really really good at handling long context by effectively utilizing its local file system as like that point of record. That being said, once you move from a traditional like readonly plugin or skill set into a full-blown extension oper operation, like I tell it to go talk to the central server, I tell it to go spin up a database or at least connect to the specific database as part of a tool. So like the a lot of the use cases that we do, we just did one around proxy voting, being able to actually go define your proxy voting systems, what it actually had what it sent to. was like even though the plugin in the surface area was codeex the actual source of record was this app server that was connected via one of the plug-in extensions it says hey write this report write the findings write the date write the the sources back to this end of it database and that database feeds the like their production application system which is outside of codeex so operating system from a user perspective is codeex uh operating system from an administration perspective and from the legal review perspective was outside of codeex so the extensions MCPS allow you to be flexible where the those different operating systems are and where the systems record are. Uh and you can either offload that to the codeex and keep things local or you can build the tools and build those avenues into particular like MCP and OOTH related you know permission sets by pretty easily dropping an MCP where you need to. Yeah, I would say like another like common use there is really like to especially when you're doing like specific product um processes that needs to be kind of like audited like he was

mentioning proxy voting having like kind of like a standardized like what we frequently build like kind of a standardized like logs that will allow like you know a technical user to understand you know not just the end of the analysis but how every decision every like piece of was done that's for example Like this extended log is something that we tend to store frequently and especially you know in a highly regulated framework like um industry like financial services being able to back trace every decision to the data source and the logic behind it was kind of like kind of a crucial and they like usually for example when we produce this artifact as output we also store like logs that are fully auditable for this reason as part of like the process and the plug-in. One last question maybe. >> I think we almost at the end now or >> Yeah, we are like five minutes out. >> Yeah, thank you. Thank you for spending time with us now. And then please reach out for us and thanks have a good good uh conference and thanks for coming. Thank you so much guys.