Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait — Al Chen, Cisco

When I was in 10th When I was in 10th grade, I had a grade, I had a grade, I had a math teacher, math teacher, math teacher, Mr. Gaff, who was Mr. Gaff, who was Mr. Gaff, who was known for being known for being known for being tough on tough on tough on tests. He tests. He tests. He made us made us made us write down our answers write down our answers write down our answers to the fourth decimal place to the fourth decimal place to the fourth decimal place , just in , just in , just in case we case we case we guessed the answer guessed the answer guessed the answer by accident or by accident or by accident or luck. And luck. And luck. And most of the points for most of the points for most of the points for the task were given the task were given the task were given not for the result, but for not for the result, but for not for the result, but for the proof, where it was the proof, where it was the proof, where it was clear how you clear how you clear how you arrived at it. When I arrived at it. When I arrived at it. When I started working on started working on AI oversight software, I AI oversight software, I realized that realized that realized that most of the most of the most of the compliance and compliance and compliance and risk teams I work with were like risk teams I work with were like Mr. Gaff, but with Mr. Gaff, but with multi-million dollar multi-million dollar multi-million dollar budgets. I want budgets. I want budgets. I want to say that to say that to say that implementing implementing implementing AI agents in banks takes a AI agents in banks takes a AI agents in banks takes a very, very long very, very long very, very long time. And the reason lies in time. And the reason lies in time. And the reason lies in one key one key one key word: "production." I'm word: "production." I'm word: "production." I'm not talking about not talking about not talking about prototypes in development prototypes in development prototypes in development or in the or in the or in the testing phase, or even testing phase, or even testing phase, or even internal internal internal RAG agents to RAG agents to RAG agents to improve improve employee productivity. I employee productivity. I mean the applications that mean the applications that millions of customers of millions of customers of these banks use every day. The same ones these banks use every day. The same ones these banks use every day. The same ones you you you use when you use when you use when you log in to your bank's website or log in to your bank's website or log in to your bank's website or mobile app mobile app mobile app . It is these types of . It is these types of . It is these types of cases that cases that cases that matter. My team matter. My team matter. My team is at the forefront of is at the forefront of is at the forefront of implementing not only implementing not only implementing not only our our our software, but also software, but also software, but also engineering processes engineering processes engineering processes at the largest banks in the at the largest banks in the at the largest banks in the world. I want to world. I want to world. I want to share share share the problems that clients the problems that clients the problems that clients come to us with come to us with come to us with and and and the strategies that the strategies that the strategies that shorten the path to shorten the path to shorten the path to production from a year to a

production from a year to a production from a year to a few weeks. So, few weeks. So, few weeks. So, let's first let's first let's first look at how look at how look at how standard SaaS AI is standard SaaS AI is standard SaaS AI is deployed in a deployed in a deployed in a bank. Spoiler: it doesn't bank. Spoiler: it doesn't bank. Spoiler: it doesn't work. There is work. There is work. There is outgoing traffic to outgoing traffic to outgoing traffic to the Internet, which the Internet, which can cause confidential information to can cause confidential information to leak out. When leak out. When leak out. When you deploy AI on you deploy AI on you deploy AI on your own servers in a your own servers in a your own servers in a bank without bank without bank without internet access, the process internet access, the process internet access, the process becomes a little more becomes a little more becomes a little more complicated. Today complicated. Today complicated. Today I will examine three key I will examine three key I will examine three key aspects. The first is the aspects. The first is the aspects. The first is the assembly line itself. assembly line itself. assembly line itself. How do banks How do banks How do banks load load load our software images into their our software images into their our software images into their cluster? If a region cluster? If a region cluster? If a region fails fails fails during a query, what is the during a query, what is the during a query, what is the data recovery plan? And data recovery plan? And data recovery plan? And finally, finally, finally, most importantly: how most importantly: how most importantly: how to ensure that to ensure that to ensure that confidential data confidential data confidential data does not get to does not get to does not get to internal internal internal employees or employees or employees or outside the bank's VPC? outside the bank's VPC? Compliance speed Compliance speed and CVE are inextricably and CVE are inextricably and CVE are inextricably linked. And no linked. And no platform worth its salt is platform worth its salt is complete without a plan complete without a plan complete without a plan to address that to address that to address that first hurdle— first hurdle— CVEs. I CVEs. I will explain why this will explain why this will explain why this process is so process is so process is so complicated. First, complicated. First, complicated. First, you can take one CVE and you can take one CVE and you can take one CVE and research it on the research it on the research it on the NVD, CISA, or MITRE websites.

NVD, CISA, or MITRE websites. NVD, CISA, or MITRE websites. But in reality, there are many images on But in reality, there are many images on But in reality, there are many images on our platform, like our platform, like our platform, like most others, most others, most others, and and and each of them has each of them has each of them has quite a few CVEs. And I'm quite a few CVEs. And I'm quite a few CVEs. And I'm trying to fix trying to fix trying to fix vulnerabilities not for vulnerabilities not for vulnerabilities not for one bank, but for one bank, but for one bank, but for many. Therefore, we many. Therefore, we many. Therefore, we had to create an had to create an had to create an appropriate system appropriate system appropriate system and process. The and process. The and process. The complication is complication is complication is that our banks that our banks that our banks use different use different use different versions of images, each with its own versions of images, each with its own unique CVEs. In unique CVEs. In addition, banks use addition, banks use addition, banks use various various various scanning tools that scanning tools that scanning tools that provide a report on provide a report on provide a report on existing vulnerabilities. existing vulnerabilities. Additionally, they have their Additionally, they have their own policies regarding own policies regarding own policies regarding which CVEs are which CVEs are which CVEs are critical, what their critical, what their severity level is, and what their SLA remediation timelines are. Our SLA remediation timelines are. Our images rely images rely images rely on many open-source on many open-source on many open-source projects, which contain projects, which contain projects, which contain many transitive many transitive many transitive dependencies with CVEs that dependencies with CVEs that dependencies with CVEs that our our our development team cannot development team cannot development team cannot remove. Therefore, we are remove. Therefore, we are remove. Therefore, we are constantly constantly constantly reviewing the reviewing the reviewing the release notes of these release notes of these release notes of these projects, because we have to projects, because we have to projects, because we have to provide detailed provide detailed provide detailed explanations to the explanations to the explanations to the banks' security teams as to why banks' security teams as to why banks' security teams as to why we cannot we cannot we cannot fix these fix these fix these vulnerabilities and why we vulnerabilities and why we vulnerabilities and why we depend on depend on depend on open-source developers.

open-source developers. So, once you've So, once you've identified the CVEs that your identified the CVEs that your identified the CVEs that your development team development team development team can actually fix, can actually fix, can actually fix, there are a few ways to there are a few ways to there are a few ways to fix them. You fix them. You fix them. You can upgrade the can upgrade the can upgrade the package version or package version or package version or rebuild your rebuild your rebuild your image based on the image based on the image based on the base Docker image. base Docker image. But here comes the But here comes the difficulty: some difficulty: some difficulty: some banks say they banks say they banks say they want to want to want to use use use version 3.8 of the package. And version 3.8 of the package. And version 3.8 of the package. And another bank might another bank might another bank might say, "Oh, wait say, "Oh, wait . Wait. This version is . Wait. This version is . Wait. This version is too high. There are too many too high. There are too many too high. There are too many CVEs. I want CVEs. I want CVEs. I want to stay on an to stay on an to stay on an older version of older version of older version of this package." So, this package." So, this package." So, despite all this, your despite all this, your software should software should work and work and work and function. Therefore, function. Therefore, function. Therefore, one solution could one solution could one solution could be to create a be to create a scaling system. We scaling system. We have a tool that have a tool that have a tool that does the main does the main does the main heavy lifting, prepares heavy lifting, prepares heavy lifting, prepares reports and conducts reports and conducts reports and conducts internal internal internal research. research. The result is a handy The result is a handy table that tells table that tells table that tells our team which CVEs our team which CVEs our team which CVEs need to be fixed, and need to be fixed, and need to be fixed, and also provides also provides also provides an explanation for an explanation for an explanation for the banking the banking the banking security services about what security services about what security services about what we can't change we can't change we can't change due to these transitive due to these transitive due to these transitive dependencies.

dependencies. Most scanning tools like Sysdig, Wiz, scanning tools like Sysdig, Wiz, or Prisma simply or Prisma simply or Prisma simply check if check if check if a package has been updated to a a package has been updated to a a package has been updated to a specific version. But specific version. But specific version. But they don't say anything they don't say anything they don't say anything about whether about whether about whether the exploit remains in the exploit remains in the exploit remains in your application's code. And your application's code. And your application's code. And how can you actually how can you actually how can you actually check this? One check this? One check this? One solution is to create solution is to create solution is to create a task in Harbor, which is a task in Harbor, which is a task in Harbor, which is similar to creating an similar to creating an similar to creating an environment or " environment or " sandbox" for sandbox" for sandbox" for testing the agent. testing the agent. So, within this So, within this Harbor task, you have Harbor task, you have Harbor task, you have instructions for the instructions for the instructions for the agent, an environment agent, an environment agent, an environment that contains the actual that contains the actual that contains the actual Docker images for your Docker images for your Docker images for your application, and application, and application, and tests that you can tests that you can tests that you can run on the agent run on the agent run on the agent to verify that they to verify that they to verify that they pass. pass. The instruction here is the CVE The instruction here is the CVE for this service, for this service, for this service, this package. And I give this package. And I give this package. And I give the agent an indication that the agent an indication that the agent an indication that version 2.6.2 of this version 2.6.2 of this version 2.6.2 of this package supposedly package supposedly package supposedly fixes this CVE. I fixes this CVE. I fixes this CVE. I want you to write an want you to write an want you to write an exploit based on the exploit based on the exploit based on the older version to older version to older version to check if the check if the check if the new version contains new version contains new version contains that exploit. Then that exploit. Then that exploit. Then write a test to write a test to write a test to check whether check whether check whether this this this exploit worked or not. And with exploit worked or not. And with exploit worked or not. And with Harbor, you can Harbor, you can Harbor, you can run a bunch of run a bunch of run a bunch of Docker containers or the Docker containers or the Docker containers or the Harbor, Modal, or Harbor, Modal, or Harbor, Modal, or Daytona sandboxes. Run them all in Daytona sandboxes. Run them all in Daytona sandboxes. Run them all in parallel and you parallel and you parallel and you get a multitude of get a multitude of get a multitude of trials to see trials to see trials to see how the agent performed how the agent performed how the agent performed under different under different under different tests. And I want to tests. And I want to tests. And I want to say that it is say that it is say that it is not enough to simply not enough to simply not enough to simply declare an declare an declare an update to the

update to the update to the package version to a certain package version to a certain package version to a certain number. We must number. We must number. We must provide evidence provide evidence provide evidence that the exploit is that the exploit is that the exploit is no longer present in the application code no longer present in the application code . The second pillar, . The second pillar, . The second pillar, disaster recovery disaster recovery , is not , is not , is not something like something like something like optional optional infrastructure hygiene for a bank. infrastructure hygiene for a bank. The difficulty here is that The difficulty here is that for the underlying for the underlying for the underlying database platform database platform database platform used by used by used by most most most AI AI observability tools, observability tools, observability tools, including some of the including some of the including some of the companies sponsoring companies sponsoring companies sponsoring this conference, there this conference, there this conference, there is no managed is no managed is no managed service for service for failover between failover between regions and regions and regions and data replication. data replication. Maybe someone knows Maybe someone knows what database platform what database platform what database platform I'm talking about? It's not I'm talking about? It's not I'm talking about? It's not her. So, Redis or her. So, Redis or her. So, Redis or Postgres have managed Postgres have managed Postgres have managed failover failover failover between regions, but this between regions, but this between regions, but this database database database does not have that capability. does not have that capability. ClickHouse is a great ClickHouse is a great database platform, database platform, database platform, but even their but even their fault tolerance documentation fault tolerance documentation states that there is states that there is no managed service for failover between regions. This no managed service for failover between regions. This means that banks means that banks means that banks must independently must independently must independently address address address disaster disaster disaster recovery issues. So here's a recovery issues. So here's a recovery issues. So here's a simple pattern you simple pattern you simple pattern you can apply: you can apply: you can apply: you have ClickHouse in the have ClickHouse in the have ClickHouse in the primary region, primary region, primary region, a job that a job that a job that backs up to a backs up to a backs up to a bucket, and another bucket, and another bucket, and another job that job that job that restores that bucket in the restores that bucket in the disaster disaster recovery region. Within recovery region. Within recovery region. Within this region, you this region, you this region, you can have multiple can have multiple can have multiple pods with replicas of pods with replicas of pods with replicas of this data. This entire this data. This entire this data. This entire procedure procedure procedure relies on relies on relies on ClickHouse backup—an open-source ClickHouse backup—an open-source ClickHouse backup—an open-source project project project supported by Altinity, supported by Altinity, supported by Altinity, which itself is a which itself is a which itself is a competitor to ClickHouse in

competitor to ClickHouse in competitor to ClickHouse in managing managing data failover data failover between regions. So during between regions. So during between regions. So during this procedure, ClickHouse this procedure, ClickHouse this procedure, ClickHouse backup provides logs about backup provides logs about backup provides logs about whether the whether the whether the backup backup backup and restore took place, so and restore took place, so and restore took place, so you get this you get this you get this handy table. But handy table. But handy table. But this status this status this status says nothing about the state of says nothing about the state of says nothing about the state of your data at this your data at this your data at this moment. A failed moment. A failed moment. A failed disk, a failed disk, a failed disk, a failed node, a failed node, a failed node, a failed migration, or a delayed migration, or a delayed migration, or a delayed replication—these are all replication—these are all replication—these are all issues that issues that issues that can cause your data to can cause your data to can cause your data to become outdated and become outdated and become outdated and different from what it different from what it different from what it was when the was when the was when the job was performed. job was performed. job was performed. So you can take an So you can take an So you can take an extra step: extra step: extra step: check the number of check the number of check the number of rows in the tables or rows in the tables or rows in the tables or run a check to see run a check to see run a check to see if if the replicas within the the replicas within the region are synchronized. The main region are synchronized. The main region are synchronized. The main takeaway here is that it is takeaway here is that it is takeaway here is that it is important important important to verify to verify to verify the correctness of the data, the correctness of the data, the correctness of the data, not just check not just check not just check the box indicating that the the box indicating that the backup and backup and restore task was successful. Similar restore task was successful. Similar restore task was successful. Similar to vulnerabilities (CVE), to vulnerabilities (CVE), to vulnerabilities (CVE), you can't just you can't just you can't just update the version and update the version and update the version and think the threat think the threat think the threat is gone. You must is gone. You must is gone. You must perform these perform these perform these additional checks.

additional checks. additional checks. And one more note And one more note And one more note for architects who are for architects who are for architects who are building building backup and backup and recovery systems in recovery systems in recovery systems in clusters: you also clusters: you also clusters: you also make decisions make decisions make decisions about where to about where to about where to store data, whether store data, whether store data, whether you like it or you like it or you like it or not. When you not. When you not. When you replicate data to a replicate data to a replicate data to a cloud storage (bucket), cloud storage (bucket), cloud storage (bucket), it doesn't care if it's it doesn't care if it's it doesn't care if it's in Oregon or in Oregon or in Oregon or Virginia. However, for Virginia. However, for data localization requirements, it is data localization requirements, it is important where exactly important where exactly important where exactly these these these individual records are stored. Therefore, individual records are stored. Therefore, individual records are stored. Therefore, one solution is one solution is one solution is to replicate only to replicate only to replicate only the identifiers and the identifiers and the identifiers and metadata of your metadata of your metadata of your traces and data to traces and data to traces and data to different regions. But the different regions. But the different regions. But the main tracking data itself main tracking data itself is stored within is stored within one region and does not one region and does not one region and does not leave it due to leave it due to leave it due to certain legal regulations. The certain legal regulations. The final pillar— most banks most banks most banks believe that they do not believe that they do not believe that they do not disclose any disclose any disclose any confidential confidential confidential information to their information to their information to their internal internal internal employees or employees or employees or users outside of the users outside of the users outside of the VPC. However, there VPC. However, there VPC. However, there may be may be may be open "backdoors open "backdoors ". This applies to ". This applies to ". This applies to cases where cases where cases where personally identifiable information (PII) personally identifiable information (PII) personally identifiable information (PII) and non-public and non-public and non-public confidential confidential confidential information (MNPI) need information (MNPI) need information (MNPI) need to be hidden. There are to be hidden. There are to be hidden. There are many different many different many different strategies for doing this.

strategies for doing this. strategies for doing this. You can You can You can hide hide hide information at the information at the information at the SDK level, at the gateway level, SDK level, at the gateway level, SDK level, at the gateway level, using various using various using various middleware middleware middleware . In . In . In practice, in the practice, in the user interface it user interface it looks like this: some of the looks like this: some of the customer information or customer information or confidential data is confidential data is confidential data is hidden. But what hidden. But what hidden. But what about the judges ( about the judges ( LLM-judge) you LLM-judge) you LLM-judge) you use? I use? I use? I encourage all of you encourage all of you encourage all of you to review the system to review the system to review the system queries for your LLM queries for your LLM judges and check judges and check judges and check the traces to see if the traces to see if the traces to see if they contain they contain any PII or MNPI along with the judge logic. Here is a any PII or MNPI along with the judge logic. Here is a system query for a system query for a system query for a simple judge. It simple judge. It simple judge. It provides an assessment and provides an assessment and provides an assessment and explanation of explanation of explanation of how well the how well the how well the agent performed on the agent performed on the task. But there task. But there is one line in this query is one line in this query is one line in this query that seems that seems that seems innocent, because you innocent, because you innocent, because you want a human want a human reviewer to be able reviewer to be able reviewer to be able to check whether the agent to check whether the agent to check whether the agent worked well and accurately worked well and accurately worked well and accurately . This . This . This could potentially allow a could potentially allow a could potentially allow a judge to extract and judge to extract and judge to extract and disclose PII or MNPI. disclose PII or MNPI.

So, your front end So, your front end still looks still looks still looks secure, but on the secure, but on the secure, but on the backend, the LLM- backend, the LLM- judge logic could reveal judge logic could reveal judge logic could reveal sensitive sensitive customer information. So one customer information. So one strategy is that strategy is that strategy is that your judge your judge your judge can read the can read the can read the raw raw raw trace data, but trace data, but trace data, but that that that judge's reasoning is not judge's reasoning is not judge's reasoning is not preserved. The rating, preserved. The rating, preserved. The rating, however, you want to however, you want to however, you want to leave, it is leave, it is leave, it is useful for useful for useful for evaluation, so it is evaluation, so it is evaluation, so it is always always always displayed in the displayed in the displayed in the interface. But in this interface. But in this interface. But in this other branch of other branch of other branch of reasoning, as we have reasoning, as we have reasoning, as we have seen, it can still seen, it can still seen, it can still lead to the leakage of lead to the leakage of lead to the leakage of personal data. personal data. personal data. However, However, However, you can you can you can add a check in your application code that add a check in your application code that add a check in your application code that tells the agent that tells the agent that tells the agent that this record does not contain this record does not contain this record does not contain sensitive sensitive sensitive information and information and information and can therefore be stored can therefore be stored can therefore be stored in the interface. So in the interface. So in the interface. So now you have an assessment now you have an assessment now you have an assessment and rationale for and rationale for and rationale for this entry. And if this entry. And if this entry. And if this check this check this check fails fails fails and the information is still and the information is still and the information is still confidential, you confidential, you confidential, you simply hide this simply hide this simply hide this justification in the justification in the justification in the interface. interface. interface. Most of the banks Most of the banks Most of the banks we we we work with want to work with want to work with want to see LLMs in the role of see LLMs in the role of see LLMs in the role of judges analyzing this judges analyzing this judges analyzing this confidential confidential confidential information. information. An additional An additional improvement that improvement that improvement that can be added to this can be added to this can be added to this process is process is process is to use a second to use a second to use a second LLM as a referee, where the input LLM as a referee, where the input LLM as a referee, where the input data is cleaned of data is cleaned of data is cleaned of sensitive sensitive sensitive information, so you information, so you information, so you can be sure can be sure can be sure that the referee's output data that the referee's output data that the referee's output data is clean and can be is clean and can be is clean and can be displayed in the displayed in the displayed in the interface. The main interface. The main interface. The main takeaway here is takeaway here is takeaway here is that that that data integrity data integrity data integrity and data sovereignty are and data sovereignty are and data sovereignty are not free. In not free. In not free. In this case, it is this case, it is this case, it is the cost of making the cost of making the cost of making another request to LLM another request to LLM another request to LLM for the same for the same for the same task. So, one

task. So, one task. So, one of the ways we of the ways we of the ways we build trust with build trust with build trust with security and security and security and compliance teams is not compliance teams is not compliance teams is not just saying, " just saying, " Trust us, here's Trust us, here's Trust us, here's the answer." We the answer." We the answer." We must provide must provide must provide evidence and work that evidence and work that evidence and work that demonstrates how we demonstrates how we demonstrates how we arrived at this arrived at this arrived at this answer. And many answer. And many answer. And many vendors vendors vendors will tell you, "you know, will tell you, "you know, will tell you, "you know, our production deployment time our production deployment time our production deployment time is is is a few weeks." But a few weeks." But a few weeks." But what they don't say is that what they don't say is that what they don't say is that they have they have they have n't checked this time n't checked this time n't checked this time against your against your against your infrastructure, infrastructure, infrastructure, the state of your security, the state of your security, the state of your security, and the nuances of your and the nuances of your and the nuances of your specific cluster specific cluster . So some of the signals . So some of the signals . So some of the signals we look for to we look for to we look for to reduce time reduce time reduce time to production to production are having are having are having a lot of a lot of processes set up, like processes set up, like loading images loading images loading images from the registry and from the registry and from the registry and having GPUs ready to go having GPUs ready to go having GPUs ready to go to to to perform inference. perform inference. This schedule This schedule is pushed back a is pushed back a is pushed back a few months as few months as few months as you conduct you conduct you conduct formal formal formal security audits or build a security audits or build a security audits or build a custom LLM gateway custom LLM gateway custom LLM gateway or configure RBAC. And or configure RBAC. And or configure RBAC. And that timeline that timeline that timeline is pushed back even further is pushed back even further is pushed back even further when you when you when you don't even have an don't even have an don't even have an approval process for approval process for approval process for uploading images uploading images uploading images to your registry and no to your registry and no to your registry and no clear plan for clear plan for clear plan for sourcing and purchasing sourcing and purchasing sourcing and purchasing GPUs. I have noticed that GPUs. I have noticed that GPUs. I have noticed that most of these most of these most of these factors that factors that factors that delay the schedule are delay the schedule are delay the schedule are less technical and less technical and less technical and more related to more related to more related to internal internal internal human and human and human and organizational organizational organizational challenges and challenges and challenges and alignments within the alignments within the alignments within the banks. But these are still banks. But these are still banks. But these are still important factors important factors important factors that need to be that need to be that need to be addressed. So,

addressed. So, addressed. So, today's topic is today's topic is showcasing your showcasing your showcasing your work. This is a real work. This is a real work. This is a real lever for influencing lever for influencing lever for influencing the speed of compliance with the speed of compliance with the speed of compliance with requirements when working with the requirements when working with the requirements when working with the bank. And when our bank. And when our bank. And when our banks, when one of banks, when one of banks, when one of our customers our customers our customers comes to us and comes to us and comes to us and says, "Hey, our says, "Hey, our says, "Hey, our customers are having customers are having customers are having problems." These problems." These problems." These problems become problems become problems become my my my team's problems, and we team's problems, and we team's problems, and we break them down into break them down into break them down into engineering, engineering, engineering, workflow, and workflow, and workflow, and people and people and people and organizational organizational organizational areas. The goal is areas. The goal is to automate to automate to automate as much as possible, but as much as possible, but as much as possible, but these are the issues that keep these are the issues that keep these are the issues that keep me up me up me up at night, and I want to at night, and I want to at night, and I want to ensure that ensure that ensure that our competitors don't our competitors don't outwork or outsmart us in outwork or outsmart us in any of these any of these any of these areas. I would say areas. I would say areas. I would say that you don't want to be that you don't want to be that you don't want to be just another just another just another supplier, you supplier, you supplier, you want to actually want to actually want to actually solve solve solve your customers' problems. your customers' problems. The biggest lesson The biggest lesson I learned from working with I learned from working with I learned from working with these banks and these banks and these banks and security teams security teams security teams is that there is a is that there is a lot at stake. Current lot at stake. Current accounts, mortgages, accounts, mortgages, accounts, mortgages, credit cards. In credit cards. In credit cards. In such situations, it is such situations, it is such situations, it is important to get important to get important to get answers accurate answers accurate answers accurate to the fourth decimal place to the fourth decimal place to the fourth decimal place . And there's a . And there's a . And there's a reason why a lot of reason why a lot of reason why a lot of the the the software in these software in these software in these banks is old and banks is old and banks is old and outdated. But the line outdated. But the line outdated. But the line between legacy between legacy software, software, old banking old banking old banking processes, and processes, and processes, and innovation is innovation is innovation is shifting every day, and shifting every day, and shifting every day, and it's thanks to all of you.

it's thanks to all of you. So I think you're all So I think you're all exploring different exploring different exploring different agent frameworks, agent frameworks, agent frameworks, building with new building with new building with new tools, and tools, and tools, and experimenting with GenAI experimenting with GenAI . And I would say, for those . And I would say, for those . And I would say, for those few who few who few who are concerned with the are concerned with the are concerned with the slow, boring, slow, boring, slow, boring, bureaucratic, bureaucratic, bureaucratic, methodical, methodical, methodical, confusing confusing confusing human and human and human and organizational organizational organizational challenges in banking, challenges in banking, challenges in banking, there is nothing better there is nothing better there is nothing better than implementing AI in the than implementing AI in the than implementing AI in the financial sector, financial sector, financial sector, because the stakes are because the stakes are because the stakes are too high. And yes, too high. And yes, too high. And yes, if you want if you want if you want to evaluate your own to evaluate your own to evaluate your own implementations against these implementations against these implementations against these pillars, just pillars, just pillars, just contact us. contact us. I've I've only discussed three pillars today, but, only discussed three pillars today, but, only discussed three pillars today, but, you know, there's a whole you know, there's a whole you know, there's a whole production checklist production checklist production checklist that we that we that we use that use that use that underpins underpins underpins these pillars. these pillars. Going back to my Going back to my math test in math test in math test in 10th grade, one of the 10th grade, one of the 10th grade, one of the best parts best parts best parts after the test was when my after the test was when my after the test was when my friends and I would go friends and I would go friends and I would go to the dinner table, to the dinner table, to the dinner table, try to write down try to write down try to write down all the problems we all the problems we all the problems we remembered from the test, and remembered from the test, and remembered from the test, and then we would discuss, " then we would discuss, " Hey, this is how we Hey, this is how we Hey, this is how we approached approached approached this problem or this problem or this problem or that problem." This that problem." This that problem." This allowed me allowed me allowed me to think about to think about to think about different approaches and different approaches and different approaches and ways to solve ways to solve ways to solve these problems.

these problems. Conferences like this are like Conferences like this are like that lunch that lunch that lunch table when I was in 10th table when I was in 10th table when I was in 10th grade, because I want grade, because I want grade, because I want to learn from all of you to learn from all of you to learn from all of you how you solve how you solve how you solve your your your customers' problems. Thank you very much customers' problems. Thank you very much customers' problems. Thank you very much .