1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life

A big part of my A big part of my research is, um, research is, um, research is, um, studying AI failures, studying AI failures, studying AI failures, investigating them, and investigating them, and investigating them, and then learning from then learning from then learning from them. them. them. Tanmey Tanmey Tanmey Shah joins me. Shah joins me. Shah joins me. Yes, hello everyone. I am Yes, hello everyone. I am Yes, hello everyone. I am Tanmey Shah. I am a senior Tanmey Shah. I am a senior Tanmey Shah. I am a senior quantitative modeler quantitative modeler quantitative modeler and AI researcher. This is my and AI researcher. This is my and AI researcher. This is my second time here, so it's second time here, so it's second time here, so it's great to be back. great to be back. great to be back. So I'll hand it over to So I'll hand it over to So I'll hand it over to Vishal so we can get Vishal so we can get Vishal so we can get started. started. So, we looked at the AI ​​incident database. This is a AI ​​incident database. This is a publicly available publicly available publicly available database of database of database of AI incidents. And, um, AI incidents. And, um, AI incidents. And, um, we looked at, um, we looked at, um, we looked at, um, AI failures and asked AI failures and asked AI failures and asked one simple question: one simple question: one simple question: What do these, um, failures, um, What do these, um, failures, um, What do these, um, failures, um, teach us about teach us about teach us about how to evaluate an how to evaluate an how to evaluate an AI system, and AI system, and AI system, and can we, um, can we, um, can we, um, apply that apply that apply that knowledge to evaluation at the knowledge to evaluation at the knowledge to evaluation at the system level? system level? So the first thing we So the first thing we see is that AI failures are see is that AI failures are see is that AI failures are , ahem, increasing , ahem, increasing , ahem, increasing exponentially. And exponentially. And exponentially. And if you look at the if you look at the , um, last four, , um, last four, , um, last four, um, years, um, of AI failures, um, years, um, of AI failures, um, years, um, of AI failures, especially in the era of especially in the era of especially in the era of fundamental fundamental fundamental models, you'll see models, you'll see models, you'll see that the jump is almost, um, that the jump is almost, um, that the jump is almost, um, vertical. Um, and vertical. Um, and vertical. Um, and many of us many of us many of us believe that, um, believe that, um, believe that, um, the number of the number of the number of AI use cases that AI use cases that AI use cases that we have in we have in we have in enterprises, um, enterprises, um, enterprises, um, they're, um, much they're, um, much they're, um, much larger than our larger than our larger than our ability to look at ability to look at ability to look at each one individually. Therefore, each one individually. Therefore, each one individually. Therefore, we argue that we argue that we argue that we need a we need a we need a structured structured structured way to investigate way to investigate way to investigate these AI failures. And for

these AI failures. And for these AI failures. And for that, um, we're that, um, we're that, um, we're looking at AI failures looking at AI failures looking at AI failures essentially in three essentially in three essentially in three dimensions. The first dimension dimensions. The first dimension is the development phase, is the development phase, is the development phase, when, um, the AI ​​model and/ when, um, the AI ​​model and/ or system or system or system is being developed. So, is being developed. So, is being developed. So, it could be it could be it could be issues with issues with issues with data quality, data quality, data quality, tool architecture, or tool architecture, or tool architecture, or system design. system design. Any of these things Any of these things in the development phase in the development phase in the development phase can go wrong, and can go wrong, and can go wrong, and once they once they once they do, those errors do, those errors do, those errors can spread to the can spread to the can spread to the next dimension, which next dimension, which next dimension, which is the deployment phase. is the deployment phase. Even if a model Even if a model or system has some or system has some or system has some design flaws, but design flaws, but design flaws, but if you have if you have if you have enough control enough control enough control during deployment, during deployment, during deployment, these errors can be these errors can be these errors can be contained. And that's why it's contained. And that's why it's contained. And that's why it's so important so important so important to understand what to understand what to understand what controls we have controls we have controls we have during deployment, and whether during deployment, and whether during deployment, and whether we can contain those we can contain those we can contain those failures as we get from the failures as we get from the failures as we get from the production production production deployment platform to the deployment platform to the deployment platform to the final stage, final stage, final stage, which is the context in which is the context in which is the context in which the AI ​​system which the AI ​​system which the AI ​​system is used from a is used from a is used from a business perspective.

business perspective. This can lead to This can lead to reputational risk reputational risk . This can lead to . This can lead to . This can lead to legal or legal or legal or financial risk, and financial risk, and financial risk, and this whole business this whole business context is really context is really context is really important for the important for the important for the AI ​​system in which it is AI ​​system in which it is AI ​​system in which it is used. And used. And used. And we propose that we propose that we propose that the model itself, or the the model itself, or the the model itself, or the technical power of the model, is technical power of the model, is only one only one only one part of the system. We part of the system. We part of the system. We must look at must look at must look at the system holistically. So the system holistically. So , after defining , after defining , after defining these three main these three main these three main dimensions: technical dimensions: technical dimensions: technical strength, where the system strength, where the system strength, where the system needs to be strong, needs to be strong, needs to be strong, or, conversely, where the or, conversely, where the or, conversely, where the system can system can system can break, the second break, the second break, the second dimension is how we dimension is how we dimension is how we ensure that there are ensure that there are deployment controls . And the third is the business . And the third is the business context in which the context in which the AI ​​system is used. We AI ​​system is used. We divide each of these divide each of these divide each of these main dimensions into main dimensions into main dimensions into several sub-dimensions. several sub-dimensions.

So, in terms of So, in terms of technical strength, technical strength, technical strength, right? There may be six right? There may be six right? There may be six main sub-dimensions main sub-dimensions main sub-dimensions that can affect the that can affect the that can affect the technical strength of an technical strength of an technical strength of an AI system. First, AI system. First, AI system. First, it may be it may be it may be related to the quality of the related to the quality of the related to the quality of the data we data we data we use to use to use to build the AI ​​system. build the AI ​​system. build the AI ​​system. How the system was How the system was How the system was designed for the designed for the designed for the appropriate appropriate appropriate task it is task it is task it is supposed to perform. supposed to perform. How is How is predictive validity predictive validity predictive validity or analysis of or analysis of AI system performance performed? Is AI system performance performed? Is this system working as we this system working as we this system working as we intended? Hmm, there may intended? Hmm, there may intended? Hmm, there may be checks be checks be checks related to related to related to fairness. fairness. Failure modes Failure modes essentially consider where, essentially consider where, essentially consider where, when, how, and for whom a when, how, and for whom a system can system can fail. So, fail. So, fail. So, essentially, it's a map of all the essentially, it's a map of all the essentially, it's a map of all the failure modes of the failure modes of the failure modes of the system. And finally, system. And finally, system. And finally, can we explain can we explain , explain decisions , explain decisions , explain decisions to regulators or to regulators or to regulators or other bodies so that we other bodies so that we other bodies so that we know how the system know how the system know how the system makes its decisions? makes its decisions? So, these six So, these six sub-dimensions are essentially sub-dimensions are essentially sub-dimensions are essentially part of the technical part of the technical part of the technical power of an power of an power of an artificial intelligence system. The artificial intelligence system. The next four next four essentially look at how essentially look at how essentially look at how we use we use we use the model. What the model. What the model. What powers have we powers have we powers have we given to AI, right? Is given to AI, right? Is AI there to AI there to inform us, or does inform us, or does inform us, or does it act on our it act on our it act on our behalf? What behalf? What decision-making powers do we decision-making powers do we attribute to AI?

attribute to AI? Secondly, human Secondly, human Secondly, human mediation. mediation. mediation. How effective is How effective is How effective is a person in the process of a a person in the process of a a person in the process of a cycle, right? This is cycle, right? This is cycle, right? This is really important. really important. really important. Third, Third, Third, influence control. Who actually influence control. Who actually influence control. Who actually uses the system uses the system ? Is it internal ? Is it internal ? Is it internal employees so employees so employees so they can see the error they can see the error they can see the error and contain it, and contain it, and contain it, or is it exposed to or is it exposed to external customers external customers so any small so any small so any small problem can problem can problem can spread quickly. spread quickly. spread quickly. And finally, what kind of And finally, what kind of And finally, what kind of monitoring, uh, monitoring, uh, monitoring, uh, control do we have so control do we have so control do we have so that we can see some that we can see some that we can see some problems in problems in problems in production, production, production, detect them and be able to detect them and be able to detect them and be able to contain them. Finally, contain them. Finally, contain them. Finally, at the financial at the financial at the financial impact or impact or impact or business impact stage, we see these business impact stage, we see these business impact stage, we see these three main three main three main sub-dimensions. What sub-dimensions. What sub-dimensions. What financial impact financial impact financial impact will the error have? will the error have? Could there be legal Could there be legal consequences and how many consequences and how many consequences and how many users will be users will be users will be affected? This is the affected? This is the affected? This is the last sub-dimension of last sub-dimension of last sub-dimension of the scale. So, we the scale. So, we the scale. So, we look at it as look at it as look at it as one framework that one framework that one framework that has 13 dimensions and three has 13 dimensions and three has 13 dimensions and three main questions, as main questions, as main questions, as we explained.

we explained. we explained. Once we define Once we define Once we define these 13 sub-dimensions and look these 13 sub-dimensions and look these 13 sub-dimensions and look back back back at at at AI incidents AI incidents AI incidents , we , we , we find that almost 98% of AI failures can be captured find that almost 98% of AI failures can be captured find that almost 98% of AI failures can be captured using these 13 using these 13 using these 13 sub-dimensions sub-dimensions . This means that . This means that AI failures are related to AI failures are related to AI failures are related to some of these dimensions some of these dimensions , in some cases to , in some cases to , in some cases to several of these several of these several of these dimensions, and in some dimensions, and in some dimensions, and in some cases to several of cases to several of cases to several of them. But these dimensions are them. But these dimensions are them. But these dimensions are involved. So, AI failures involved. So, AI failures involved. So, AI failures have structures. We have structures. We have structures. We need to need to need to reflect on them and act reflect on them and act reflect on them and act accordingly to these accordingly to these accordingly to these sub-dimensions. As I sub-dimensions. As I sub-dimensions. As I mentioned, some of these mentioned, some of these mentioned, some of these subdimensions of failures subdimensions of failures subdimensions of failures may be may be may be present in these present in these present in these failures simultaneously. This failures simultaneously. This failures simultaneously. This means that when something means that when something means that when something goes wrong with a particular goes wrong with a particular goes wrong with a particular failure, there may be failure, there may be failure, there may be several things that several things that several things that went wrong. We went wrong. We went wrong. We found that, on found that, on found that, on average, there are six average, there are six average, there are six subdimensions of failures that subdimensions of failures that subdimensions of failures that may be may be may be present in each present in each present in each of these of these of these AI failure incidents we AI failure incidents we AI failure incidents we investigated. Now investigated. Now the question that comes to mind is: are all these the question that comes to mind is: are all these sub-dimensions equal, or are sub-dimensions equal, or are sub-dimensions equal, or are some more important some more important some more important than others? And the answer than others? And the answer than others? And the answer is that is that is that some of the sub-dimensions some of the sub-dimensions some of the sub-dimensions may be more may be more may be more critical than others. And what critical than others. And what critical than others. And what I'm showing you I'm showing you I'm showing you here on the graph on the here on the graph on the here on the graph on the Y-axis is the relationship with Y-axis is the relationship with Y-axis is the relationship with harm when these harm when these harm when these sub-dimensions are present. And

sub-dimensions are present. And sub-dimensions are present. And on the X-axis we have on the X-axis we have on the X-axis we have the frequency of presence of the frequency of presence of the frequency of presence of these sub-dimensions. What these sub-dimensions. What these sub-dimensions. What we see in the upper left we see in the upper left we see in the upper left corner of corner of corner of the graph are the graph are the graph are sub-dimensions such as human sub-dimensions such as human sub-dimensions such as human agency or agency or agency or human efficiency human efficiency human efficiency in the cycle, in the cycle, in the cycle, harm detection, i.e. do we have harm detection, i.e. do we have monitoring control. On the X-axis, monitoring control. On the X-axis, we see things like we see things like we see things like the scale and design of the the scale and design of the the scale and design of the system. Although they are system. Although they are system. Although they are present in many present in many present in many cases, they are not as cases, they are not as cases, they are not as strongly associated with strongly associated with strongly associated with harm as some of the harm as some of the harm as some of the previous previous previous sub-dimensions I sub-dimensions I sub-dimensions I mentioned. So when mentioned. So when mentioned. So when we really we really we really look at look at look at artificial intelligence systems, artificial intelligence systems, artificial intelligence systems, some of these some of these some of these sub-dimensions can sub-dimensions can sub-dimensions can be really be really be really critical. Therefore, we critical. Therefore, we critical. Therefore, we need to need to need to prioritize these prioritize these prioritize these subdimensions by subdimensions by subdimensions by potential for potential for potential for harm and then harm and then harm and then allocate appropriate allocate appropriate allocate appropriate resources to resources to resources to research these research these research these subdimensions, rather than subdimensions, rather than subdimensions, rather than treating all treating all treating all failure subdimensions failure subdimensions failure subdimensions equally. And when we equally. And when we equally. And when we combine these combine these combine these sub-dimensions together, we sub-dimensions together, we sub-dimensions together, we see that the see that the see that the controls we controls we controls we have during have during have during deployment are more deployment are more deployment are more critically related to critically related to critically related to damage, followed by damage, followed by damage, followed by business impact, and business impact, and business impact, and then technical failure.

then technical failure. So this graph So this graph essentially tells us that essentially tells us that essentially tells us that once we take care once we take care once we take care of of of deployment controls for a deployment controls for a deployment controls for a high- high- high- risk system, it gives us a risk system, it gives us a risk system, it gives us a certain level of certain level of certain level of confidence that confidence that confidence that we can take care we can take care we can take care of failures in advance, of failures in advance, of failures in advance, as opposed to as opposed to as opposed to just considering a just considering a just considering a technical failure. So technical failure. So , all of these sub-dimensions are , all of these sub-dimensions are , all of these sub-dimensions are important, but important, but important, but deployment controls are deployment controls are deployment controls are much more important. And much more important. And in the next step, in the next step, we developed a we developed a we developed a materiality scale, meaning materiality scale, meaning materiality scale, meaning if we can if we can if we can understand which understand which AI use cases are AI use cases are riskier riskier riskier compared to others, compared to others, compared to others, we can we can we can prioritize these prioritize these AI use cases AI use cases based on materiality, based on materiality, based on materiality, because materiality because materiality because materiality directly ranks the directly ranks the directly ranks the harm harm harm associated with these associated with these AI use cases. This AI use cases. This means that we can means that we can means that we can spend, spend, spend, let's say, knowing let's say, knowing let's say, knowing which which which AI use cases are AI use cases are AI use cases are more significant, then more significant, then more significant, then we can allocate we can allocate we can allocate more resources to more resources to more resources to much more significant much more significant AI use cases than compared to AI use cases than compared to some other some other AI use cases that AI use cases that may not be as may not be as may not be as significant. With that, I would significant. With that, I would significant. With that, I would hand over to Tanmei, hand over to Tanmei, hand over to Tanmei, who would explain how we who would explain how we who would explain how we can can can use this use this use this knowledge to build knowledge to build knowledge to build assessments at a system assessments at a system assessments at a system level.

level. level. Thank you, Vishal. So, Thank you, Vishal. So, Thank you, Vishal. So, as far as we've seen, as far as we've seen, as far as we've seen, what these failures teach us what these failures teach us what these failures teach us , so now , so now , so now let's use let's use let's use these lessons to build these lessons to build these lessons to build assessments at the system assessments at the system assessments at the system level. So, first, level. So, first, level. So, first, we broke down risk into we broke down risk into we broke down risk into four components: how four components: how four components: how often the model fails often the model fails , whether that failure , whether that failure , whether that failure reaches a real reaches a real reaches a real solution, whether it solution, whether it solution, whether it causes or causes or causes or goes goes goes undetected, and undetected, and undetected, and how serious the how serious the how serious the consequences are. So, in consequences are. So, in consequences are. So, in this scenario, this scenario, deployment setup is important. What deployment setup is important. What level of authority does level of authority does level of authority does the system have? Does anyone the system have? Does anyone the system have? Does anyone check her source check her source check her source data? Can data? Can data? Can monitoring detect monitoring detect monitoring detect errors before errors before errors before they cause damage? they cause damage? So, this same model So, this same model can create can create can create different risks depending different risks depending different risks depending on how we on how we on how we deploy it. So, we deploy it. So, we deploy it. So, we mean that mean that mean that if the risk depends if the risk depends if the risk depends on the entire on the entire on the entire deployment system, our deployment system, our deployment system, our assessment should assessment should assessment should cover this setting as well cover this setting as well cover this setting as well . This . This . This brings us to the brings us to the brings us to the next question.

next question. next question. What should we test What should we test What should we test in an agent system? in an agent system? You see, in You see, in traditional traditional traditional machine learning, machine learning, machine learning, we already know how to we already know how to we already know how to challenge models. challenge models. challenge models. We test their We test their We test their assumptions, inputs, assumptions, inputs, assumptions, inputs, and behavior under and behavior under and behavior under stress. Now that stress. Now that stress. Now that we extend this we extend this we extend this discipline to the entire discipline to the entire discipline to the entire deployment system, for deployment system, for deployment system, for example, does example, does example, does the search return the the search return the the search return the correct information correct information ? Or what happens ? Or what happens ? Or what happens when a tool when a tool when a tool fails? Does fails? Does fails? Does the person detect a the person detect a the person detect a mistake? Does the agent mistake? Does the agent mistake? Does the agent stay within stay within stay within its permissions or limits its permissions or limits ? And can the system ? And can the system ? And can the system finally recover? finally recover? finally recover? So, the logic here So, the logic here So, the logic here remains remains remains unchanged. The boundaries of what unchanged. The boundaries of what unchanged. The boundaries of what we test are we test are we test are expanding. So, expanding. So, expanding. So, now the question now the question now the question is, how is, how is, how should we test an should we test an should we test an agent system? So agent system? So , on the right, we have the , on the right, we have the , on the right, we have the familiar traditional familiar traditional familiar traditional machine learning. We machine learning. We machine learning. We test the model on test the model on test the model on hidden data, hidden data, hidden data, calculate calculate calculate metrics such as metrics such as metrics such as accuracy and AUC ROC, and accuracy and AUC ROC, and accuracy and AUC ROC, and use this use this use this evidence to evidence to evidence to justify justify justify deployment. But we deployment. But we deployment. But we still need to still need to still need to verify the data and verify the data and verify the data and assumptions assumptions assumptions underlying these underlying these underlying these figures. But on figures. But on figures. But on the left we have an the left we have an the left we have an agent system, which agent system, which agent system, which adds new adds new adds new complexity. So, a complexity. So, a complexity. So, a deployment system deployment system deployment system has many possible has many possible has many possible inputs, inputs, tool responses, and tool responses, and sequences of actions, sequences of actions, sequences of actions, or trajectories, or or trajectories, or or trajectories, or deployments, whatever you deployments, whatever you deployments, whatever you describe. So, we describe. So, we describe. So, we test the selected test the selected test the selected scenarios. Sometimes another scenarios. Sometimes another scenarios. Sometimes another AI, er, agent, or AI AI, er, agent, or AI judge generates these judge generates these judge generates these tests or evaluates the tests or evaluates the tests or evaluates the result. So, this

result. So, this result. So, this judge could be judge could be judge could be wrong, and the test wrong, and the test wrong, and the test could miss could miss could miss important failures. So, this is where important failures. So, this is where important failures. So, this is where we need to we need to we need to check whether check whether check whether the estimator is reliable, what the estimator is reliable, what the estimator is reliable, what the evidence covers, and the evidence covers, and the evidence covers, and whether whether whether it applies to the system it applies to the system it applies to the system we are actually we are actually we are actually deploying. That's why deploying. That's why deploying. That's why the evaluator is part of the the evaluator is part of the the evaluator is part of the system we system we system we manage. So now I'll manage. So now I'll manage. So now I'll turn it over to Vishal to turn it over to Vishal to turn it over to Vishal to explain how we explain how we explain how we can check if can check if can check if an assessment is an assessment is an assessment is reliable, and what to do reliable, and what to do reliable, and what to do when it's not. when it's not. Thank you, Tanmei. So Thank you, Tanmei. So we saw evaluation we saw evaluation we saw evaluation at the system level at the system level at the system level for the evaluator and for the evaluator and for the evaluator and also, you know, for the also, you know, for the also, you know, for the entire AI system. entire AI system. The question is The question is , how should we , how should we , how should we actually... And when I was actually... And when I was actually... And when I was looking at some of the looking at some of the looking at some of the previous previous previous research, research, research, especially in the field of especially in the field of especially in the field of cognitive cognitive cognitive neuroscience, I found neuroscience, I found neuroscience, I found that, um, this scientist that, um, this scientist that, um, this scientist Peter Weison, he's the one Peter Weison, he's the one Peter Weison, he's the one who coined this term who coined this term confirmation bias. And, confirmation bias. And, essentially, he did a bunch of essentially, he did a bunch of essentially, he did a bunch of experiments, but experiments, but experiments, but essentially, um, in this essentially, um, in this essentially, um, in this case, the series that you case, the series that you case, the series that you see is 2 4 6, right? He see is 2 4 6, right? He see is 2 4 6, right? He asked people asked people asked people to predict the next to predict the next to predict the next number, and most often number, and most often number, and most often they predicted 8 10, they predicted 8 10, they predicted 8 10, for example, even for example, even for example, even numbers. But, um, those numbers. But, um, those numbers. But, um, those answers were answers were answers were wrong wrong wrong because, essentially, the because, essentially, the because, essentially, the correct number correct number correct number was any was any was any increasing number.

increasing number. increasing number. Essentially, this proves that Essentially, this proves that Essentially, this proves that all people, like all people, like all people, like most people, um, in most people, um, in most people, um, in this case 79% of people this case 79% of people who tested who tested who tested wrong, um, wrong, um, wrong, um, may have some kind of may have some kind of confirmation bias. And confirmation bias. And AI systems, as we AI systems, as we AI systems, as we know, um, know, um, know, um, learn from learn from learn from human-generated data human-generated data . So, he . So, he . So, he suggested that suggested that any assessment or any assessment or any evaluation that any evaluation that any evaluation that needs to be done needs to be done needs to be done should be refuting, should be refuting, should be refuting, not confirming. And not confirming. And not confirming. And this is exactly the way this is exactly the way this is exactly the way it should be done. And it should be done. And it should be done. And we take this lesson from we take this lesson from we take this lesson from his research and his research and his research and propose that the propose that the propose that the evaluation system should evaluation system should evaluation system should be be be failure-oriented. So, if failure-oriented. So, if failure-oriented. So, if evaluation systems are evaluation systems are failure-oriented, what would that failure-oriented, what would that mean? This mean? This mean? This would mean that you would mean that you would mean that you would start by looking at would start by looking at would start by looking at your system's statements, your system's statements, your system's statements, testing it and testing it and testing it and trying to break it, and trying to break it, and trying to break it, and then identifying where the then identifying where the then identifying where the system's statements system's statements system's statements break, right? break, right? And what are you doing to And what are you doing to fix it? Either you, uh fix it? Either you, uh , don't let the , don't let the , don't let the AI ​​system work, AI ​​system work, AI ​​system work, you know, in an area you know, in an area you know, in an area where, uh, there are glitches, or you where, uh, there are glitches, or you where, uh, there are glitches, or you come up with, uh, come up with, uh, come up with, uh, backup systems, right backup systems, right backup systems, right ? Which ones are you ? Which ones are you ? Which ones are you sure about? And you sure about? And you sure about? And you keep doing keep doing keep doing it iteratively, uh, it iteratively, uh, it iteratively, uh, until you reach a until you reach a until you reach a level of confidence, level of confidence, level of confidence, right? This is, in fact, the right? This is, in fact, the right? This is, in fact, the first first principles-based approach we principles-based approach we propose for propose for any assessment system.

any assessment system. So, essentially, you So, essentially, you take your take your take your AI systems, you do this AI systems, you do this AI systems, you do this build-break- build-break- fix cycle, fix cycle, fix cycle, you understand where the you understand where the you understand where the AI ​​systems can be AI ​​systems can be AI ​​systems can be trusted in terms of trusted in terms of trusted in terms of their, uh, end their, uh, end their, uh, end results, and you results, and you results, and you map out, uh, map out, uh, map out, uh, uncertain areas uncertain areas uncertain areas and high- and high- and high- risk areas where you can't risk areas where you can't risk areas where you can't be sure be sure be sure of the of the of the system's performance, and you system's performance, and you system's performance, and you map out the whole map out the whole map out the whole failure map and failure map and failure map and route it route it route it based on reliability and based on reliability and based on reliability and materiality, which materiality, which materiality, which means that especially means that especially means that especially if you're if you're if you're using AI using AI using AI for any system for any system for any system with a high with a high with a high risk rating, risk rating, risk rating, make sure that your make sure that your make sure that your AI is AI is AI is only used for, uh, that only used for, uh, that only used for, uh, that area that you're area that you're area that you're confident in. And with that, I'll confident in. And with that, I'll confident in. And with that, I'll hand it over to Tanmei, whoever, hand it over to Tanmei, whoever, hand it over to Tanmei, whoever, whoever, whoever, whoever, whoever, whoever, whoever, applies this applies this applies this approach to a real, approach to a real, approach to a real, uh, AI failure case uh, AI failure case uh, AI failure case . . Okay, uh, yeah. So, uh , yeah, uh, let's , yeah, uh, let's , yeah, uh, let's make this framework make this framework make this framework concrete and concrete and concrete and see how a narrow see how a narrow see how a narrow assessment can assessment can assessment can miss failures, uh, miss failures, uh, miss failures, uh, that are important in the that are important in the that are important in the real world. So real world. So , I'll quickly tell you , I'll quickly tell you , I'll quickly tell you the story behind the story behind the story behind this. So, the client this. So, the client this. So, the client needed to needed to needed to travel after a travel after a travel after a death in the family. He death in the family. He death in the family. He asked asked asked the airline's chatbot about a the airline's chatbot about a the airline's chatbot about a discounted fare. The discounted fare. The chatbot said he chatbot said he chatbot said he could buy a could buy a could buy a regular ticket and regular ticket and regular ticket and get a discount get a discount get a discount later within 90 later within 90 later within 90 days. So, he days. So, he days. So, he relied on that relied on that relied on that advice, but advice, but advice, but the airline later the airline later the airline later refused because refused because refused because its policy, uh, didn't its policy, uh, didn't its policy, uh, didn't allow that

allow that allow that request. Ultimately, request. Ultimately, request. Ultimately, the tribunal found the tribunal found the tribunal found the airline the airline the airline liable. So, liable. So, liable. So, in this example we in this example we in this example we see a chain of see a chain of see a chain of failures. So, it's failures. So, it's failures. So, it's crystal clear crystal clear . So, the wrong . So, the wrong . So, the wrong advice reaches advice reaches advice reaches the client, the client acts the client, the client acts the client, the client acts on it, and on it, and on it, and the organization the organization the organization faces the faces the faces the consequences. So consequences. So consequences. So if we take if we take if we take our "build, our "build, our "build, break, fix, and break, fix, and break, fix, and break again" approach, uh, it break again" approach, uh, it break again" approach, uh, it could detect the failure could detect the failure could detect the failure before deployment. before deployment. before deployment. So, ideally, we would So, ideally, we would So, ideally, we would start with a clear start with a clear start with a clear requirement, for example, that requirement, for example, that requirement, for example, that advice must advice must advice must comply with comply with comply with approved approved approved policy. Then we would policy. Then we would policy. Then we would ask the chatbot a ask the chatbot a ask the chatbot a question about question about question about refunds refunds refunds after the trip. If after the trip. If after the trip. If these tests revealed these tests revealed these tests revealed incorrect advice, incorrect advice, incorrect advice, then we could then we could then we could add add add policy review or policy review or human escalation. human escalation. Then we would Then we would Then we would test again to test again to test again to see if see if see if these controls caught the these controls caught the these controls caught the errors before errors before errors before they reached the customer. they reached the customer. they reached the customer. So, this is our So, this is our So, this is our practical structure. practical structure. practical structure. We must identify We must identify We must identify the claim, actively the claim, actively the claim, actively challenge it, and challenge it, and challenge it, and use use use the evidence to make the evidence to make the evidence to make changes and make a changes and make a deployment decision. So deployment decision. So finally, I'll hand it over to finally, I'll hand it over to finally, I'll hand it over to Vishal to put Vishal to put Vishal to put all the pieces together and come to a all the pieces together and come to a all the pieces together and come to a conclusion.

conclusion. Uh, this is our last Uh, this is our last slide. So, essentially, what slide. So, essentially, what slide. So, essentially, what we mean is that we mean is that we mean is that your assessments should be your assessments should be your assessments should be based on based on based on risk, not risk, not risk, not frequency. You should frequency. You should frequency. You should consider your entire consider your entire consider your entire system and then system and then system and then break it down into major break it down into major break it down into major components such as components such as components such as technical, er, failures, technical, er, failures, deployment controls, and deployment controls, and business context. And business context. And business context. And then assess then assess then assess the materiality of your the materiality of your AI use cases with a AI use cases with a certain, uh, scale certain, uh, scale , uh, and that could be the , uh, and that could be the , uh, and that could be the scale of risk scale of risk scale of risk specific to specific to specific to your company, if your company, if your company, if that risk could that risk could that risk could change. And change. And once you do that, once you do that, once you do that, apply system- apply system- level stress testing level stress testing level stress testing to the to the to the AI ​​use case. And AI ​​use case. And AI ​​use case. And once you are once you are once you are confident about confident about confident about the region, then the region, then the region, then make a decision make a decision make a decision about deployment. about deployment. about deployment. Otherwise, it Otherwise, it Otherwise, it may be too may be too may be too expensive. Our expensive. Our expensive. Our final conclusion: final conclusion: final conclusion: evaluate the entire evaluate the entire evaluate the entire AI system based on risk, AI system based on risk, AI system based on risk, not just one not just one not just one component. component. component. Thank you very much for your time and Thank you very much for your time and Thank you very much for your time and attention. You can attention. You can attention. You can contact us contact us contact us externally. We will be happy externally. We will be happy externally. We will be happy to answer to answer any questions you may have any questions you may have any questions you may have later.