Rebuilding CI for AI-Scale Commit Volume — Gautam Korlam, Sonar

Hello everyone. Congratulations on Hello everyone. Congratulations on my report. my report. my report. Today we'll Today we'll Today we'll talk about how talk about how talk about how we're rebuilding CI we're rebuilding CI we're rebuilding CI for AI-scale commit volumes for AI-scale commit volumes for AI-scale commit volumes . This is . This is . This is based on based on based on years of years of years of experience working with experience working with build and merge infrastructure, build and merge infrastructure, which I have been involved with for which I have been involved with for which I have been involved with for over a decade. I over a decade. I over a decade. I want to introduce you want to introduce you want to introduce you to the new “ to the new “ to the new “ bottlenecks” we are bottlenecks” we are bottlenecks” we are seeing in seeing in seeing in software production, and in particular, how does this software production, and in particular, how does this software production, and in particular, how does this apply to banks and apply to banks and apply to banks and highly highly highly regulated industries? How do we regulated industries? How do we regulated industries? How do we stay stay stay compliant compliant compliant and and and auditable while maintaining auditable while maintaining auditable while maintaining high high high speed? So let's speed? So let's speed? So let's begin. My name is begin. My name is begin. My name is Gautam. I am the Gautam. I am the Gautam. I am the co-founder and co-founder and co-founder and CTO of CTO of CTO of Gitar, and recently Gitar, and recently joined Sonar as a joined Sonar as a lead engineer following an acquisition. Before lead engineer following an acquisition. Before lead engineer following an acquisition. Before Gitar, I Gitar, I Gitar, I worked at Uber for almost a decade worked at Uber for almost a decade worked at Uber for almost a decade on on on build and merge infrastructure for build and merge infrastructure for build and merge infrastructure for monorepositories. monorepositories. monorepositories. That is, everything: from build That is, everything: from build , CI/CD, IDE—you , CI/CD, IDE—you , CI/CD, IDE—you understand, to understand, to understand, to ensure ensure developer productivity. And now developer productivity. And now in Gitar as part of Sonar, I in Gitar as part of Sonar, I in Gitar as part of Sonar, I try to try to try to make sure that the code make sure that the code make sure that the code that is committed that is committed that is committed is tested, and all the is tested, and all the is tested, and all the code written by code written by code written by agents is of high agents is of high agents is of high quality before quality before quality before integration. So integration. So integration. So before we begin, I want to before we begin, I want to before we begin, I want to outline a little outline a little outline a little historical context.

historical context. historical context. At Uber, we had At Uber, we had At Uber, we had AI AI agent-scale commit volumes before the agent-scale commit volumes before the agent-scale commit volumes before the agents themselves existed. This agents themselves existed. This agents themselves existed. This means that means that means that we had one we had one we had one huge huge huge monorepository and monorepository and monorepository and made thousands of changes made thousands of changes made thousands of changes every day. With very every day. With very every day. With very long builds, long builds, long builds, this meant that if this meant that if this meant that if we were to process everything we were to process everything we were to process everything sequentially one by one sequentially one by one sequentially one by one so as not to break the so as not to break the so as not to break the master branch, it would master branch, it would master branch, it would take too take too take too long—up to 20 long—up to 20 long—up to 20 days—to days—to days—to make sure everything was make sure everything was make sure everything was working correctly. And working correctly. And working correctly. And if we skipped if we skipped if we skipped everything and let everyone everything and let everyone everything and let everyone merge their changes one merge their changes one merge their changes one by one, we would constantly by one, we would constantly by one, we would constantly see errors because see errors because see errors because the changes the changes the changes would conflict, and even if CI was would conflict, and even if CI was would conflict, and even if CI was successful for each one successful for each one successful for each one individually, together they individually, together they individually, together they could break everything. could break everything. could break everything. So our iOS So our iOS mono repository would be mono repository would be mono repository would be constantly " constantly " red", and it was red", and it was red", and it was not productive at all not productive at all not productive at all for for for our developers. our developers. To fix this, we To fix this, we created a system created a system created a system called Summit Queue, called Summit Queue, called Summit Queue, which we have a separate article about, and which we have a separate article about, and which we have a separate article about, and tested all tested all tested all possible variants of possible variants of possible variants of incoming commits incoming commits , reducing , reducing , reducing wait times through wait times through wait times through parallel parallel parallel testing. We testing. We testing. We could also predict what would could also predict what would could also predict what would break by break by break by testing all testing all testing all possible combinations.

possible combinations. possible combinations. So, as soon as we So, as soon as we So, as soon as we learn that a PR may learn that a PR may learn that a PR may cause a crash, we cause a crash, we cause a crash, we remove it from remove it from remove it from the queue and the queue and the queue and notify the author notify the author notify the author that they need to that they need to that they need to fix it before fix it before fix it before resubmitting resubmitting resubmitting after a rebase. after a rebase. after a rebase. Finally, to Finally, to Finally, to keep the keep the keep the trunk stable trunk stable , we , we , we only allow merges when only allow merges when only allow merges when all combinations of all combinations of all combinations of new commits new commits new commits pass validation. pass validation. pass validation. So it's worth So it's worth So it's worth making sure that the making sure that the making sure that the effort spent effort spent effort spent justifies the justifies the justifies the stability of our stability of our stability of our main branch, and that main branch, and that main branch, and that still works at Uber. I'm still works at Uber. I'm still works at Uber. I'm sure sure sure we have merge queues now, we have merge queues now, we have merge queues now, which GitHub which GitHub which GitHub and other code hosts have implemented and other code hosts have implemented , but there , but there , but there wasn't a system like that back then, so wasn't a system like that back then, so wasn't a system like that back then, so we developed one. The we developed one. The we developed one. The problem with this problem with this problem with this approach is approach is approach is that all of our that all of our that all of our queues, whether CI or queues, whether CI or queues, whether CI or code review, are a code review, are a code review, are a centralized " centralized " bottleneck" in the bottleneck" in the software development lifecycle. All of these software development lifecycle. All of these queues were queues were queues were designed for designed for designed for human speed, and human speed, and human speed, and when you build them when you build them when you build them for that speed, for that speed, for that speed, they can't they can't they can't handle it when handle it when handle it when agents show up. agents show up. Because people were making changes Because people were making changes several times an several times an several times an hour, we could hour, we could hour, we could do testing and do testing and do testing and validation one at a time. validation one at a time.

Now, with agents, Now, with agents, developers write developers write developers write several PRs a day, and several PRs a day, and several PRs a day, and they are not always willing to they are not always willing to they are not always willing to wait in line. They wait in line. They wait in line. They all want to commit all want to commit all want to commit at the same time, which at the same time, which at the same time, which means that the CI and review queues are means that the CI and review queues are piling up and piling up and under under under pressure. And after that, pressure. And after that, pressure. And after that, due to software factory templates due to software factory templates and and background agents background agents background agents running various running various running various workflows, workflows, workflows, suddenly everything starts suddenly everything starts suddenly everything starts to slow down a lot to slow down a lot , because the " , because the " , because the " bottlenecks" bottlenecks" bottlenecks" are still CI and are still CI and are still CI and verification, which in verification, which in verification, which in this case are this case are this case are overloaded. So what does overloaded. So what does overloaded. So what does this mean for this mean for this mean for modern teams? To modern teams? To modern teams? To put that into perspective: our put that into perspective: our put that into perspective: our own own own commit count has grown almost commit count has grown almost commit count has grown almost tenfold since tenfold since we started we started we started using GitHub using GitHub using GitHub about 3 years ago. about 3 years ago. about 3 years ago. And this is simply And this is simply And this is simply incredible with the incredible with the incredible with the same same same team size. We are still team size. We are still team size. We are still very small. But we are very small. But we are very small. But we are able to implement able to implement able to implement much more code. And much more code. And much more code. And we even see this in we even see this in we even see this in industry trends on industry trends on industry trends on GitHub. This graph GitHub. This graph GitHub. This graph shows the number of shows the number of shows the number of PRs that we PRs that we PRs that we believe are passing believe are passing GitHub review over time. If GitHub review over time. If you look at the you look at the you look at the growth trajectory growth trajectory , it's quite , it's quite , it's quite exponential.

exponential. exponential. Almost every month we Almost every month we Almost every month we have a 25% increase in the have a 25% increase in the have a 25% increase in the volume of PRs that volume of PRs that volume of PRs that need to be checked, need to be checked, need to be checked, fixed in CI, etc. fixed in CI, etc. I think this is I think this is supported by supported by supported by data from data from data from industry players like GitHub, which are industry players like GitHub, which are industry players like GitHub, which are also recording a also recording a also recording a stunning stunning stunning increase in the number of increase in the number of increase in the number of PRs. And this is what awaits PRs. And this is what awaits PRs. And this is what awaits every team now. If every team now. If every team now. If our small our small our small team sees this, I'm team sees this, I'm team sees this, I'm sure that large sure that large sure that large teams and banks teams and banks teams and banks face it face it face it much more as they much more as they much more as they go through their go through their go through their AI implementation journey. So AI implementation journey. So , the problem I was , the problem I was , the problem I was talking about, right? If talking about, right? If talking about, right? If agents open a PR agents open a PR agents open a PR and you force them and you force them and you force them to check everything through a to check everything through a to check everything through a CI system, individually CI system, individually CI system, individually they may pass, they may pass, they may pass, but when you but when you but when you merge them into the merge them into the merge them into the master branch, your master branch, your master branch, your code may still code may still code may still break. So, how to break. So, how to break. So, how to solve this? How solve this? How solve this? How to maintain order to maintain order to maintain order at such a high at such a high at such a high speed of work? Is speed of work? Is speed of work? Is n't that right? This is very n't that right? This is very n't that right? This is very important, similar to how important, similar to how important, similar to how we solve the we solve the we solve the queue problem. queue problem. queue problem. The changes themselves are not tested against each The changes themselves are not tested against each The changes themselves are not tested against each other. other. other. Therefore, changes are needed Therefore, changes are needed Therefore, changes are needed either in the CI system or in the either in the CI system or in the either in the CI system or in the code architecture and code architecture and code architecture and queues to make them queues to make them queues to make them smarter smarter smarter to address this to address this to address this issue. Another issue. Another issue. Another problem we problem we problem we see with everyone see with everyone see with everyone trying to trying to trying to implement “ implement “ software factories”—where software factories”—where agents write all the agents write all the agents write all the code to a specification code to a specification code to a specification and humans only and humans only and humans only test and review it— test and review it— is that the is that the modern production modern production modern production pipeline was pipeline was pipeline was designed for designed for designed for human speed.

human speed. This means that as This means that as agents make agents make agents make more changes, each more changes, each more changes, each part of the pipeline part of the pipeline part of the pipeline slows down. slows down. slows down. For example, during For example, during For example, during build and testing, build and testing, build and testing, you may have many you may have many you may have many re-builds re-builds re-builds due to due to due to compilation errors or compilation errors or compilation errors or integration tests. There integration tests. There integration tests. There may be may be may be unstable tests unstable tests unstable tests that cannot be that cannot be that cannot be reproduced locally, reproduced locally, reproduced locally, and it just and it just and it just slows everything down. Since you slows everything down. Since you slows everything down. Since you have to have to have to rerun them, rerun them, rerun them, this this this can take hours on complex projects. can take hours on complex projects. can take hours on complex projects. Add to this a Add to this a Add to this a review and review and review and approval stage. Many approval stage. Many approval stage. Many of you probably already of you probably already of you probably already use AI for use AI for code preview. code preview. I hope you don't I hope you don't I hope you don't do every review do every review do every review solely with solely with solely with human effort. Ideally, you human effort. Ideally, you human effort. Ideally, you should should should use AI use AI use AI as the first stage of as the first stage of as the first stage of verification. If you verification. If you verification. If you don't have such a system, don't have such a system, don't have such a system, reviewers become a reviewers become a reviewers become a bottleneck in the bottleneck in the bottleneck in the queue. Even after queue. Even after queue. Even after review, there may be review, there may be review, there may be additional criteria additional criteria additional criteria for full for full for full approval, because now approval, because now approval, because now in the industry many in the industry many in the industry many people simply people simply people simply automatically put automatically put automatically put "stamps" because they can't "stamps" because they can't "stamps" because they can't keep up with the large keep up with the large keep up with the large amount of PR. They amount of PR. They amount of PR. They rely on other rely on other rely on other signals to simply signals to simply signals to simply cope with the flow. cope with the flow. cope with the flow. So, the process has to So, the process has to So, the process has to change. Another change. Another change. Another problem we problem we problem we have now is that there are have now is that there are too too many such many such many such checks being performed. We checks being performed. We checks being performed. We keep a lot of keep a lot of keep a lot of records records records that expire.

that expire. that expire. So, your old logs So, your old logs So, your old logs disappear when disappear when their retention period expires. their retention period expires. There is no note about There is no note about what exactly was checked, what exactly was checked, what exactly was checked, there is only a stamp of there is only a stamp of there is only a stamp of approval. Therefore, approval. Therefore, approval. Therefore, some some some accounting of these things is needed for accounting of these things is needed for accounting of these things is needed for audits and compliance checks audits and compliance checks . Another . Another problem we problem we problem we see is see is see is that as the that as the that as the amount of amount of amount of code increases, the time to code increases, the time to code increases, the time to fix also fix also fix also creates a lot of creates a lot of creates a lot of overhead. On overhead. On overhead. On our platform, we our platform, we our platform, we see that at least see that at least see that at least 13% of PRs had recorded 13% of PRs had recorded 13% of PRs had recorded CI failures. This is one or CI failures. This is one or CI failures. This is one or more CI failures, which more CI failures, which more CI failures, which means a separate means a separate means a separate work cycle: return to the work cycle: return to the work cycle: return to the developer's machine developer's machine developer's machine or analyze the logs by or analyze the logs by or analyze the logs by the agent, find the agent, find the agent, find the cause of the failure, the cause of the failure, the cause of the failure, fix it, fix it, fix it, send a new send a new send a new commit, and iterate until commit, and iterate until commit, and iterate until success. And this is just CI, success. And this is just CI, success. And this is just CI, because usually because usually because usually the reviewer does not the reviewer does not the reviewer does not look at PR until CI look at PR until CI look at PR until CI becomes "green", because becomes "green", because becomes "green", because it is the first line of it is the first line of it is the first line of defense. This also defense. This also defense. This also slows down the slows down the slows down the review process because it can't review process because it can't review process because it can't happen in parallel— happen in parallel— we usually want to we usually want to we usually want to see the finished see the finished see the finished result. And every result. And every result. And every lab now lab now lab now pays for CI and pays for CI and pays for CI and verification. So, if verification. So, if verification. So, if comments come in comments come in comments come in for review and you for review and you for review and you do another do another do another iteration, you iteration, you iteration, you run CI again. And you run CI again. And you run CI again. And you seem to repeat this seem to repeat this seem to repeat this cycle. So how do we cycle. So how do we cycle. So how do we improve this? What can be done improve this? What can be done improve this? What can be done to make this to make this to make this problem manageable problem manageable problem manageable while agents while agents while agents generate more generate more generate more code? We found this, code? We found this, code? We found this, apparently, and on GitHub so far apparently, and on GitHub so far . The most obvious

. The most obvious is to make your CI is to make your CI is to make your CI really fast. This really fast. This really fast. This means that it is worth means that it is worth means that it is worth removing expensive removing expensive removing expensive checks, moving checks, moving checks, moving them to the post- them to the post- them to the post- merge stage, and making the merge stage, and making the merge stage, and making the CI step itself very fast CI step itself very fast CI step itself very fast so that fixes are more so that fixes are more so that fixes are more efficient. You efficient. You efficient. You can can can parallelize parallelize parallelize tasks, tasks, tasks, use use use sharding, and caching. sharding, and caching. sharding, and caching. We will also talk We will also talk We will also talk about some of the methods we about some of the methods we about some of the methods we used ourselves used ourselves . You also need to . You also need to . You also need to make sure that make sure that make sure that the verification is the verification is the verification is done quickly. done quickly. Not only Not only deep reviews are needed, but deep reviews are needed, but deep reviews are needed, but also multiple stages that also multiple stages that also multiple stages that give developers a give developers a give developers a quick signal to quick signal to quick signal to understand where understand where understand where deep deep deep analysis is needed and where to analysis is needed and where to entrust AI with approval entrust AI with approval entrust AI with approval and merging for simple and merging for simple and merging for simple changes. And for checks changes. And for checks changes. And for checks that are already running in CI, that are already running in CI, that are already running in CI, if you can if you can if you can shift them "to the left shift them "to the left ", it's much cheaper. ", it's much cheaper. ", it's much cheaper. And this is actually very And this is actually very And this is actually very important for important for important for the checks that the checks that the checks that occur even occur even occur even after CI. For example, after CI. For example, after CI. For example, you might have you might have you might have expensive regression expensive regression expensive regression tests or things that tests or things that tests or things that could fail in could fail in could fail in production. It is not production. It is not production. It is not always possible to always possible to always possible to move everything to the move everything to the move everything to the early stages, but if you early stages, but if you early stages, but if you can, you can, you can, you reduce the number of reduce the number of reduce the number of bottlenecks in bottlenecks in bottlenecks in the future. And the future. And the future. And locally, agents in locally, agents in locally, agents in any case have any case have any case have more context for more context for more context for solving the problems they solving the problems they solving the problems they face.

face. face. You also need to You also need to You also need to have policies in place have policies in place have policies in place . This is very . This is very . This is very important for important for important for regulated industries regulated industries . Here's the story from Backtrace: . Here's the story from Backtrace: . Here's the story from Backtrace: we lost our license we lost our license we lost our license to operate in London to operate in London to operate in London due to a software glitch due to a software glitch that that allowed drivers allowed drivers allowed drivers to get on the to get on the to get on the platform without all the platform without all the platform without all the proper checks. And proper checks. And proper checks. And this happened because of a this happened because of a this happened because of a change in the state of the change in the state of the change in the state of the function flag. So we function flag. So we function flag. So we had to implement a had to implement a had to implement a process where process where process where every change that every change that every change that affects London and affects London and affects London and goes through the SDLC goes through the SDLC goes through the SDLC needs to be needs to be needs to be validated validated validated like, "Okay, this is a like, "Okay, this is a like, "Okay, this is a change that affects change that affects change that affects London." If so, London." If so, London." If so, what control measures what control measures what control measures are in place? And we were able to get are in place? And we were able to get are in place? And we were able to get our our our license back. Many license back. Many license back. Many of you may be facing of you may be facing of you may be facing similar similar similar issues where issues where issues where agents make it difficult agents make it difficult agents make it difficult to understand what is to understand what is to understand what is happening in your happening in your happening in your codebase. So how do codebase. So how do codebase. So how do you make sure your you make sure your you make sure your policies are actually policies are actually policies are actually being followed? You being followed? You being followed? You need a way need a way need a way to enforce to enforce to enforce these rules these rules these rules outside of CI. The outside of CI. The outside of CI. The fact that code fact that code fact that code compiles does not compiles does not compiles does not mean that it mean that it mean that it conforms to a particular conforms to a particular conforms to a particular specification. It could specification. It could specification. It could be HIPAA or, say, be HIPAA or, say, be HIPAA or, say, DORA. There are many DORA. There are many DORA. There are many other policies that other policies that other policies that agents will not agents will not agents will not know about know about know about unless they unless they provide them themselves. It is also provide them themselves. It is also worth making sure worth making sure worth making sure that you record the fact of that you record the fact of that you record the fact of implementation for implementation for implementation for audit purposes. Now a short audit purposes. Now a short audit purposes. Now a short story about how we story about how we story about how we accelerated our CI. We accelerated our CI. We accelerated our CI. We had a had a had a baseline that was, I baseline that was, I baseline that was, I would say, not bad, would say, not bad, would say, not bad, but it was very annoying but it was very annoying but it was very annoying when you tried to when you tried to when you tried to ship more ship more ship more code. Our CI took code. Our CI took code. Our CI took about 10 minutes, and we about 10 minutes, and we about 10 minutes, and we wanted to significantly wanted to significantly wanted to significantly reduce that time. We reduce that time. We reduce that time. We started investigating started investigating started investigating how we could cache how we could cache how we could cache output results. We output results. We output results. We use Rust use Rust use Rust for our for our for our build system. So we started

build system. So we started build system. So we started caching as much caching as much caching as much data as possible and data as possible and data as possible and reusing it reusing it reusing it . We . We . We also looked for ways to also looked for ways to also looked for ways to reduce the reduce the reduce the load on the load on the load on the compiler. So we compiler. So we compiler. So we added a fix added a fix added a fix to make the to make the to make the compilation part more compilation part more cacheable and less likely to cacheable and less likely to fail. fail. By adding even more By adding even more fixes, including fixes, including fixes, including sharding and eliminating sharding and eliminating sharding and eliminating such such such overhead, we were able to overhead, we were able to overhead, we were able to reduce CI time reduce CI time reduce CI time to almost a minute for to almost a minute for to almost a minute for incremental incremental incremental steps. This means that steps. This means that steps. This means that once the “cold” once the “cold” once the “cold” CI warms up, CI warms up, CI warms up, incremental incremental incremental builds become very, builds become very, builds become very, very fast, and we very fast, and we very fast, and we can ship can ship can ship significantly more code. significantly more code. significantly more code. Another problem is Another problem is Another problem is code validation. So, code validation. So, code validation. So, even though CI is “ even though CI is “ green,” it’s interesting that green,” it’s interesting that green,” it’s interesting that when we measured when we measured when we measured our P90 time from our P90 time from our P90 time from PR creation to PR creation to PR creation to production, Uber production, Uber spent most of its time on spent most of its time on verification. And even verification. And even verification. And even today, to be honest today, to be honest today, to be honest , most , most , most teams spend teams spend teams spend more time on more time on more time on code reviews and code reviews and code reviews and other types of testing other types of testing other types of testing before they dare to before they dare to before they dare to merge code into the merge code into the merge code into the main branch. So main branch. So , this is something that needs to , this is something that needs to , this is something that needs to be addressed be addressed be addressed in parallel with in parallel with in parallel with CI optimization. Because what's CI optimization. Because what's CI optimization. Because what's happening now is that happening now is that happening now is that agents agents agents are generating code, and are generating code, and are generating code, and at the same time, AI agents at the same time, AI agents at the same time, AI agents are finding problems in are finding problems in are finding problems in your your your code review system. These code review system. These code review system. These automated automated automated checks also checks also checks also produce findings produce findings produce findings that you now that you now that you now need to review need to review need to review and fix. So you and fix. So you and fix. So you either have to either have to either have to load load load the context locally the context locally the context locally for your agent or for your agent or for your agent or have an agent in CI that have an agent in CI that have an agent in CI that iterates and fixes iterates and fixes iterates and fixes bugs. And humans bugs. And humans bugs. And humans will be able to review will be able to review will be able to review what's left what's left what's left after the AI after the AI after the AI has made its first has made its first has made its first pass. And this should be pass. And this should be pass. And this should be part of the equation part of the equation part of the equation so that you don't waste so that you don't waste so that you don't waste valuable

valuable valuable human time on the ever- human time on the ever- human time on the ever- increasing amount of PR increasing amount of PR increasing amount of PR generated by generated by generated by agents. Here is an example agents. Here is an example agents. Here is an example from our experience. We from our experience. We from our experience. We refactored our refactored our refactored our codebase to codebase to codebase to speed up CI. We speed up CI. We speed up CI. We used Next.js used Next.js used Next.js to parallelize to parallelize to parallelize tests, and we had a tests, and we had a tests, and we had a shared database shared database shared database that we ran that we ran that we ran once so all our once so all our once so all our tests could tests could reuse it. But the reuse it. But the problem was problem was problem was that we didn't provide that we didn't provide that we didn't provide certain safeguards, certain safeguards, certain safeguards, so during heavy so during heavy so during heavy CI load, CI load, CI load, cross- cross- cross- contamination occurred and the tests contamination occurred and the tests contamination occurred and the tests became became became unstable. Gitter, unstable. Gitter, unstable. Gitter, as a as a as a code reviewer, discovered this in code reviewer, discovered this in code reviewer, discovered this in our PR before our PR before our PR before we merged it. we merged it. we merged it. So we didn't have to So we didn't have to So we didn't have to waste time waste time waste time fixing it later. fixing it later. fixing it later. Gitter also fixed this Gitter also fixed this Gitter also fixed this issue for us issue for us issue for us as as as he suggested a solution he suggested a solution , committed it to the , committed it to the , committed it to the commit, and added the commit, and added the commit, and added the engineer as a engineer as a engineer as a contributor. Thanks to this, contributor. Thanks to this, contributor. Thanks to this, we were able to we were able to we were able to prevent costly prevent costly prevent costly mistakes before the mistakes before the mistakes before the code even hit the code even hit the code even hit the main branch. Another main branch. Another main branch. Another thing we do at thing we do at thing we do at GitLab, and that we GitLab, and that we GitLab, and that we see more and see more and see more and more of more of more of our customers starting to do, is our customers starting to do, is our customers starting to do, is use policies use policies use policies and criteria to and criteria to and criteria to decide when it's decide when it's decide when it's appropriate to appropriate to appropriate to approve changes at all.

approve changes at all. There are many changes There are many changes that are routine, that are routine, that are routine, low- low- low- risk, like risk, like risk, like documentation or documentation or dependency updates. Of course, dependency updates. Of course, if they don't break if they don't break if they don't break the API and aren't the API and aren't the API and aren't security patches. Even security patches. Even security patches. Even low- low- low- risk changes related to risk changes related to risk changes related to our tools our tools can potentially can potentially can potentially be approved be approved be approved automatically. GitLab automatically. GitLab automatically. GitLab supports these supports these supports these capabilities out of capabilities out of capabilities out of the box, and you the box, and you the box, and you can also add can also add can also add your own your own your own plain language rules to plain language rules to plain language rules to use use use to manage to manage to manage approvals and approvals and approvals and merges. We see that merges. We see that merges. We see that teams that actively teams that actively teams that actively use AI use AI use AI are starting are starting are starting to implement it more and more to implement it more and more to implement it more and more often. I think on often. I think on often. I think on our platform, we're our platform, we're our platform, we're seeing a rapid seeing a rapid seeing a rapid increase in the increase in the increase in the use of use of use of auto-approval and auto-approval and auto-approval and auto-merge in auto-merge in auto-merge in teams, even teams, even teams, even large ones with large ones with large ones with over 100 or even 2,000 over 100 or even 2,000 over 100 or even 2,000 developers. Here is an developers. Here is an developers. Here is an example of an example of an example of an approval policy. So if approval policy. So if approval policy. So if it's a it's a it's a dependency update or a dependency update or a dependency update or a documentation change, documentation change, documentation change, we have to we have to we have to approve it. Any approve it. Any approve it. Any changes to sensitive changes to sensitive changes to sensitive parts of the system, such parts of the system, such as payments or as payments or as payments or authentication, authentication, authentication, may require may require may require additional additional additional human approval. And, human approval. And, human approval. And, of course, you don't want the of course, you don't want the of course, you don't want the agent who made agent who made agent who made the changes to approve them themselves the changes to approve them themselves the changes to approve them themselves , because that , because that , because that goes against the principles of goes against the principles of goes against the principles of zero trust. So zero trust. So zero trust. So in this case, when in this case, when in this case, when agents create PRs, agents create PRs, agents create PRs, they are immediately they are immediately they are immediately approved. When a approved. When a approved. When a new new new change appears that does not change appears that does not change appears that does not meet the criteria meet the criteria , we have to pass it on to , we have to pass it on to , we have to pass it on to another person another person another person responsible for responsible for responsible for approval. This can be approval. This can be approval. This can be done using done using done using your own agents, but your own agents, but your own agents, but GitLab also supports GitLab also supports GitLab also supports it out of the box if it out of the box if it out of the box if you want to try it.

you want to try it. Another problem Another problem with relying solely on CI with relying solely on CI with relying solely on CI today is today is today is that, as I that, as I that, as I said, successful CI doesn't said, successful CI doesn't said, successful CI doesn't always mean always mean always mean the code meets the code meets the code meets the requirements. So, in this the requirements. So, in this the requirements. So, in this case, CI is case, CI is case, CI is successful and we have successful and we have successful and we have some metadata about some metadata about some metadata about it. When it. When it. When re-running, an re-running, an re-running, an unstable test unstable test unstable test may throw an error, may throw an error, may throw an error, meaning that when meaning that when meaning that when you do a force you do a force you do a force push to fix it, push to fix it, push to fix it, the branch now points to a the branch now points to a the branch now points to a different piece of code. different piece of code. The problem here is The problem here is that over time, most that over time, most that over time, most CI data becomes outdated. CI data becomes outdated. CI data becomes outdated. So if you don't have So if you don't have So if you don't have a record of what a record of what a record of what happened and what was happened and what was happened and what was fixed, you won't fixed, you won't fixed, you won't be able to ensure be able to ensure compliance or compliance or verify what verify what verify what actually happened. actually happened. actually happened. That's why I think That's why I think That's why I think things like things like things like agent trajectories agent trajectories agent trajectories and the results after and the results after and the results after each push are very each push are very each push are very important. So, important. So, any information any information any information you can record in a you can record in a you can record in a PR or other PR or other PR or other accounting system will become a accounting system will become a accounting system will become a source of truth when source of truth when source of truth when your artifacts your artifacts your artifacts become outdated. The control become outdated. The control become outdated. The control here is to here is to here is to make sure that make sure that make sure that the change has been approved and the change has been approved and the change has been approved and tested by someone tested by someone tested by someone other than the person who other than the person who other than the person who wrote it. If you wrote it. If you wrote it. If you don't have this don't have this don't have this information, it can information, it can information, it can be a problem. Here is an be a problem. Here is an be a problem. Here is an example. We have a example. We have a example. We have a pipeline that pipeline that pipeline that ended in error ended in error ended in error and one that ended and one that ended and one that ended successfully. But the difference is successfully. But the difference is successfully. But the difference is that if you that if you that if you properly properly properly capture the cause of capture the cause of capture the cause of the failure, and someone else the failure, and someone else the failure, and someone else can approve it because can approve it because can approve it because it's a known type of it's a known type of it's a known type of error saved error saved error saved with with with the changes you made, then it the changes you made, then it the changes you made, then it passes your passes your passes your checks.

checks. checks. Unlike the situation where there is Unlike the situation where there is Unlike the situation where there is no record and you no record and you no record and you blindly trust blindly trust blindly trust the agent sending the the agent sending the the agent sending the code because the test code because the test code because the test could have been of poor quality. There could have been of poor quality. There could have been of poor quality. There was no was no was no record of all record of all compliance checks performed compliance checks performed . So, in more . So, in more . So, in more sensitive industries, it's sensitive industries, it's sensitive industries, it's important to have all of these important to have all of these important to have all of these decisions decisions decisions captured as captured as captured as rules or rules or compliance checks in compliance checks in your PR or in your PR or in your PR or in addition to addition to addition to code reviews. So what code reviews. So what code reviews. So what records do we need to records do we need to records do we need to keep? Here are some keep? Here are some keep? Here are some examples of what we examples of what we examples of what we store in Git that store in Git that store in Git that I think are I think are I think are illustrative. For others illustrative. For others illustrative. For others it may look it may look it may look different. For example, different. For example, different. For example, which which which coding agent created coding agent created coding agent created this PR? Where does this PR? Where does this PR? Where does this PR come from? What this PR come from? What this PR come from? What session was its session was its session was its source? And who source? And who source? And who initiated it? Could this have initiated it? Could this have initiated it? Could this have been the launch of an been the launch of an been the launch of an automated automated automated workflow or workflow or workflow or just an engineer just an engineer just an engineer sending a request? And for sending a request? And for sending a request? And for each of these each of these each of these CI runs, which CI runs, which CI runs, which commit was commit was commit was confirmed by CI? It is confirmed by CI? It is confirmed by CI? It is also important also important also important to keep a to keep a to keep a test report, not test report, not test report, not just a status. You should just a status. You should just a status. You should make sure that make sure that make sure that each bug you find each bug you find each bug you find has its own has its own has its own record: how was it resolved, was record: how was it resolved, was record: how was it resolved, was it it it rejected, was it rejected, was it rejected, was it worked on, or was it worked on, or was it worked on, or was it fixed in PR? It also fixed in PR? It also fixed in PR? It also requires that requires that requires that approval be given by approval be given by approval be given by someone other than the bot someone other than the bot someone other than the bot that the that the that the same person activated. And same person activated. And same person activated. And make sure that make sure that any cancellations of any cancellations of any cancellations of inspections are clearly inspections are clearly inspections are clearly justified. So justified. So justified. So if a build is "red if a build is "red " or a code coverage check is missed " or a code coverage check is missed , it should be , it should be clear clear clear why the check why the check why the check failed and why we are failed and why we are failed and why we are okay with it in okay with it in okay with it in such a scenario. You such a scenario. You such a scenario. You also need to also need to also need to make sure that the code being make sure that the code being make sure that the code being merged merged merged contains all of this, and is not

contains all of this, and is not contains all of this, and is not just a small just a small just a small change on top of it. change on top of it. change on top of it. This way, all This way, all This way, all checks checks checks are performed every are performed every are performed every time. In this case, time. In this case, time. In this case, we need to we need to we need to ensure that ensure that ensure that agents cannot agents cannot agents cannot approve their own approve their own approve their own changes. The engineer who changes. The engineer who changes. The engineer who initiated the request initiated the request initiated the request cannot also be cannot also be cannot also be the author. If you the author. If you the author. If you implement these implement these implement these practices, you can practices, you can practices, you can eliminate bottlenecks eliminate bottlenecks eliminate bottlenecks as faster CI as faster CI as faster CI will speed up builds, will speed up builds, will speed up builds, making your making your making your workflow workflow workflow much more convenient. And much more convenient. And much more convenient. And by moving by moving by moving some checks to some checks to some checks to earlier stages, you earlier stages, you earlier stages, you can also significantly can also significantly can also significantly reduce the reduce the testing load on your CI system. If you system. If you system. If you can can can automate automate automate the review as much as possible, involving the review as much as possible, involving the review as much as possible, involving humans only in humans only in humans only in exceptional cases exceptional cases exceptional cases or even or even or even automatically automatically automatically approving approving approving low- low- low- risk internal changes, you will risk internal changes, you will risk internal changes, you will remove additional remove additional remove additional barriers. This will reduce barriers. This will reduce barriers. This will reduce the number of tasks the number of tasks the number of tasks requiring the participation of requiring the participation of requiring the participation of specialists and allow specialists and allow specialists and allow your "software factory" your "software factory" your "software factory" to operate at full to operate at full to operate at full capacity. And of course, capacity. And of course, capacity. And of course, after every after every code approval and merge, you want to have a code approval and merge, you want to have a record of what record of what record of what happened. After all, as happened. After all, as the scale of development the scale of development grows, without grows, without grows, without understanding the events of understanding the events of understanding the events of the past, you will lose the the past, you will lose the the past, you will lose the context that is context that is context that is extremely important extremely important extremely important for the continuous for the continuous for the continuous improvement of your improvement of your software. That's all software. That's all from my side. I would be really from my side. I would be really from my side. I would be really happy if you happy if you happy if you tried Guitar and tried Guitar and tried Guitar and saw how you can saw how you can saw how you can improve the work of improve the work of improve the work of your "software factories". That's your "software factories". That's your "software factories". That's all, thank you very much all, thank you very much all, thank you very much for your attention.