Weight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli

Hello everyone. Uh Hello everyone. Uh Thank you for coming and Thank you for coming and Thank you for coming and uh uh uh I'll start the talk now. So I'll start the talk now. So I'll start the talk now. So This talk is uh around a paper This talk is uh around a paper This talk is uh around a paper that I did um that I did um that I did um which is which is which is very simple. very simple. very simple. The proposition is very clear. The proposition is very clear. The proposition is very clear. It's basically It's basically It's basically two lines of algebra that make uh the two lines of algebra that make uh the two lines of algebra that make uh the RMS norm layer in transformers RMS norm layer in transformers RMS norm layer in transformers cheaper, quicker, and kind of improve cheaper, quicker, and kind of improve cheaper, quicker, and kind of improve improve it as like a layer improve it as like a layer improve it as like a layer in the transformer architecture. in the transformer architecture. in the transformer architecture. Similar to how layer norm Similar to how layer norm Similar to how layer norm once used to be the standard and then it once used to be the standard and then it once used to be the standard and then it was substituted by RMS norm. was substituted by RMS norm. was substituted by RMS norm. This follows along um this way of This follows along um this way of This follows along um this way of thinking. thinking. thinking. And I got the chance to kind of meet And I got the chance to kind of meet And I got the chance to kind of meet some people from the open source world some people from the open source world some people from the open source world and um I co-authored this paper together and um I co-authored this paper together and um I co-authored this paper together with uh Niels Graf who was the with uh Niels Graf who was the with uh Niels Graf who was the kind of the creator of this.

kind of the creator of this. kind of the creator of this. And the work follows from there. And the work follows from there. And the work follows from there. So this is uh presented on archive. You So this is uh presented on archive. You So this is uh presented on archive. You can have a look, read it, test it out. can have a look, read it, test it out. can have a look, read it, test it out. There is a repo as well. There is a repo as well. There is a repo as well. And the concept the And the concept the And the concept the let's say the idea and the way of let's say the idea and the way of let's say the idea and the way of thinking it's easiest to explain with thinking it's easiest to explain with thinking it's easiest to explain with maybe flash attention. So in a similar maybe flash attention. So in a similar maybe flash attention. So in a similar way of how um flash attention kind of way of how um flash attention kind of way of how um flash attention kind of weights until there's a multiplication weights until there's a multiplication weights until there's a multiplication and tries to limit this uh and tries to limit this uh and tries to limit this uh communications between memory so that communications between memory so that communications between memory so that the whole process is faster. This is the whole process is faster. This is the whole process is faster. This is kind of a similar thought along those kind of a similar thought along those kind of a similar thought along those lines and it does certain improvements lines and it does certain improvements lines and it does certain improvements that make that make that make the RMS the RMS the RMS norm process norm process norm process much quicker and much quicker and much quicker and in in effect improve the whole in in effect improve the whole in in effect improve the whole transformer. transformer. transformer. And one question is okay, why RMS norm And one question is okay, why RMS norm And one question is okay, why RMS norm since that layer does almost none of the since that layer does almost none of the since that layer does almost none of the math?

math? math? And that's true. So the share of the And that's true. So the share of the And that's true. So the share of the kind of math portion if you look at it kind of math portion if you look at it kind of math portion if you look at it is quite small. is quite small. is quite small. However, the However, the However, the clock time or wall time as they say is clock time or wall time as they say is clock time or wall time as they say is quite big and quite big and quite big and for example in one decode step. So right for example in one decode step. So right for example in one decode step. So right when like inference is performed when like inference is performed when like inference is performed the RMS norm can be started like 33 the RMS norm can be started like 33 the RMS norm can be started like 33 times. Of course, it depends on the times. Of course, it depends on the times. Of course, it depends on the model and so on. In the paper, you have model and so on. In the paper, you have model and so on. In the paper, you have the specific models and how this was the specific models and how this was the specific models and how this was tested. tested. tested. Um Um Um and the question is and the question is and the question is how this can be improved and how how this can be improved and how how this can be improved and how this wait for the matrix multiplication this wait for the matrix multiplication this wait for the matrix multiplication can be kind of avoided. can be kind of avoided. can be kind of avoided. And the reason why this is slow is And the reason why this is slow is And the reason why this is slow is because the GPUs are not because the GPUs are not because the GPUs are not slow or bad at math, but they're bad at slow or bad at math, but they're bad at slow or bad at math, but they're bad at everything else around the actual math. everything else around the actual math. everything else around the actual math. So that means starting the work, the So that means starting the work, the So that means starting the work, the actual work. So actual work. So actual work. So for example, starting the process for example, starting the process for example, starting the process as it happens in some of the experiments as it happens in some of the experiments as it happens in some of the experiments 33 times that takes a long time 33 times that takes a long time 33 times that takes a long time and for example, fusing and for example, fusing and for example, fusing um um um each normalization into the matrix each normalization into the matrix each normalization into the matrix multiplication can help avoid this. Also multiplication can help avoid this. Also multiplication can help avoid this. Also doing weight folding can help in kind of doing weight folding can help in kind of doing weight folding can help in kind of moving data between moving data between moving data between memory and um memory and um memory and um that's a process that's also slow for that's a process that's also slow for that's a process that's also slow for GPUs.

GPUs. GPUs. And also waiting. Uh so, for example, And also waiting. Uh so, for example, And also waiting. Uh so, for example, deferring the division that's done in deferring the division that's done in deferring the division that's done in the RMS norm layer is also a way to the RMS norm layer is also a way to the RMS norm layer is also a way to avoid this waiting step. So, basically, avoid this waiting step. So, basically, avoid this waiting step. So, basically, what this paper does is it improves all what this paper does is it improves all what this paper does is it improves all these three aspects by doing a few these three aspects by doing a few these three aspects by doing a few algebraic tricks in the way RMS norm is algebraic tricks in the way RMS norm is algebraic tricks in the way RMS norm is computed. computed. computed. That's it. That's it. And math-wise, these are the tricks. And math-wise, these are the tricks. Uh it's mainly around the first two Uh it's mainly around the first two Uh it's mainly around the first two propositions. One is weightless propositions. One is weightless propositions. One is weightless normalization. Uh you can see that here. normalization. Uh you can see that here. normalization. Uh you can see that here. Um and deferred normalization, so um Um and deferred normalization, so um Um and deferred normalization, so um that's the second one. And now in more that's the second one. And now in more that's the second one. And now in more newer architectures, there is a newer architectures, there is a newer architectures, there is a situation where um RMS can be kind of situation where um RMS can be kind of situation where um RMS can be kind of can appear twice. Uh for example, in can appear twice. Uh for example, in can appear twice. Uh for example, in Gemma 4, this happens. So, Gemma 4, this happens. So, Gemma 4, this happens. So, canceling the pre-normalization also canceling the pre-normalization also canceling the pre-normalization also works. Um and all of this is works. Um and all of this is works. Um and all of this is algebraically proven in the paper.

And the first proposition is this where And the first proposition is this where kind of the the gain and the weight fold kind of the the gain and the weight fold kind of the the gain and the weight fold folded to one matrix W, uh you can see folded to one matrix W, uh you can see folded to one matrix W, uh you can see here with an asterisk. here with an asterisk. here with an asterisk. And that is computed offline, similar to And that is computed offline, similar to And that is computed offline, similar to how maybe in flash attention, you how maybe in flash attention, you how maybe in flash attention, you compute some stuff on the side so that compute some stuff on the side so that compute some stuff on the side so that there is no uh communication between there is no uh communication between there is no uh communication between memory all the time. So, this is one memory all the time. So, this is one memory all the time. So, this is one step that's kind of um done, this weight step that's kind of um done, this weight step that's kind of um done, this weight folding. folding. folding. And the other step is um deferring um And the other step is um deferring um And the other step is um deferring um the scalar the scalar divide of the the scalar the scalar divide of the the scalar the scalar divide of the matmul so that they can be done in matmul so that they can be done in matmul so that they can be done in parallel. So, in a normal case, you parallel. So, in a normal case, you parallel. So, in a normal case, you would have to compute once, then wait, would have to compute once, then wait, would have to compute once, then wait, and compute again. In this case, the and compute again. In this case, the and compute again. In this case, the idea is to kind of split this so that it idea is to kind of split this so that it idea is to kind of split this so that it can be parallelized. can be parallelized. can be parallelized. And the third one, which is kind of a And the third one, which is kind of a And the third one, which is kind of a version of this is that version of this is that version of this is that um um um there is kind of if there are two there is kind of if there are two there is kind of if there are two um because this is scale invariant, one um because this is scale invariant, one um because this is scale invariant, one of them can be dropped and this still of them can be dropped and this still of them can be dropped and this still works.

works. works. And this is applicable to newer models And this is applicable to newer models And this is applicable to newer models um that can have this architecture and um that can have this architecture and um that can have this architecture and implementation. So, in order to make this happen in real So, in order to make this happen in real life, especially life, especially life, especially this proposition number two, this proposition number two, this proposition number two, um um um so for this one, for example, it's easy. so for this one, for example, it's easy. so for this one, for example, it's easy. There is a repo called Transformer There is a repo called Transformer There is a repo called Transformer Tricks. You can just apply this to any Tricks. You can just apply this to any Tricks. You can just apply this to any model and it works. model and it works. model and it works. But in order to do this, there is some But in order to do this, there is some But in order to do this, there is some kernel work. So, it's not as kernel work. So, it's not as kernel work. So, it's not as straightforward to do. straightforward to do. straightforward to do. So, in order for me to do that, So, in order for me to do that, So, in order for me to do that, I I I was implementing this and I was implementing this and I was implementing this and I came out with this experiment once. came out with this experiment once. came out with this experiment once. So, it looks okay in general, where it's So, it looks okay in general, where it's So, it looks okay in general, where it's like, "Okay, the prompt is the like, "Okay, the prompt is the like, "Okay, the prompt is the Transformer architecture revolutionally Transformer architecture revolutionally Transformer architecture revolutionally revolutionized NLP because and then revolutionized NLP because and then revolutionized NLP because and then there is some kind of expected output." there is some kind of expected output." there is some kind of expected output." But in the output I got, I saw this But in the output I got, I saw this But in the output I got, I saw this repetition and one-step lag, as you can repetition and one-step lag, as you can repetition and one-step lag, as you can see here, the word because appears see here, the word because appears see here, the word because appears again. And again. And again. And there was something happening with the there was something happening with the there was something happening with the GPU streams and I was trying to figure GPU streams and I was trying to figure GPU streams and I was trying to figure out what was happening.

out what was happening. out what was happening. And I was getting this one-step lag and And I was getting this one-step lag and And I was getting this one-step lag and kind of um outputs that were from the kind of um outputs that were from the kind of um outputs that were from the past in a way. past in a way. past in a way. Um and in debugging all of this, I Um and in debugging all of this, I Um and in debugging all of this, I realized that um realized that um realized that um in the process of building something in the process of building something in the process of building something like this, so like this, so like this, so as I explained the proposition two or as I explained the proposition two or as I explained the proposition two or deferring these two operations, deferring these two operations, deferring these two operations, um in CUDA, you can do two things. You um in CUDA, you can do two things. You um in CUDA, you can do two things. You can do like tensor cores that do one can do like tensor cores that do one can do like tensor cores that do one part of the matrix multiplication and part of the matrix multiplication and part of the matrix multiplication and you can do CUDA cores that kind of run you can do CUDA cores that kind of run you can do CUDA cores that kind of run stuff like element-wise operations, stuff like element-wise operations, stuff like element-wise operations, reductions, square roots, and so on. So, reductions, square roots, and so on. So, reductions, square roots, and so on. So, the idea was to the idea was to the idea was to do this in parallel and get the benefit do this in parallel and get the benefit do this in parallel and get the benefit of what I was explaining in the paper to of what I was explaining in the paper to of what I was explaining in the paper to actually test out this concept. actually test out this concept. actually test out this concept. So, this is how it was supposed to look So, this is how it was supposed to look So, this is how it was supposed to look like. So, there is if you do things like. So, there is if you do things like. So, there is if you do things sequentially, there is this idle waiting sequentially, there is this idle waiting sequentially, there is this idle waiting time when you when the vector unit time when you when the vector unit time when you when the vector unit computes the RMS and scaling, and then computes the RMS and scaling, and then computes the RMS and scaling, and then there is a matrix multiplication. So, there is a matrix multiplication. So, there is a matrix multiplication. So, the idea was okay, with flash norm, the idea was okay, with flash norm, the idea was okay, with flash norm, which is the technique in the paper, which is the technique in the paper, which is the technique in the paper, you're supposed to do those both in you're supposed to do those both in you're supposed to do those both in parallel. So, the matrix unit computes parallel. So, the matrix unit computes parallel. So, the matrix unit computes the matmul and the vector unit computes the matmul and the vector unit computes the matmul and the vector unit computes the RMS. So, in that way you save uh the RMS. So, in that way you save uh the RMS. So, in that way you save uh time. However, you cannot just do this time. However, you cannot just do this time. However, you cannot just do this in Python, you have to go a bit lower.

in Python, you have to go a bit lower. in Python, you have to go a bit lower. And I did that And I did that And I did that with CUDA code like this. with CUDA code like this. with CUDA code like this. And this looked in general okay at my uh And this looked in general okay at my uh And this looked in general okay at my uh at that time. at that time. at that time. However, However, However, um um um I realized that I did something slightly I realized that I did something slightly I realized that I did something slightly wrong. wrong. wrong. And that thing was that And that thing was that And that thing was that the join in the end, where you're the join in the end, where you're the join in the end, where you're supposed to join the two streams, was supposed to join the two streams, was supposed to join the two streams, was implicit in my case. implicit in my case. implicit in my case. And when I tested this out, the unit And when I tested this out, the unit And when I tested this out, the unit tests worked, the quality seemed tests worked, the quality seemed tests worked, the quality seemed similar, like perplexity testing, and so similar, like perplexity testing, and so similar, like perplexity testing, and so on, because it's just like um similar on, because it's just like um similar on, because it's just like um similar generation, but over long generation, I generation, but over long generation, I generation, but over long generation, I was able to see this problem. So, I had was able to see this problem. So, I had was able to see this problem. So, I had no idea what this was. no idea what this was. no idea what this was. And the reason was that when I was doing And the reason was that when I was doing And the reason was that when I was doing this uh implicit um this uh implicit um this uh implicit um join, join, join, basically, one of the streams hadn't basically, one of the streams hadn't basically, one of the streams hadn't finished the work, so I got race finished the work, so I got race finished the work, so I got race conditions that kind of read the past conditions that kind of read the past conditions that kind of read the past from the unfinished matrix from the unfinished matrix from the unfinished matrix multiplication. So, the idea that I had multiplication. So, the idea that I had multiplication. So, the idea that I had to fix this was um to fix this was um to fix this was um around the fact that I had to be around the fact that I had to be around the fact that I had to be explicit about the join and wait until explicit about the join and wait until explicit about the join and wait until one of the operations is finished so one of the operations is finished so one of the operations is finished so that I'm certain that when I join I'm that I'm certain that when I join I'm that I'm certain that when I join I'm not reading from the past.

not reading from the past. not reading from the past. So, that was the realization um in this So, that was the realization um in this So, that was the realization um in this exploration of CUDA streams. exploration of CUDA streams. exploration of CUDA streams. And this is how I had things done. So, And this is how I had things done. So, And this is how I had things done. So, the join was implicit. So, the post um the join was implicit. So, the post um the join was implicit. So, the post um scale read like an old uh buffer value. scale read like an old uh buffer value. scale read like an old uh buffer value. And how this is fixed is with this where And how this is fixed is with this where And how this is fixed is with this where basically you need to mark the end of basically you need to mark the end of basically you need to mark the end of the matrix multiplication, then mark the the matrix multiplication, then mark the the matrix multiplication, then mark the end of the RMS, end of the RMS, end of the RMS, and then and then and then post scale wait for post scale wait for post scale wait for the first stream and then wait for the the first stream and then wait for the the first stream and then wait for the second stream. second stream. second stream. And that fixed the bug and made kind of And that fixed the bug and made kind of And that fixed the bug and made kind of the paper work and the model speak the paper work and the model speak the paper work and the model speak forwards instead of backwards. forwards instead of backwards. forwards instead of backwards. And And And that was the cool maybe academic that was the cool maybe academic that was the cool maybe academic perspective, but I also wanted to try perspective, but I also wanted to try perspective, but I also wanted to try things, right? Deploy this, test it out, things, right? Deploy this, test it out, things, right? Deploy this, test it out, see how I can make it work um see how I can make it work um see how I can make it work um in maybe a more production setting. And in maybe a more production setting. And in maybe a more production setting. And you can also read the paper and see all you can also read the paper and see all you can also read the paper and see all the tests. Um the tests. Um the tests. Um some are done most are done around llama some are done most are done around llama some are done most are done around llama models, but like this works for other models, but like this works for other models, but like this works for other architectures as well.

architectures as well. architectures as well. Um so, what you can do for this specific Um so, what you can do for this specific Um so, what you can do for this specific paper is um for example, the weight paper is um for example, the weight paper is um for example, the weight folding that I explained the folding that I explained the folding that I explained the pre-position one, you can just do it pre-position one, you can just do it pre-position one, you can just do it with with with um some code in the repo um some code in the repo um some code in the repo that's like flash you say flashify and that's like flash you say flashify and that's like flash you say flashify and it does that. However, with this second it does that. However, with this second it does that. However, with this second thing that I mentioned, you need to do a thing that I mentioned, you need to do a thing that I mentioned, you need to do a bit of kernel work if you want to do bit of kernel work if you want to do bit of kernel work if you want to do that uh like I explained in my example. that uh like I explained in my example. that uh like I explained in my example. And these are some results that are And these are some results that are And these are some results that are based on llama models and there are based on llama models and there are based on llama models and there are different kind of details that you can different kind of details that you can different kind of details that you can have a look at as well as well. Like have a look at as well as well. Like have a look at as well as well. Like what happens if you do only the third what happens if you do only the third what happens if you do only the third normalization, what happens if you do a normalization, what happens if you do a normalization, what happens if you do a full fused kernel. Um so there are a lot full fused kernel. Um so there are a lot full fused kernel. Um so there are a lot of experiments of going lower here to of experiments of going lower here to of experiments of going lower here to test all the propositions, and this have test all the propositions, and this have test all the propositions, and this have been our results been our results been our results um in different, let's say, levels of um um in different, let's say, levels of um um in different, let's say, levels of um scrutiny and detail. scrutiny and detail. scrutiny and detail. But even the simple one with like weight But even the simple one with like weight But even the simple one with like weight folding um folding um folding um shows some improvement. shows some improvement. shows some improvement. And this also works with like the And this also works with like the And this also works with like the day-to-day tools that you use in the day-to-day tools that you use in the day-to-day tools that you use in the models. It's not like you have to models. It's not like you have to models. It's not like you have to reinvent the wheel or, you know, do reinvent the wheel or, you know, do reinvent the wheel or, you know, do things from scratch. So it works with uh things from scratch. So it works with uh things from scratch. So it works with uh torch compile torch compile torch compile um because the it's kind of like a new um because the it's kind of like a new um because the it's kind of like a new checkpoint, and that's it. Flash checkpoint, and that's it. Flash checkpoint, and that's it. Flash attention does similar tricks at a attention does similar tricks at a attention does similar tricks at a different layer, and also it works with different layer, and also it works with different layer, and also it works with quantized models. So it's totally cool quantized models. So it's totally cool quantized models. So it's totally cool to actually apply this, and you can get to actually apply this, and you can get to actually apply this, and you can get a model that has this cool new a model that has this cool new a model that has this cool new normalization layer.

normalization layer. normalization layer. And where you can get this And where you can get this And where you can get this um um um details and code to actually run this is details and code to actually run this is details and code to actually run this is this transformer tricks repo. So this transformer tricks repo. So this transformer tricks repo. So uh it has different algebraic tricks uh it has different algebraic tricks uh it has different algebraic tricks like I explained, as well as this paper like I explained, as well as this paper like I explained, as well as this paper that I mentioned. that I mentioned. that I mentioned. And also there is the GitHub uh not the And also there is the GitHub uh not the And also there is the GitHub uh not the GitHub, but the Hugging Face uh model GitHub, but the Hugging Face uh model GitHub, but the Hugging Face uh model repo where I've done this with some repo where I've done this with some repo where I've done this with some models, and you can have a Hugging Face models, and you can have a Hugging Face models, and you can have a Hugging Face link to that model and test it out. link to that model and test it out. link to that model and test it out. Um and what you also can do with this Um and what you also can do with this Um and what you also can do with this Hugging Face models is to deploy them in Hugging Face models is to deploy them in Hugging Face models is to deploy them in production. And so when I was thinking production. And so when I was thinking production. And so when I was thinking about doing this, um about doing this, um about doing this, um I realized that, okay, now that, let's I realized that, okay, now that, let's I realized that, okay, now that, let's say, the science is done and there is a say, the science is done and there is a say, the science is done and there is a link to a Hugging Face model, link to a Hugging Face model, link to a Hugging Face model, um Superlinked's uh inference engine was um Superlinked's uh inference engine was um Superlinked's uh inference engine was a cool way to actually deploy any um a cool way to actually deploy any um a cool way to actually deploy any um uh Hugging Face model, and we've done uh Hugging Face model, and we've done uh Hugging Face model, and we've done this at hackathons where people would this at hackathons where people would this at hackathons where people would bring like a custom Hugging Face model bring like a custom Hugging Face model bring like a custom Hugging Face model or checkpoint that they have with their or checkpoint that they have with their or checkpoint that they have with their fine-tuned stuff, and you can test out fine-tuned stuff, and you can test out fine-tuned stuff, and you can test out like even if you have some version of like even if you have some version of like even if you have some version of this algebraic tricks that you want to this algebraic tricks that you want to this algebraic tricks that you want to improve a model and test on test your improve a model and test on test your improve a model and test on test your own research ideas, you can actually try own research ideas, you can actually try own research ideas, you can actually try that out and have a deployed version on that out and have a deployed version on that out and have a deployed version on a cluster of this model and not have to a cluster of this model and not have to a cluster of this model and not have to worry about this glue code around worry about this glue code around worry about this glue code around deploying models.

deploying models. deploying models. So, that's um So, that's um So, that's um pretty cool. And the the point is that pretty cool. And the the point is that pretty cool. And the the point is that if you have the full cluster open source if you have the full cluster open source if you have the full cluster open source and the model inference open source, you and the model inference open source, you and the model inference open source, you can actually test out this kind of can actually test out this kind of can actually test out this kind of maybe maybe maybe more novel research ideas where more novel research ideas where more novel research ideas where if you want to if you want to if you want to do do do kernel manipulation or kernel manipulation or kernel manipulation or flash norm and things like that, it's flash norm and things like that, it's flash norm and things like that, it's much more difficult to do that do this much more difficult to do that do this much more difficult to do that do this at a rented endpoint where you don't own at a rented endpoint where you don't own at a rented endpoint where you don't own the inference. It's you want something the inference. It's you want something the inference. It's you want something that's portable and flexible to actually that's portable and flexible to actually that's portable and flexible to actually allow you to do this stuff, but it's allow you to do this stuff, but it's allow you to do this stuff, but it's also production ready enough so that you also production ready enough so that you also production ready enough so that you can test things out at scale. And you can, for example, use site to And you can, for example, use site to combine this with other models like, as combine this with other models like, as combine this with other models like, as you can see in the top left, there is you can see in the top left, there is you can see in the top left, there is you can have this flashified models with you can have this flashified models with you can have this flashified models with different other models to do agentic different other models to do agentic different other models to do agentic tasks if you want and kind of do that tasks if you want and kind of do that tasks if you want and kind of do that end-to-end bigger use case. end-to-end bigger use case. end-to-end bigger use case. And the way site works is this And the way site works is this And the way site works is this production cluster helps you deploy the production cluster helps you deploy the production cluster helps you deploy the models, so you can have a look at site's models, so you can have a look at site's models, so you can have a look at site's repo as well for more details on this.

repo as well for more details on this. repo as well for more details on this. Um and also there is a smarter queuing Um and also there is a smarter queuing Um and also there is a smarter queuing mechanism that helps you, especially if mechanism that helps you, especially if mechanism that helps you, especially if you work with smaller models cuz when you work with smaller models cuz when you work with smaller models cuz when doing the flash norm stuff, I worked doing the flash norm stuff, I worked doing the flash norm stuff, I worked with like smaller llama models and also with like smaller llama models and also with like smaller llama models and also with small agents from hugging face. So, with small agents from hugging face. So, with small agents from hugging face. So, having a way to deploy having a way to deploy having a way to deploy smaller models that can also work on smaller models that can also work on smaller models that can also work on like the same GPU so that you don't have like the same GPU so that you don't have like the same GPU so that you don't have to to to spend your money on GPU cost, but spend your money on GPU cost, but spend your money on GPU cost, but actually kind of actually kind of actually kind of switch models around, especially smaller switch models around, especially smaller switch models around, especially smaller models. It was quite useful. models. It was quite useful. models. It was quite useful. And you can also control the model And you can also control the model And you can also control the model configs through an API as well as the configs through an API as well as the configs through an API as well as the cluster, which is also pretty convenient cluster, which is also pretty convenient cluster, which is also pretty convenient without having like an infra guy without having like an infra guy without having like an infra guy supporting you in your open source supporting you in your open source supporting you in your open source research. So, that's cool as well. research. So, that's cool as well. research. So, that's cool as well. Um, and you own your cloud, which is Um, and you own your cloud, which is Um, and you own your cloud, which is useful if you want open weights, open useful if you want open weights, open useful if you want open weights, open models, open source. models, open source. models, open source. And there's also like a catalog that Sci And there's also like a catalog that Sci And there's also like a catalog that Sci has of different models, um, not just has of different models, um, not just has of different models, um, not just the ones I mentioned, but you can have a the ones I mentioned, but you can have a the ones I mentioned, but you can have a look. There's also re-ranking embedding look. There's also re-ranking embedding look. There's also re-ranking embedding models if you're building something models if you're building something models if you're building something along those lines.

along those lines. along those lines. And with that I'm kind of finishing this And with that I'm kind of finishing this And with that I'm kind of finishing this story of my research journey where story of my research journey where story of my research journey where I co-authored this paper, I co-authored this paper, I co-authored this paper, um, around the technique that improves um, around the technique that improves um, around the technique that improves the transformer, but also found a way the transformer, but also found a way the transformer, but also found a way kind of to bring this to, let's say, kind of to bring this to, let's say, kind of to bring this to, let's say, production and test it out and find a production and test it out and find a production and test it out and find a way to play around with this open source way to play around with this open source way to play around with this open source models. And feel free to contact me on models. And feel free to contact me on models. And feel free to contact me on LinkedIn, maybe if you have any LinkedIn, maybe if you have any LinkedIn, maybe if you have any questions or contributions. A lot of questions or contributions. A lot of questions or contributions. A lot of this stuff that I've mentioned, like this stuff that I've mentioned, like this stuff that I've mentioned, like some of them are PRs on like vLLM or on some of them are PRs on like vLLM or on some of them are PRs on like vLLM or on Hugging Face. You might find them all Hugging Face. You might find them all Hugging Face. You might find them all around. You can also see the check out around. You can also see the check out around. You can also see the check out the paper. That's the archive link that the paper. That's the archive link that the paper. That's the archive link that you have there. Um, and you also have you have there. Um, and you also have you have there. Um, and you also have the Sci repo and my LinkedIn. the Sci repo and my LinkedIn. the Sci repo and my LinkedIn. Um, so Um, so Um, so thank you very much for attending. thank you very much for attending. thank you very much for attending. >> [applause] >> [applause] >> [applause] [cheering] >> And you can catch me for questions. >> And you can catch me for questions. We'll be here, close by.