This AI lands DevOps job (and it runs on k8s)

Hey, what's up? I built my own AI application with actual users and of course it is running on Kubernetes. And in this video I'm going to show you what I've built and how it works. So let's jump in. The application I built is called CubeCraft AI and this is an application that my students can use to help them get support when I am not online. So as you may know, I'm a senior DevOps engineer who specializes into helping people into six-figure tech careers. So most of my customers are IT professionals like support engineers, software engineers, and sysadmins. And I've been doing this full-time for nearly 3 years. And I do this through the community. So people join the platform, they are in the community, they are asking questions, and they are interacting with each other. And in the bottom you can see that we have over 5,000 foreign posts that are built up over the years. At the same time, I also host multiple mentorship calls every single week. So over these years, I've built up I think it's around 300 hours of me mentoring other engineers live on the call. So whenever someone new comes in and they ask a question, it's actually very likely that this question has been asked before. So rather than having to wait for my answer, I wanted to give them something that has access to all of these posts, everything I've ever produced, all of these mentorship calls, so that they can see if this question has been asked

before and get their answer more quickly. So when I log in to the application, it's already showing me a few suggestions and these rotate as if you refresh. So if I refresh this now, you're going to see other suggestions here. So for example, how do I debug a pod in crash loop back off? That's a pretty cool question to ask. So this is then going to go through, as you see in the bottom, the school community, the courses, the Q&A calls, and all the YouTube videos I've ever made. It is then going to select the relevant material from that, and it's going to base the answer with on that, and it's going to give you all of the source material. So, what someone someone first of all gets an answer when a pod lands in Crash Override, start with the logs in the events. So, this is exactly what I would say and what I usually say in these Q&A calls. It's giving you some code examples.

But, the really cool thing is is then it's going to show all of the forum posts that have to do with this particular issue. So, all of the all of the troubleshooting posts that we have, it is going to then um show the relevant posts. You can click the link, and then you can see, "Oh, this was discussed earlier. Like, this is 2 years ago." But, then people, as you can see here, people have been sharing lots of wisdom here that this would have never been resurfaced. Like, nobody is reading posts from 2024 when they're in the community. But, when they ask the AI, then these posts will get resurfaced, and then you can you get to uh capitalize on that knowledge, that corpus of knowledge that is already been shared inside of the community. So, this is not necessarily to like remove myself out of the operations or whatever. Like, I am still online in the community every single day, as you see on my profile. Uh every single day answering questions.

It's not about that. It's more about like if they are in the moment, they are stuck on something, and they want to get an answer to something, then they can they can ask the AI first and get unstuck right away. And it's actually working pretty damn good. So, let's do another question. Should I use Helm releases for home lab apps? So, this These questions are based on like specific takes that I have because I have a very particular take on this. So, I'm going to see what it's going to say here. Let's see. For self-hosted apps in a home lab, I generally skip Helm releases and write the manifest myself. This is exactly what I recommend. So, this is this is actually so good that it it like I I trained this to to answer in my voice and based on my recommendations. And then and then we can see where are these coming from, right? So, we have for first of all, the course reference material and then a few posts. So, for example, this one.

Here, why not use Helm releases for regular apps in the home lab? And then I I I give this answer here. And again, this is done from January 2025. So, it's really really really good. Um it's I'm really satisfied with how the app is working so far and lots of my my customers are getting lots of value out of it. I just was on an event recently and several people came up to me and said, "I really love the app, man. Well done." So, that's the app I built and now let's get into how it actually works. So, to start this off, first of all, I should say that I am not a software engineer. I am a senior DevOps engineer. I'm really good at infrastructure and and running applications, but I have I have only limited experience with writing apps or building apps. Like I've I've I've written thousands of lines of bash code and Python code to build like small custom APIs or internal CLI tools, that kind of thing in my own projects, but I've never really built an end-to-end application on my own.

And then, of course, with the rise of the of AI and and what they call vibe coding, I I I'm I prefer the term agentic engineering these days, but uh with the rise of these tools, I was actually able to have this idea, and it took me a about 6 months. Of course, I wasn't working full-time on this, but off and on, it took me about 6 months to get to this stage. But by by simply working with the models and constantly iterating, giving it feedback, giving it feedback, focusing on security, I was actually able to build this whole application, and it's looking pretty smooth. But so so just to say, I won't make any claims of being a good software engineer or anything like that, because I'm not. But this is how the application works. So, there's a front-end application, and this was based on the What is it called? The Vercel uh Vercel AI chat app. Is that Is that what it said?

Uh here. I think I think this is the one. Yeah, the AI SDK Next.js app router. I I think I I based it on on on this one. So, I took that for the front end. So, this is what you see here. This is the portal, and then I I I removed a lot of functionality out of that. I I I stripped it down to what actually actually what I needed, and I made some changes. So, for example, there is this usage estimate that is down here, and I like to have it in tokens rather than something like abstractions, that kind of thing. Of course, it has my branding and everything. So, that is the front end, and it's based on that, but yeah, it has the nuts and bolts of that, but I I changed a lot, and I took a lot out a lot of the original stuff. So, this front end, of course, is running as a container, and we're going to get into Kubernetes, etc., later. But then, the contain the front end is then going to talk to the rag back end.

Rag back end. That's how I called this this application. And yes, this is a rag application. And yeah, people like to like these days they say, "Oh, rag is not is useless or that's not really used anymore." That kind of thing or it a rag app is not a real AI app. But I don't I don't agree. Like I had a specific use case. This is what I needed it to do. I needed to access the information that I have available, this corpus of knowledge, and I needed to give an answer based on that. And it is performing that function really well. So, I I don't see any reason to look beyond if it's actually achieving what I need it to achieve. So, the rag back-end, this is a So, this was Next.js, we said. Next. js. The front-end, this this is actually a Python FastAPI.

And the reason why I use Python here is because Python is the language that I am most competent in myself. So, when I write code myself, I would always use Python. I did learn Go back in the day, but in this case, like I don't see any issues with my application so far. I might change it to a different I might change it to Go later, but for now, like you know, CubeCraft has 833 members and I think about 200 people have logged in to the application so far. It has been online for almost 2 weeks now. So, of course, not not everyone is interested in using it. Some of these users haven't logged in in a while. I have about 200 active users right now. So, I mean, in that sense, performance is it is fine. Like if I were to scale this to millions of users, which is of course never going to happen, then we're going to have a different conversation. But for now, Python FastAPI is more than capable of handling the volume that I need it to handle.

So, to keep things simple though, um you know, I'm a a DevOps engineer. I love running my own Postgres databases, et cetera. But I knew one thing when I was setting out to do this that I wanted this to be Postgres based. Now, at this stage, like one thing is to make the front end, the back end, and to work with it, and to get something you can you you can use. But then there's the layer of, "Okay, so now I have an application. How am I going to give access to people, and how am I going to limit the access for people?" So, how can I make sure that everyone who is watching this YouTube video cannot just log in to the application and use it for free? Because this application is worth The information here is worth a lot of money if you consider that this can help you land a six-figure tech career, right? So, if you make more if you go from making 80k a year to 200k year, let's let's say that's an increase of 100k conservatively. If you repeat that for 10 years, that's over a million bucks, right? So, this information can help you get a tech career, and we're really good at it, by the way, if you look at the win section.

Um just in the past week, we have landed one, two three four, five six uh seven, seven jobs in the past 9 days. So, yeah, this stuff really works. We're really good at what we do. And I don't want people I like this cannot be public information. People will have to invest in order to get access to this information. So, that is the what what that was the first challenge. But, the and the and the second thing I know is that I love Postgres. I love running Postgres databases. So, I of course needed I also needed a database. Now, the perfect service for this then is Supabase. Here, this is a really cool project. It gives you a Postgres database. It gives you a login server. And it it gives you everything you need to run a simple CRUD application like this.

So, I I I like I was considering like setting up Keycloak and running my own database, etc. But, at this stage, that is not where you want to focus on. Like when you build software, when you build something, you want to get something running first. You test it with some users. This is what I did. I I had a very simple application first. I invited 50 of my members to test it to see if they would think it is useful. They said it was useful, so I I figured, "Okay, I want to scale this up." And then you want something to be up and running as soon as possible. So, then using a service like Supabase is then a really good option because it helps you get up and running really quick. So, the RAG backend then queries the Supabase database. Supabase Postgres DB. So, in this database, I have stored all of these the these transcripts, the forum posts, etc.

All of that is stored as embeddings. And because they're stored as embeddings, the RAG application is very efficient at retrieving that information and matching it to the correct chunks of the information that is actually relevant to the question that the user is answering. So, that is how it works. And then of course, this is all um um And yeah, so the the authentication happens at the front. So, here there will be a Let's say call it a box around it. And this would be then front front end off. So, the only user-facing application is actual front end. This off is then also handled through Supabase. So, when someone wants to log in, they first have to Well, it's not the Supabase Postgres DB, but you get the idea. Someone wants to log in, they need to enter their signing code.

So, we can actually test this. If I go to chat.cubecraftlabs.io, you won't be able to log in. And if I just do imaginary at email.com, send the code, then it will say that email is not on the invite list. And if you want to get access to this, you can click that link and apply to join Cubecraft, because that's the only way that you will get access to this. So, that is working That is how big picture how it works. So, Supabase really makes sense in this kind of situation. And when you are As I said, you when you are building an application, you want to get something up and running as fast as possible. And even though I am perfectly capable of setting up Keycloak, of setting up a self-hosted Postgres cluster on my Kubernetes cluster, I can easily do that, but that is adding layers of maintenance, right? And at this stage, I am interested in getting it up and running as quickly as possible, and to see the validity of this application. Like, I might not be running this after 3 months if my users are not using it and it's not providing value. Right now, the indication is really good. People really love it, and they're getting value out of it, but you don't know. So, it's makes much more sense to just pay, I don't know, I think I'm paying 25 bucks a month.

This gives me a database that's fully managed, backed up. I can always restore it. And it gives me authentication with a really neat API, which is a bit of a pain in the ass to set up yourself if you're going to do it from scratch. And for those 25 bucks a month, I am definitely getting back, I I would say at least an hour of maintenance a month. And if And And of course, it would have taken me maybe, I don't know, maybe 20 hours to build this. So, for now, this is a really good trade-off. And then later, if the app is successful in the sense that my users are using it, getting value out of it, then I might consider if I want to have that maintenance burden. I can consider taking out some of these pieces, like for example, the Supabase Postgres DB. I can self-host it by database. I can set up my own auth and and take it in-house. But, you you first want to get something up and running.

Get it Use a paid service for that. Get it up and running. Like relieve yourself of maintenance and setup workload, and then later you can decide on what to take back. So, that's my thinking around this. And of course, the last piece of this is the LLM. So, what am I actually using to formulate the answers here? So, I actually started this project because I wanted to learn more about local AI on Kubernetes. So, initially, and if I if I open the cluster, when you see the pods here, you will see that there is a Gemma 26B KServe router, and there's a KServe uh pod running. So, I was I I actually was using this project to learn KServe and I have a node here with a 4090 GPU and I I had that in the cluster and I was initially just doing this with local inference, so running a local AI model.

However I and I got that that to work, but I found that the context window that I could get on my GPU unfortunately was a bit too small, so I need to either add more GPUs to this or maybe use some of these network attached GPU solutions. So, for now I decided to just use the normal LLM APIs. So, I could use OpenAI, ChatGPT of course. I could use Anthropic, but I actually chose to use Grok. So, right at this moment I'm the the application is calling Grok 4.6. It is getting lots of good it's getting really good results on this. I haven't actually even tried any of the others. The reason why I went for Grok is well, first of all, because I like to try try unusual things. So, the the main the main the first knee-jerk reaction is to go straight to OpenAI or straight to Anthropic and I build everything with Claude by the way, Anthropic Claude.

I use Opus and Fable for this, but the interesting thing here is that XAI, so the company by Elon Musk, they have these huge compute clusters, but they don't have many people using their models yet. So, Grok currently is quite competitively priced. So, the the quality you get per dollar, let's say, is much higher than what than what I I at Anthropic, I would say, because using Anthropic to answer these questions, uh I found that what that was quite um quite more expensive. And I just think it's fun to use something that is not mainstream. So, that's the reason for that, but I am fully in I'm fully intending to bring back my local inference because that is what I'm building my skill set towards. And um I I it was a lot of fun to just do it that way. But with SD upscaling and I needed more um LLM compute, then I figured, "Okay, I will just use a frontier model for now."

So, that's what I'm using right now. Next, we're going to get into how am I actually running this then? So, now we know the basic architecture of the application. Now, let's get into get into where this is actually running. So, me being Misha van den Berg, of course, this is running on Kubernetes. And I'm running this on a three-node Kubernetes cluster that is currently running as three virtual machines um that are located in Germany at Hetzner. So, there's three control planes. I'm using uh Talos Linux to provision the Kubernetes cluster. And I am running the application as follows. If I show you the cluster here, I can get the namespaces. So, you will see that I have several namespaces. Here, there's the CubeCraft AI portal. This is like the internal admin portal that me and my team use to uh manage access to this, where I can remove users, et cetera. Then, there is the staging environment, and then there's the um prod environment. So, if I do K config set context CubeCraft AI namespace grok prod So, if I now do K get pods, I get all of the pods that are running in the current namespace.

So, you will see that then the rag back end and the chat UI are simply two separate deployments. These are both running in two replicas. And because I'm running three servers and those are all control planes. So, in this case, I'm allowing workloads to be scheduled on my control planes. In this case, I think this is completely fine because the load is not that high. But because it's like this, I can I I also make sure that one server is always online. So, if I do K get PDB, then you will see that I'm running a pod disruption budget. And this means that there's going to be always one replica available at minimum. So, if I were to run a update, for example, or if I do a K get pods a wide, so here we see that these pods are spread out on several nodes, right?

So, we see that for example, the Hetzner control plane 02 has both a rag back end and chat UI on there. So, because the application is running in multiple replicas and I have made sure that there are pod disruption budgets available on the cluster, this means that whenever I'm doing an update or something like that, it is going to intelligently figure out on which node it should put it. And there in that way, I can ensure that the application is always online. So, I'm going to do a separate video on where I go deeper into why I chose Kubernetes for this specifically. But this is one of the key factors that this just makes it really easy for something to be online at all times. Plus I already have the Kubernetes skills. So, for me it is actually not so big so such a big of a deal to be running this. Now, finally, of course, I'm running this as an application with actual end users, so I have a pretty elaborate alerting setup. And one of the advantages of using Kubernetes for this is that you get really good monitoring tooling for free right out of the box.

Like, of course, I had to install this. This is not part of Talos Linux. But at this point, these these applications are so they are working so well that um it's it's really easy to get this up and running. So, I have my my app my alerts here. I will get Slack messages on my phone when something goes wrong. I've also built a bunch of custom dashboards. So, for example, I have some really nice insights if I want to, for example, take a look at the usage and cost. So, when I look at this dashboard, you can see the cost that I have so far and how much how many tokens are being used per day. I can also see which users are the biggest token users here, for example. We request per day by by status. And I have a couple of other really cool dashboards here well that give me some user insights, etc. And this is one of the reasons why I really love using the cloud native ecosystem because it just makes it so easy to build this kind of stuff on top of it, right?

So, for the in the past 24 hours, 25 chats have been sent. We're doing about six chats an hour, 100 chats in the la- in in the past um 7 days, etc. So, I get all of the important information. I get alerting. I can set up alerting for when the cost goes out of hand, etc. And it's working really well. So, that is how the application works. If you're interested in becoming a DevOps engineer, check out the free road map that's in the description below. You can fill out your information and it will tell you which stage you're in and what you need to do to get out of that stage. It's completely for free and if you do want to get access to CubeCraft AI and the career accelerator, there's some information below this video as well. Thank you so much for watching and I'll see you in the next one.