Which Proxmox Storage Should You Use?

Hey guys, today I'd like to show and Hey guys, today I'd like to show and discuss all the different storage discuss all the different storage discuss all the different storage options we have on Proxbox because I options we have on Proxbox because I options we have on Proxbox because I think this is a quite interesting topic. think this is a quite interesting topic. think this is a quite interesting topic. We just briefly discussed here on this We just briefly discussed here on this We just briefly discussed here on this channel like in one of my videos about channel like in one of my videos about channel like in one of my videos about my Proxmox seph cluster project, but my Proxmox seph cluster project, but my Proxmox seph cluster project, but some of the comments on this video were some of the comments on this video were some of the comments on this video were divided. Some of you called Seph a true divided. Some of you called Seph a true divided. Some of you called Seph a true game changer while others said, "Well, game changer while others said, "Well, game changer while others said, "Well, it is too complicated. It is too it is too complicated. It is too it is too complicated. It is too expensive because you need three nodes expensive because you need three nodes expensive because you need three nodes minimum." And several of you have minimum." And several of you have minimum." And several of you have replaced that with ZFS replication or replaced that with ZFS replication or replaced that with ZFS replication or are still using NFS on a NAS device. And are still using NFS on a NAS device. And are still using NFS on a NAS device. And especially if you're new to Proxmox or especially if you're new to Proxmox or especially if you're new to Proxmox or home labbing in general, all the home labbing in general, all the home labbing in general, all the different storage options might be even different storage options might be even different storage options might be even more confusing and you might wonder what more confusing and you might wonder what more confusing and you might wonder what should you use for your home lab setup. should you use for your home lab setup. should you use for your home lab setup. It's mostly always a compromise between It's mostly always a compromise between It's mostly always a compromise between simplicity, cost, availability, simplicity, cost, availability, simplicity, cost, availability, performance, and how much you actually performance, and how much you actually performance, and how much you actually want to get for your home lab. So I want to get for your home lab. So I want to get for your home lab. So I think let's talk about all of this. think let's talk about all of this. think let's talk about all of this. We'll go through all the different We'll go through all the different We'll go through all the different storage options step by step and I'm storage options step by step and I'm storage options step by step and I'm telling you exactly what my opinion and telling you exactly what my opinion and telling you exactly what my opinion and experience is on those. So, let's get experience is on those. So, let's get experience is on those. So, let's get started. So, the best way to start this started. So, the best way to start this started. So, the best way to start this video is by just taking a look at the video is by just taking a look at the video is by just taking a look at the official Proxmox documentation because official Proxmox documentation because official Proxmox documentation because here we got all the different storage here we got all the different storage here we got all the different storage options explained and shown in a simple options explained and shown in a simple options explained and shown in a simple chart. And Proxmox divides storage types chart. And Proxmox divides storage types chart. And Proxmox divides storage types in two categories file level storage and in two categories file level storage and in two categories file level storage and block level storage. It is important to block level storage. It is important to block level storage. It is important to mention that the virtual machine itself mention that the virtual machine itself mention that the virtual machine itself always sees a virtual block device. So always sees a virtual block device. So always sees a virtual block device. So that is basically a disk that its

that is basically a disk that its that is basically a disk that its operating system can partition, format operating system can partition, format operating system can partition, format and just use like a physical drive. But and just use like a physical drive. But and just use like a physical drive. But Proxbox itself can store that virtual Proxbox itself can store that virtual Proxbox itself can store that virtual disc differently on the hypervisor and disc differently on the hypervisor and disc differently on the hypervisor and there we have the file level storage. So there we have the file level storage. So there we have the file level storage. So these are basically virtual machine these are basically virtual machine these are basically virtual machine discs stored as a normal file on the discs stored as a normal file on the discs stored as a normal file on the system which is used by local directory system which is used by local directory system which is used by local directory storage NFS CFS and this is by the way storage NFS CFS and this is by the way storage NFS CFS and this is by the way also used for storing ISOs storing also used for storing ISOs storing also used for storing ISOs storing backup files and then we have the block backup files and then we have the block backup files and then we have the block level storage. So there is no normal level storage. So there is no normal level storage. So there is no normal virtual machine disk stored as a file. virtual machine disk stored as a file. virtual machine disk stored as a file. Instead, that storage allocates a Instead, that storage allocates a Instead, that storage allocates a logical volume or a block image for the logical volume or a block image for the logical volume or a block image for the virtual machine such as LVM, LVM thin, virtual machine such as LVM, LVM thin, virtual machine such as LVM, LVM thin, ZFS, ZV, SER, RBD and ICSI or ICSI use ZFS, ZV, SER, RBD and ICSI or ICSI use ZFS, ZV, SER, RBD and ICSI or ICSI use that model as well. Here in the chart, that model as well. Here in the chart, that model as well. Here in the chart, it's all described what are the it's all described what are the it's all described what are the different levels. ZFS for example can different levels. ZFS for example can different levels. ZFS for example can use both file and block level storage. use both file and block level storage. use both file and block level storage. Whether this is shared so that means the Whether this is shared so that means the Whether this is shared so that means the same storage can be accessed by multiple same storage can be accessed by multiple same storage can be accessed by multiple Proxmox node at the same time. This is Proxmox node at the same time. This is Proxmox node at the same time. This is important if you have a Proxmox cluster important if you have a Proxmox cluster important if you have a Proxmox cluster and of course most of them support and of course most of them support and of course most of them support snapshots and they are also mostly all snapshots and they are also mostly all snapshots and they are also mostly all stable except this one here BDRFS. Now stable except this one here BDRFS. Now stable except this one here BDRFS. Now this Proxbox calls a technology preview this Proxbox calls a technology preview this Proxbox calls a technology preview with best effort support. So you can use with best effort support. So you can use with best effort support. So you can use it but honestly if you want to hear my it but honestly if you want to hear my it but honestly if you want to hear my opinion you should not use that. You opinion you should not use that. You opinion you should not use that. You should probably just use ZFS. That's a should probably just use ZFS. That's a should probably just use ZFS. That's a better option. Now the Proxmark storage better option. Now the Proxmark storage better option. Now the Proxmark storage generally is configured under the data generally is configured under the data generally is configured under the data center storage and then you can add all

center storage and then you can add all center storage and then you can add all these different storage options these different storage options these different storage options directory LVM LVM FIN and all the others directory LVM LVM FIN and all the others directory LVM LVM FIN and all the others as well. Now here I'd like to start with as well. Now here I'd like to start with as well. Now here I'd like to start with the simplest that is the directory the simplest that is the directory the simplest that is the directory storage. So that is basically a normal storage. So that is basically a normal storage. So that is basically a normal directory inside the mounted file system directory inside the mounted file system directory inside the mounted file system of the Proxbox hypervisor and Proxmox of the Proxbox hypervisor and Proxmox of the Proxbox hypervisor and Proxmox already uses this storage type by already uses this storage type by already uses this storage type by default for its local storage for ISO default for its local storage for ISO default for its local storage for ISO images, container templates or backups. images, container templates or backups. images, container templates or backups. However, you can also use a directory However, you can also use a directory However, you can also use a directory storage if you really want that to hold storage if you really want that to hold storage if you really want that to hold virtual machine disk. So that is when virtual machine disk. So that is when virtual machine disk. So that is when you enable the disk image content type. you enable the disk image content type. you enable the disk image content type. Here for example I've created a new Here for example I've created a new Here for example I've created a new directory storage attach this to my directory storage attach this to my directory storage attach this to my second proxmox node and then when we second proxmox node and then when we second proxmox node and then when we open this here you can see that is the open this here you can see that is the open this here you can see that is the directory demo when you go to virtual directory demo when you go to virtual directory demo when you go to virtual machine discs you will see that Q code 2 machine discs you will see that Q code 2 machine discs you will see that Q code 2 file so this is attached to that virtual file so this is attached to that virtual file so this is attached to that virtual machine temp test one now before we move machine temp test one now before we move machine temp test one now before we move on and discuss all the other options I'd on and discuss all the other options I'd on and discuss all the other options I'd like to talk about one thing that you like to talk about one thing that you like to talk about one thing that you need regardless of which storage you need regardless of which storage you need regardless of which storage you choose and that is a safe way to access choose and that is a safe way to access choose and that is a safe way to access your Proxmox server from outside when your Proxmox server from outside when your Proxmox server from outside when you're not at home. And for this, my you're not at home. And for this, my you're not at home. And for this, my first choice is Twing, the sponsor of first choice is Twing, the sponsor of first choice is Twing, the sponsor of today's video. Twate is great for these today's video. Twate is great for these today's video. Twate is great for these situations, just logging into my HomeLab situations, just logging into my HomeLab situations, just logging into my HomeLab servers over SSH or opening the Proxmox servers over SSH or opening the Proxmox servers over SSH or opening the Proxmox web UI. It is what we call ZTNA intact, web UI. It is what we call ZTNA intact, web UI. It is what we call ZTNA intact, a zero trust network access platform a zero trust network access platform a zero trust network access platform that establishes secure connections that establishes secure connections that establishes secure connections between all of my devices where every

between all of my devices where every between all of my devices where every request has to be verified and request has to be verified and request has to be verified and authorized otherwise Twinate won't let authorized otherwise Twinate won't let authorized otherwise Twinate won't let the request go through. And the cool the request go through. And the cool the request go through. And the cool thing is that this technology even works thing is that this technology even works thing is that this technology even works through Nut devices and firewalls through Nut devices and firewalls through Nut devices and firewalls without any port forwardings or firewall without any port forwardings or firewall without any port forwardings or firewall exceptions required. So you can just log exceptions required. So you can just log exceptions required. So you can just log into Twin Gate and securely access all into Twin Gate and securely access all into Twin Gate and securely access all of your systems no matter where you are of your systems no matter where you are of your systems no matter where you are or where you're connecting from. If or where you're connecting from. If or where you're connecting from. If you'd like to try it out, then check out you'd like to try it out, then check out you'd like to try it out, then check out my tutorials on how to install Twinate my tutorials on how to install Twinate my tutorials on how to install Twinate in your network and how to integrate it in your network and how to integrate it in your network and how to integrate it into Docker, Kubernetes or your DevOps into Docker, Kubernetes or your DevOps into Docker, Kubernetes or your DevOps environments. So start making your environments. So start making your environments. So start making your network more secure and accessible with network more secure and accessible with network more secure and accessible with Twing. Of course, I leave you link in Twing. Of course, I leave you link in Twing. Of course, I leave you link in the description box down below. the description box down below. the description box down below. By default, Proxmox uses LVM thin. So By default, Proxmox uses LVM thin. So By default, Proxmox uses LVM thin. So LVM stands for logical volume manager LVM stands for logical volume manager LVM stands for logical volume manager and that stores every virtual machine and that stores every virtual machine and that stores every virtual machine disc at its own fixed size block volume disc at its own fixed size block volume disc at its own fixed size block volume instead of a simple file. The complete instead of a simple file. The complete instead of a simple file. The complete space is reserved as soon as you create space is reserved as soon as you create space is reserved as soon as you create that virtual disc on an LVM storage. So that virtual disc on an LVM storage. So that virtual disc on an LVM storage. So that is also the difference between the that is also the difference between the that is also the difference between the normal LVM and the LVM thin. So when you normal LVM and the LVM thin. So when you normal LVM and the LVM thin. So when you create a virtual machine disc on an LVM create a virtual machine disc on an LVM create a virtual machine disc on an LVM storage that has for example the virtual storage that has for example the virtual storage that has for example the virtual machine disc gets 40 GB disc then LVM machine disc gets 40 GB disc then LVM machine disc gets 40 GB disc then LVM reserves all that 40 GB immediately even reserves all that 40 GB immediately even reserves all that 40 GB immediately even when the guest has only written 5 GB when the guest has only written 5 GB when the guest has only written 5 GB inside the guest operating system. That inside the guest operating system. That inside the guest operating system. That is also the reason why you should is also the reason why you should is also the reason why you should certainly prefer LVM thin because when certainly prefer LVM thin because when certainly prefer LVM thin because when you then provision a new disk, it only you then provision a new disk, it only you then provision a new disk, it only allocates the space that the guest allocates the space that the guest allocates the space that the guest operating system has written to it. And

operating system has written to it. And operating system has written to it. And that is probably one of the catches that is probably one of the catches that is probably one of the catches here. You can assign more virtual here. You can assign more virtual here. You can assign more virtual storage than the pool physically storage than the pool physically storage than the pool physically contains. So that means if you create contains. So that means if you create contains. So that means if you create virtual machines, one with 40 GB, one virtual machines, one with 40 GB, one virtual machines, one with 40 GB, one with 100 GB, 200 GB and you exceed that with 100 GB, 200 GB and you exceed that with 100 GB, 200 GB and you exceed that storage that is physically available on storage that is physically available on storage that is physically available on the physical disc, then you might get in the physical disc, then you might get in the physical disc, then you might get in trouble. So you need to calculate this trouble. So you need to calculate this trouble. So you need to calculate this properly and you need to be aware that properly and you need to be aware that properly and you need to be aware that if you're creating virtual machines with if you're creating virtual machines with if you're creating virtual machines with larger discs that you still have enough larger discs that you still have enough larger discs that you still have enough space on your local disk where you space on your local disk where you space on your local disk where you create that LVM storage on. Now to create that LVM storage on. Now to create that LVM storage on. Now to create such an LVM thin storage, you create such an LVM thin storage, you create such an LVM thin storage, you need to first of all go to the Proxmox need to first of all go to the Proxmox need to first of all go to the Proxmox node and go to disks here. So if you node and go to disks here. So if you node and go to disks here. So if you find the physical disk in here, you can find the physical disk in here, you can find the physical disk in here, you can go to LVM thin and then you can create a go to LVM thin and then you can create a go to LVM thin and then you can create a thin pool. So then you need to select thin pool. So then you need to select thin pool. So then you need to select the disk. Currently I have no disk the disk. Currently I have no disk the disk. Currently I have no disk available. So I only have one and that available. So I only have one and that available. So I only have one and that is used by the volume group PVE. But is used by the volume group PVE. But is used by the volume group PVE. But normally you select your disk here, normally you select your disk here, normally you select your disk here, create a name for this thin pool and create a name for this thin pool and create a name for this thin pool and then you can click on add storage to then you can click on add storage to then you can click on add storage to automatically add this to your data automatically add this to your data automatically add this to your data center storage or you can leave it as it center storage or you can leave it as it center storage or you can leave it as it is. Create these LVM thin pools on all is. Create these LVM thin pools on all is. Create these LVM thin pools on all the Proxmox nodes selecting your the Proxmox nodes selecting your the Proxmox nodes selecting your physical disks and then go to data physical disks and then go to data physical disks and then go to data center storage and create a new LVM thin center storage and create a new LVM thin center storage and create a new LVM thin storage. select that volume group that storage. select that volume group that storage. select that volume group that you have created and then you can also you have created and then you can also you have created and then you can also enable that on each of these nodes.

enable that on each of these nodes. enable that on each of these nodes. Again, by default, Proxbox uses this and Again, by default, Proxbox uses this and Again, by default, Proxbox uses this and it creates a storage type local LVM that it creates a storage type local LVM that it creates a storage type local LVM that is used for disk images and containers. is used for disk images and containers. is used for disk images and containers. So, this is normally uh by default So, this is normally uh by default So, this is normally uh by default created when you add a single Proxmox created when you add a single Proxmox created when you add a single Proxmox node with a physical disk. So it is not node with a physical disk. So it is not node with a physical disk. So it is not shared and it also although it supports shared and it also although it supports shared and it also although it supports snapshots and backups it does not have snapshots and backups it does not have snapshots and backups it does not have any redundancy by itself. So when you any redundancy by itself. So when you any redundancy by itself. So when you create that on a physical disc and that create that on a physical disc and that create that on a physical disc and that is only an SSD or an NVME drive. This is is only an SSD or an NVME drive. This is is only an SSD or an NVME drive. This is a single point of failure. You can still a single point of failure. You can still a single point of failure. You can still make this redundant in professional make this redundant in professional make this redundant in professional server systems by using a hardware rate. server systems by using a hardware rate. server systems by using a hardware rate. So this is a physical rate card where So this is a physical rate card where So this is a physical rate card where you connect multiple drives. However, I you connect multiple drives. However, I you connect multiple drives. However, I suppose that not many of you will have a suppose that not many of you will have a suppose that not many of you will have a professional enterprise server where you professional enterprise server where you professional enterprise server where you want to use or can use hardware rate. want to use or can use hardware rate. want to use or can use hardware rate. Certainly, there are even motherboards Certainly, there are even motherboards Certainly, there are even motherboards on consumer hardware that support rate on consumer hardware that support rate on consumer hardware that support rate options. But in my opinion, it is much options. But in my opinion, it is much options. But in my opinion, it is much better to use a software rate and not better to use a software rate and not better to use a software rate and not use hardware rate. And that is by using use hardware rate. And that is by using use hardware rate. And that is by using the next storage option that we have the next storage option that we have the next storage option that we have ZFS. Because ZFS manages the local discs ZFS. Because ZFS manages the local discs ZFS. Because ZFS manages the local discs and their redundancy directly while and their redundancy directly while and their redundancy directly while Proxmbox stores the virtual machine Proxmbox stores the virtual machine Proxmbox stores the virtual machine discs as block devices called Z volumes.

discs as block devices called Z volumes. discs as block devices called Z volumes. And that provides lot of great features And that provides lot of great features And that provides lot of great features like ZFS has check sums. It has like ZFS has check sums. It has like ZFS has check sums. It has compression to make the virtual machine compression to make the virtual machine compression to make the virtual machine discs consume less storage on the discs consume less storage on the discs consume less storage on the physical devices. It supports snapshots physical devices. It supports snapshots physical devices. It supports snapshots rollbacks. It supports software defined rollbacks. It supports software defined rollbacks. It supports software defined redundancy through mirrors or RAZ Z. So redundancy through mirrors or RAZ Z. So redundancy through mirrors or RAZ Z. So that means that you can bundle multiple that means that you can bundle multiple that means that you can bundle multiple hard drives together to one larger hard drives together to one larger hard drives together to one larger storage pool and when one drive fails storage pool and when one drive fails storage pool and when one drive fails that is no problem. All the data is that is no problem. All the data is that is no problem. All the data is still there. You can just replace that still there. You can just replace that still there. You can just replace that damaged drive and you don't have to roll damaged drive and you don't have to roll damaged drive and you don't have to roll back or restore anything that for back or restore anything that for back or restore anything that for example I'm using on my first Proxmox example I'm using on my first Proxmox example I'm using on my first Proxmox server. Here you can see I'm using rate server. Here you can see I'm using rate server. Here you can see I'm using rate Z2 with five hard drives. So these are Z2 with five hard drives. So these are Z2 with five hard drives. So these are just magnetic hard drives connected to just magnetic hard drives connected to just magnetic hard drives connected to the storage controller or you can also the storage controller or you can also the storage controller or you can also connect that directly on the motherboard connect that directly on the motherboard connect that directly on the motherboard of the computer and then you can create of the computer and then you can create of the computer and then you can create a new ZFS storage pool. Give it a name, a new ZFS storage pool. Give it a name, a new ZFS storage pool. Give it a name, select the discs and then select your select the discs and then select your select the discs and then select your rate level. You can use a single disk rate level. You can use a single disk rate level. You can use a single disk with ZFS. So even if you do not need with ZFS. So even if you do not need with ZFS. So even if you do not need redundancy at all, but you still want to redundancy at all, but you still want to redundancy at all, but you still want to use the features of compression, use the features of compression, use the features of compression, snapshots and so on, you can create a snapshots and so on, you can create a snapshots and so on, you can create a single disc. can also mirror one disc to single disc. can also mirror one disc to single disc. can also mirror one disc to another of the identical size. So that another of the identical size. So that another of the identical size. So that means if you lose one disc, the other means if you lose one disc, the other means if you lose one disc, the other still has the same data. However, this still has the same data. However, this still has the same data. However, this is not the most efficient one because is not the most efficient one because is not the most efficient one because you're sacrificing one disc completely you're sacrificing one disc completely you're sacrificing one disc completely because you're mirroring the data. So in because you're mirroring the data. So in because you're mirroring the data. So in my opinion, the best options are rate Z.

my opinion, the best options are rate Z. my opinion, the best options are rate Z. So this leaves one disc for par. A So this leaves one disc for par. A So this leaves one disc for par. A minimum of three discs are required, but minimum of three discs are required, but minimum of three discs are required, but you're only sacrificing the storage you're only sacrificing the storage you're only sacrificing the storage capacity of one drive. So if each of the capacity of one drive. So if each of the capacity of one drive. So if each of the discs have one terabyte, you have two discs have one terabyte, you have two discs have one terabyte, you have two terabytes in total. And you can also add terabytes in total. And you can also add terabytes in total. And you can also add a second drive as a parity with RAID Z2 a second drive as a parity with RAID Z2 a second drive as a parity with RAID Z2 and a third parity with RAID Z3. That and a third parity with RAID Z3. That and a third parity with RAID Z3. That only makes sense, of course, if you got only makes sense, of course, if you got only makes sense, of course, if you got more discs. So then you have a fall more discs. So then you have a fall more discs. So then you have a fall tolerance of two or three discs meaning tolerance of two or three discs meaning tolerance of two or three discs meaning two discs at the same time can fail two discs at the same time can fail two discs at the same time can fail which can happen sometimes because some which can happen sometimes because some which can happen sometimes because some storage controllers they are somehow storage controllers they are somehow storage controllers they are somehow bundling the different uh interfaces on bundling the different uh interfaces on bundling the different uh interfaces on on a chip that supports two ports and on a chip that supports two ports and on a chip that supports two ports and this is really good option because it this is really good option because it this is really good option because it provides you data integrity compression provides you data integrity compression provides you data integrity compression snapshots clones local disc redundancy snapshots clones local disc redundancy snapshots clones local disc redundancy without a separate storage server. It's without a separate storage server. It's without a separate storage server. It's really really great. No, this is really really great. No, this is really really great. No, this is certainly by far the best option that I certainly by far the best option that I certainly by far the best option that I would choose on a single server and it would choose on a single server and it would choose on a single server and it can still be a good option if you got a can still be a good option if you got a can still be a good option if you got a second or third Proxmox node in a second or third Proxmox node in a second or third Proxmox node in a cluster. The main issue with that setup cluster. The main issue with that setup cluster. The main issue with that setup is that if you're running a cluster and is that if you're running a cluster and is that if you're running a cluster and you need to migrate virtual machines you need to migrate virtual machines you need to migrate virtual machines from one node to another um that need to from one node to another um that need to from one node to another um that need to copy the data first and when that's a copy the data first and when that's a copy the data first and when that's a ZFS storage or an LVM thin storage that ZFS storage or an LVM thin storage that ZFS storage or an LVM thin storage that only exists on a on a disk on that only exists on a on a disk on that only exists on a on a disk on that physical server all the data needs to be physical server all the data needs to be physical server all the data needs to be copied to the other node. The same is copied to the other node. The same is copied to the other node. The same is when that one node dies completely and when that one node dies completely and when that one node dies completely and you cannot access the data from that you cannot access the data from that you cannot access the data from that server. So you need to somehow repair

server. So you need to somehow repair server. So you need to somehow repair the server to get the virtual machines the server to get the virtual machines the server to get the virtual machines uh back online again. As soon as you uh back online again. As soon as you uh back online again. As soon as you start adding a second or a third start adding a second or a third start adding a second or a third Proxmbox node to your setup, ZFS or LVM Proxmbox node to your setup, ZFS or LVM Proxmbox node to your setup, ZFS or LVM thin are not enough and you need to thin are not enough and you need to thin are not enough and you need to think about how do you deal with think about how do you deal with think about how do you deal with redundancy and how do you copy the data redundancy and how do you copy the data redundancy and how do you copy the data or make this available to the other or make this available to the other or make this available to the other nodes. And one option most of you have nodes. And one option most of you have nodes. And one option most of you have recommended in the comments of uh my recommended in the comments of uh my recommended in the comments of uh my self storage video, you should use self storage video, you should use self storage video, you should use storage replication. That is something storage replication. That is something storage replication. That is something that I personally have not tried out that I personally have not tried out that I personally have not tried out yet. I'm going to be honest with you. I yet. I'm going to be honest with you. I yet. I'm going to be honest with you. I just did a few tests here to see how just did a few tests here to see how just did a few tests here to see how it's generally working. But that has it's generally working. But that has it's generally working. But that has never been a reliable and good option in never been a reliable and good option in never been a reliable and good option in my opinion at least. Now just to explain my opinion at least. Now just to explain my opinion at least. Now just to explain how replication works, Proxmox how replication works, Proxmox how replication works, Proxmox configures replication by virtual configures replication by virtual configures replication by virtual machines or containers. If you select machines or containers. If you select machines or containers. If you select this year Proxmox server 7 and go to this year Proxmox server 7 and go to this year Proxmox server 7 and go to replication, you can add a replication replication, you can add a replication replication, you can add a replication and select a target node like Proxmox and select a target node like Proxmox and select a target node like Proxmox pro one or two and then say like every pro one or two and then say like every pro one or two and then say like every 30 minutes or every 15 minutes. That's 30 minutes or every 15 minutes. That's 30 minutes or every 15 minutes. That's by the way the minimum you can select.

by the way the minimum you can select. by the way the minimum you can select. It should copy the disk to the other It should copy the disk to the other It should copy the disk to the other node. And if you're using ZFS, that is a node. And if you're using ZFS, that is a node. And if you're using ZFS, that is a good way to do it because ZFS allows you good way to do it because ZFS allows you good way to do it because ZFS allows you to first of all run one large initial to first of all run one large initial to first of all run one large initial copy to the other node and then only copy to the other node and then only copy to the other node and then only copy incremental updates. That only copy incremental updates. That only copy incremental updates. That only needs to copy every 50 minutes just a needs to copy every 50 minutes just a needs to copy every 50 minutes just a few megabytes for example. And that's few megabytes for example. And that's few megabytes for example. And that's really fast. And when one node dies, the really fast. And when one node dies, the really fast. And when one node dies, the other node has the data from probably in other node has the data from probably in other node has the data from probably in the worst case uh the last 50 minutes the worst case uh the last 50 minutes the worst case uh the last 50 minutes ago. Now, that of course is also a ago. Now, that of course is also a ago. Now, that of course is also a viable option if you only have like I'd viable option if you only have like I'd viable option if you only have like I'd say two nodes and you want to use ZFS say two nodes and you want to use ZFS say two nodes and you want to use ZFS because it has all the great features because it has all the great features because it has all the great features and you still want to have some fault and you still want to have some fault and you still want to have some fault tolerance or redundancy on the other tolerance or redundancy on the other tolerance or redundancy on the other nodes. However, I also want to say that nodes. However, I also want to say that nodes. However, I also want to say that this is in my opinion not a true high this is in my opinion not a true high this is in my opinion not a true high availability because if you're only availability because if you're only availability because if you're only copying every 50 minutes, you can lose copying every 50 minutes, you can lose copying every 50 minutes, you can lose when something changes during that time. when something changes during that time. when something changes during that time. So that's why I think replication is So that's why I think replication is So that's why I think replication is somewhat a practical middle ground for somewhat a practical middle ground for somewhat a practical middle ground for smaller setups like homelabs or so if smaller setups like homelabs or so if smaller setups like homelabs or so if you are starting with a two node Proxmox you are starting with a two node Proxmox you are starting with a two node Proxmox cluster and you have no other good cluster and you have no other good cluster and you have no other good option to solve that problem I think option to solve that problem I think option to solve that problem I think replication can work pretty well but if replication can work pretty well but if replication can work pretty well but if you want to configure true high you want to configure true high you want to configure true high availability availability availability if the virtual machine should if the virtual machine should if the virtual machine should automatically restart after node failure automatically restart after node failure automatically restart after node failure this is certainly not the right option.

this is certainly not the right option. this is certainly not the right option. So that means instead of using a local So that means instead of using a local So that means instead of using a local storage that is only available to one storage that is only available to one storage that is only available to one node and where you need to copy the data node and where you need to copy the data node and where you need to copy the data over another node, you use a shared over another node, you use a shared over another node, you use a shared storage that is available to all the storage that is available to all the storage that is available to all the Proxmox nodes at the same time. So using Proxmox nodes at the same time. So using Proxmox nodes at the same time. So using only one storage location and all the only one storage location and all the only one storage location and all the nodes can access that location. One of nodes can access that location. One of nodes can access that location. One of the options to do that is NFS. With NFS, the options to do that is NFS. With NFS, the options to do that is NFS. With NFS, the Proxmox server connects to a remote the Proxmox server connects to a remote the Proxmox server connects to a remote server where the data is stored. server where the data is stored. server where the data is stored. Ideally, that would be a NAS device that Ideally, that would be a NAS device that Ideally, that would be a NAS device that has larger discs where you can of course has larger discs where you can of course has larger discs where you can of course also use EFS or other redundant storage also use EFS or other redundant storage also use EFS or other redundant storage options and then run the virtual options and then run the virtual options and then run the virtual machines from there. For example, if machines from there. For example, if machines from there. For example, if you're running unrade or a true nest you're running unrade or a true nest you're running unrade or a true nest storage server, there you can easily storage server, there you can easily storage server, there you can easily make this available to the Proxmox host. make this available to the Proxmox host. make this available to the Proxmox host. And then in Proxmox you just go to data And then in Proxmox you just go to data And then in Proxmox you just go to data center storage create and add a new NFS center storage create and add a new NFS center storage create and add a new NFS storage volume. You just need to give storage volume. You just need to give storage volume. You just need to give this a name and then select the IP this a name and then select the IP this a name and then select the IP address of your NAS device. The export address of your NAS device. The export address of your NAS device. The export that is available. For example, if I try that is available. For example, if I try that is available. For example, if I try that on my server, you should see that that on my server, you should see that that on my server, you should see that is the mount volume that you want to is the mount volume that you want to is the mount volume that you want to make available to your Proxmox node. And make available to your Proxmox node. And make available to your Proxmox node. And then you can also use that for disk then you can also use that for disk then you can also use that for disk images. You can use that for ISO images, images. You can use that for ISO images, images. You can use that for ISO images, container templates and so on. and then container templates and so on. and then container templates and so on. and then select which nodes should have access to select which nodes should have access to select which nodes should have access to that NFS central storage location. Of that NFS central storage location. Of that NFS central storage location. Of course, in a cluster, it makes the most course, in a cluster, it makes the most course, in a cluster, it makes the most sense to select all the free nodes. Then sense to select all the free nodes. Then sense to select all the free nodes. Then all the free nodes can access this

all the free nodes can access this all the free nodes can access this storage at the same time and you can storage at the same time and you can storage at the same time and you can create virtual machines running the create virtual machines running the create virtual machines running the disks from that storage and then you can disks from that storage and then you can disks from that storage and then you can easily migrate them over because when easily migrate them over because when easily migrate them over because when you do a live migration, Proxmbox only you do a live migration, Proxmbox only you do a live migration, Proxmbox only needs to copy the memory state from one needs to copy the memory state from one needs to copy the memory state from one node to another but the disk is node to another but the disk is node to another but the disk is available immediately. So you don't have available immediately. So you don't have available immediately. So you don't have to change this and this is really great. to change this and this is really great. to change this and this is really great. It's much simpler than running a SE It's much simpler than running a SE It's much simpler than running a SE storage when a reliable NAS system storage when a reliable NAS system storage when a reliable NAS system already exists. However, this also has already exists. However, this also has already exists. However, this also has its pros and cons of course. Yeah, its pros and cons of course. Yeah, its pros and cons of course. Yeah, because this heavily depends on a single because this heavily depends on a single because this heavily depends on a single point of failure or two single points of point of failure or two single points of point of failure or two single points of failure if in the worst case because it failure if in the worst case because it failure if in the worst case because it depends on the NAS device. If that NAS depends on the NAS device. If that NAS depends on the NAS device. If that NAS device has an outage for whatever device has an outage for whatever device has an outage for whatever reason, all your nodes cannot access the reason, all your nodes cannot access the reason, all your nodes cannot access the virtual machine discs anymore. And the virtual machine discs anymore. And the virtual machine discs anymore. And the second is you're constantly copying data second is you're constantly copying data second is you're constantly copying data over the network because when you need over the network because when you need over the network because when you need to start the virtual machines and the to start the virtual machines and the to start the virtual machines and the virtual machine disk itself is stored on virtual machine disk itself is stored on virtual machine disk itself is stored on a remote device and that virtual machine a remote device and that virtual machine a remote device and that virtual machine needs to write data to the disk what needs to write data to the disk what needs to write data to the disk what it's doing constantly then the data it's doing constantly then the data it's doing constantly then the data always needs to be written over the always needs to be written over the always needs to be written over the network. So that requires some fast network. So that requires some fast network. So that requires some fast hardware both on the NAS device. You hardware both on the NAS device. You hardware both on the NAS device. You certainly can run this with magnetic certainly can run this with magnetic certainly can run this with magnetic hard drives but it's not enjoyable to be hard drives but it's not enjoyable to be hard drives but it's not enjoyable to be honest. So, you at least need a NAS honest. So, you at least need a NAS honest. So, you at least need a NAS device that has SSDs or better NVMEs.

device that has SSDs or better NVMEs. device that has SSDs or better NVMEs. And then you need to have at least 10 GB And then you need to have at least 10 GB And then you need to have at least 10 GB networking or better even more. You can networking or better even more. You can networking or better even more. You can also use SMB or CIFS. That's also use SMB or CIFS. That's also use SMB or CIFS. That's particularly interesting for backups and particularly interesting for backups and particularly interesting for backups and installation media. You could also use installation media. You could also use installation media. You could also use CSI or SCASI. This is also a shared CSI or SCASI. This is also a shared CSI or SCASI. This is also a shared block storage. When you have a NAS that block storage. When you have a NAS that block storage. When you have a NAS that supports it, it's certainly not supports it, it's certainly not supports it, it's certainly not supported on all NAS operating systems. supported on all NAS operating systems. supported on all NAS operating systems. For example, I think that unrade did not For example, I think that unrade did not For example, I think that unrade did not had it. At least a few years ago, it did had it. At least a few years ago, it did had it. At least a few years ago, it did not had it. But Trun for example had not had it. But Trun for example had not had it. But Trun for example had ISI. This is mostly used in professional ISI. This is mostly used in professional ISI. This is mostly used in professional environments where you have existing environments where you have existing environments where you have existing SANS or capable NAS devices providing SANS or capable NAS devices providing SANS or capable NAS devices providing it. Then this is probably the better it. Then this is probably the better it. Then this is probably the better option. But any of the three are option. But any of the three are option. But any of the three are basically the same type of storage. basically the same type of storage. basically the same type of storage. Mostly in HomeLabs, I would not use them Mostly in HomeLabs, I would not use them Mostly in HomeLabs, I would not use them for running virtual machine discs. But for running virtual machine discs. But for running virtual machine discs. But you can still use them for as a backup you can still use them for as a backup you can still use them for as a backup destination for example that is really destination for example that is really destination for example that is really good in my opinion. So if you have a good in my opinion. So if you have a good in my opinion. So if you have a nest connected with NFS just store the nest connected with NFS just store the nest connected with NFS just store the Proxmox backups there that is probably Proxmox backups there that is probably Proxmox backups there that is probably the best option or the best use case for the best option or the best use case for the best option or the best use case for running NFS SMB CIFS or ICSI. All right running NFS SMB CIFS or ICSI. All right running NFS SMB CIFS or ICSI. All right so now we still have one and that is RBD so now we still have one and that is RBD so now we still have one and that is RBD and that is requiring a Proxmox SE and that is requiring a Proxmox SE and that is requiring a Proxmox SE cluster. So this is a shared block cluster. So this is a shared block cluster. So this is a shared block storage distributed across several nodes storage distributed across several nodes storage distributed across several nodes starting with a minimum size of three starting with a minimum size of three starting with a minimum size of three and that can be accessed by any of the and that can be accessed by any of the and that can be accessed by any of the nodes at the same time. In this setup, nodes at the same time. In this setup, nodes at the same time. In this setup, each node has one OSD. So that's the

each node has one OSD. So that's the each node has one OSD. So that's the small storage demon managing exactly one small storage demon managing exactly one small storage demon managing exactly one NVME discks. So here for example on all NVME discks. So here for example on all NVME discks. So here for example on all the free Proxmox nodes if you go to the free Proxmox nodes if you go to the free Proxmox nodes if you go to physical disks you can see that here I physical disks you can see that here I physical disks you can see that here I have added one NVMe with two terabytes have added one NVMe with two terabytes have added one NVMe with two terabytes that is existing on all the free nodes that is existing on all the free nodes that is existing on all the free nodes with an RBD image. So when you go to with an RBD image. So when you go to with an RBD image. So when you go to seph you can also see and check if seph you can also see and check if seph you can also see and check if everything is uh running fine how the everything is uh running fine how the everything is uh running fine how the performance is and so on and you can of performance is and so on and you can of performance is and so on and you can of course also use that for disk images and course also use that for disk images and course also use that for disk images and containers. Now, I won't go into too containers. Now, I won't go into too containers. Now, I won't go into too much detail here because I've explained much detail here because I've explained much detail here because I've explained this in depth in my SEF storage project this in depth in my SEF storage project this in depth in my SEF storage project video. I leave that in the description. video. I leave that in the description. video. I leave that in the description. If you're interested in a Free Node If you're interested in a Free Node If you're interested in a Free Node Proxbox cluster with a shared Proxbox cluster with a shared Proxbox cluster with a shared distributed storage, then you should distributed storage, then you should distributed storage, then you should check out that video. But this is a check out that video. But this is a check out that video. But this is a setup that I'm currently running. Of setup that I'm currently running. Of setup that I'm currently running. Of course, this has high hardware course, this has high hardware course, this has high hardware requirements. You need three Proxmox requirements. You need three Proxmox requirements. You need three Proxmox nodes at minimum. All of them need a nodes at minimum. All of them need a nodes at minimum. All of them need a separate physical drive and that is the separate physical drive and that is the separate physical drive and that is the minimum requirement. Better you have minimum requirement. Better you have minimum requirement. Better you have more and you are basically sacrificing more and you are basically sacrificing more and you are basically sacrificing the entire storage for for that because the entire storage for for that because the entire storage for for that because it always needs to distribute the same it always needs to distribute the same it always needs to distribute the same data to all of these nodes and data to all of these nodes and data to all of these nodes and constantly writing updates or sharing constantly writing updates or sharing constantly writing updates or sharing the updates in the network. So as you the updates in the network. So as you the updates in the network. So as you can imagine this also adds high can imagine this also adds high can imagine this also adds high requirement on the network back end. It requirement on the network back end. It requirement on the network back end. It also is recommended to use a separate also is recommended to use a separate also is recommended to use a separate network with a minimum of 10 GB speed network with a minimum of 10 GB speed network with a minimum of 10 GB speed for just that synchronization or for just that synchronization or for just that synchronization or distribution traffic and then use a

distribution traffic and then use a distribution traffic and then use a separate network for the front end. So separate network for the front end. So separate network for the front end. So when you're copying virtual machines, when you're copying virtual machines, when you're copying virtual machines, there's a separate HA link in Proxmox there's a separate HA link in Proxmox there's a separate HA link in Proxmox that should also have a respectable that should also have a respectable that should also have a respectable speed. So in best case scenarios you speed. So in best case scenarios you speed. So in best case scenarios you need two 10 GB network interfaces for need two 10 GB network interfaces for need two 10 GB network interfaces for one for HA uh one for SE and yeah if you one for HA uh one for SE and yeah if you one for HA uh one for SE and yeah if you were to go totally crazy you can also were to go totally crazy you can also were to go totally crazy you can also use a third one for other application use a third one for other application use a third one for other application traffic. So as you can imagine it gets traffic. So as you can imagine it gets traffic. So as you can imagine it gets quite expensive you [snorts] need lots quite expensive you [snorts] need lots quite expensive you [snorts] need lots of network interfaces high network of network interfaces high network of network interfaces high network speeds you need the switches that speeds you need the switches that speeds you need the switches that supports that network speed and provide supports that network speed and provide supports that network speed and provide all the different interfaces. So this is all the different interfaces. So this is all the different interfaces. So this is an option that works surprisingly well. an option that works surprisingly well. an option that works surprisingly well. I honestly had no problems with it. I'm I honestly had no problems with it. I'm I honestly had no problems with it. I'm running this for a few months right now running this for a few months right now running this for a few months right now and and it's working great. I can easily and and it's working great. I can easily and and it's working great. I can easily migrate virtual machines in just a few migrate virtual machines in just a few migrate virtual machines in just a few seconds. I can start any virtual machine seconds. I can start any virtual machine seconds. I can start any virtual machine on any node. I can shut down one of the on any node. I can shut down one of the on any node. I can shut down one of the nodes when I need to do maintenance nodes when I need to do maintenance nodes when I need to do maintenance doing Proxmox updates and still continue doing Proxmox updates and still continue doing Proxmox updates and still continue running the same machine or most of running the same machine or most of running the same machine or most of these machines on other nodes. So that I these machines on other nodes. So that I these machines on other nodes. So that I basically have zero downtime. And if one basically have zero downtime. And if one basically have zero downtime. And if one node fails, I can easily just keep node fails, I can easily just keep node fails, I can easily just keep running my entire home lab. I can create running my entire home lab. I can create running my entire home lab. I can create HA groups. The virtual machine can start HA groups. The virtual machine can start HA groups. The virtual machine can start up on another node completely automatic.

up on another node completely automatic. up on another node completely automatic. [snorts] So this is really a a solid [snorts] So this is really a a solid [snorts] So this is really a a solid high available and full torrent scenario high available and full torrent scenario high available and full torrent scenario for managing a shared distributed for managing a shared distributed for managing a shared distributed storage. But again, it's quite storage. But again, it's quite storage. But again, it's quite expensive. It also you probably need expensive. It also you probably need expensive. It also you probably need some time to get started with it and to some time to get started with it and to some time to get started with it and to understand how it actually works and understand how it actually works and understand how it actually works and what all the different parts are that what all the different parts are that what all the different parts are that you need. So I certainly would not you need. So I certainly would not you need. So I certainly would not recommend this for each and every setup. recommend this for each and every setup. recommend this for each and every setup. So yes, these are all the storage So yes, these are all the storage So yes, these are all the storage options or in my opinion all the storage options or in my opinion all the storage options or in my opinion all the storage options that actually matter in Proxmox. options that actually matter in Proxmox. options that actually matter in Proxmox. I know Proxmbox storage can be quite I know Proxmbox storage can be quite I know Proxmbox storage can be quite confusing. There are all the different confusing. There are all the different confusing. There are all the different options and for me personally it was options and for me personally it was options and for me personally it was really hard to actually get all the really hard to actually get all the really hard to actually get all the equipment to test all of this properly equipment to test all of this properly equipment to test all of this properly side by side. Try out ZFS. I needed a side by side. Try out ZFS. I needed a side by side. Try out ZFS. I needed a bunch of hard drives. I needed three bunch of hard drives. I needed three bunch of hard drives. I needed three nodes and networking and all of that nodes and networking and all of that nodes and networking and all of that stuff. But if I would summarize it, I stuff. But if I would summarize it, I stuff. But if I would summarize it, I think for a single Proxmox node, if think for a single Proxmox node, if think for a single Proxmox node, if you're just running one server, choose you're just running one server, choose you're just running one server, choose ZFS. There's actually no reason to not ZFS. There's actually no reason to not ZFS. There's actually no reason to not do this. And I certainly would recommend do this. And I certainly would recommend do this. And I certainly would recommend you at least get a second fast drive you at least get a second fast drive you at least get a second fast drive such as a second SSD or better a second such as a second SSD or better a second such as a second SSD or better a second NVME and create a ZFS mirror because NVME and create a ZFS mirror because NVME and create a ZFS mirror because that allows you to have all the great that allows you to have all the great that allows you to have all the great features like check some uh snapshots, features like check some uh snapshots, features like check some uh snapshots, roll backs and redundancy. If one drive roll backs and redundancy. If one drive roll backs and redundancy. If one drive fails, the second one is still the data.

fails, the second one is still the data. fails, the second one is still the data. You can keep your server up and running. You can keep your server up and running. You can keep your server up and running. And if you add a second or a third And if you add a second or a third And if you add a second or a third Proxmbox node later, you can still keep Proxmbox node later, you can still keep Proxmbox node later, you can still keep running ZFS, you just add a replication running ZFS, you just add a replication running ZFS, you just add a replication job and then copy the data over the job and then copy the data over the job and then copy the data over the network every 15 minutes. That is network every 15 minutes. That is network every 15 minutes. That is completely fine. And of course, still completely fine. And of course, still completely fine. And of course, still you can lose some data, but I I think you can lose some data, but I I think you can lose some data, but I I think you should have a separate backup job you should have a separate backup job you should have a separate backup job anyway. So remember, the storage options anyway. So remember, the storage options anyway. So remember, the storage options are never replacing a backup at all. And are never replacing a backup at all. And are never replacing a backup at all. And if you're getting into Proxmox if you're getting into Proxmox if you're getting into Proxmox clustering, well, you can start with clustering, well, you can start with clustering, well, you can start with ZFS. I would certainly not use NFS for ZFS. I would certainly not use NFS for ZFS. I would certainly not use NFS for running virtual machine discs anymore. running virtual machine discs anymore. running virtual machine discs anymore. That is in my opinion not a reliable That is in my opinion not a reliable That is in my opinion not a reliable setup and the hardware requirements are setup and the hardware requirements are setup and the hardware requirements are just too high. You can still use NFS as just too high. You can still use NFS as just too high. You can still use NFS as a backup location. That is completely a backup location. That is completely a backup location. That is completely fine. But in my opinion, if you're fine. But in my opinion, if you're fine. But in my opinion, if you're serious about Proxmox clustering, the serious about Proxmox clustering, the serious about Proxmox clustering, the one and only storage option that really one and only storage option that really one and only storage option that really makes sense in that scenario is using makes sense in that scenario is using makes sense in that scenario is using Seph. I've had no issues with this. It's Seph. I've had no issues with this. It's Seph. I've had no issues with this. It's running really well and with fast running really well and with fast running really well and with fast network speeds, 10 Gbit network, even if network speeds, 10 Gbit network, even if network speeds, 10 Gbit network, even if I'm just cheating a little and I'm not I'm just cheating a little and I'm not I'm just cheating a little and I'm not using a separate network interface for using a separate network interface for using a separate network interface for just SE. I'm using the same 10 GB just SE. I'm using the same 10 GB just SE. I'm using the same 10 GB network interface for HA traffic, SE network interface for HA traffic, SE network interface for HA traffic, SE traffic and application traffic and it's traffic and application traffic and it's traffic and application traffic and it's still working great. I not any major still working great. I not any major still working great. I not any major issues or slow performance or anything issues or slow performance or anything issues or slow performance or anything like this. So, so if you're serious like this. So, so if you're serious like this. So, so if you're serious about getting started with Proxmox about getting started with Proxmox about getting started with Proxmox clustering, get a three node Proxmox clustering, get a three node Proxmox clustering, get a three node Proxmox cluster installed seph. that's going to cluster installed seph. that's going to cluster installed seph. that's going to be the best option for that. All right, be the best option for that. All right, be the best option for that. All right, guys. So, I hope I could still help you

guys. So, I hope I could still help you guys. So, I hope I could still help you with my experience and with all the with my experience and with all the with my experience and with all the projects that I've done so far in projects that I've done so far in projects that I've done so far in Proxmox clustering. I hope this video Proxmox clustering. I hope this video Proxmox clustering. I hope this video made it all clear. If you still have made it all clear. If you still have made it all clear. If you still have open questions, then please join our open questions, then please join our open questions, then please join our Discord. Let's continue the discussion. Discord. Let's continue the discussion. Discord. Let's continue the discussion. Leave me some comments. And as always, Leave me some comments. And as always, Leave me some comments. And as always, thank you so much for watching. A big thank you so much for watching. A big thank you so much for watching. A big thanks goes out to all of the members thanks goes out to all of the members thanks goes out to all of the members supporting my channel. You guys are supporting my channel. You guys are supporting my channel. You guys are really amazing. And I catch you in the really amazing. And I catch you in the really amazing. And I catch you in the next video. Take care. Bye-bye.