Hey y'all,

Have any of you used Proxmox within a clustered environment? In the new year I am thinking about buying a few cheap Dell Optiplexes from Amazon and making a 3 node Proxmox cluster, getting some large spinning discs too and making a nice HA environment to have some local VM's.

Anyone done this before? Any gotcha's to look out for?

---
|14Best regards,
|11Ch|03rist|11ia|15n |11a|03ka |11Me|03aTLoT|11io|15N

|07── |08[|10eml|08] |15ml@erb.pw |07── |08[|10web|08] |15www.erb.pw |07───┐ |07── |08[|09fsx|08] |1521:1/158 |07── |08[|11tqw|08] |151337:1/101 |07┬──┘ |07── |08[|12rtn|08] |1580:774/81 |07─┬ |08[|14fdn|08] |152:250/5 |07───┘
|07── |08[|10ark|08] |1510:104/2 |07─┘

--- Mystic BBS v1.12 A47 2020/12/04 (Linux/64)
* Origin: thE qUAntUm wOrmhOlE, rAmsgAtE, uK. bbs.erb.pw (1337:1/101)

Re: Proxmox clustering
By: MeaTLoTioN to All on Fri Dec 18 2020 01:52 am

Have any of you used Proxmox within a clustered environment? In the new year I am thinking about
buying a few cheap Dell Optiplexes from Amazon and making a 3 node Proxmox cluster, getting some large
spinning discs too and making a nice HA environment to have some local VM's.

At my old workplace we ran our production environment off proxmox clusters. Over 600 VMs running across dozens of VM hosts in regionally distributed datacenters. The cluster features are awesome, you can migrate VMs across hardware and move their IPs without shutting them down. But all that takes a bit of work.

I wish I could give you more help -- but the linux admins were in charge of it. :-) I just know is was cool.



|01bobbobbobbob|09bob|03bob|11bob|03bob|09bob|01bobbobbob |01robrobrobrob|09rob|03rob|11rob|03rob|09rob|01robrobrob |07


... Confidence is the feeling you had before you knew better.
--- SBBSecho 3.11-Linux
* Origin: Halls of Valhalla =-= Happy Holidays (1337:3/135)

Anyone done this before? Any gotcha's to look out for?

I did :)

Try to get your VMs on SSDs as much as possible, then mount spinning disks as additional storage for the data, otherwise the IO will be crazy high at
times. Plan your RAID based on need as well, avoid RAID-5 for OS drives, it's OK for data tho most of the times.

If you plan on having multiple machines running similar OSes, using ZFS with plenty of RAM and deduplication enabled will save a lot of space but will
cost memory. Compression can also help for IO mitigation (since it
read/writes less on the disk) but with an additional cost in CPU.

If you want to have it run in cluster mode, you need a central storage
solution as well, lots of trial and error there as well. I ended up using
a Fiber attached SAN but you may be good with iSCSI or even NFS/SMB.

Proxmox is really a cool platform tho, been using it since v3 at least and never looked back. I surprise myself when I get to an ESXi platform to try to do something that works under Proxmox but unavail in ESX unless you pay the
big bucks (if it can).

I'd check for workstations instead of Optiplex tho, something like the HP
Z400. Better machine specs and in the refurb market they are pretty cheap and super silent as well.

If you can afford the noise, rackmount servers are the best value for performance but fans are noisy...

Clustered virtualization is a science in itself so it works the way you
intend it to work. Also, make a lot of testing before putting something in "prod". Try disconnecting stuff, shutdown hosts, etc and see if it behaves
like you want it to do...

Have fun!

|15 ▀ ▐ |15StackFault |08<|03.|11.|15P|11h|03EN|11o|15M|11.|03.|08>
|11 ▌ ▀ |11The Bottomless Abyss BBS
|03 ▀ ▌▀ |03ssh|08.|072222 |08/ |03telnet|08.|072023 |08/ |03https
|08 ▄■▐ |08bbs|07.|08bottomlessabyss|07.|08net

--- Mystic BBS v1.12 A45 2020/02/18 (Linux/64)
* Origin: The Bottomless Abyss BBS * bbs.bottomlessabyss.net (1337:3/106)

Re: Re: Proxmox clustering
By: StackFault to MeaTLoTioN on Mon Dec 28 2020 08:34 am

Howdy,

Clustered virtualization is a science in itself so it works the way you intend it to work. Also, make a lot of testing before putting something in "prod". Try disconnecting stuff,
shutdown hosts, etc and see if it behaves like you want it to do...

I agree, it is.

It depends on your goals - if you are doing this just for the heck of it (and to learn :) then awesome.

But if you are wanting to make an "application" highly available, then docker/kube is the way to go. Still a learning
curve, but the environment takes care of most of the requirements and you can use commodity hardware.

You still need a shared storage environment, that can span multiple hosts (and those hosts can be virtual) - which is
where glusterfs (I found problematic, and I am not sure of its development status), ceph, NFS, etc can help. I would prefer gluster/ceph over NFS, since "NFS" is not HA and when it goes down it takes everything with it (never been a fan of NFS).

Personally I use a commercial software package that provides HA storage (because I work for the company that makes it,
and I can put it down to skills :) With it nodes providing the storage can go offline, as long as there are a major of
nodes still active and atleast 1 of thoses nodes with the filesystem.

On my intel server, I use VMware to make multiple VMs and use the commercial software for docker, but for the Pi, I'll still use VMware to make multiple guests and play with gluster again.

...δεσ∩

... Hindsight is an exact science.
--- SBBSecho 3.11-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (1337:2/101)