Discourse was down yesterday when my proxmox server died. It started up briefly before turning off. It never posted or output to the screen.
I did not have high availability enabled but it is part of a proxmox cluster. Under proxmox configurations for all vms and containers in the cluster are mirrored onto every node by corosync and pmxcfs.
All VMs are run on back-end storage mostly on NFS but some needing better performance are on iscsi.
It took a little while but I finally found that vms can be restarted by, for example, logging into the good_node and moving 100.conf from /etc/pve/nodes/bad_node/qemu-server to /etc/pve/nodes/good_node/qemu-server. This enabled the vm to be brought up either through the command line or the GUI.
128GB RAM individual host with 4TB of SSD storage I use for package mirrors for Alpine Linux, Debian, PyPy, and a few other things. This host also runs a few other services such as Jellyfin, The *Seer stack, NZBGET, and so on;
2x 32GB hosts in a two-host cluster, but really there’s zero need for them to be clustered. Plus without a third node (the 128GB node was originally the third node) the cluster doesn’t have quorum – these are for experimenting with random things
A fourth and final Proxmox server that I don’t think is running … lol I don’t recall
Great to see a win and hear a positive story! I’m really curious as to how portable Proxmox VMs are now - Of course moving between hosts makes sense, but out of curiosity, I really want to try and move something from Proxmox to UTM (both qemu under the hood) and see how that goes…
That’s where I’m at right now, @mcrilly. I’ve spun up my new (second hand) home Proxmox server a month or so ago and have two former Proxmox servers here looking for something to do. I had pondered experimenting with a cluster but “there’s zero need for them to be clustered” given what I’m using Proxmox for, and one of my goals for home is to keep power usage to an absolute minimum. It somewhat defeats the purpose of having low power hardware running several VMs if I just add a bunch more hardware back into the mix!
I am curious about the idea of having a cluster or even a high availability setup spanning home and a VPS somewhere (or two for quorum). I understand the latter will introduce some challenges, particularly regarding storage. Do you have any experience with Ceph and/or experience running a cluster that spans multiple physical sites?
I think this discussion encapsulates the homelab experience. It’s great to experiment but if you really don’t care about whether a particular VM stays up or not it’s hard to justify running multiple servers and bothering with clusters and high availability.
For my part I am running proxmox on several second hand servers. The one that died on Wednesday has a motherboard date of manufacture from 2013. Others are similar. In the last few days I have learnt a lot about deteriorating mobos, dead RAM slots, keeping one’s backup routines simple but regular, monitoring hard drive space, corrupted filesystems and dust; lots and lots about lots of dust.
As a result I have moved this Discourse server back to the cloud. As a homelabber it seemed important to eat your own dog food but it turned out that this one was past its use by date.
@Belfry I am still thinking about a HLB cross site cluster. It might be something we can talk about Monday week.
Quoting for visibility! I’m trying to get better at backing up things both for general DR scenarios, and also before just making that one little inconsequential change to the configuration and inevitably breaking the lot. Finally learned that it’s easier to restore a backup than to retrace steps from scratch over and over (that encapsulates a lot of my homelab experience, unfortunately)!
My own homelab attitude is one of casual self-education and experimentation. Sometimes that might involve complex redundant configurations well beyond a “sensible” home use case, sometimes that might mean throwing something onto a spare machine and not really caring about availability or data integrity while you’re tinkering, sometimes that involves running something on a VPS rather than at home, sometimes that might mean getting the soldering iron out to replace the capacitors on that motherboard from 2013, and sometimes it means pushing configurations or hardware to the limit merely to see what happens. @zeeclor I’d argue you are still “eat[ing] your own dog food” - Discourse is still up with no data loss, and you learned a bunch of stuff along the way (aging hardware, operating systems and drives, environmental factors such as dust, migrating data to/from cloud hosting, etc.). Thanks for hosting it for the group.
I’m still working on freeing up that night and will hopefully be able to confirm by mid-next week. It’d be great to catch up!
For everybody’s information, email notifications have been turned back on for all users. It may have defaulted to off when the server was moved last week.
Also note the Jitsi server is turned off until just prior to the next online meeting to preserve system resources.
I came across a useful app for Proxmox on the week-end. In essence Proxmenux is a series of command line scripts that take the hassle out of doing some simple (and some complicated) tasks.
If you have a proxmox server it’s probably easist just to run the script and see what’s there.
If you haven’t, the video by Brandon Lee gives a good overview.
I deleted some Windows 10 VMs earlier this year and my Windows 11 is no longer booting so I thought I’d try the Windows scripts. I opted for the Windows 11 insider edition but you could go dev or canary if you are keen. They also have Windows Server and Win 10 builds at UUPDump. The install wasn’t completely seamless but we got there in the end.
Similarly, they have scripts for a variety of NASes. I wasn’t quite sure how to start with them but I recall that Matt Fletcher had a Synology VM running a servarr setup and I think @mcrilly does the same.
I use it in production. Really need to get around to buying that subscription for work…
I have a homelab cluster of DL360s (G8/9 iirc) with 18TB of ceph pool backed by flash (old Samsung SATA SSDs). I don’t use it anymore - when you need that much grunt it’s great, but there was 6 months where it was idling to run a single VM!
I’ve successfully migrated QEMU /virt-manager → proxmox before. Since there’s not much in the way of secret sauce (it is just QEMU after all) I imagine that going the other way is straightforward, especially when you’re writing out .qcow or ,raw disk images.
It sounds like you’ve got a lot more knowledge of how QEMU works under the hood - have you ever been able to emulate another architecture using Proxmox?
I use UTM (another QEMU based project) to emulate Motorola 68000s occasionally. I looked into some of the Debian packages to see if I could possibly add some bits onto Proxmox (qemu-system-m68k) and manually create the VM with a m68k using qm. How much of “full” (probably the wrong term) QEMU does Proxmox expose?
(Yes, I’m aware that I could potentially break the Proxmox machine trying to shoehorn 68000 emulation into it, but I’m so curious to see if it is possible ).
Not really - we use QEMU-user a bit in Gentoo but I’ve never tried to touch it in proxmox. It would be fun to be able to access alt-arch VMs via the proxmox GUI though…
Brandon Lee has a recent video and write up about it.
It was a straight forward installation under docker and the docker link error that he mentions has now been fixed. I created the .env file that he needed to get it running but that may also be unnecessary now.
I watched a good video from Techno Tim the other night showing the feature off and discussing the mechanics as well as the pros/cons. Any thoughts from the Docker aficionados?
).
