In the early hours of Monday morning my Geekom A7 died. It took out a talos master and worker, and my email and git servers. It does not power up at all despite changing adapters.
All these apps were running as virtual machines or in virtual machines.
I have a spare server in my proxmox cluster and powered it up. The VMs hard drives are on shared storage over NFS.
Running
mv /etc/pve/nodes/<deadnode>/qemu-server/* /etc/pve/nodes/<good-node>/qemu-server
on any node in the cluster shows the “lost” vms in the GUI. They can then be fired up and we are backup and running.
I removed the dead node with
pvecm delnode <deadnode>
Corosync is the application under proxmox that provides some of this magic. If I had had high availability enabled the recovery would have been automatic. However my applications are not time critical and email servers are functionally “poste restante”.
Now I just have to get a new machine out of Geekom.