Homelab PXE Booting

This is a (long, sorry!) follow on from the Debian & Docker on the Seagate Personal Cloud thread. I thought it’d be better to split out the PXE boot side of things and make it a bit more generic as that will undoubtedly be more useful to others thinking about implementing PXE at home themselves.

If you’re not familiar with Preboot eXecution Environment, it’s a way of booting a computer from a network rather than local boot media (e.g., HDD/SSD/CD/DVD/USB/Floppy). It’s been around for a while - I first saw it used on a Novell NetWare, 10BASE2 network a long time ago…

The PXE basic theory of operation is that when a network card makes a request for an IP/subnet mask/gateway/DNS servers via DHCP, it can also receive a TFTP server IP and file name for a client to fetch and then boot from.

I’ve always wanted to have a local PXE server here that served a menu up with various operating systems ready to go. I currently use a fishing tackle box full of USB drives and a CD wallet, but that’s a pain, and I never seem to put them back after I use them. Docker based projects such as netboot.xyz package up this functionality in a really nice way. I don’t have the option of running (easily) on that Seagate Personal Cloud device, but I really wanted to use the Personal Cloud hardware for this task, so I had to put together something to do PXE boot myself. Hopefully the below notes help others understand the theory and perhaps some of the traps I encountered along the way. I don’t intend the below to be a complete tutorial at all, but merely some notes about my own implementation to hopefully assist others. Using something like netboot.xyz is almost certainly going to do a better job of this than I did!


My goals with this were:

  • Have a physical switch port at my workbench that served files over PXE to whatever client was plugged into it
  • Implemented on a physically separate device rather than a VM
  • Totally isolated from my home network (as I am occasionally working on friend/family members’ malware ridden machines to help them out)
  • Does not interfere with the existing network in any way
  • Serve 32 bit, 64 bit, BIOS, and UEFI clients as universally as possible
  • Some sort of a menu, so that I don’t have to change a single boot file entry in a configuration file each time I want to boot something different
  • Serve Windows and Windows based (i.e., WinPE) and *nix based operating systems

Optionally:

  • Serve ARM64 clients (out of scope for this time around, but I am planning to come back to this)
  • Avoid having to pull the Seagate Personal Cloud apart again once it’s back together
  • Files are locally hosted, with the option to boot files on the fly from the internet

Those goals led into the way I set things up:

  • Debian 13 on the Personal Cloud (see the other thread)
  • Using a 60GB SSD rather than the one in that thread (as it was a good way to recycle a SSD in the parts bin and still gave me space to serve 10-30 ISOs of various sizes)
  • The Personal Cloud itself was to request a DHCP address (so I didn’t have to worry about fixed IPs - if my network architecture changes, the ‘PXE server’ effortlessly changes with it)
  • All the PXE magic happens on a separate VLAN and subnet - gives me the physical switch port in my homelab I want, and isolates things from the rest of the network
  • iPXE is used as a jump start to boot via TFTP, and it can then load ISOs over HTTP
  • IPv4 only, and files served over HTTP only (not HTTPS) to simplify implementation
  • DHCP server functions are handled via my existing network infrastructure, not by the Personal Cloud

Earlier this week I set up a separate VLAN and subnet, in my case 10.7.93.0/29 (7, 9, 3 mapping to P, X, E on a telephone keypad - I can easily identify what that traffic is!). My MikroTik router issues a static DHCP lease to the Personal Cloud now called pxeboot of 10.7.93.2 - more on DHCP later in the post. I won’t go into the VLAN config here, but the requisite ports were set up to give me a dedicated port on my homelab switch for PXE and also allow traffic tagged with 793 to go to that network (e.g., for Proxmox VMs).

The pxeboot device runs tftpd-hpa and lighttpd, both of which have armhf packages in Debian 13. I opted for lighttpd because I want to try and keep overhead really low on that ARMv7 device, and because php is still available if I want to experiment with delivering some more advanced menus via PXE later. Something like Caddy or any other HTTP server would work just fine too.

iPXE can chainload, where a small file is served via PXE to load iPXE, and then iPXE can perform more advanced functionality (e.g., pull ISOs over HTTP). iPXE seems like one of those amazing projects which is infinitely flexible. I was setting up to compile my own versions, but they already provide undionly.kpxe for BIOS machines and ipxe.uefi for UEFI machines which were sufficient for what I wanted to achieve. Both of these files were placed in /srv/tftp.

I’ll skip straight to the last step - serving the files over HTTP as I’m sure you’ll all be familiar with that step. I configured lighttpd to allow directory listings, and dumped my ISOs into onto the pxeboot machine to be served. The ISOs are accessible at http://10.7.93.2/<filename>. I also created an ipxe directory which I’ll discuss shortly.

TFTP server set up :white_check_mark:, HTTP server set up :white_check_mark:. The last pieces of the puzzle are configuring DHCP and setting up a menu for iPXE to display.

DHCP options 66 and 67 serve the IP for the TFTP server and the boot file name to retrieve, respectively. In this case, option 66 is pretty easy 10.7.93.2. Option 67 is where things get slightly more complicated.

I want to serve one file to BIOS clients, one to UEFI clients, and a third to iPXE clients. Many DHCP servers can modify their responses based on the client’s request. There’s plenty of documentation on how to do this using various DHCP servers, so I’ll keep this relatively generic. Essentially we want a way to identify whether the “device” making the request at the DHCP Discover & DHCP Request stages of the transaction booted via BIOS, UEFI, or is the iPXE firmware.

Big disclaimer: this is how I implemented it after looking through a ton of resources online (and troubleshooting with Wireshark) but it’s almost certainly not the optimal way. I had to work around a few MikroTik and UEFI quirks to get this working.

Working backwards, iPXE clients are relatively easy to identify, as they send DHCP Option 77 (User Class) as the string “iPXE”. The DHCP server can be configured to reply with DHCP Option 67 of a URI rather than a file to retrieve via TFTP, as iPXE understands what to do with that.

BIOS clients are identified by them either sending DHCP Option 93 (Client System Architecture) as 0, or not sending it all (because they pre date that option existing). These clients get undionly.kpxe, which is the iPXE BIOS firmware. The iPXE firmware then boots, and makes a DHCP request of its own, which is identified as iPXE and served a URI, as above.

BIOS and iPXE clients worked straight away - very cool!

UEFI was a giant pain in the butt, and where I spent most of my day. I’ve got no doubt that recent versions of UEFI are well behaved (and in fact can apparently boot from URIs too - untested by me), but I’m trying to implement something as close to universal as possible, so I’ve got to handle a few bugs. In theory, UEFI clients should send DHCP Option 93 with a value of 7. I couldn’t get my MikroTik router to reliably work with that, so I opted to match based on DHCP Option 60 (Vendor Class Identifier) which is a string sent by the client. I had odd issues with that too, I had to change to matching based on the hex for Arch:00007 which appears in that string for UEFI clients. For anyone in MikroTik land playing at home - it would match based on the hex and not on the string. The full command I used was /ip dhcp-server matcher add address-pool=pxeboot code=60 matching-type=substring name=UEFI-client option-set=pxe-uefi server=pxeboot value=0x417263683a3030303037 where 0x417263683a3030303037 = Arch:00007. I’ve got no idea if this is a bug in RouterOS or not, but for whatever reason the string would never match but the hex would :person_shrugging:.

UEFI clients were now being reliably identified, but again I ran into some quirks with the way some test machines implemented UEFI. Some clients silently ignore UEFI file names and only boot from bootx64.efi. I’ve run into this before a few times, as the Wyse 3040s I use on some other projects (details in other threads) have this annoying quirk. I renamed ipxe.efi to bootx64.efi (+ changed the DHCP Option 67 value for UEFI clients) and some clients that weren’t working started to work. Probably could have symlinked it, but oh well. More UEFI clients were working, but not all.

Back to Wireshark for a closer look. The UEFI client that were failing to boot were requesting bootx64.efi (which is really ipxe.efi, just renamed) from the TFTP server with an additional something at the end.

Very long story short, this is apparently a bug in some implementations of UEFI rather than a MikroTik issue. From what I understand, the MikroTik is strictly following the standard, but these clients are interpreting the terminator at the end of the DHCP Offer as a part of the boot file name as that’s the last thing being sent. Looks like my options were to send something after Option 67 (making sure it’s not the last thing in the DHCP Offer), rename/symlink the file to include the 0xFF at the end, or add a null terminator at the end of DHCP Option 67 for UEFI clients. I went for the latter, and bootx64.efi became 0x626f6f747836342e65666900 (full command /ip dhcp-server optionadd code=67 name=uefi-pxe-bootfile value=0x626f6f747836342e65666900). Success! UEFI clients boot iPXE, and then iPXE makes its own DHCP request and gets a URI to boot.


The last piece of the puzzle is the iPXE menu. I’ve stated several times that iPXE clients (identified via DHCP Option 77) are getting served a URI. In my case, it’s literally a DHCP Option 67 (boot file) of http://10.7.93.2/ipxe/menu.ipxe

The contents of this file are below:

#!ipxe

menu iPXE ISO Boot Menu
item debian   Boot Debian 13.1 amd64 Netinst ISO
item kali     Boot Kali 2025.2 Netinst ISO
item windows10_64  Boot Windows 10 64-bit Installer ISO
item windows10_32  Boot Windows 10 32-bit Installer ISO
item ubcd     Boot Ultimate Boot CD 5.3.9
item shell    iPXE shell
item reboot   Reboot
choose --default ubcd --timeout 10000 target && goto ${target}

:debian
sanboot --no-describe http://10.7.93.2/debian-13.1.0-amd64-netinst.iso || goto failed

:kali
sanboot --no-describe http://10.7.93.2/kali-linux-2025.2-installer-netinst-amd64.iso || goto failed

:windows10_64
sanboot --no-describe http://10.7.93.2/Win10_22H2_English_x64v1.iso || goto failed

:windows10_32
sanboot --no-describe http://10.7.93.2/Win10_22H2_English_x32v1.iso || goto failed

:ubcd
sanboot --no-describe http://10.7.93.2/ubcd539.iso || goto failed
:shell
shell

:reboot
reboot

:failed
echo Boot failed, returning to menu...
sleep 3
goto start

Most of the file is probably relatively straightforward to the HLB community, but I’ll point out sanboot which tells iPXE to boot from a SAN/iSCSI/HTTP target. In my case, it pulls the ISOs being served via lighttpd as discussed above.

The menu comes up on the client:

Tested successfully on VMs, and real UEFI and BIOS hardware of varying quality. The default option is to boot into Ultimate Boot CD after 10 seconds. I’ll probably reorder it to the top when I add a few other options. Both 64 and 32 bit Windows boot, as do Debian and Kali. I’m planning to add a few other ISOs to the menu as I begin to use it more.

Finally, I’ll flag the iPXE shell option on the bottom. That will drop the user into the iPXE shell, where commands such as chain and boot can be issued. One such use would be to boot from any http source on the internet (chain/boot in iPXE documentation).


For the sake of documentation completeness, I’ve been dropping ISOs to the pxeboot unit either via direct wget download, or using SCP from my MacBook. I toyed with setting up Samba, but decided against it as I want to keep the overhead low on the Personal Cloud and the ISOs are going to remain relatively static. Similarly, I haven’t bothered trying to set up any Web UI to add menu entries (like netboot.xyz’s functionality) as hand editing it won’t be a huge imposition for how infrequently it will change.

Still very much a work in progress, but hopefully that helps others and perhaps inspires people to give PXE a go in their own homelab. I will admit, during testing this afternoon it was very cool to plug any device into that port on the switch and be presented with the menu!


Edit 12/9: Tested a Linux and Windows install this morning and both failed early into the process. sanboot looks like it’s not the right way to do this - I’ll come back to the thread and update it once I get the menu side of things sorted out.

It’s a while since I pxebooted but back in the early noughties we used it to boot an image that comprised little more than a kernel and rdesktop. This in turn automatically launched a connection to a remote Windows VM. (Domain specific software was all Windows based. :confounded_face:) It was lightning fast but the kernel was compiled for a particular hardware set. This made it less flexible and when new equipment came along we abandoned it rather than creating more images.

@Belfry, as I understand it, you’re creating an options menu for almost any machine to boot a suitable iso image when plugged into a designated port on the router. That sounds cool but I am not clear what you do after that. Have I got that straight or have I missed something?

Spot on. I freely concede that this is nothing more than a very convoluted solution to a problem of my own making.

My current workflow is:

  1. Have a need to perform a task (e.g., repair/reinstall Windows on a family member’s computer, or set up a new Debian workstation for myself).

  2. Fish around on my work bench and main office desk through the 30+ unlabelled/poorly labelled/mislabelled USB flash drives floating around, in a futile attempt to find the specific one required to perform that task, which I just know is somewhere

  3. Either find the correct USB drive but then be unsure whether it’s the right version (e.g., “It’s Windows 10, but is it 22H2 or is it 21H2? Is this 32 or 64 bit? When did I last use this :thinking:?”) OR not find it at all because I accidentally left it in the machine I handed back to the last person I did a repair for, or it’s floating around in the bottom of my laptop bag, or under a seat in my car, etc.

  4. Inevitably give up and decide to pick a random USB drive from the ones laying around, download a fresh ISO of whatever I need right this minute, and write it to the USB drive (still unlabelled, or proudly sporting its previous and now incorrect label).

  5. Boot the USB drive a half hour or more after I initially decided to start the task I wanted to start.

Entirely unavoidable if I were to properly label the USB drives, not lose them, have some sort of actual system in place, and so on.

PXE boot and menu is cooler :smiling_face_with_sunglasses::smiling_face_with_sunglasses::smiling_face_with_sunglasses:. If I can have my top 10-20 ISOs ready to roll, and do nothing more than plug a computer into a network port then it’s going to be saving me a lot of time and mucking around compared to my current [self-inflicted] workflow.

Ahhh, got it now. The essential problem is that sticky labels are hard to attach to a USB stick. :head_shaking_horizontally:

On a more serious note, we have mentioned Ventoy here previously. I have serveral isos on a Ventoy USB stick but they are not always seen by the new computer. Such was the case when I tried some of @jdownie’s immutables.

You mentioned the Iodd-ST300 and 400 in that thread. You’re current solution covers a lot of what it provides but runs on repurposed hardware.

Such is the true spirit of a homelabber.

In case you haven’t seen my edit this morning, I’ll open by saying that sanboot didn’t work. It works in the sense that it’ll boot the ISOs (and UBCD and its tools work really well), but I discovered in some more thorough testing early this morning that both the Windows and Linux installers fail early in the process as they can’t find the source files. Looking further through the docs, sanboot definitely isn’t the right way to do this and I’ve got some other avenues to explore, so I’ll have to come back to the iPXE menu side of things when I get more time. Sharing the stuff that doesn’t work is just as important as sharing the stuff that does!

My intent with this thread is definitely not a tutorial on a roll your own PXE boot ISO setup, but more of some notes to share some theory and my own discoveries along the way in the hope that it helps someone else or even prompts them to share their own projects. I can repurpose some hardware that was in the e-waste pile, get down in the weeds with some network protocol stuff (which I really enjoy) and also learn a new tool - iPXE. Maybe someone out there in HLB land sees it and goes “oh no, don’t use sanboot!” and can help me out, maybe it encourages someone else to setup Ventoy or netboot.xyz in their own homelabs, or maybe another homelabber sees the thread and thinks “Isolated network? That would solve a totally separate problem I have” and starts a thread on VLANs and subnetting. Hopefully I can share some more successes in the coming weeks as I work my way through the iPXE menu side of things!

Exactly! :joy:. My reasons for doing PXE are definitely a case of a solution looking for a problem.

A little off topic - I mentioned it briefly in passing at the meeting earlier this week, but one of my best purchases of recent years was a Brother QL-700, on sale for around $70. My work areas have already begun to look like Ned Flanders’ holiday house, and there’s no reason I couldn’t buy some small labels and do up a well labeled set of USB drives, but where’s the fun in that? :joy:

I’m very happy with my ST-400. I originally purchased it with the intent of having a way to have encrypted offline backups at home (e.g., sync it once a month or so and otherwise leave it in a drawer), but it has been absorbed into the same workflow as above and now it’s just another thing that I may or may not find on the work bench when I want to boot an ISO. Freeing that device up for its intended purpose is another reason why I need to find a better alternative to host bootable ISOs.

I have a tip here, on a smooth side of the usb-key roll on a strip of papermate, 30 seconds for it to dry, then write your label. Repeat to change or scrape it off. Very durable. :slight_smile:

1 Like

I finally had some time to come back to the PXE project this week. After a bit of time debugging the iPXE menus in mid-September, I realised that I was reinventing the wheel to a large extent, particularly given that I was digging through the netboot.xyz GitHub repo to understand the iPXE menu and boot process anyway. At that point, I may as well pivot back to the actual netboot.xyz project and use 64-bit ARM hardware.

While working on an unrelated project a fortnight ago, I ended up at Jaycar for some odds and ends and saw that they had the Radxa ROCK 4C+ on clearance for $99. One impulse buy later, I had a shiny new ARMv8 board based on the Rockchip RK3399. I’ve since added a PoE hat and 128GB eMMC chip to the build, and repurposed a Raspberry Pi 4 case (the ROCK 4C+ is pretty close but definitely not identical to a Pi 4). There were a few interesting rabbit holes along the way getting Debian going with the ROCK 4C+ board, and I can do a full writeup of the hardware side of things in another thread if there is any interest.

The separate subnet and DHCP nitty-gritty was already set up weeks ago, and the latest ARMv8 Docker and netboot.xyz went onto the ROCK 4C+ easily, with the only minor snags being iptables-legacy and --net=host related). I’ve done a several VM/BIOS/UEFI test boots, and one full Debian 13 netinst on real hardware. I’ll circle back to add a few custom boot entries at some stage for some specialised software I need to use occasionally, but otherwise the PXE side of it is done.

I’ve now got a low-power, physically small, PoE powered (i.e., single cable to the unit) box that can live on its own VLAN+subnet and provide boot services to the workbench as necessary :white_check_mark:. The Seagate Personal Cloud which inspired this thread will probably end up with the 1TB SSD back in it, and used as a local media server for when I re-rip the CDs to FLAC at some stage.

Definitely give netboot.xyz a go if you’re looking to solve a similar problem - keeping on top of rapidly multiplying and unlabelled USB boot drives :joy:.