Homelab Certificate Management

General
1

Hi all,

Thought I’d throw this one to the group to get some ideas as I suspect this is one of those areas where one can ask a half dozen people the same question and get ten different answers!

Like a lot of people, I get self-signed certificate warnings when accessing a lot of internal self-hosted services and devices. It would be nice to make a small quality of life improvement by changing these certificates over to ones issued by a recognised third-party CA in order to remove these warnings. I’m talking specifically about internal services/devices only accessible via my internal network, and not externally facing services which already have recognised TLS certificates attached.

I can see several approaches available, but nothing has really jumped out at me yet as being an optimal solution. A few thoughts (not necessarily pros and cons) on the options swimming around in my head are below.

ACME protocol and Let’s Encrypt (or equivalent):

  • Seems to have good support in Proxmox (although for Let’s Encrypt only, not other equivalent services)
  • Seems to be some sort of support for Home Assistant (albeit via DuckDNS or third-party addons)
  • Doesn’t help for other devices that can’t run ACME (e.g., network hardware such as wireless access points)
  • Using these certificates everywhere on my LAN will require hosting proxies of some sort (e.g., Traefik or Nginix Proxy Manager) for the devices or services without built in ACME/certbot support
  • Could get messy quickly with a possible mix of DNS validation, HTTP validation (also requires exposing services to the internet which I’d rather avoid)
  • Hostnames of internal services will get exposed via certificate transparency logs or DNS entries

Proxy everything via Traefik/Nginx Proxy Manager:

  • Some of the above points still apply (e.g., certificate transparency logs)
  • Extra setup and admin for another service at home
  • Won’t solve specific edge cases (e.g., can’t load this certificate onto my wireless controller/access points to serve HTTPS captive portal for guests) without a ton of mucking around (e.g., manually downloading certificates and uploading them to devices at or before expiry)

Roll my own and host my own CA:

  • Extra flexibility for cert usage and validity periods
  • No DNS/HTTP validation/other exposure to the public internet
  • Requires adding self-issued root certificate to all devices (not always an option - e.g., guest devices on guest SSID for visitors, or giving a visitor temporary access to Home Assistant while they stay for a few nights)
  • Extra setup and admin for another service at home, more so than using Traefik/NPM

Buy a domain wildcard certificate:

  • Big $$$$ which I’m not willing to spend on what is really only a mild “quality of life” improvement
  • Potentially a lot of manual intervention coming as certificate validity periods begin to drop, especially for devices that need me to manually replace certificates (i.e., switches, wireless access points)
  • More difficult to automate when compared to ACME/certbot

Any thoughts from the brains trust? I’ll admit this isn’t really a big deal and is more of a “nice to have”. It’s a bit of a blind spot for me knowledge wise, as over the decades I’ve never had to go much deeper than “generate CSR → upload to issuer → download certificate → install on web server”.

Open to any and all discussion or knowledge from the group!

Cheers,

Belfry

Caddy - "The Ultimate Server"
2

Well, i’m equal parts tight arse and maverick on this one. I’m rolling my own. It’s been an important part of my k3s cluster and my self hosted docker registry. If you’d like a bit of a tour of what i’m doing, i can set up a little Jitsi meeting one night for you (and anybody else that’s interested) and we can see what i’ve done. I’m sure what i’ve done can be improved, so i’m keen to be put on the spot with a question that i can’t answer.

3

Thanks, @jdownie. I recall seeing that when you were showing off your k3s cluster recently and would like to catch up at some stage to get a bit of a run through of how it all works.

Unfortunately, I don’t think self-hosting will tick all the boxes. One example is that my wireless controller serves a captive portal to guests (typical of any sort of public wifi). On a desktop machine this flags with the typical self-signed certificate warning, but on Android devices it just flat out refuses to present the captive portal at all and there’s no internet access despite the wifi being connected. Android seems to require a “real” certificate to even present the end user with the option to continue.

Typing that out has triggered another thought bubble of sorts - is it possible to purchase an “intermediate certificate” to allow an end user to self-host and sign their own certificates with a full chain up to a recognised root CA? A very very quick search as I write this seems to suggest there’s zero chance of that happening, but you may have some secret sauce somewhere. Are your self-signed certs in your own certificates somehow part of a chain of trust back to a recognised root somewhere?

4

Nah, my rootCA is not signed by any higher authority. I have to install and trust it on all of my clients. That includes Windows laptops, Linux laptops, Linux servers and Kubernetes nodes. I also install it on my Android phone, so when i want to use my home apps on my phone, i bring up tailscale and browse to my self hosted apps, and the rootCA installed on my phone does what it’s supposed to do.

Using your own certificates and CA gives you an appreciation for how much configuration and setup is taken care of for you because of that register of trusted CAs that is already globally distributed on popular operating systems and browsers.

Would installing your rootCA on your Android phone attend to the issue you were describing?

Another way i might be able to help… I have been meaning to publish the first of a series of git repositories on our gitea host. My thinking is that each of these repositories would have a “theme”. The first would be “Orchestration”, and the second would be “Scripting”. For all of these “themed” repositories, the root folder would container a folder for each of us that wants to participate. So, I’d publish some of my handy scripts for Orchestration under it’s jdownie folder. You could then copy what you want to play with into your Belfry folder, and when you have questions i could add changes and comments to your files, commit and push. We’d conduct most of the conversation up here in Discourse, but we’d have a way to share files in a trackable way. The first thing i’d put in there would be my scripts for creating a rootCA along with signed certificates. With those scripts, i add a new hostname to the hosts file, and run a docker compose up. I get a new .crt and .key file in the out folder. I’m just using mkcert, but i’ve written some scripting to make the process that simple.

After seeing your meshtastic gadget the other week, i got mine up and running. It’s sitting in the window of my son’s bedroom. There still aren’t any peers in range, but maybe one day. I’ve given it a static ip address on my LAN by reserving it on my router. I’ve registered that ip address as meshtastic.downie.net.au publically. You can ping that hostname to see it’s address on my home network, but you won’t have line of sight to it. My next move (when i get time) is to add meshtastic to my hosts file, run docker compose up to get a certificate, and then install that certificate (somehow) on the esp32 sitting in Jack’s bedroom.

Anyway, if you’d like to see how i’ve been doing it, i can turn my attention to getting that repository up and running.

5

Unfortunately, I can’t control a guest’s device(s) nor is it reasonable for me to install my certificates on them. (I also appreciate that’s not reasonable for a home user have a captive portal for their completely isolated Guest SSID/VLAN but that’s all part of the fun of learning about and mucking around with enterprise gear in the homelab).

I am really curious to see how it all works. I have some limited experience with the concepts from a Windows Active Directory CA environment a very long time ago, but haven’t seen it in action in a *nix one.

Doing similar crossed my mind, but I’ve always run a Split DNS setup. Not at all strictly necessary, but it’s the same rationale as the Certificate Transparency discussion in my first post (see Side Effects section in the Wikipedia article) in that I’m trying to keep all details of internal services entirely out of the eye of the public internet, without exception. It’s utter overkill, but security is somewhere I do try to “eat the dog food”, to yet again borrow a phrase from others in the group :joy:!

(Also I’ll make sure to leave my meshtastic unit on and charged and see if something comes up!)

Having more of a ponder about it all over the weekend, I think I need to take a step back and better define what the problem is and which of my own constraints I want to put on the outcome, which will no doubt better lead me to a solution.

I think your scripting might be part of the mix too, so I’m interested in getting a walkthrough at some stage. There’s a few other interesting ingredients I’ve come across since my original post too so I’ll share them with the group in case they’re of interest to others:

  • This project by Smallstep, which I haven’t tried out yet, looks like very polished a “Private CA in a box”
  • ZeroSSL as a potential alternative to Let’s Encrypt. Full ACME protocol support, support for wildcard certificates, and a web portal as an alternative to ACME.

I can possibly see myself using a combination of wildcard certificates from ZeroSSL and some automation to grab a single wildcard certificate issued by ZeroSSL/Let’s Encrypt’s service to an ACME client, and push that wildcard cert out (SCP?) to devices that can’t run ACME such as wireless APs (or in your case, the meshtastic box). Or just download the certificate via the web portal every ~89 days myself. Maybe combine it with a private CA for other internal use or issuing client certificates… no idea just yet.

Back to being a thought bubble for now, but I am interested in seeing how you manage your CA and which parts of your implementation might be useful for me too.

6

James, I am confused.

meshtastic.downie.net.au is a class C local IP address. I am not sure what a public DNS record for it achieves. Am I missing something?

dguest@minforantor:~$ ping downie.net.au
PING downie.net.au (116.90.56.17) 56(84) bytes of data.
64 bytes from downie.net.au (116.90.56.17): icmp_seq=1 ttl=57 time=19.8 ms
^C
— downie.net.au ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 19.773/19.773/19.773/0.000 ms
dguest@minforantor:~$ ping meshtastic.downie.net.au
PING meshtastic.downie.net.au (192.168.68.40) 56(84) bytes of data.
^C
— meshtastic.downie.net.au ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

7

I think the intent is that @jdownie can have a third-party hosting his DNS elsewhere and have the benefits of having a DNS record in place when connected to his home network. However, anyone using that hostname externally will not be able to access the device as it’s an RFC1918 private IP.

Please jump in if that’s not correct, @jdownie!

8

That’s a nice way of dressing up “lazy” @Belfry :wink:

Those hostname entries are of no use to anybody outside of my network. I just get to pretend that my publically hosted apps and internally hosted apps are all on the same network. When I’m out and about I just need to bring up tailscale to access the internally hosted ones.

It’s completely at odds with @Belfry’s “eat your own dog food” philosophy. I should be more concerned with advertising the specific octets of my hosts and services, but that stuff would be easy enough to discover if somebody had access to my Lan anyway. :man_shrugging:

9

Ah, tailscale. I told you I was confused. I had not heard of tailscale’s split DNS before and will look into it.

As regards homelab certificate management, I use caddy as my reverse proxy. I found it reliable and relatively easy to setup. Traefik seemed a bit daunting to me but looks to be the reverse proxy of choice for kubernetes clusters.

Caddy automates the download of new certificates prior to the current one’s expiration date. I bind caddy volumes for my docker containers and hence a current certificate for a particular domain is always available at ./data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/ Browsers and thunderbird do not complain when you visit resources that use those valid certificate.

Mailcow has this for Caddy v2 but other reverse proxy configuration are also listed.

Might that be a solution?

10

:joy: I don’t consider that “lazy”. Even if it were, there’s nothing wrong with lazy if it’s “good enough” and it’s doing the job you need it to!

Thanks for bringing up Caddy, @zeeclor. I’d heard of the project before (only referenced in passing by another project), never really looked into it, and subsequently forgot all about it. I can potentially see that as being one of the ingredients in the mix somewhere - maybe some self-signed certificates for service to service use, client identification, and services only my devices will access, and a series of public root CA signed certificates or a single wildcard certificate via Caddy for things that are accessed by others. I think the last piece of the puzzle will be to see how difficult it is to automate replacing certificates on various devices (e.g., some sort of a python/bash script that takes the certificate issued to Caddy and uploads it to a wireless access point every 14 days).

I won’t have time to come back to this and have a look at any docs in detail for at least few weeks, but I appreciate the input from you both. I’ll also take some time to better define the problem and any constraints I need to consider.

1 Like
11

Following on from the experiments with Caddy yesterday, I gave acme.sh a try this afternoon. Good excuse to try out Debian 13 for the first time too!

acme.sh handled a few test domains flawlessly, including an IDN and a wild card certificate! For my tests I used the “manual DNS” mode and Let’s Encrypt’s staging environment. It was easy to do and a broad overview of my test experience is documented below with the identifying details redacted.

acme.sh --issue -d *.domainname.tld --dns --staging --yes-I-know-dns-manual-mode-enough-go-ahead-please
Result:

[Fri 15 Aug 2025 17:20:06 AEST] Using ACME_DIRECTORY: https://acme-staging-v02.api.letsencrypt.org/directory
[Fri 15 Aug 2025 17:20:07 AEST] Using CA: https://acme-staging-v02.api.letsencrypt.org/directory
[Fri 15 Aug 2025 17:20:08 AEST] Creating domain key
[Fri 15 Aug 2025 17:20:08 AEST] The domain key is here: /home/xxx/.acme.sh/*.xxx_etld/*.domainname.tld.key
[Fri 15 Aug 2025 17:20:08 AEST] Single domain='*.domainname.tld'
[Fri 15 Aug 2025 17:20:12 AEST] Getting webroot for domain='*.domainname.tld'
[Fri 15 Aug 2025 17:20:13 AEST] Add the following TXT record:
[Fri 15 Aug 2025 17:20:13 AEST] Domain: '_acme-challenge.domainname.tld'
[Fri 15 Aug 2025 17:20:13 AEST] TXT value: ‘xxxxxxxxxx’
[Fri 15 Aug 2025 17:20:13 AEST] Please make sure to prepend '_acme-challenge.' to your domain
[Fri 15 Aug 2025 17:20:13 AEST] so that the resulting subdomain is: _acme-challenge.domainname.tld
[Fri 15 Aug 2025 17:20:13 AEST] Please add the TXT records to the domains, and re-run with --renew.
[Fri 15 Aug 2025 17:20:13 AEST] Please check log file for more details: /home/xxx/.acme.sh/acme.sh.log

After adding the DNS entry, waiting 2-3 minutes, and confirming it was there with dig, I issued acme.sh --issue -d *.domainname.tld --dns --staging --yes-I-know-dns-manual-mode-enough-go-ahead-please --renew as instructed.

Result:

[Fri 15 Aug 2025 17:31:11 AEST] Using ACME_DIRECTORY: https://acme-staging-v02.api.letsencrypt.org/directory
[Fri 15 Aug 2025 17:31:11 AEST] The domain '*.domainname.tld' seems to already have an ECC cert, let's use it.
[Fri 15 Aug 2025 17:31:11 AEST] Renewing: '*.domainname.tld'
[Fri 15 Aug 2025 17:31:11 AEST] Renewing using Le_API=https://acme-v02.api.letsencrypt.org/directory
[Fri 15 Aug 2025 17:31:13 AEST] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Fri 15 Aug 2025 17:31:14 AEST] Single domain='*.domainname.tld'
[Fri 15 Aug 2025 17:31:14 AEST] Verifying: *.domainname.tld
[Fri 15 Aug 2025 17:31:17 AEST] Pending. The CA is processing your order, please wait. (1/30)
[Fri 15 Aug 2025 17:31:21 AEST] Success
[Fri 15 Aug 2025 17:31:21 AEST] Verification finished, beginning signing.
[Fri 15 Aug 2025 17:31:21 AEST] Let's finalize the order.
[Fri 15 Aug 2025 17:31:21 AEST] Le_OrderFinalize='https://acme-v02.api.letsencrypt.org/acme/finalize/xxxxx’
[Fri 15 Aug 2025 17:31:23 AEST] Downloading cert.
[Fri 15 Aug 2025 17:31:23 AEST] Le_LinkCert='https://acme-v02.api.letsencrypt.org/acme/cert/xxxxx’
[Fri 15 Aug 2025 17:31:24 AEST] Cert success.
-----BEGIN CERTIFICATE-----
*snip*
-----END CERTIFICATE-----
[Fri 15 Aug 2025 17:31:24 AEST] Your cert is in: /home/xxx/.acme.sh/*.domainname.tld_ecc/*.domainname.tld.cer
[Fri 15 Aug 2025 17:31:25 AEST] Your cert key is in: /home/xxx/.acme.sh/*.domainname.tld_ecc/*.domainname.tld.key
[Fri 15 Aug 2025 17:31:25 AEST] The intermediate CA cert is in: /home/xxx/.acme.sh/*.domainname.tld_ecc/ca.cer
[Fri 15 Aug 2025 17:31:25 AEST] And the full-chain cert is in: /home/xxx/.acme.sh/*.domainname.tld_ecc/fullchain.cer

Other than the fact that I was issued a “real” certificate despite me using the staging environment (oops), it was flawless. I’d say that may have happened due to my messing about with Caddy last night using the same domain name(s) for my tests, and already having a few “real” certificates from Let’s Encrypt out there? Not sure.

DNS API looks pretty easy to implement, although I specifically didn’t test it tonight. acme.sh should be very easy to script up as a cronjob to renew every month, and a follow on script to push my wildcard cert out to whatever devices need it seems like it’ll be pretty straightforward as well. I wasn’t aware of the acme.sh project when I wrote my original post, but it ticks all the boxes from my original discussion, with none of the compromises. Hopefully that helps others too.

12

Well @Belfry , now you’ve got me diving down the rabbit hole of my certificates because I haven’t had to deal with them in so long.

Everything I run is behind Traefik in K8s. I have configured cert-manager to use my free Cloudflare DNS to get certificates issued via Ingress Manifests. All certificates are acquired via Cert-Manager, and I never have to worry (it does the creation of the ACME DNS TXT Challenges).

Internally, AdGuard Home redirects all internal (and VPN) traffic bound for my domain back home to the K8s servers.

Home Assistant is a little different because I pay for the Nabu Casa subscription (I use it so much, only fair), in saying that, my Cloudflare DNS redirects my Home Assistant address to the supplied address from Nabu Casa. This allows for ACME challenges and certificates.

Here’s the fun part, Cloudflare doesn’t have my A address record, it’s over on duckdns.org, which is free and automatically updated by my router and Home Assistant. I recall doing it this way because duckdns.org didn’t allow you to show ownership for the domain, whereas my Cloudflare account does (plus the TXT Challenges).

The result: I have a free wildcard domain on Cloudflare, which handles all the certificate challenges and DNS redirects, while duckdns.org allows me to update my A record address so that services can find me.

Some Google foo says you might be able to get the Traefik Proxy to serve to other IP addresses outside of K8s using Ingress, Service and Endpoint manifests. This may give you an automated Cert refresher without too much hassle but I haven’t tested it.

Maybe this helps someone, maybe it doesn’t ha ha

1 Like
13

Nice setup! AGH is an outstanding product. I used to use that up until quite recently and it’s a great bit of kit. I only stopped using it due to a router change (new router has functionality for downloading blocklists and turning them into static DNS entries built in), not because of any issue with AGH itself. Traefik is on the list to play with at some stage as it seems to be very well regarded. I’m yet to dip my toe into the k8s/k3s waters, but that is one place where Traefik seems to shine too. I lean heavily on the Cloudflare free tier for my “production” stuff at home (The usual DNS/proxy/WAF, plus their Workers/Zero Trust/Access services), so your cert and DNS setup isn’t all that dissimilar to mine overall.

In the dynamic IP days, I used to have the base domain on Cloudflare CNAMEd to somewhere else in the same way that you’re using duckdns. One of the things I really like about Cloudflare is that it’ll CNAME flatten on a root domain for some of these interesting edge cases! Although I’ve got static IPv4s available now, I’ve been fully IPv6 native at home* for well over 10 years and have started just using a Cloudflare AAAA record direct to the IPv6 address of the VM/device in question. Configuring the IPv6 firewall to allow traffic through to whatever HTTP/HTTPS service from Cloudflare’s IP ranges and enabling proxying in the dashboard achieves the same thing - domains get issued a certificate, services are accessible from IPv4 only (and IPv6) networks via Cloudflare’s proxy, not accessible directly, and I haven’t had to concern myself with running Caddy/Traefik/Nginx/traversing NAT, etc. In fact, nothing to do at all beyond configuring some firewall rules and a DNS entry. That type of setup never used to work but was added 2? 3? years ago so if you’ve got IPv6 available and working, that could be another option. Any dynamic IPv4 part (which I assume duckdns is doing) may not be required any more, although I appreciate that duckdns is well integrated into Home Assistant so it’s probably easier to keep using it!

If something is a little less robust than I’d like to have out there yet, or is something I’m putting together temporarily for testing (e.g., the n8n thread the other week), I’ll slap a Cloudflare Worker in front of it that does nothing more than Basic Authentication (but over HTTPS) to briefly keep “internet riff raff” out, while maintaining the benefits of that single AAAA record and Cloudflare doing all the certificate management for me.

Given that how much I do rely on Cloudflare, it’s nice to explore some of the alternatives to solve the same sort of problems and also solve some new ones. For example, the “AAAA record in Cloudflare” setup doesn’t help solve the problem of certificates on things such as wireless access points/controllers or switches, etc. that are deliberately not exposed to the internet and never will be. I do need to roll my own internal certificates at some stage for these sort of use cases. Likewise, if Cloudflare ever decide to remove or start charging for some functionality I start using, then it’ll be good to be familiar with acme.sh/Caddy/Traefik/etc. and be able to migrate over relatively painlessly.

I’ll have to have a chat with you about the Nabu Casa sub at some stage. It’s something I’ve looked into but not gone down the route of - not because I don’t think it’s worth it, but because I think I’d find it more satisfying to put together a secure remote access solution myself :blush:. It’s low on the priority list but I’ll get around to it one day (which is admittedly the same thing I say about every odd job at home). It’d be interesting to hear what the real world experience is like for ~$100 odd a year.

*Well, except all those weird and wonderful smart home/IoT devices that seem to be continually multiplying and still live on their own heavily locked down IPv4 only SSID/VLAN where I can keep a very close eye on them :joy:.

14

I’ll tack this onto the end of the thread after tonight’s talk. I didn’t explicitly mention them by name in the brief demo, but a couple of free search engines for Certificate Transparency logs are crt.sh and Merklemap. I’ve found crt.sh has been playing up over the last few days so you may need to reload pages once or twice to get them to load.