My Homelab Got Hacked - A Postmortem
phunky.cafe
A postmortem on the hack of my self-hosted Forgejo instance and a reminder to patch your software.

Welp. My Forgejo instance got popped with an RCE two days ago by CVE-2026-60004. Luckily, I noticed the following morning and had the day free to figure out what happened. Let’s dive in!

As a homelab enthusiast, I found this a very interesting post. Here are my take aways from the post that I’m implementing myself:

  • Miner detection. I’ve updated monitoring rules to now watch the CPU on my hosts. If the same thing happened to me I would not have been alerted at all as I’m doing simple up / down monitoring. Fixed.
  • Access logging. I turned on access logging for my homelab Caddy instances.
  • Log retention. I have increased the amount and retention of my logging. The hope is this will help me reconstruct what happened after a breach.
  • Logs offsite. The VPS access logs now ride along with the normal backup process, which runs hourly. The homelab side still only gets caught by the weekly VM backup, so that’s next.
  • Closed an open signup. My webtrees instance (genealogy) had self registration enabled, which is the same door this guy got hit through. Oops. Fixed.
  • Built a tool. log-inventory.sh, so “could I actually reconstruct what happened” is a command I run instead of a thing I assume.
  • MadPsyentist@lemmy.nzEnglish
    1·
    2 days ago

    Right, i think we are talking about 2 different but very close things. Jet’s post is talking about a setup similar to mine, which is a wildcard certificate which LetsEncrypt only issue with a DNS-01 challenge, which, as you stated, requires the TXT record and not a text file on the server that the HTTP-01 challenge requires. Yes the HTTP-01 challenge would fail for an internal ip because letsencrypt wouldn’t be able to reach your site to check the acme challenge text file.

    But for sanity’s sake you can setup a wildcard cert for your homelab. Traefik makes it pretty painless and now you are not having to configure DNS records for your subdomains. For added security your A record on your dns can point to your internal IP address of your reverse proxy which gives everything routed through your proxy TLS but no-one outside your network can access it.

    • KubeRoot@discuss.tchncs.deEnglish
      2·
      2 days ago

      Jet says the following

      If you want https it’s gotta be a external DNS record anyway, might as well point to internal ips

      My point is, the A record, the one pointing to an IP address, doesn’t need to be public/external, it’s not needed for the wildcard cert, so it can be kept internal.

      I’m mentioning the other challenge trying to make it clear why the record matters for that, and not for a wildcard cert, sorry if that just made things more confusing.