Some systems are experiencing issues

About This Site

Announcements and status/progress updates about upgrades will be posted here.

This website runs on a dedicated host and will be up even if everything else is down.

Past Incidents

13th July 2026

nginx-frontweb Update rolled back: nginx-frontweb

An automatic update of nginx-frontweb was rolled back (failed the post-update health check; previous version restored). The service is running the previous version.

  • Automated RCA (incident responder) — REPEAT of incident #363 (06:02 today). Root cause unchanged: upstream regression in nginxproxy/docker-gen latest tag. At ~10:00 UTC upstream CI pushed ANOTHER master build to latest (local image 7e689bcec391). Its new digest was not in the updater failed_targets list (which stores exact digests), so the 12:42 update retried and pulled it. The build still contains the docker-entrypoint.sh regression from docker-gen PR #761: under set -eu the entrypoint probes the container command with command -v; our compose service passes a flag-style command (-notify-sighup nginx-proxy -watch ...), the probe returns non-zero, the entrypoint dies before its fallback and the container crash-loops. nginx-frontweb has no health_url, so the container-state probe failed the full 300s window (12:43:28 to 12:48:48) and the updater correctly rolled back; healthy again 12:51:11. acme-companion was the same digest that failed this morning — innocent bystander pulled alongside. CURRENT STATE: nginx-frontweb STABLE on previous images, all three containers up; no action taken on the service, diagnosis only. RECOMMENDATION: pin nginxproxy/docker-gen:0.17.2 and nginxproxy/acme-companion:2.8.1 in docker-compose.yml instead of latest — every new upstream master push mints a new digest that bypasses failed_targets, so this pull/crash/rollback cycle will repeat each 6-hourly check until pinned or fixed upstream (docker-gen PR #761 / issue #628). Optionally add a health_url for nginx-frontweb and a per-service cooldown after repeated rollbacks.

    • nginx-frontweb Update rolled back: nginx-frontweb

    An automatic update of nginx-frontweb was rolled back (failed the post-update health check; previous version restored). The service is running the previous version.

  • Automated RCA (incident responder) — nginx-frontweb update rolled back at 06:11 CEST; service is STABLE on the previous version, no user-facing impact beyond the update window.

    Root cause: upstream nginx-proxy/docker-gen published a master/CI build to the latest tag at 00:36 UTC today (0.17.2-6-g256bda4, not a tagged release). It includes PR 761, which rewrote app/docker-entrypoint.sh: the new code assigns the result of command -v on the first CMD argument to a variable while running under set -eu. Our compose service passes flag-style arguments (-notify-sighup nginx-proxy -watch ...), so command -v returns non-zero and set -e aborts the entrypoint before its docker-gen fallback line can run. The nginx-proxy-gen container exited immediately and crash-looped; the updater health probe (all containers running) failed for the full 300s window and it correctly rolled back. The simultaneously pulled acme-companion build (v2.8.1-4-g10d98eb) contains only test changes plus an internal docker-gen version bump and was almost certainly an innocent bystander, rolled back together with docker-gen.

    Actions taken: diagnosis only. Nothing was restarted, deleted, or edited. The updater already restored the previous images (docker-gen 0.17.2-3-gea3bb98, acme-companion v2.8.1) and recorded the failed digests in state.json so the same digests will not be retried.

    Recommendation: (1) pin nginxproxy/docker-gen (and ideally acme-companion) to versioned tags, e.g. nginxproxy/docker-gen:0.17.2, instead of latest — upstream pushes untagged master builds to latest; (2) optionally report the regression upstream (docker-gen PR 761 / issue 628: entrypoint aborts under set -e when CMD starts with a flag) — until it is fixed upstream, every new latest digest will fail and roll back again; (3) close this incident when convenient — the service itself is healthy.

    • 6th July 2026

    dragonhive-nextcloud Service needs attention: dragonhive-nextcloud

    Resolved: the Nextcloud update needed longer than the health check allowed for its internal database upgrade. The update was completed manually (rolled forward to 34.0.1) and the service is fully operational again. Total downtime was about 11 minutes.

    5th July 2026

    dragonhive-slowstream Update failed: dragonhive-slowstream

    False alarm: the update was skipped because of local config changes; the service was never modified and remained fully operational the whole time.

    12th June 2026

    running manual updates on the primary host

    we're updating the primary host, and all the services. this will cause some brief downtime, so this statuspage will keep track of that.

  • everything should be up and running again. if you have any issues, feel free to poke https://mastodon.derg.nz/@dhstatus or anthropy on your preferred channels.

  • aaand it's back! okay let me finish the service updates and bring everything back up

  • rebooting again, let's see if it works now.

  • ..debian for whatever reason decided to install grub-efi instead of grub-pc, even though I use legacy bios boot (because it's a hetzner server). I never requested this, but I'm now fixing it so it should work again. also curious grub still allows you to do grub-install and such while the wrong version is installed? but oh well

  • yea it looks like it's bootlooping, but unsure why, let me get back into the rescue distro

  • rescue system came up fine, so the machine is fine. going to try something else and see if that boots. I guess something was wrong with grub after all, but unsure what because the config looks okay?

  • no response yet. I'm going to netboot into a rescue image to see what's wrong.

  • reset sent. let's wait a few minutes to see if it comes back

  • going to powercycle/reset the server through the hetzner control panel once we hit 15 minutes past midnight.

  • hmm, reboot is taking a bit longer than expected, though this can happen if something is hanging on shutdown or boot.

  • apt complained about grub so I'm verifying everything is okay there, it seems like it is though. .. rebooting machine now.

  • updates are running smoothly so far. reboot will happen as soon as those are done.

    • Related Mastodon account