Announcements and status/progress updates about upgrades will be posted here.
This website runs on a dedicated host and will be up even if everything else is down.
Other Components
cachet-docker_postgres-cachet
Operational
cachet-docker_cachet
Operational
dergnz-semicity
Operational
dergnz-semiracing
Operational
dergnz-unowngen
Operational
dergnz-mastodon
Operational
dragonhive-gitliz
Operational
dragonhive-fractalhoard
Operational
dragonhive-drgnshooty
Operational
dragonhive-commet
Operational
dragonhive-VHF
Operational
dragonhive-element-call
Operational
dragonhive-coturn
Operational
dragonhive-coatrack
Operational
dragonhive-bashorg-quotes-api
Operational
dragonhive-hoardware
Operational
dragonhive-livekit
Operational
dragonhive-owntracks
Operational
dragonhive-shellserver01
Operational
dragonhive-icecast
Operational
dragonhive-lk-jwt
Operational
dragonhive-owncast
Operational
dragonhive-logme
Operational
dragonhive-hyditor
Operational
dragonhive-nextcloud
Operational
dragonhive-seafile
Operational
dragonhive-matrix
Operational
nginx-frontweb
Operational
dragonhive-slowstream
Operational
youwouldntverbanounbot
Operational
dragonhive-gitlab-runner
Operational
dragonhive-gitlab
Operational
dergnz-semibox
Operational
Past Incidents
6th August 2026
dergnz-semiboxUpdate rolled back: dergnz-semibox
An automatic update of dergnz-semibox was rolled back (failed the post-update health check; previous version restored). The service is running the previous version.
13th July 2026
nginx-frontwebUpdate rolled back: nginx-frontweb
An automatic update of nginx-frontweb was rolled back (failed the post-update health check; previous version restored). The service is running the previous version.
Automated RCA (incident responder) — REPEAT of incident #363 (06:02 today). Root cause unchanged: upstream regression in nginxproxy/docker-gen latest tag. At ~10:00 UTC upstream CI pushed ANOTHER master build to latest (local image 7e689bcec391). Its new digest was not in the updater failed_targets list (which stores exact digests), so the 12:42 update retried and pulled it. The build still contains the docker-entrypoint.sh regression from docker-gen PR #761: under set -eu the entrypoint probes the container command with command -v; our compose service passes a flag-style command (-notify-sighup nginx-proxy -watch ...), the probe returns non-zero, the entrypoint dies before its fallback and the container crash-loops. nginx-frontweb has no health_url, so the container-state probe failed the full 300s window (12:43:28 to 12:48:48) and the updater correctly rolled back; healthy again 12:51:11. acme-companion was the same digest that failed this morning — innocent bystander pulled alongside. CURRENT STATE: nginx-frontweb STABLE on previous images, all three containers up; no action taken on the service, diagnosis only. RECOMMENDATION: pin nginxproxy/docker-gen:0.17.2 and nginxproxy/acme-companion:2.8.1 in docker-compose.yml instead of latest — every new upstream master push mints a new digest that bypasses failed_targets, so this pull/crash/rollback cycle will repeat each 6-hourly check until pinned or fixed upstream (docker-gen PR #761 / issue #628). Optionally add a health_url for nginx-frontweb and a per-service cooldown after repeated rollbacks.
nginx-frontwebUpdate rolled back: nginx-frontweb
An automatic update of nginx-frontweb was rolled back (failed the post-update health check; previous version restored). The service is running the previous version.
Automated RCA (incident responder) — nginx-frontweb update rolled back at 06:11 CEST; service is STABLE on the previous version, no user-facing impact beyond the update window.
Root cause: upstream nginx-proxy/docker-gen published a master/CI build to the latest tag at 00:36 UTC today (0.17.2-6-g256bda4, not a tagged release). It includes PR 761, which rewrote app/docker-entrypoint.sh: the new code assigns the result of command -v on the first CMD argument to a variable while running under set -eu. Our compose service passes flag-style arguments (-notify-sighup nginx-proxy -watch ...), so command -v returns non-zero and set -e aborts the entrypoint before its docker-gen fallback line can run. The nginx-proxy-gen container exited immediately and crash-looped; the updater health probe (all containers running) failed for the full 300s window and it correctly rolled back. The simultaneously pulled acme-companion build (v2.8.1-4-g10d98eb) contains only test changes plus an internal docker-gen version bump and was almost certainly an innocent bystander, rolled back together with docker-gen.
Actions taken: diagnosis only. Nothing was restarted, deleted, or edited. The updater already restored the previous images (docker-gen 0.17.2-3-gea3bb98, acme-companion v2.8.1) and recorded the failed digests in state.json so the same digests will not be retried.
Recommendation: (1) pin nginxproxy/docker-gen (and ideally acme-companion) to versioned tags, e.g. nginxproxy/docker-gen:0.17.2, instead of latest — upstream pushes untagged master builds to latest; (2) optionally report the regression upstream (docker-gen PR 761 / issue 628: entrypoint aborts under set -e when CMD starts with a flag) — until it is fixed upstream, every new latest digest will fail and roll back again; (3) close this incident when convenient — the service itself is healthy.
Resolved: the Nextcloud update needed longer than the health check allowed for its internal database upgrade. The update was completed manually (rolled forward to 34.0.1) and the service is fully operational again. Total downtime was about 11 minutes.