Some systems are experiencing issues

About This Site

Announcements and status/progress updates about upgrades will be posted here.

This website runs on a dedicated host and will be up even if everything else is down.

Past Incidents

5th September 2026

dragonhive-seafile Seafile upgraded to 13.0 — brief outages tonight

Seafile (seafile.dragonhive.net) was upgraded tonight from 11.0.13 to 12.0.14 and then to 13.0.25, with database backups and a ZFS snapshot taken before each step. Seafile is now on the current major version, using Redis instead of memcached, and its 27 known critical vulnerabilities from the old image are gone.

Downtime: Seafile was unavailable roughly 01:11–01:18 CEST for the 12.0 step (a database health-check mismatch cost about four of those minutes) and about 01:26–01:28 for the 13.0 step.

Separately, at 01:27 a Docker engine package upgrade restarted the container engine on the host. All services restarted; most were back within a minute, but the reverse proxy and one small site did not come back on their own and were restarted by hand, so every site was unreachable for roughly three to five minutes around 01:30–01:33. The engine is now configured to keep containers running across engine restarts so this cannot repeat.

Also completed today: host firewall, sandboxed automation, vulnerability scanning of every update, and unprivileged containers for most services. Everything is up.

4th September 2026

dragonhive-gitliz Outage: dragonhive-gitliz

dragonhive-gitliz recovered after about 24 minutes of downtime.

  • Incident responder: root cause is gitliz's own in-app rate limiter, not a crash. Container was running (no OOM, 0 restarts) but the health URL returned HTTP 429. The app keys its brute-force block on request.remote_addr, which behind the nginx proxy is always the proxy IP 172.22.0.34, so all clients share one bucket. A burst of 4xx at 15:10:59 (likely a scanner) tripped the >10/min limit. Because the after_request hook also counts the 429 itself as a 4xx and re-extends blocked_until by 10 min on every request, the once-a-minute health probe kept the block alive indefinitely (self-perpetuating in-memory state, same mechanism as the 13:37 and yesterday's 13:52 outages). ACTION: restarted gitliz once at 15:32:36 to clear the in-memory block table; gunicorn came up cleanly with 4 workers. Health verified HTTP 200 twice (15:32 and ~15:34), no new block lines since. Follow-up needed in code (gitliz/web/app_factory.py ~L79-107): use X-Forwarded-For/ProxyFix so the real client IP is keyed, and do not count 429 responses toward the block or extend blocked_until while already blocked. Until fixed, this will recur whenever any scanner sends >10 4xx/min through the proxy.

    • dragonhive-gitliz Outage: dragonhive-gitliz

    dragonhive-gitliz recovered after about 19 minutes of downtime.

    3rd September 2026

    dragonhive-gitliz Outage: dragonhive-gitliz

    dragonhive-gitliz recovered after about 25 minutes of downtime.

    6th August 2026

    dergnz-semibox Update rolled back: dergnz-semibox

    An automatic update of dergnz-semibox was rolled back (failed the post-update health check; previous version restored). The service is running the previous version.

    13th July 2026

    nginx-frontweb Update rolled back: nginx-frontweb

    An automatic update of nginx-frontweb was rolled back (failed the post-update health check; previous version restored). The service is running the previous version.

  • Automated RCA (incident responder) — REPEAT of incident #363 (06:02 today). Root cause unchanged: upstream regression in nginxproxy/docker-gen latest tag. At ~10:00 UTC upstream CI pushed ANOTHER master build to latest (local image 7e689bcec391). Its new digest was not in the updater failed_targets list (which stores exact digests), so the 12:42 update retried and pulled it. The build still contains the docker-entrypoint.sh regression from docker-gen PR #761: under set -eu the entrypoint probes the container command with command -v; our compose service passes a flag-style command (-notify-sighup nginx-proxy -watch ...), the probe returns non-zero, the entrypoint dies before its fallback and the container crash-loops. nginx-frontweb has no health_url, so the container-state probe failed the full 300s window (12:43:28 to 12:48:48) and the updater correctly rolled back; healthy again 12:51:11. acme-companion was the same digest that failed this morning — innocent bystander pulled alongside. CURRENT STATE: nginx-frontweb STABLE on previous images, all three containers up; no action taken on the service, diagnosis only. RECOMMENDATION: pin nginxproxy/docker-gen:0.17.2 and nginxproxy/acme-companion:2.8.1 in docker-compose.yml instead of latest — every new upstream master push mints a new digest that bypasses failed_targets, so this pull/crash/rollback cycle will repeat each 6-hourly check until pinned or fixed upstream (docker-gen PR #761 / issue #628). Optionally add a health_url for nginx-frontweb and a per-service cooldown after repeated rollbacks.

    • nginx-frontweb Update rolled back: nginx-frontweb

    An automatic update of nginx-frontweb was rolled back (failed the post-update health check; previous version restored). The service is running the previous version.

  • Automated RCA (incident responder) — nginx-frontweb update rolled back at 06:11 CEST; service is STABLE on the previous version, no user-facing impact beyond the update window.

    Root cause: upstream nginx-proxy/docker-gen published a master/CI build to the latest tag at 00:36 UTC today (0.17.2-6-g256bda4, not a tagged release). It includes PR 761, which rewrote app/docker-entrypoint.sh: the new code assigns the result of command -v on the first CMD argument to a variable while running under set -eu. Our compose service passes flag-style arguments (-notify-sighup nginx-proxy -watch ...), so command -v returns non-zero and set -e aborts the entrypoint before its docker-gen fallback line can run. The nginx-proxy-gen container exited immediately and crash-looped; the updater health probe (all containers running) failed for the full 300s window and it correctly rolled back. The simultaneously pulled acme-companion build (v2.8.1-4-g10d98eb) contains only test changes plus an internal docker-gen version bump and was almost certainly an innocent bystander, rolled back together with docker-gen.

    Actions taken: diagnosis only. Nothing was restarted, deleted, or edited. The updater already restored the previous images (docker-gen 0.17.2-3-gea3bb98, acme-companion v2.8.1) and recorded the failed digests in state.json so the same digests will not be retried.

    Recommendation: (1) pin nginxproxy/docker-gen (and ideally acme-companion) to versioned tags, e.g. nginxproxy/docker-gen:0.17.2, instead of latest — upstream pushes untagged master builds to latest; (2) optionally report the regression upstream (docker-gen PR 761 / issue 628: entrypoint aborts under set -e when CMD starts with a flag) — until it is fixed upstream, every new latest digest will fail and roll back again; (3) close this incident when convenient — the service itself is healthy.

    • Related Mastodon account