Some systems are experiencing issues

About This Site

Announcements and status/progress updates about upgrades will be posted here.

This website runs on a dedicated host and will be up even if everything else is down.

Past Incidents

3rd November 2026

dragonhive-coatrack Outage: dragonhive-coatrack

dragonhive-coatrack has been failing health checks since 03:51 (server time). Investigating. Request load: traffic at 03:53: 0.2 req/min, 1 5xx, 0% bots, top client 45.148.10.120 (1).

  • Responder findings (04:12, outage mode). NO restart/up/exec performed.

    Root cause: the coatrack app container (image dragonhive-coatrack-coatrack) is crash-looping (restarts=40, exit 1, oomkilled=false). Every start fails in /app/migrate.py -> sqlalchemy create_engine with "ModuleNotFoundError: No module named 'psycopg'". The DATABASE_URL uses SQLAlchemy's psycopg (v3) dialect, but the built image doesn't include the psycopg package. This most likely comes from upstream commit 660060ce -> 25369277, which was pulled and built at 03:34 (a dependency/driver change in requirements or the DB URL). The rollback at 03:40 didn't recover: the restored container still fails the same way. Either the rollback reused the newly built image tag, or it didn't rebuild from the previous commit.

    Postgres (coatrack-postgres) is up and healthy. The app reaches it fine ("PostgreSQL is ready!"). There are no disk, OOM or DB problems.

    Why I didn't act: a restart re-runs the same image and fails the same way. Emergency exec is out of scope (exec_services/host paths are empty). Installing packages into a running container would also be lost on recreation. Health checked twice: HTTP 000 / curl exit 60 (TLS certificate verification failure). This is probably the reverse proxy's fallback because the backend is down, but a human should confirm that the cert itself hasn't expired.

    Human fix: in the dragonhive-coatrack source, add psycopg (e.g. psycopg[binary]) to requirements, or set DATABASE_URL to the installed driver (e.g. postgresql+psycopg2:// if psycopg2 is present). Then rebuild (docker compose build --no-cache coatrack && docker compose up -d). Alternatively, check out 660060ce, rebuild, and pin it until upstream is fixed. NOTE: the DB password is printed in plaintext in the app logs by migrate.py. Consider rotating it and reporting this upstream.

    • dragonhive-coatrack Service needs attention: dragonhive-coatrack

    Automatic update of dragonhive-coatrack failed: failed the post-update health check and the rollback also failed. Automated diagnosis is starting; manual intervention may be required.

  • Diagnosis (no repair attempted). coatrack is crash-looping (exit 1, 14+ restarts, not OOM). postgres is healthy and reachable: the entrypoint gets to "PostgreSQL is ready". The failure is in /app/migrate.py: create_engine() loads SQLAlchemy's postgresql+psycopg (psycopg v3) dialect, and import psycopg raises ModuleNotFoundError. So the locally built image dragonhive-coatrack-coatrack has a dependency mismatch: the code or config wants psycopg3, but the image doesn't install it (it may only ship psycopg2, or nothing). This most likely comes from upstream commit 660060ce -> 25369277. The rollback did not help. The current container was created at 03:45:37, after the rollback, and still fails the same way. My guess is the rollback reused the same locally tagged image, or didn't rebuild from the old commit, so the broken build is still running. A human should check this. Why I didn't act: a restart or up would rerun the same image and hit the same ImportError. Running pip install inside the container isn't a service-native fix, wouldn't survive a restart, and no exec scope is configured anyway. Suggested fix: either rebuild from the previous commit 660060ce (git checkout 660060ce && docker compose build --no-cache coatrack && docker compose up -d), or fix requirements upstream by adding psycopg[binary] or switching the URL/driver back to psycopg2. The database was not touched. Side finding: migrate.py prints the full DATABASE_URL, password included, to the container logs. That's worth fixing upstream, and if the logs have been shipped anywhere, consider rotating the password.

    • 29th October 2026

    Brief restart of all services (network proxy maintenance)

    The container network layer is being restarted to clear a resource leak. Every service is unavailable for roughly 2-5 minutes, then comes back on its own.

  • All services are back (21:43). The restart took about 9 minutes instead of the planned 2-5 because six containers did not come back on their own and were started by hand; that is being fixed so the next one is short.

    • 15th September 2026

    dergnz-semibox Outage: dergnz-semibox

    dergnz-semibox recovered after about 132 minutes of downtime.

    dergnz-semibox Service needs attention: dergnz-semibox

    Automatic update of dergnz-semibox failed: failed the post-update health check and the rollback also failed. Automated diagnosis is starting; manual intervention may be required.

    5th September 2026

    Platform maintenance: moving all services to an unprivileged container runtime

    As the final step of this week's security hardening, every DragonHive / derg.nz service is being moved from a root-level container runtime to an unprivileged one.

    During this window each service restarts once. Expect short, intermittent outages: typically 1-5 minutes per service, up to ~10 minutes for GitLab, Mastodon, Nextcloud and Seafile. No data is modified, and the previous setup is kept intact for an instant rollback.

    This entry stays open until the migration is complete and verified; progress updates follow below.

  • All 31 services were moved to the unprivileged container runtime and re-verified after the updater's first cycle (18 services re-created onto freshly pulled images, all healthy). Nothing was rolled back. From now on, feature upgrades of Mastodon, Seafile and Nextcloud will be announced on this page a day in advance, with a summary of what changes.

  • All 31 services have moved to the unprivileged runtime and pass their health checks. The automatic updater now runs there too; its first cycle (20:50) re-creates a number of containers onto freshly pulled copies of the same images, so expect one more short restart per service. This entry closes once that round is verified.

  • Progress: 27 of 31 services have moved and are verified healthy (all sites, games, calling, streaming, Matrix). Remaining: GitLab, Nextcloud, Seafile and Mastodon, which move one at a time as their images finish copying; each will be unavailable for roughly 3-10 minutes during its move.

  • Migration window started. The reverse proxy moves first, then each service in turn; a service is unreachable from the moment its old copy stops until its new copy is up (usually 1-3 minutes).

    • dragonhive-seafile Seafile upgraded to 13.0 — brief outages tonight

    Seafile (seafile.dragonhive.net) was upgraded tonight from 11.0.13 to 12.0.14 and then to 13.0.25, with database backups and a ZFS snapshot taken before each step. Seafile is now on the current major version, using Redis instead of memcached, and its 27 known critical vulnerabilities from the old image are gone.

    Downtime: Seafile was unavailable roughly 01:11–01:18 CEST for the 12.0 step (a database health-check mismatch cost about four of those minutes) and about 01:26–01:28 for the 13.0 step.

    Separately, at 01:27 a Docker engine package upgrade restarted the container engine on the host. All services restarted; most were back within a minute, but the reverse proxy and one small site did not come back on their own and were restarted by hand, so every site was unreachable for roughly three to five minutes around 01:30–01:33. The engine is now configured to keep containers running across engine restarts so this cannot repeat.

    Also completed today: host firewall, sandboxed automation, vulnerability scanning of every update, and unprivileged containers for most services. Everything is up.

    4th September 2026

    dragonhive-gitliz Outage: dragonhive-gitliz

    dragonhive-gitliz recovered after about 24 minutes of downtime.

  • Incident responder: root cause is gitliz's own in-app rate limiter, not a crash. Container was running (no OOM, 0 restarts) but the health URL returned HTTP 429. The app keys its brute-force block on request.remote_addr, which behind the nginx proxy is always the proxy IP 172.22.0.34, so all clients share one bucket. A burst of 4xx at 15:10:59 (likely a scanner) tripped the >10/min limit. Because the after_request hook also counts the 429 itself as a 4xx and re-extends blocked_until by 10 min on every request, the once-a-minute health probe kept the block alive indefinitely (self-perpetuating in-memory state, same mechanism as the 13:37 and yesterday's 13:52 outages). ACTION: restarted gitliz once at 15:32:36 to clear the in-memory block table; gunicorn came up cleanly with 4 workers. Health verified HTTP 200 twice (15:32 and ~15:34), no new block lines since. Follow-up needed in code (gitliz/web/app_factory.py ~L79-107): use X-Forwarded-For/ProxyFix so the real client IP is keyed, and do not count 429 responses toward the block or extend blocked_until while already blocked. Until fixed, this will recur whenever any scanner sends >10 4xx/min through the proxy.

    • dragonhive-gitliz Outage: dragonhive-gitliz

    dragonhive-gitliz recovered after about 19 minutes of downtime.

    Related Mastodon account