Announcements and status/progress updates about upgrades will be posted here.
This website runs on a dedicated host and will be up even if everything else is down.
Other Components
cachet-docker_postgres-cachet
Operational
cachet-docker_cachet
Operational
dergnz-unowngen
Operational
dergnz-mastodon
Operational
dergnz-semicity
Operational
dergnz-semiracing
Operational
dragonhive-commet
Operational
dragonhive-hoardware
Operational
dragonhive-bashorg-quotes-api
Operational
dragonhive-drgnshooty
Operational
dragonhive-coatrack
Operational
dragonhive-gitliz
Operational
dragonhive-fractalhoard
Operational
dragonhive-VHF
Operational
dragonhive-coturn
Operational
dragonhive-element-call
Operational
dragonhive-hyditor
Operational
dragonhive-owncast
Operational
dragonhive-logme
Operational
dragonhive-matrix
Operational
dragonhive-lk-jwt
Operational
dragonhive-nextcloud
Operational
dragonhive-icecast
Operational
dragonhive-owntracks
Operational
dragonhive-livekit
Operational
dragonhive-seafile
Operational
dragonhive-shellserver01
Operational
nginx-frontweb
Operational
dragonhive-slowstream
Operational
youwouldntverbanounbot
Operational
dragonhive-gitlab-runner
Operational
dragonhive-gitlab
Operational
dergnz-semibox
Operational
dergnz-semitower
Operational
faelden-dergnz
Operational
dergnz-semispelunker
Operational
Past Incidents
3rd November 2026
dragonhive-coatrackOutage: dragonhive-coatrack
dragonhive-coatrack recovered after about 2351 minutes of downtime.
Responder findings (04:12, outage mode). NO restart/up/exec performed.
Root cause: the coatrack app container (image dragonhive-coatrack-coatrack) is crash-looping (restarts=40, exit 1, oomkilled=false). Every start fails in /app/migrate.py -> sqlalchemy create_engine with "ModuleNotFoundError: No module named 'psycopg'". The DATABASE_URL uses SQLAlchemy's psycopg (v3) dialect, but the built image doesn't include the psycopg package. This most likely comes from upstream commit 660060ce -> 25369277, which was pulled and built at 03:34 (a dependency/driver change in requirements or the DB URL). The rollback at 03:40 didn't recover: the restored container still fails the same way. Either the rollback reused the newly built image tag, or it didn't rebuild from the previous commit.
Postgres (coatrack-postgres) is up and healthy. The app reaches it fine ("PostgreSQL is ready!"). There are no disk, OOM or DB problems.
Why I didn't act: a restart re-runs the same image and fails the same way. Emergency exec is out of scope (exec_services/host paths are empty). Installing packages into a running container would also be lost on recreation. Health checked twice: HTTP 000 / curl exit 60 (TLS certificate verification failure). This is probably the reverse proxy's fallback because the backend is down, but a human should confirm that the cert itself hasn't expired.
Human fix: in the dragonhive-coatrack source, add psycopg (e.g. psycopg[binary]) to requirements, or set DATABASE_URL to the installed driver (e.g. postgresql+psycopg2:// if psycopg2 is present). Then rebuild (docker compose build --no-cache coatrack && docker compose up -d). Alternatively, check out 660060ce, rebuild, and pin it until upstream is fixed. NOTE: the DB password is printed in plaintext in the app logs by migrate.py. Consider rotating it and reporting this upstream.
Automatic update of dragonhive-coatrack failed: failed the post-update health check and the rollback also failed. Automated diagnosis is starting; manual intervention may be required.
Diagnosis (no repair attempted). coatrack is crash-looping (exit 1, 14+ restarts, not OOM). postgres is healthy and reachable: the entrypoint gets to "PostgreSQL is ready". The failure is in /app/migrate.py: create_engine() loads SQLAlchemy's postgresql+psycopg (psycopg v3) dialect, and import psycopg raises ModuleNotFoundError. So the locally built image dragonhive-coatrack-coatrack has a dependency mismatch: the code or config wants psycopg3, but the image doesn't install it (it may only ship psycopg2, or nothing). This most likely comes from upstream commit 660060ce -> 25369277. The rollback did not help. The current container was created at 03:45:37, after the rollback, and still fails the same way. My guess is the rollback reused the same locally tagged image, or didn't rebuild from the old commit, so the broken build is still running. A human should check this. Why I didn't act: a restart or up would rerun the same image and hit the same ImportError. Running pip install inside the container isn't a service-native fix, wouldn't survive a restart, and no exec scope is configured anyway. Suggested fix: either rebuild from the previous commit 660060ce (git checkout 660060ce && docker compose build --no-cache coatrack && docker compose up -d), or fix requirements upstream by adding psycopg[binary] or switching the URL/driver back to psycopg2. The database was not touched. Side finding: migrate.py prints the full DATABASE_URL, password included, to the container logs. That's worth fixing upstream, and if the logs have been shipped anywhere, consider rotating the password.
29th October 2026
Brief restart of all services (network proxy maintenance)
The container network layer is being restarted to clear a resource leak. Every service is unavailable for roughly 2-5 minutes, then comes back on its own.
All services are back (21:43). The restart took about 9 minutes instead of the planned 2-5 because six containers did not come back on their own and were started by hand; that is being fixed so the next one is short.
15th September 2026
dergnz-semiboxOutage: dergnz-semibox
dergnz-semibox recovered after about 132 minutes of downtime.
Automatic update of dergnz-semibox failed: failed the post-update health check and the rollback also failed. Automated diagnosis is starting; manual intervention may be required.