Past Incidents

All 31 services were moved to the unprivileged container runtime and re-verified after the updater's first cycle (18 services re-created onto freshly pulled images, all healthy). Nothing was rolled back. From now on, feature upgrades of Mastodon, Seafile and Nextcloud will be announced on this page a day in advance, with a summary of what changes.

All 31 services have moved to the unprivileged runtime and pass their health checks. The automatic updater now runs there too; its first cycle (20:50) re-creates a number of containers onto freshly pulled copies of the same images, so expect one more short restart per service. This entry closes once that round is verified.

Progress: 27 of 31 services have moved and are verified healthy (all sites, games, calling, streaming, Matrix). Remaining: GitLab, Nextcloud, Seafile and Mastodon, which move one at a time as their images finish copying; each will be unavailable for roughly 3-10 minutes during its move.

Migration window started. The reverse proxy moves first, then each service in turn; a service is unreachable from the moment its old copy stops until its new copy is up (usually 1-3 minutes).

Incident responder: root cause is gitliz's own in-app rate limiter, not a crash. Container was running (no OOM, 0 restarts) but the health URL returned HTTP 429. The app keys its brute-force block on request.remote_addr, which behind the nginx proxy is always the proxy IP 172.22.0.34, so all clients share one bucket. A burst of 4xx at 15:10:59 (likely a scanner) tripped the >10/min limit. Because the after_request hook also counts the 429 itself as a 4xx and re-extends blocked_until by 10 min on every request, the once-a-minute health probe kept the block alive indefinitely (self-perpetuating in-memory state, same mechanism as the 13:37 and yesterday's 13:52 outages). ACTION: restarted gitliz once at 15:32:36 to clear the in-memory block table; gunicorn came up cleanly with 4 workers. Health verified HTTP 200 twice (15:32 and ~15:34), no new block lines since. Follow-up needed in code (gitliz/web/app_factory.py ~L79-107): use X-Forwarded-For/ProxyFix so the real client IP is keyed, and do not count 429 responses toward the block or extend blocked_until while already blocked. Until fixed, this will recur whenever any scanner sends >10 4xx/min through the proxy.

Automated RCA (incident responder) — REPEAT of incident #363 (06:02 today). Root cause unchanged: upstream regression in nginxproxy/docker-gen latest tag. At ~10:00 UTC upstream CI pushed ANOTHER master build to latest (local image 7e689bcec391). Its new digest was not in the updater failed_targets list (which stores exact digests), so the 12:42 update retried and pulled it. The build still contains the docker-entrypoint.sh regression from docker-gen PR #761: under set -eu the entrypoint probes the container command with command -v; our compose service passes a flag-style command (-notify-sighup nginx-proxy -watch ...), the probe returns non-zero, the entrypoint dies before its fallback and the container crash-loops. nginx-frontweb has no health_url, so the container-state probe failed the full 300s window (12:43:28 to 12:48:48) and the updater correctly rolled back; healthy again 12:51:11. acme-companion was the same digest that failed this morning — innocent bystander pulled alongside. CURRENT STATE: nginx-frontweb STABLE on previous images, all three containers up; no action taken on the service, diagnosis only. RECOMMENDATION: pin nginxproxy/docker-gen:0.17.2 and nginxproxy/acme-companion:2.8.1 in docker-compose.yml instead of latest — every new upstream master push mints a new digest that bypasses failed_targets, so this pull/crash/rollback cycle will repeat each 6-hourly check until pinned or fixed upstream (docker-gen PR #761 / issue #628). Optionally add a health_url for nginx-frontweb and a per-service cooldown after repeated rollbacks.