Past Incidents

All 31 services were moved to the unprivileged container runtime and re-verified after the updater's first cycle (18 services re-created onto freshly pulled images, all healthy). Nothing was rolled back. From now on, feature upgrades of Mastodon, Seafile and Nextcloud will be announced on this page a day in advance, with a summary of what changes.

All 31 services have moved to the unprivileged runtime and pass their health checks. The automatic updater now runs there too; its first cycle (20:50) re-creates a number of containers onto freshly pulled copies of the same images, so expect one more short restart per service. This entry closes once that round is verified.

Progress: 27 of 31 services have moved and are verified healthy (all sites, games, calling, streaming, Matrix). Remaining: GitLab, Nextcloud, Seafile and Mastodon, which move one at a time as their images finish copying; each will be unavailable for roughly 3-10 minutes during its move.

Migration window started. The reverse proxy moves first, then each service in turn; a service is unreachable from the moment its old copy stops until its new copy is up (usually 1-3 minutes).

Incident responder: root cause is gitliz's own in-app rate limiter, not a crash. Container was running (no OOM, 0 restarts) but the health URL returned HTTP 429. The app keys its brute-force block on request.remote_addr, which behind the nginx proxy is always the proxy IP 172.22.0.34, so all clients share one bucket. A burst of 4xx at 15:10:59 (likely a scanner) tripped the >10/min limit. Because the after_request hook also counts the 429 itself as a 4xx and re-extends blocked_until by 10 min on every request, the once-a-minute health probe kept the block alive indefinitely (self-perpetuating in-memory state, same mechanism as the 13:37 and yesterday's 13:52 outages). ACTION: restarted gitliz once at 15:32:36 to clear the in-memory block table; gunicorn came up cleanly with 4 workers. Health verified HTTP 200 twice (15:32 and ~15:34), no new block lines since. Follow-up needed in code (gitliz/web/app_factory.py ~L79-107): use X-Forwarded-For/ProxyFix so the real client IP is keyed, and do not count 429 responses toward the block or extend blocked_until while already blocked. Until fixed, this will recur whenever any scanner sends >10 4xx/min through the proxy.