Outage: dragonhive-gitliz Friday 4th September 2026 15:15:14


dragonhive-gitliz recovered after about 24 minutes of downtime.

Incident responder: root cause is gitliz's own in-app rate limiter, not a crash. Container was running (no OOM, 0 restarts) but the health URL returned HTTP 429. The app keys its brute-force block on request.remote_addr, which behind the nginx proxy is always the proxy IP 172.22.0.34, so all clients share one bucket. A burst of 4xx at 15:10:59 (likely a scanner) tripped the >10/min limit. Because the after_request hook also counts the 429 itself as a 4xx and re-extends blocked_until by 10 min on every request, the once-a-minute health probe kept the block alive indefinitely (self-perpetuating in-memory state, same mechanism as the 13:37 and yesterday's 13:52 outages). ACTION: restarted gitliz once at 15:32:36 to clear the in-memory block table; gunicorn came up cleanly with 4 workers. Health verified HTTP 200 twice (15:32 and ~15:34), no new block lines since. Follow-up needed in code (gitliz/web/app_factory.py ~L79-107): use X-Forwarded-For/ProxyFix so the real client IP is keyed, and do not count 429 responses toward the block or extend blocked_until while already blocked. Until fixed, this will recur whenever any scanner sends >10 4xx/min through the proxy.

Related Mastodon account