Some systems are experiencing issues

About This Site

Announcements and status/progress updates about upgrades will be posted here.

This website runs on a dedicated host and will be up even if everything else is down.

Past Incidents

19th October 2025

things cannot connect to their backends for whatever reason

welp, not sure what docker changed but the frontend loadbalancer can't see any of the backend services right now for whatever reason.

standby, we're working on it

  • yea, everything is fine now.

    post mortem: I think what happened, is that after the reboot, docker-gen ran before things settled, working with stale information somehow and creating a bad config with addresses nginx couldn't reach, and then got confused somehow.

  • .... after repeatedly restarting the frontend loadbalancer it now suddenly connects to the containers again.

    looks like things are working? but I'm keeping an eye on it

  • still investigating. it's very strange, the backend network is there, all the containers are in it, yet none of the services appear to be reachable from the frontend loadbalancer?

    istg if fundamental tools could stop radically changing every update that'd be really swell

  • 11th September 2025

    Mastodon down because Backup is running

    I just realized the mastodon backup didn't successfully complete because the ZFS backup is running .. ... might cancel it but it could mess with things so first trying to see how far into the process it is

  • alright fixed, but I'm still messing with the webserver so some disturbance can still happen

  • hmm, started it but now I think nginx is upset because of some unrelated config

  • Related Mastodon account