Announcements and status/progress updates about upgrades will be posted here.
This website runs on a dedicated host and will be up even if everything else is down.
We're updating Mastodon and other services. I don't expect trouble but considering Mastodon will be offline multiple times I figured I'd have a statuspage open for the progress updates. See below for the updates.
Gitlab has also been restarted, meaning this update cycle has come to an end. Closing this statuspage, feel free to reach out if you run into any issues
mastodon migrations complete, booting in progress
Migrations are still in progress, everything looking good so far. I'm glad to see the script seems to be able to deal with the major version upgrade.. but let's wait with celebrating until it's all done
And we're on Mastodon 4.4.8! seems like everything worked fine, though the upgrade process is very slow because of how busy the server is. I'm firing off the upgrade process for 4.5.
Note that, because of this being a major upgrade, I do somewhat expect some migration pains, the last few times they had changed how Mastodon is launched an I had to edit the docker config.. but we'll see whatever breaks and fix it.
mastodon update is in progress; database is being backed up by the script, then it'll update everything, restart everything and run the migrations, and it'll be back .. for a bit, because I'll do the 4.5 update right after.
hmm, still busy, I think mastodon is being scraped, which is making it heavier than normal. I'll proceed with the Mastodon updates, that should also alleviate the load, then I'll do Gitlab last
Waiting for Nextcloud to calm down so I can restart Gitlab without it throwing a fit (it will literally fail to start the container if the database doesn't come up within 30 seconds)
frontend loadbalancer is being restarted so we're briefly down, after this I'll start with gitlab
Matrix has been restarted, Nextcloud is next.. almost every other application has been updated. After Gitlab is done I'll start the Mastodon upgrade to 4.4.8, and then 4.5 afterwards
welp, not sure what docker changed but the frontend loadbalancer can't see any of the backend services right now for whatever reason.
standby, we're working on it
yea, everything is fine now.
post mortem: I think what happened, is that after the reboot, docker-gen ran before things settled, working with stale information somehow and creating a bad config with addresses nginx couldn't reach, and then got confused somehow.
.... after repeatedly restarting the frontend loadbalancer it now suddenly connects to the containers again.
looks like things are working? but I'm keeping an eye on it
still investigating. it's very strange, the backend network is there, all the containers are in it, yet none of the services appear to be reachable from the frontend loadbalancer?
istg if fundamental tools could stop radically changing every update that'd be really swell