Announcements and status/progress updates about upgrades will be posted here.
This website runs on a dedicated host and will be up even if everything else is down.
fucking fucking gitlab fuck motherfucking fuck ugh fuck everything is broken fuck .. standby, fixing is in progress, but various containers are in a broken state and need manual unfucking
everything except for gitlab is back up. if you run into problems let me know.
alright, base framework is back up, website is back up and a bunch of services like nextcloud, seafile, matrix.. currently trying to upgrade mastodon since 4.5.2 is out and I might as well take care of that right away
and I guess I'll keep gitlab offline for now until I decide what to do with it
This is planned to start at 3PM, 40 minutes from now.
If you have any questions, feel free to reach out to Anthropy
I believe everything is done now. Let me know if you run into any problems!
update is done, mastodon should be coming back up
database backup still in progress.
Everything is back up, except for Mastodon because I realized 4.5.1 got released a few days ago, so I'm taking care of that right away. This will take an additional 10-20 minutes though.
It's back up! Now to start the services again..
upgrade complete, rebooting again, this will take a few minutes
main upgrade is running, so far so good.
alright, took everything down, ran a first set of updates to get it to the latest packages for the current version (bookworm), and now rebooting, so that I can continue with the actual distro upgrade next
Waiting for Nextcloud to finish its cron run, after that I'll start shutting down the services
We're updating Mastodon and other services. I don't expect trouble but considering Mastodon will be offline multiple times I figured I'd have a statuspage open for the progress updates. See below for the updates.
Gitlab has also been restarted, meaning this update cycle has come to an end. Closing this statuspage, feel free to reach out if you run into any issues
mastodon migrations complete, booting in progress
Migrations are still in progress, everything looking good so far. I'm glad to see the script seems to be able to deal with the major version upgrade.. but let's wait with celebrating until it's all done
And we're on Mastodon 4.4.8! seems like everything worked fine, though the upgrade process is very slow because of how busy the server is. I'm firing off the upgrade process for 4.5.
Note that, because of this being a major upgrade, I do somewhat expect some migration pains, the last few times they had changed how Mastodon is launched an I had to edit the docker config.. but we'll see whatever breaks and fix it.
mastodon update is in progress; database is being backed up by the script, then it'll update everything, restart everything and run the migrations, and it'll be back .. for a bit, because I'll do the 4.5 update right after.
hmm, still busy, I think mastodon is being scraped, which is making it heavier than normal. I'll proceed with the Mastodon updates, that should also alleviate the load, then I'll do Gitlab last
Waiting for Nextcloud to calm down so I can restart Gitlab without it throwing a fit (it will literally fail to start the container if the database doesn't come up within 30 seconds)
frontend loadbalancer is being restarted so we're briefly down, after this I'll start with gitlab
Matrix has been restarted, Nextcloud is next.. almost every other application has been updated. After Gitlab is done I'll start the Mastodon upgrade to 4.4.8, and then 4.5 afterwards
welp, not sure what docker changed but the frontend loadbalancer can't see any of the backend services right now for whatever reason.
standby, we're working on it
yea, everything is fine now.
post mortem: I think what happened, is that after the reboot, docker-gen ran before things settled, working with stale information somehow and creating a bad config with addresses nginx couldn't reach, and then got confused somehow.
.... after repeatedly restarting the frontend loadbalancer it now suddenly connects to the containers again.
looks like things are working? but I'm keeping an eye on it
still investigating. it's very strange, the backend network is there, all the containers are in it, yet none of the services appear to be reachable from the frontend loadbalancer?
istg if fundamental tools could stop radically changing every update that'd be really swell