Some systems are experiencing issues

About This Site

This status page will automatically update as long as MonitorLizard runs on the main server. It may mark services as having an outage, it may post updates about running backups, updates about running updates, or other relevant information.

This status page will be up even if the main server isn't, but if the main server is down, this page may show outdated data (fix is WIP).

Automated Mastodon Status Messages (runs on main server)

Past Incidents

28th April 2025

mastodon down due to zfs/docker issue

apparently docker gets severely upset when you try to restart containers while a ZFS send/receive is running ..

WELL FINE I GUESS I'LL STOP AND RESTART THE ENTIRE BACKUP THAT'S BEEN RUNNING FOR DAYS THEN? fucking docker I swear to god

  • Mastodon is back up, backup has been restarted from scratch .. sigh

    • 20th April 2025

    updating mastodon to 4.3.7

    I'm going to update Mastodon to the latest version, it may be briefly down.

  • everything done and dusted.

    I think this should be the last downtime for the coming while. knocks on wood

  • starting the update now

    • 19th April 2025

    switching over to new docker config

    This is still in progress but taking a little longer than expected, hence this tracking incident.

    Everything appears to work so far but it needs time to download and start all the different images from scratch again.

    only when the frontend loadbalancer is up again we'll know if it's working as intended.

  • everything should be up again, let me know if there are still any issues

  • matrix is up and running, mastodon is starting up next

    • 18th April 2025

    I accidentally mastodon while trying to fix a broken config

    sorry my bad I'm working on it.

    why this broke it? no idea. Then again also not sure why it was working with the config defined in a place it definitely didn't belong.

    I'm not even sure what's broken right now, it should just work, why are computers like this oml

  • ah yes, in dockerland strings shouldn't be defined with quotes, because then the string will contain the quotes.

    Oh well, at least the streaming errors are gone now! everything should operate as normal now

    • Gitlab is upset about the new server

    Unsure what's going on. When moving to the new server it didn't want to run its config file anymore, and when fixing that it refused to come back up at all. Currently investigating what's wrong.

  • Gitlab randomly decided to

    • no longer listen on port 80 (which I mean I guess 443 is fair too even if internal-only)
    • break prometheus (had to delete its data)
    • change oauth URLs to broken ones (easy to fix, also fixed the web IDE)

    ..but everything has been fixed, and it now appears to be running again. If you run into any issues let me know.

    • 17th April 2025

    Switchover to new main server

    After the transfer completed, all that remains is syncing and switching over individual services! This will start now.

    Downtime is expected while the last syncs complete. Updates on the progress will be posted here.

  • Everything is up and running again except for Gitlab. I'll open a new statuspage to track that issue

  • everything got synced and seems to be functioning! I've adjusted the config files where needed, and I'm running one automated update + restart across all services to confirm everything is working now (and update everything)

    After this everything should be back up and running as normal.

  • It's still busy with the Mastodon files.. it's about 5TB of tiny files, it's transferred a few dozen million already but I have no idea how much more it needs to do.

    ..So I'll probably check back tomorrow morning (in about 9-10 hours from now) to see how it's going, and if Mastodon or any other services have been synced, I can turn them back on.

  • Still going steadily as planned, docker folder has finished, the main services storage folder has gone through 13M files so far and synced 26G of additional small changes, can't tell how much is left but it shouldn't take forever considering the main server has nothing better to do than syncing stuff right now

  • Ah, it's not even DNS this time, but Docker that's making it hard to do partial switchovers.. I can't really do that if I plan to move over Docker's daemon data folder because it'll be inconsistent.

    So I guess I'll just.. take everything offline and do one last sync for my main services, then bring them all online at once. Sorry for the inconvenience.

  • Matrix has been taken offline and will be synced over first.

    We're going to try and allow each service to be up while it waits for its turn, but due to the DNS being all CNAMEs it might be that everything goes down at once at some point. We'll see how it goes, updates will be provided here regardless.

    • Related Mastodon account