Some systems are experiencing issues

About This Site

This status page will automatically update as long as MonitorLizard runs on the main server. It may mark services as having an outage, it may post updates about running backups, updates about running updates, or other relevant information.

This status page will be up even if the main server isn't, but if the main server is down, this page may show outdated data (fix is WIP).

Automated Mastodon Status Messages (runs on main server)

Stickied Incidents

21st March 2024

nginx frontend randomly crashing/quitting

Nginx appears to randomly.. gracefully quit and sit there doing nothing. I'm still investigating, I'll keep this incident open so that if everything is randomly offline again you'll know why.

Past Incidents

24th November 2024

Backup Process on 2024-11-24 03:00:04

Starting backup of /storage to ./mainstorage/borg on a remote server with BorgBackup. Max retries allowed: 5.

23rd November 2024

Starting prune

Prune completed successfully.

Backup Process on 2024-11-23 03:00:01

Backup completed successfully:

Archive name: 2024-11-23_03:00:02 Archive fingerprint: 60c6dac9f7af2f2ec00ab97f6bc251f8d969c80d160230803427bfced67d3c06 Time (start): Sat, 2024-11-23 03:00:10 Time (end): Sat, 2024-11-23 21:32:49 Duration: 18 hours 32 minutes 38.94 seconds Number of files: 86559950 Utilization of max. archive size: 40%

                   Original size      Compressed size    Deduplicated size

This archive: 25.43 TB 19.43 TB 9.43 GB All archives: 3.06 PB 2.30 PB 14.88 TB

                   Unique chunks         Total chunks

Chunk index: 68065499 11115703975

22nd November 2024

Starting prune

Prune completed successfully.

Backup Process on 2024-11-22 03:00:17

Backup completed successfully:

Archive name: 2024-11-22_03:00:18 Archive fingerprint: 5875bb218a9bcb2527036a1cf9a64bfa075f551a9a181bca81b36d4f8959b95e Time (start): Fri, 2024-11-22 03:00:23 Time (end): Fri, 2024-11-22 19:49:33 Duration: 16 hours 49 minutes 10.24 seconds Number of files: 86540812 Utilization of max. archive size: 40%

                   Original size      Compressed size    Deduplicated size

This archive: 25.43 TB 19.43 TB 8.71 GB All archives: 3.04 PB 2.28 PB 14.87 TB

                   Unique chunks         Total chunks

Chunk index: 67963921 11021175662

Starting prune

Prune completed successfully.

21st November 2024

Backup Process on 2024-11-21 05:43:52

Backup completed successfully:

Archive name: 2024-11-21_05:43:53 Archive fingerprint: 3c4a8fdea2da0da6443f1931735f0491502f3c5dda372c079a267a5e9c5ff82c Time (start): Thu, 2024-11-21 05:43:57 Time (end): Thu, 2024-11-21 23:56:42 Duration: 18 hours 12 minutes 45.18 seconds Number of files: 86535049 Utilization of max. archive size: 40%

                   Original size      Compressed size    Deduplicated size

This archive: 25.42 TB 19.43 TB 8.76 GB All archives: 3.01 PB 2.26 PB 14.86 TB

                   Unique chunks         Total chunks

Chunk index: 67887565 10926667645

Starting prune

Prune completed successfully.

20th November 2024

Backup Process on 2024-11-20 12:59:28

Backup completed successfully:

Archive name: 2024-11-20_12:59:29 Archive fingerprint: 131a7a29e9118c4b9490a4fac659de0e4fd866dffd4ac3830c81eb848b2aec48 Time (start): Wed, 2024-11-20 12:59:32 Time (end): Thu, 2024-11-21 05:38:33 Duration: 16 hours 39 minutes 0.86 seconds Number of files: 86529839 Utilization of max. archive size: 40%

                   Original size      Compressed size    Deduplicated size

This archive: 25.42 TB 19.42 TB 8.64 GB All archives: 2.99 PB 2.24 PB 14.85 TB

                   Unique chunks         Total chunks

Chunk index: 67803145 10832165588

Starting prune

Prune completed successfully.

19th November 2024

Backup Process on 2024-11-19 18:44:25

Backup completed successfully:

Archive name: 2024-11-19_18:44:26 Archive fingerprint: e5286131b3abe6a9ea48086ba20db03e3a67ad875092c154db0239aec4256081 Time (start): Tue, 2024-11-19 18:44:29 Time (end): Wed, 2024-11-20 12:54:05 Duration: 18 hours 9 minutes 35.47 seconds Number of files: 86525801 Utilization of max. archive size: 40%

                   Original size      Compressed size    Deduplicated size

This archive: 25.42 TB 19.42 TB 16.28 GB All archives: 2.96 PB 2.22 PB 14.84 TB

                   Unique chunks         Total chunks

Chunk index: 67727829 10737669472

Starting prune

Prune completed successfully.

17th November 2024

Backup Process on 2024-11-17 03:00:17

Backup completed successfully:

Archive name: 2024-11-17_03:00:17 Archive fingerprint: b98c3703aed1e1b35aee5702d1986cf9f47a9f1a26dd0a9e068e83c9e9ac5a5c Time (start): Sun, 2024-11-17 03:00:21 Time (end): Tue, 2024-11-19 18:38:49 Duration: 2 days 15 hours 38 minutes 28.26 seconds Number of files: 86512014 Utilization of max. archive size: 40%

                   Original size      Compressed size    Deduplicated size

This archive: 25.41 TB 19.41 TB 12.76 GB All archives: 2.94 PB 2.20 PB 14.83 TB

                   Unique chunks         Total chunks

Chunk index: 67579202 10643178533

Starting prune

Prune completed successfully.

16th November 2024

Backup Process on 2024-11-16 05:51:33

Backup completed successfully:

Archive name: 2024-11-16_05:51:33 Archive fingerprint: 2a51b3ba75e1164ee7298c968fc24e8bd5dee6743fb9a12a19184c2e7a106ec9 Time (start): Sat, 2024-11-16 05:51:37 Time (end): Sun, 2024-11-17 00:08:29 Duration: 18 hours 16 minutes 52.50 seconds Number of files: 86499728 Utilization of max. archive size: 40%

                   Original size      Compressed size    Deduplicated size

This archive: 25.41 TB 19.41 TB 12.47 GB All archives: 2.91 PB 2.18 PB 14.81 TB

                   Unique chunks         Total chunks

Chunk index: 67423765 10548703487

Starting prune

Prune completed successfully.

15th November 2024

Backup Process on 2024-11-15 13:51:29

Backup completed successfully:

Archive name: 2024-11-15_13:51:29 Archive fingerprint: 0ac4f715243665fd6ebb62022f4f30e2e0168d47e08470cc3c8014a99c27802b Time (start): Fri, 2024-11-15 13:51:32 Time (end): Sat, 2024-11-16 05:46:27 Duration: 15 hours 54 minutes 55.17 seconds Number of files: 86452503 Utilization of max. archive size: 40%

                   Original size      Compressed size    Deduplicated size

This archive: 25.40 TB 19.40 TB 10.75 GB All archives: 2.89 PB 2.16 PB 14.80 TB

                   Unique chunks         Total chunks

Chunk index: 67383872 10454241367

Starting prune

Prune completed successfully.

14th November 2024

Backup Process on 2024-11-14 19:00:22

Backup completed successfully:

Archive name: 2024-11-14_19:00:24 Archive fingerprint: ba33cd16595f009b51e5df28790b889a1daa48874cecc9b2f5b9e4e585f8b6c4 Time (start): Thu, 2024-11-14 19:00:32 Time (end): Fri, 2024-11-15 13:46:35 Duration: 18 hours 46 minutes 3.03 seconds Number of files: 86438255 Utilization of max. archive size: 40%

                   Original size      Compressed size    Deduplicated size

This archive: 25.40 TB 19.40 TB 29.22 GB All archives: 2.86 PB 2.14 PB 14.79 TB

                   Unique chunks         Total chunks

Chunk index: 67227163 10359828364

10th November 2024

Maintenance, updates, and other things

A few maintenance tasks need to be completed, in addition to some work on Monitorlizard.

This will cause intermittent downtime, but this status page here will stay up, so feel free to keep this open and refresh this to get live updates.

  • Offsite backup done, Monitoringlizard has been reinitiated, normal daily backups will continue starting 3AM

  • After having updated the remote backup server, and having migrated BCacheFS to the latest version, the offsite backup is now running. Mild latency is to be expected; as with any backup

  • server has rebooted and services are slowly starting up. After that's done I'll initiate the offsite backup and make some changes to Monitorlizard

  • everything appears to be done, rebooting the server now

  • Containers have all been updated, but Nextcloud is still running its upgrade process. After that's done I'll reboot probably

  • I've canceled the running backups, I'm now running a full update of all the containers, and will reboot the server once that's done. During the updates, mastodon and other services will be briefly down.

  • Backup Process on 2024-11-10 03:00:03

    Starting backup of /storage to ./mainstorage/borg on a remote server with BorgBackup. Max retries allowed: 5.

    9th November 2024

    Starting prune

    Prune completed successfully.

    Backup Process on 2024-11-09 03:00:19

    Backup completed successfully:

    Archive name: 2024-11-09_03:00:20 Archive fingerprint: f28d74907a9f65e0b88a975bafd9999c0c9148bcde15ae8d1f48e1f1b0f72754 Time (start): Sat, 2024-11-09 03:00:24 Time (end): Sat, 2024-11-09 22:24:12 Duration: 19 hours 23 minutes 47.72 seconds Number of files: 86244392 Utilization of max. archive size: 40%

                       Original size      Compressed size    Deduplicated size
    

    This archive: 25.38 TB 19.38 TB 7.59 GB All archives: 2.83 PB 2.12 PB 14.76 TB

                       Unique chunks         Total chunks
    

    Chunk index: 66980767 10259719688

    8th November 2024

    Starting prune

    Prune completed successfully.

    Backup Process on 2024-11-08 03:00:08

    Backup completed successfully:

    Archive name: 2024-11-08_03:00:08 Archive fingerprint: 74981e194f39d2e61f971161bfd249ff761721f78c902db86b7c4d9febe6a92a Time (start): Fri, 2024-11-08 03:00:15 Time (end): Fri, 2024-11-08 20:01:27 Duration: 17 hours 1 minutes 12.17 seconds Number of files: 86232973 Utilization of max. archive size: 40%

                       Original size      Compressed size    Deduplicated size
    

    This archive: 25.38 TB 19.38 TB 11.23 GB All archives: 2.81 PB 2.10 PB 14.75 TB

                       Unique chunks         Total chunks
    

    Chunk index: 66959251 10165518294

    Starting prune

    Prune completed successfully.

    7th November 2024

    Backup Process on 2024-11-07 03:00:15

    Backup completed successfully:

    Archive name: 2024-11-07_05:20:33 Archive fingerprint: 677115d41ef441c4f4c4722aaf7c763ef10fdf4cfb79670f98b2433348aefa8b Time (start): Thu, 2024-11-07 05:20:37 Time (end): Fri, 2024-11-08 00:35:05 Duration: 19 hours 14 minutes 27.55 seconds Number of files: 86225668 Utilization of max. archive size: 40%

                       Original size      Compressed size    Deduplicated size
    

    This archive: 25.38 TB 19.38 TB 12.59 GB All archives: 2.78 PB 2.08 PB 14.74 TB

                       Unique chunks         Total chunks
    

    Chunk index: 66820145 10071329456

    6th November 2024

    Starting prune

    Prune failed.

    Backup Process on 2024-11-06 10:39:10

    Backup failed after 5 attempts due to persistent issues.

    Starting prune

    Prune failed.

    5th November 2024

    Backup Process on 2024-11-05 19:02:16

    Backup failed after 5 attempts due to persistent issues.

    Starting prune

    Prune completed successfully.

    3rd November 2024

    Backup Process on 2024-11-03 03:00:02

    Backup completed successfully:

    Archive name: 2024-11-03_03:00:03 Archive fingerprint: c7211ae9b655b9e4b781664aa559aaa9686725313cb4b2c47691d16c3e37608b Time (start): Sun, 2024-11-03 03:00:07 Time (end): Tue, 2024-11-05 18:56:33 Duration: 2 days 15 hours 56 minutes 26.13 seconds Number of files: 86124736 Utilization of max. archive size: 40%

                       Original size      Compressed size    Deduplicated size
    

    This archive: 25.36 TB 19.36 TB 8.00 GB All archives: 2.75 PB 2.05 PB 14.71 TB

                       Unique chunks         Total chunks
    

    Chunk index: 66527956 9950872015

    Starting prune

    Prune completed successfully.

    2nd November 2024

    Backup Process on 2024-11-02 06:47:32

    Backup completed successfully:

    Archive name: 2024-11-02_06:47:33 Archive fingerprint: af4e9c6eed452567cf2c871aff1fce285b586a1528641a412114e0eaf0a96be0 Time (start): Sat, 2024-11-02 06:47:36 Time (end): Sun, 2024-11-03 01:52:30 Duration: 19 hours 4 minutes 54.19 seconds Number of files: 86118090 Utilization of max. archive size: 40%

                       Original size      Compressed size    Deduplicated size
    

    This archive: 25.36 TB 19.36 TB 11.63 GB All archives: 2.72 PB 2.03 PB 14.70 TB

                       Unique chunks         Total chunks
    

    Chunk index: 66477211 9856795869

    Starting prune

    Prune completed successfully.

    1st November 2024

    Backup Process on 2024-11-01 00:52:40

    Backup completed successfully:

    Archive name: 2024-11-01_00:52:41 Archive fingerprint: 549d62a7bb7b91a23f93a98c18f44c46768ad38403b7866c2e8f043632a8dbb1 Time (start): Fri, 2024-11-01 00:52:47 Time (end): Sat, 2024-11-02 06:42:05 Duration: 1 days 5 hours 49 minutes 18.41 seconds Number of files: 86097122 Utilization of max. archive size: 40%

                       Original size      Compressed size    Deduplicated size
    

    This archive: 25.35 TB 19.36 TB 628.59 GB All archives: 2.69 PB 2.01 PB 14.69 TB

                       Unique chunks         Total chunks
    

    Chunk index: 66314397 9762727375

    23rd October 2024

    Updating Mastodon

    Mastodon 4.3.1 has been released 2 days ago. We will be updating the server momentarily; it's only a minor upgrade and we don't expect it to take long.

  • update done (was already done)

  • 20th October 2024

    a raid scrub is running

    if you experience 50x errors in mastodon or slowness on gitlab, beware that for the next ~12 hours a scrub is running which will affect performance. Mastodon doesn't seem to like the slow I/O and is throwing sporadic errors, please try again a few times if you receive errors.

    8th October 2024

    Mastodon 4.3 released -- Updating at 10PM GMT+1

    Considering Mastodon 4.3 has been officially released, we will be updating tonight at 10PM GMT+1

    Expected downtime is about 20 minutes.. plus up to a (few) hour(s) if it breaks, but we'll see, I have faith in The Scripts™️

    More info can be found here:

    • https://mastodon.social/@Mastodon/113272025176822566
    • https://blog.joinmastodon.org/2024/10/mastodon-4.3/
  • mastodon: up me: wait but streaming container: still dead mastodon: everything including streaming works me: I- .. I'm just going to go to bed for now and investigate what the bitch docker is fucking about now

  • finished building, but still streaming won't start because it can't find the files it needs, what the

  • waiting for something to finish building past midnight because you have to run some other things after it and you can't go to bed yet because your little server thingy will be down is such a mood

  • think I got the fix for the streaming container, currently rebuilding it and restarting everything again after.

    edit: for those curious, I had to change the build parameters in the docker-compose.yml, if you run into the same issue, make sure to check that part

  • alright now the backend is running smoothly again, but the Streaming container is still Severely Upset about life

  • the migrations went pretty quick, going to give starting it a try again

  • database restore in postgres 17 complete, that version upgrade should hopefully last a while. now to figure out and run the database migrations

  • database dumping in progress. I should probably also run the appropriate database migrations after before bringing it up again, now that I remembered that I also have this page https://derg.nz/dragonhive/web/sysadmin/kb/blog/2023/10/02/how-to-fix-mastodon.html

  • ugh fuck postgres needs to be updated because the one currently running is not sufficient for the latest mastodon

    I'm so happy I wrote a guide for myself in 2022 https://derg.nz/kb/sysadmin/2022/10/15/upgrading-postgres-the-easy-way.html

  • almost there.. but something is wrong with the streaming servers

  • the good news: the upgrade succeeded! the bad news: even more random changes that I couldn't find documentation about but is now yelling at me through the logs. We're getting closer though

  • updates compiling smoothly, expecting the reboot to automatically happen in about 5 minutes

  • Running the updates again now, the scripts have been updated to accommodate major version changes, let's see how it goes

  • so as mentioned on @dhstatus mastodon, it didn't work, for some reason even though the configs are pointing to a newer image, it still seems to show the older one. I'll look further into this tomorrow.

  • installation going smoothly thusfar, at this rate it'll be back up in 5-10 minutes .. but knocks on nearest wooden object

  • Starting the mastodon update now, expected downtime 20 minutes, but we'll update as it goes.

    Starting the update scripts now

  • Related Mastodon account