Service Outage Announcement: 7 Oct 2024

Manoj · December 20, 2024, 10:30pm

Gaël’s update on the outage dated 20 Dec:

Finally some better news: after several failed new attempts to access files contents, we finally succeeded and made some first copy tests which have been successful (so far) for users’ data. The bad news is that there is still a filesystem inconsistency that prevents us to fully stabilize the cluster to make it fully up and running again. The external consultants are trying to understand why and have contacted the CephFS developers for advice. First feedback is that there could be a software bug that results into an unexpected filesystem state. This explaination still needs to be confirmed but it would explain why we failed to reenable the storage cluster for more than 2 months by using the standard recovery procedures. What’s next: we’re conducting more extraction tests (copying and opening files) to validate that we can now fully access all data. Once those test are completed, the current plan is to copy all the data to a new and clean filesystem and replug to murena.io .

Meanwhile, we have made significant progress with the backup. We’re still struggling with 1/3 of the backups, but for the other 2/3, we have put in place a service that will offer to users to download their own archive of data. We intended to launch this service early this week but we had to deal with a few issues, which postponed the opening of this process to today. Some users have started to receive an email with a link and a procedure to download their own archive. The process is still slow because we want to ensure that it fully works as there can be some edge cases downloading and decrypting huge archives. Then we will accelerate the rate of sending during the next days. Note: individual archives will be protected by a password that is available in murena.io password application for each user (i.e. NOT the user’s password).