Service Outage Announcement: 7 Oct 2024

Great, thanks for you quick answer.

Is the MurenaRecovery folder going to contain the same backup (some) users were emailed? The one I was emailed was from March and was missing data, Iā€™m hoping this recovery is at least more recent and has less data missing.

Hoera, Hoera, Hoera, sync works again!

2 Likes

The recovery folder will have the same files as sent in the link.

Hi Manoj,

Ok, thank you. I can conform that works.

Since file access to Murena was reopened recently Iā€™m running into a continued Nextcloud error for each file attempting to sync. Each file on my phone shows a progress bar as if the file is being sent up to the cloud, but at the last moment I get a message back that says

Upload failed You are not permitted to upload to this directory

This all started with the restart of Murena file sync. Is there something I need to do reset?

Thanks,

Q!

Regain your privacy! Adopt /e/OS the deGoogled mobile OS and online servicesphone

Hi Manoj

Is there any ETA until when you guys expect sync to be running for all accounts?

As others have reported Iā€™m neither able to see the MurenaRecovery folder in my account - but I donā€™t care too much about it as I expect the backup to be way too old anyway. Thatā€™s why I created a full backup of my Nextcloud folder.

However, not being able to sync blocks me from proceeding.

I checked with the team on the MurenaRecovery folder. As yet it has not been enabled for users and is not visible. This will be done once the process of opening up the access is complete for all users. We expect this to take at least 1-2 weeks.
We are going about the restoration process step by step. This is to avoid complications that might arise from enabling all services and third party applications at one shot. As some applications tend to delete or update older files and can cause issues with data already on the drive.
Regarding this fact not being mentioned in the guide I have discussed with the team, shared the concerns of the users. The team will update the guide ASAP to reflect this.
Applications like Notes are still not enabled. The team is working on this. Will update on the progress. As mentioned we will enable or restart application and services one step at a time.

6 Likes

Notes app should start appearing for users who can see the files icon in the dashboard. Sync will not work for now, will update.

1 Like

Can confirm that Notes now appears within my dashboard, with syncing not working from phone -

Hi,
Now I can access ā€˜filesā€™ via web browser at least some directories and see content of NextCloud.
But when I try connect via nextcloud after I enter credentials in Web browser connect is refused and access does not work. On NextCloud PC I get ā€œresponse Access from NextCloud clients is blockedā€
Is this new problem or nextcloud is still not resolved ?
Should we expect mail (per user) when system (for that user) should be fully functional ?
Thanks!

Hi @LucasS
This is normal for the moment; see above, Manoj wrote ā€œSync will not work for now, will updateā€. Also, see the FAQ section here: Backup my files after the Murena Workspace outage

I think weā€™re nearly there, weā€™ll just have to keep coming back until it suddenly works. For some reason this company has decided to keep all announcements as vague as possibleā€¦

1 Like

I know for sync but I can not even login via nextcloud.
But I can via web. But not if nextcloud calls web for login.Therefore I can not even backup files first as described on link. Server is not in maintenance mode anymore. How do know whether it should work now? Or it is just me with problem? Anyhow, thanks for try helping.

Can try logout and login again via nextcloud? I had smilar issue in nextcloud than I logout and now I canā€™t log in via nextcloud agin. So I wonder if we have same problems, and you are not really logged in and thats why you get fault you see?

As i described in a previous post my backup was already 15 weeks old before the incident happened and as you now say that the recovery folder has the same contents as the archive link sent by email: Is there any successful attempt to restore data from the cluster to get fresher states of the user files? Or are the 15 weeks of data changes definitly lost? And why was there no backup made in between?

We have linked a guide on your dashboard which mentions this. Third party apps are not connected as they may delete your existing files. Please check the guide for details.

In this guide on andriod you ask to click on nextcloud in Files app. Unless I am logged into murena.io via nextcloud there is no Nextcould in Files?

We are working on retrieving as much data as possible from the time of the link to the date of the outage.

well that sounds very diplomatic but technacilly that could also mean ā€œall data is lostā€.

I would like to repeat the more important question: What is so special about that 15 weeks old backup prior to the outage. Why is it 15 weeks old, why was there no fresher backup made? Was that really a backup by definition?

And as there is still no post mortem, what was the decision making process behind this 4 month outtage? Could have been the services operational again with other measures? What if you have decided to cut the losses and nuke and pave the cluster instead of trying to recover data for 4 months with a really unsatisfying outcome (still no running service and backup so old they are almost useless)?

I have the feeling, the Murena leadership is not honest and is more damaging the projects reputation that the outage itself.

I mean there are plenty of fruitless announcements of what was attempted to to, but there is no explanation why and how things happened and who is responsible. As long as that is not made public, this outage will happen again in some form or another. The next thing would be a data breach, which Murena has a slight track record of too (users could access other peoples data). That means loosing trust. Now you have to justify for everything as assuming you are incompetent is just a logical assumtion based on the outage.

Sorry, with ā€œyouā€ I mean the Murena project administration and not you personally.
Iā€™m aware my posts sound very aggressive, my apologies for that. My frustration is speaking there.

In the very first update Gaƫl put out the reason behind the outage. There was a failure in one of the storage disks. This was because we had infra which was old and scheduled to be upgraded.
Unfortunately the outage hit us and damaged a few of the disks.

There is no way we can predict if there has been any data loss in that sector. There may be or there may be not a loss of data. Unless the team fully gets that data back we will not be able to get a correct picture. As you may be aware, we depend on the contributions from users for our infra. Needless to say we are not flush with funds. On the other hand we do have a huge number of users who avail our services free of cost. Some of the paying users have closed their accounts because of this outage. We do not blame them.

If anyone wants to cast aspersions on the management there is nothing that I or anyone can do. /e/OS is an attempt to get rid of the tyranny of big tech. We continue in our humble way of working towards that goal. We have upgraded our systems now. Data and services are being restored one by one. The team has with the resources available to it worked day and night to reach this status. We will continue with our efforts.

20 Likes