"Your data may be corrupt": /e/os booting into recovery and crashing on startup after running out of disk space

/e/OS Devices Fairphone
1

I am on a Fairphone 3 and run /e/os 1.9.

Background

I recently started to run out of disk space each time one of my apps (Signal messenger) started to create a new backup, as my free space was not large enough to accommodate a new backup file. Yesterday I turned off the phone when I saw that the “little storage left” notification came up and the backup was running, and put a brand new micro SD card in the phone that I had meant to set up since a while. After powering up the phone again, the notification for low storage was (non-surprisingly) still showing, alongside with one for the new SD card. When I opened the SD card I saw that a lot of empty folders had been creating, mirroring the ones in my internal storage. The phone also started to vibrate repeatedly and showed “No log (file) could be created” in a small popup on the bottom of the screen. I decided to ignore that and try to move some files from the internal storage to the SD card to free up some space. After a short while, the phone crashed and rebooted into recovery.

Problem

When I power up the phone, it boots into recovery, showing this screen:

Recovery 
Version 1.9 () 
Active slot: a

Can't load Android system. Your data 
may be corrupt. If you continue to get 
this message, you may need to perform 
a factory data reset and erase all 
user data stored on this device.

> Try again
Factory data reset

I can also navigate back into a recovery menu. When I select try again, the phone starts up as normal. I get then to unlock my SIM. However, as soon as try to unlock the normal lock screen for the first time, the phone will crash.

If I just try to unlock the screen, an empty home screen will show for a moment, before my normal home screen loads (I use lawnchair as a launcher, which is not /e/os default). Then, sometimes a popup comes saying ??? launcher crashes all the time, partially obscured by an other popup saying Power off. The phone will then reboot into the recovery.

If I try to directly go to the settings page by swiping down and clicking the settings shortcut, that page is shown for a second or two before the phone also powers off and reboots into recovery.

If I click on the “Out of storage” notification, the same thing happens.

I would really like to prevent losing my data.

What I have tried so far

It appears like I can’t do anything after unlocking my phone, as it will reboot directly.

I have gone through the recovery and fastboot menus, trying to find anything that could be of help. I was able to activate ADB and browse around in the file system using adb shell, but none of my personal files where accessible and there was apparently a lot of space available on that partition. I was also able to mount system from an option in the recovery. Even this partition showed only 73% disk usage. It was also read-only, so I was not able to remove any files. The sdcard folder in both / and /mnt/system/ was empty.

The recovery also gives me the option to install an update, e.g. via adb sideload. I have not tried that, since I don’t know how to do so and if it will give me access to my personal data.

What to do now?

If this would be a computer, I would use a live linux system to run fsck and similar tools to identify and repair the corrupted data and to free up some disk space. Is it possible to do something similar with my phone, for example using adb?

Is there any way to access/recover and possibly back up my personal data?

2

So, you have ADB set up on the computer side, it seems.
If you connected to your phone before via ADB when /e/OS was booted, and if you granted this access to the computer then when prompted on the phone, and if you chose to remember this decision (the prompt includes this option) … then you should be able to access the phone via ADB before unlocking the SIM. And with this, you would get access to your data.

There is no further update currently. In case you’re on plain 1.9, the available 1.9.1 would be the same for all we know.

For trying to install the same version you would still have to know whether you are running a dev or a stable build to be able to use the correct install file.
Furthermore, install files for use with ADB sideload aren’t publicly listed on the FP3 download pages, you would have to get the corresponding download links out of here (dev) or here (stable).

3

Thank you for answer!

Yes, I have ADB.
Do I understand that this relies on me having done this steps prior to the problem appearing? Because now I cannot get to the menu to activate USB debugging before the phone reboots.

Just a minor detail: I can unlock the SIM fine. But as soon as I unlock the screen for the first time, it reboots.

I used the easy installer, from what I understood this is the stable branch. After that I have applied a couple of OTA updates, but I assume they don’t switch branch.

Does installing the same version with ADB sideload have a change to let me recover the data, or is that impossible through this way?

From a tip in a different forum, I have been trying to enable usb-debugging so I can issue an adb shell stop command, possibly stopping the phone from rebooting. But since the phone falls into the reboot loop so quickly I have not been able to do so. Do you have any ideas on how to either activate usb-debugging or stopping the phone from rebooting?

4

Yes.

Correct.

Correct.

Installing an OS update via the OTA updater or via ADB Sideload does not affect the data partition holding the user data at all.
User data will be deleted by using the Easy Installer, by issuing command line commands to wipe the data partition (aka userdata), by unlocking the bootloader (forcing a factory reset for security reasons) or by locking the bootloader (forcing a factory reset for security reasons).

Did you try to boot into safe mode? The shutdown menu should be available before unlocking the SIM card …

1 Like
5

I’m having the exact same issue. Fairphone 3 with /e/ OS 1.12.3.
Not exactly the same background, but same warning on lack of internal storage space. And same usage of Signal with daily backups on the phone.

Exactly like @santa-klaus , when I boot the device, I can type the SIM card PIN, then unlock the lock screen. But, right after that, it quickly displays that the launcher as crashed (it’s the standard launcher in my case: BlissLauncher, I did not change it), then restarts in recovery mode, that says:

Version 1.12
Active Slot: a
Can't load Android system. Your data my be corrupt. If you continue to get this message, you may need to perform a factory data reset and erase all user data stored on this device.
Try again / Factory data reset

If I tap “try again”, it reboots and ends up in the same state.

adb was not enabled on my FP3, so I can’t use it. To me, it’s generally recommended to disable it for security reasons (when you don’t need it). So I only enable it when needed (but can’t do that now as I can’t reach the setting)
The bootloader is locked (/e/ OS had been installed with EasyInstaller)

I’ve tried the safe mode (good idea), but it unfortunately does not help. It’s the same behavior: the launcher crashes and the phone reboots.

@santa-klaus did you manage to recover your data somehow?

6

I’ve also tried to access the settings app directly (from the “quick settings” at the top), or by asking to edit the emergency information.
It starts the settings app, but very quickly the launcher crashes and the phone reboots. It does not let me scroll in the settings at all

I suspect that the lack of internal storage space is the root cause of the launcher crash.
But I don’t see how to remove anything to recover some space. And I don’t find a workaround for now.

I’ve seen a similar report in Fairphone shuts down after entering pin and goes into recovery mode but in their case, adb was enabled.

@AnotherElk : any other ideas?

NB: if confirmed, this looks like a serious bug to me. Should we report it on gitlab?

7

I participated in the linked thread - you can free up space from the /e/ recovery (even on a stable, locked install) - granted, it is not straightforward and only comes easy for regular cli users.

2 Likes
8

Ah, sorry to hear that you ran into the same problem. I was eventually able to recover my data. I also ended up in the same situation again later (yes, that’s on me), and managed to recover again. I think draining the battery helped, but it did not automatically cause the reboots to stop. I just basically tried over and over to activate adb in the settings to be able to back up the data, and at some point the boot loop stopped. This happened in both cases.

By the way, I haven’t had the problem again when running out of space lately, neither after updating to 1.12.3 nor before. Bit that might be coincidence/luck.

2 Likes
9

Many thanks for your answers @tcecyk and @santa-klaus

@tcecyk : I’m familiar with the CLI, and would probably manage to do what is mentioned in your thread if I had an adb shell. But I don’t see how to use adb from /e/ recovery:

When in /e/ recovery, if I run adb devices, it says the device is unauthorized:

$ adb devices
List of devices attached
A209PY0V0202	unauthorized

I did not dare to tap “Factory data reset”: is there a menu behind it that allows to enable adb?

@santa-klaus : how did you manage to even reach the setting page to enable adb on your device? In my case, the launcher crashes so quickly that I can not reach it.

10

it’s behind “Advanced”. You’ll need to hop to another thread in the linked thread to get to the cli discoveries. I only gave some tips, users eventually oriented themselves. I assume the initial poster and your issue are due to space. If @santa-klaus says repeated reboots helped - maybe this is a /tmp cleanup that android does, removing a formerly downloaded .img? I speculate. Good luck

2 Likes
11

Solved!

I’m very grateful to both of you for your help. :pray:

The “Advanced” menu was not obvious to find in the recovery because, when the error message (of the recovery) is displayed, it only gives the options “Try again” or “Factory data reset”.
BUT there is an arrow on top left on the screen. That’s what I was missing: If you tap on it, then you have the advanced menu, where you can enable adb.

Then I used the command-lines in thread How to mount encrypted /data partition in recovery mode? to find the biggest files. I guessed they were the Signal backup files (and I was right). So I just removed 2 of them, and the phone could boot properly.

Such a relief!

4 Likes
12

I am glad to hear that you were able to solve the problem! The path you chose sounds more reliable than just trying to get through the boot loop 30 times in a row until you get lucky, which is what I did.

To answer your prior question - in case anybody else is reading this later - I got to the option to activate adb by brute force, i.e. just trying it over and over again. I got the impression that low battery and battery saver slowed the phone down and let me get further before it crashed, airplane mode might also be worth trying. But when I finally got through I actually didn’t need to hurry, the boot loop just got stuck and I was able to use my phone normally again, which allowed me to delete the incomplete backup file and free up some space.

13

I think we should report that issue somehow to Murena.
Internal storage being filled should not make the Launcher crash.

In which gitlab project do you think it should be reported? https://gitlab.e.foundation/e/backlog ?

Edit: just found https://gitlab.e.foundation/e/backlog/-/issues/6769 that seems to be our issue

14

It is accepted, I think, that internal storage needs to be allowed certain headroom. Some devices may be more sensitive than others. I have seen it said that 85% is a sensible top limit to be seen in Settings > storage before the user thinks of taking action. Furthermore the native “device is getting full” warning is reported to allow little time to take action before a significant breakdown.

I would be interested to know what sort of numbers you were each seeing in this particular measure when these events happened.

15

It seems to me that there should be a safeguard. A bit like, on Linux, ext* filesystems reserve (by default) 5% of disk space to root user.
It sounds fair that some things might not work if internal storage is full.
However, it should not prevent the system to boot (the launcher should not crash in this scenario), to (at least) let the user choose which files to remove.

I unfortunately did not check the disk space of partitions of my phone before removing big files. Sorry, I can’t give you numbers. I was too excited and in a hurry to fix the phone…

16

Hello, I just want to report that I had the same or very similar problem with e.OS on Fairphone 3 today. I use Signal with automatic nightly backup. The low disk space warning started showing just a few days ago and I did not take it very seriously. I noticed that the Signal backup failed yesterday, but the phone kept working. I was planning to do a clean up soon. Today, the phone crashed a few seconds after the Signal backup started, and it booted into recovery. When I tried I normal boot, I could enter SIM pin and e.OS pin, but just after entering e.OS pin the phone shutted down and rebooted like described by others.

Thanks to your tips, I was able to recover:

  • Press back button in Recovery and enable adb
  • Download Android SDK Platform Tools ZIP-file to my Windows PC, extract and
  • run adb shell
  • In the shell, mount the encrypted userdata and search for big files:

mkdir -p /mnt/userdata
mount /dev/block/bootdevice/by-name/userdata /mnt/userdata/
cd /mnt/userdata
find . -type f -size +1000M

  • I found three 4GB files. From the time stamp and size I guessed it must be the Signal backup files

cd /mnt/userdata/media/0/6mYlOAlMZSpGmEFPQ+NQSB/

FP3:/mnt/userdata/media/0/6mYlOAlMZSpGmEFPQ+NQSB # ls -l
#total 11819828
-rw-rw-r-- 1 media_rw media_rw 4112204390 2024-05-08 01:11 _+PUxYaUpHxYqeDg7GQFEJPqOtSVPZE5c
-rw-rw-r-- 1 media_rw media_rw 4119633920 2024-05-11 23:44 _6dlwmfhlImbpUs6i69EBi7zx3wl10ptb
-rw-rw-r-- 1 media_rw media_rw 4113810646 2024-05-09 00:58 _op3ODN,RCuUsWstppIIfJbKLfh6jo8J,

  • Now delete a file and hope it is really just a redundant backup and nothing very valuable

rm _6dlwmfhlImbpUs6i69EBi7zx3wl10ptb

  • Unmount

cd /
umount /mnt/userdata
exit

  • Reboot phone. Now it does not crash anymore.

Whew, I think this is a bit too exciting for a ordinary case of device storage running full. I guess there should be some more protection against this…

2 Likes
17

Me too. The same problem. I was trying to send a video to a friend on WhatsApp when there was little disk space left. And on the last moment it gone to recovery and reboot.

ADB SHELL gives error, so I have not door inside my phone.

C:\Users\MyPC>adb devices
List of devices attached
1c92lsghl54ece sideload

C:\Users\NMyPC>adb shell
error: closed

18

@nilsl and @E.FondGalaxyNote10Pl : which version of /e/OS was on your device when you had the issue?
They say on https://gitlab.e.foundation/e/backlog/-/issues/6769 that it has been fixed, at least 3 months ago.

If your version was newer than that, you should comment the gitlab ticket, and maybe create a new one

1 Like
19

adb devices is reporting your recovery as being in sideload mode, in which you can’t do adb shell (only adb sideload to install a ZIP file).

When in the /e/OS recovery, you need to do Advanced - Enable ADB … then adb devices should report it as recovery, and you should be able to continue with adb shell.

3 Likes
20

Hi! Thanks for answer.
I seen his post about it.
I was on 1.18. And now I have last one 2.0-dev…
So I installed the latest firmware 2.0. Now the phone does not go into recovery immidiatly like before. Instead, it repeatedly requires you to enter the unlock code. After several attempts to start the system, it flies into recovery with the same entry as before.

I tried to install the update on adb sdk manager (from here without succes https://developer.android.com/tools/sdkmanager) to run the command through adb, but to no avail.

When entering adb shell an error appears.

I read that I need to update the SDK manager, but how? It’s not written anywhere.

I installed Android Studio. I made update of this programm and all tools inside with a hope that inside this programm will be as a part of it last sdk manager. But no. Nothing.

There is no explanation on the website either.
BUT

I find this excellent explication for Windows and other OS
https://technastic.com/android-sdk-platform-tools-download/#how-to-set-up-android-sdk-platform-tools

Just choose your OS, install Latest ADB Platform-Tools and follow Setup link