[HOWTO] End-to-end Encrypted Full Backup and Restore Solution for /e/OS using Termux and restic

eOSWiki
, , ,
1

In the wake of the upcoming /e/OS T release, I thought it would be handy to share a new Android backup solution I recently discovered and which is still fairly unknown, I guess :smile:

About this backup solution

  • We make use of restic here, a standalone command-line backup tool which generally works on all major platforms (Linux, BSD, Mac, Windows WSL, and Android)
  • restic is free Open Source software and backups are end-to-end encrypted per default
  • it allows you to store your encrypted backups both locally or on a remote server (in my guide below I will show you how to backup to sd card, own server, and Murena Cloud / Nextcloud, but there are even more storage types)
  • backups are de-duplicated, incremental and compressed, which saves you a lot of time, space, and data usage
  • as restic is a independent tool this backup solution does NOT depend on adb, adb backup, adb root, root debugging or custom shell scripts
  • it allows you to backup all the data on your device if you have root, so you don’t have to deal with backups being incomplete as in the case of SeedVault when certain apps have set a “do not backup” flag
  • restoring data works in two ways: specifying a certain path on Android to restore to or you mount a backup snapshot on your desktop file system and just copy files out of there (there is also a open source restic file explorer software to simplify this, see Restic Browser in honorable mentions below)

Downsides of this backup solution

  • Without root you are only able to backup internal storage (of personal and work profiles) and sd card
  • We have to utilize the command line (Termux), if you are fine with that, this guide should be pretty straight foward for you :slight_smile:
  • restic consumes more RAM the bigger your backup repository gets (I guess we are talking about TBs here, my FP4 has 8 GB RAM, and worked perfectly fine so far)
  • <Your feedback is needed here>

I successfully tested this method on a FP4 with /e/OS R (Android 11) and S (Android 12).

Please note that the app data directories given in the examples here could differ depending on your specific device. The given sd card and internal storage paths should match accross most devices, I hope.

For more details on the usage of restic please refer to the restic documentation. I won’t go into topics like tags, backup rotation, key management, etc. here.

Honorable mentions

  • This guide uses restic natively via command line. If you prefer to use a GUI, there is a unofficial experimental Restic Backup Android App on F-Droid (I won’t cover this app here as it is quite limited as of now and therefore doesn’t offer the same capabilites on which we rely in this guide)
  • Restic Browser provides you with a GUI to browse and restore your restic backup repositories on your Windows/Linux/MacOS desktop machine
  • For those ppl who prefer a similar encrypted FLOSS backup solution: Instead of restic you can also use BorgBackup via Termux. It can be installed using the command: pkg install borgbackup

Prerequisites

  • Termux installed (install via F-Droid)
  • Grant Termux file access permissions in app settings (else you will run into permission denied errors when running restic)
  • For full backups: root (e.g. Magisk), Termux needs to be able to run as Superuser when running restic with sudo

1 Initial Setup: Install restic via Termux

Update packages

apt update
apt upgrade

→ If there are any prompts just hit ENTER all the time

Update all termux packages

pkg upgrade

Install sudo (ROOT ONLY)

pkg install tsu

Install restic

pkg install restic

Check if restic was installed successfully

restic version

restic 0.16.0 compiled with go1.20.6 on android/arm64

2 Backing up your data

2.1 OPTION 1: Backup to sd card

:exclamation: Replace “/storage/XXXX-XXXX/” with your sd card path :exclamation:

Create backups folder on sd card

mkdir /storage/XXXX-XXXX/backups/

Initialize restic backup repository

:exclamation: This step is only needed once at first setup :exclamation:

restic init -r /storage/XXXX-XXXX/backups/

enter password for new repository:
enter password again:
created restic repository 36fbfa9fb8 at /storage/XXXX-XXXX/backups/

Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost.

Backup internal storage (without root) (default/personal profile)

restic -r /storage/XXXX-XXXX/backups backup /storage/emulated/0/ --verbose

Backup app data (with root) (default/personal profile)

sudo restic -r /storage/XXXX-XXXX/backups backup /data/data/ /data/user/0/ /data/user_de/0/ --verbose

Show all backups

restic -r /storage/XXXX-XXXX/backups snapshots

Restore backup (latest backup)

restic -r /storage/XXXX-XXXX/backups restore --target /storage/emulated/0/my-restore-directory latest

Browse your backups / Mount backup repository to file system (for example on your desktop machine)

:exclamation: In order to use this restic needs also to be installed on your desktop machine :exclamation:

mkdir /mnt/restic
restic -r /your/sdcard/path/backups mount /mnt/restic

If you prefer a GUI for browsing your backups on your desktop machine: Use Restic Browser.

2.2 OPTION 2: Backup to Murena Cloud / your own Nextcloud via WebDAV

:exclamation: restic doesn’t support backing up to Murena Cloud / Nextcloud instances (WebDAV) out of the box, it needs rclone for that :exclamation:

Update packages

apt update
apt upgrade

→ If there are any prompts just hit ENTER all the time

Update all termux packages

pkg upgrade

Install rclone

pkg install rclone

Configure rclone

rclone config

Create new remote

2023/09/23 11:00:04 NOTICE: Config file “/data/data/com.termux/files/home/.config/rclone/rclone.conf” not found - using defaults
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q>

→ n

Specify name of your backup destination

name>

→ MurenaWebDAV / NextcloudWebDAV (or whatever name you like)

Specify type of your backup server

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.

→ Choose number corresponding to WebDAV

Specify backup-server URL

Option url.
URL of http host to connect to.
E.g. https://example.com.
Enter a value.
url>

→ Log into Murena Cloud / Nextcloud, navigate to Files → open menu → Files settings → copy WebDAV path from there (alternatively, use the URL below and replace USERNAME with your username)

→ https://murena.io/remote.php/dav/files/USERNAME/

Choose WebDAV Type

Option vendor.
Name of the WebDAV site/service/software you are using.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / Fastmail Files
\ (fastmail)
2 / Nextcloud
\ (nextcloud)
3 / Owncloud
\ (owncloud)
4 / Sharepoint Online, authenticated by Microsoft account
\ (sharepoint)
5 / Sharepoint with NTLM authentication, usually self-hosted or on-premises
\ (sharepoint-ntlm)
6 / Other site/service or software
\ (other)
vendor>

→ Choose number corresponding to Nextcloud

Specify Murena Cloud / Nextcloud username

Option user.
User name.
In case NTLM authentication is used, the username should be in the format ‘Domain\User’.
Enter a value. Press Enter to leave empty.
user>

→ Enter your Murena Cloud / Nextcloud USERNAME

Specify Murena Cloud / Nextcloud password

:exclamation: For security reasons it is recommended to use a dedicated App password :exclamation:

→ Log into Murena Cloud / Nextcloud, navigate to Settings → Security → scroll down to Devices & sessions → Fill in a app name of your liking, e.g. “restic-Android” → Create new app password → copy to clipboard

Option pass.
Password.
Choose an alternative below. Press Enter for the default (n).
y) Yes, type in my own password
g) Generate random password
n) No, leave this optional password blank (default)
y/g/n>

→ y

Enter the password:
password:
Confirm the password:
password:

→ Enter your Murena Cloud / Nextcloud PASSWORD

Skip Bearer token process

Option bearer_token.
Bearer token instead of user/pass (e.g. a Macaroon).
Enter a value. Press Enter to leave empty.
bearer_token>

→ ENTER (to skip)

Save configuration

Edit advanced config?
y) Yes
n) No (default)
y/n>

→ n

Confirm your configuration

Configuration complete.
Options:

→ y

Your added cloud server will be listed as current remote

Current remotes:

Name Type
==== ====
MurenaWebDAV webdav

Finally, you can quit rclone

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q>

→ q

Test WebDAV connection to cloud via rclone

:exclamation: Don’t miss the colon “:” after the name) :exclamation:

rclone lsd MurenaWebDAV:

→ Should display files and folders you have inside your Murena Cloud / Nextcloud:

2023/09/23 11:19:58 NOTICE: Chunks temporary upload directory: https://murena.io/remote.php/dav/uploads/USERNAME/
-1 2023-09-09 10:33:55 -1 Documents
-1 2023-09-09 10:33:54 -1 Images
-1 2023-09-09 10:33:56 -1 Templates

Initialize restic backup repository

:exclamation: This step is only needed once at first setup :exclamation:

restic -r rclone:MurenaWebDAV:/backups init

Backup internal storage (without root) (default/personal profile)

restic -r rclone:MurenaWebDAV:/backups backup /storage/emulated/0/ --verbose

Backup app data (with root) (default/personal profile)

sudo restic -r rclone:MurenaWebDAV:/backups backup /data/data/ /data/user/0/ /data/user_de/0/ --verbose

Show all backups

restic -r rclone:MurenaWebDAV:/backups snapshots

Restore backup (latest backup)

restic -r rclone:MurenaWebDAV:/backups restore --target /storage/emulated/0/my-restore-directory latest

Browse your backups / Mount backup repository to file system (for example on your desktop machine)

:exclamation: In order to use this restic needs also to be installed on your desktop machine :exclamation:

mkdir /mnt/restic
restic -r rclone:MurenaWebDAV:/backups mount /mnt/restic

If you prefer a GUI for browsing your backups on your desktop machine: Use Restic Browser.

2.3 OPTION 3: Backup to own server via SFTP/SSH

:exclamation: needs restic and OpenSSH to be installed on both sides (Android & server) :exclamation:

Install OpenSSH

pkg install openssh

Create a dedicated user for restic on your server

:exclamation: Replace HOST and USER below with the ones of your server :exclamation:

Initialize restic backup repository

:exclamation: This step is only needed once at first setup :exclamation:

restic -r sftp:USER@HOST:backups init

On first connection you will be prompted once to verify the key fingerprint of your server

The authenticity of host ‘XXX.XXX.X.XX (XXX.XXX.X.XX)’ can’t be established. ED25519 key fingerprint is SHA256:XXXXXXXXXX…
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

→ yes

Backup internal storage (without root) (default/personal profile)

restic -r sftp:USER@HOST:backups backup /storage/emulated/0/ --verbose

Backup app data (with root) (default/personal profile)

sudo restic -r sftp:USER@HOST:backups backup /data/data/ /data/user/0/ /data/user_de/0/ --verbose

Show all backups

restic -r sftp:USER@HOST:backups snapshots

Restore backup (latest backup)

restic -r sftp:USER@HOST:backups restore --target /storage/emulated/0/my-restore-directory latest

Browse your backups / Mount backup repository to file system (for example on your desktop machine)

:exclamation: In order to use this restic needs also to be installed on your desktop machine :exclamation:

mkdir /mnt/restic
restic -r sftp:USER@HOST:backups mount /mnt/restic

If you prefer a GUI for browsing your backups on your desktop machine: Use Restic Browser.

3 Advanced usage

Backup work profile (without root)

Install Termux in work profile or clone to work profile using Shelter/Island/Insular

Run following commands from Termux in work profile:

Backup internal storage of work profile to sd card

restic -r /storage/XXXX-XXXX/backups backup /storage/emulated/10/ --verbose

Backup internal storage of work profile to Murena Cloud / Nextcloud

:exclamation: To make this work first repeat rclone configuration steps from guide above with Termux in work profile :exclamation:

restic -r rclone:MurenaWebDAV:/backups backup /storage/emulated/10/ --verbose

Backup internal storage of work profile to server via SFTP/SSH

restic -r sftp:USER@HOST:backups backup /storage/emulated/10/ --verbose

Backup work profile (with root)

No need to install Termux to work profile, just use sudo and specify work profile path (/data/media/10/)

Backup internal storage of work profile to sd card (root)

sudo restic -r /storage/XXXX-XXXX/backups backup /data/media/10/ --verbose

Backup internal storage of work profile to Murena Cloud / Nextcloud (root)

:exclamation: to make this work first repeat rclone configuration steps from guide above using: sudo rclone config :exclamation:

sudo restic -r rclone:MurenaWebDAV:/backups backup /data/media/10/ --verbose

Backup internal storage of work profile to SSH server (root)

sudo restic -r sftp:USER@HOST:backups backup /data/media/10/ --verbose
14 Likes
2

That’s really interesting and I just tried it (not on /e/ though), but on my non-rooted phone I get a “permission denied” in termix if I try /storage/emulated/0/. As a consequence, the backup doesn’t really contain anything.

Edit: my bad, I hadn’t granted “file access” to termux yet and it didn’t ask, so I enabled in the the app settings.

3

Thx, I just added that bit to the guide, so that we hopefully got that permission requirement covered now :+1:

4 Likes
4

I’m stuck at the step for initializing the backup repository. It says:
Fatal: create key in repository at /storage/F01A5CD41A5C9984/System-Backups/RealmeGT/restic-backup failed: sync /storage/F01A5CD41A5C9984/System-Backups/RealmeGT/restic-backup/keys/3525ec52c31e3a12622d289628508ab9ebc4dfbdfefb0bfdbfaeacbdee609aa9-tmp-2782534080: operation not permitted

I have no idea why, I’m rooted, I used tsu, and I granted Termux all permissions

5

Without having some background info this is kinda hard to solve.

  • On what Android, Magisk, Termux, and restic version are you?
  • Did you enable Termux as Superuser in Magisk?

From what I can see you try to backup on SD card. If you are rooted anyway, does it work if you run the initial commands with sudo?:

# Create backups folder on sd card
sudo mkdir -p /storage/F01A5CD41A5C9984/System-Backups/RealmeGT/restic-backup/

# Initialize restic backup repository (NOTE: This step is only needed once at first setup)
sudo restic init -r /storage/F01A5CD41A5C9984/System-Backups/RealmeGT/restic-backup/

It could be on certain devices that the path to SD card differs from the given /storage/ example. To check if you can access your SD card from Termux, just try to run ls -l /path/to/your/sdcard/ there. If Termux returns the list of the content on your SD card you should be also able to initalize a restic backup repository there. Of course you should also make sure that the backup folder exists first and then initialize the restic repo in it.

1 Like
6

Hi, many thanks for your reply

  • Android 13, RMX2202_13.1.0.1630(EX01)
  • Magisk 26.3
  • Termux 0.118.0
  • Restic: 0.16.2
    Termux has Superuser access

I don’t know if it’s important, but this is not an SD card but an external hard drive connected via the OTG port.

Adding sudo to the commands makes no difference. Although Magisk notifies me Termux has been given superuser permissions, it still fails with the same error. ls -l works normally and lists everything. I have full access to everything on the drive from my phone and can edit and delete stuff as I please, both in Termux and in managers like Root Explorer. But it always fails with the same error message

7

Sorry for the late follow-up but it seems like that the source of your problem is indeed the fact that you are using your external hard drive connected via OTG. While researching that topic I stumbled over a lot of ppl having issues with Termux and USB storage.

In the Termux Wiki on internal and external storage it is stated that Termux will only write data into the app-private folder of external storage, e.g. /storage/F01A5CD41A5C9984/Android/data/com.termux. Maybe this will work but I’m not that confident because if you are rooted it shouldn’t matter which folder you choose:

# Create backups folder on sd card
sudo mkdir -p /storage/F01A5CD41A5C9984/Android/data/com.termux/restic-backup/

# Initialize restic backup repository (NOTE: This step is only needed once at first setup)
sudo restic init -r /storage/F01A5CD41A5C9984/Android/data/com.termux/restic-backup/

I tried to test it myself with a USB storage drive but couldn’t even figure out how to access it from within Termux because the mounted USB drive wouldn’t show up there, even though it is accessable via file manager. So for now I cant provide a definite solution to make this backup method work on connected USB storage drives, sorry.

2 Likes
8

Thank you very much for your time, research and effort, it’s really appreciated.

Is storing the backup on the phone itself the only solution, then? It strikes me as a bit counter-intuitive as that means you can only use half the storage space you otherwise could if you want to back it up

9

My experience (not this exact method) is that one would require an OTG USB drive rather than a standard HD to communicate with Android.

10

If you use a sd card (via sd card slot) instead of USB OTG it should work fine. If sd card doesnt work for you there are the two other methods: cloud via WebDAV (see 2.2 OPTION 2) and SSH (see 2.2 OPTION 3). I personally use the SSH method to backup to my local NextcloudPi NAS (which has a USB-SSD drive connected to it) and to a rented storage box at Hetzner which offers restic support. This way I achive the 3-2-1 backup rule: haveing 3 copies of data (phone, local NAS, external storage box) on 2 different media with 1 copy being off-site :slight_smile:

1 Like
11

If it works with SSH I might try that, thank you

12

This is excellent! Thanks for the in-depth guide which saved me a lot of time. :slight_smile:

13

Question about this method: What’s included in this backup?

Photos, music, documents etc. (i.e. “normal” stuff I can see and copy about with the on-device file browser) is very likely in, but I also see a bunch of folders that appear to belong to different apps, though certainly not all of them – is there a way to know what exactly I’m getting, and whether this saves me from backing some apps’ data and settings up separately? If yes, would I be able to copy back that data onto a clean sytsem (same /e/OS version, same phone), re-install the respective apps and have them back in the state I had them before? Could I even get my overall system configuration back this way?

Or is the only way to be sure to export settings and data from each app (if they support that) separately, import it after re-installation, and to write down all my non-default system settings and set them manually? There seems to be no clear word about this, and if that is so, I’m wondering what this “full” backup does for me that I don’t get with a backup of just the generic media folders (.i.e. photos, videos, documents, music…). To be clear: I’m pretty sure that there is an added benefit (after all, there is a lot of additional stuff being backed up). I’m just trying to work out what exactly it is, when/how much I can rely on it, etc…

14

As indicated in my initial guide this method allows to do a full backup if you have rooted your device. “Full” not in terms of 1:1 but in the sense of having all your individual data backed up and being able to restore them correctly (in one way or another). 1:1 backups or byte-to-byte identical images wont work on Android (or at least you wont be able to restore them, at least to my knowledge) and full backups like the ones via TWRP are also a thing from the past (at least for my device).

But you touch a good point here, as every backup solution only makes sense when one is able to make a proper restore. So let me address your questions:

Yes, this kind of data is most commonly stored on internal storage and sd card (if there is any). And yes, you could copy them off your device manually by plugging your phone into your PC or just sync them to other devices with one of various sync/cloud apps.

Well, as you maybe can imagine, apps are developed by different people and therefore data can be stored in different places. By backing up internal storage (+ sd card, if you use one) and all app data from default/personal profile (+ work profile, if you have one in use), the backup method described here tries to catch all those places.

To adress it more technically (if anyone knows better, please correct me): When I initially researched how to backup Android, I compiled a list of what paths Android uses to store data and how to access them. Then I tried to validate this information by exploring those paths on my own device (FP4) using root and Termux. If you want to make 100% sure what you actually backup, you have to do this as well, I guess. And, as I wrote in my initial introduction of this solution: app data directories ~could~ differ depending on your specific device and I only looked into Android 11 and above (Android 14 as of now) on my own device.

List of paths Android uses to store data:

  • Internal Storage (main profile)

    • /data/media/0/
      • internal storage of default profile (user 0)
      • → needs ROOT
      • alternative symlink #1: /storage/emulated/0/
        • → without root
      • alternative symlink #2: /sdcard/
        • → without root

  • Internal Storage (work profile)

    • /data/media/10/
      • → needs ROOT
      • alternative symlink: /storage/emulated/10/
        • → access from within work profile (without root)

  • SD Card

    • /storage/XXXX-XXXX/

  • App data

    • /data/user/0/

      • contains the apps data from default profile (user 0)
      • e.g. /data/user/0/com.app.whatever
      • alternative symlink: /data/data/
      • → all paths need ROOT

    • /data/user_de/0/

      • contains device protected data from default profile (user 0)
      • e.g. /data/user_de/0/com.app.whatever

    • /data/user/10/

      • contains the apps data from work profile (user 10)

      • /data/user_de/10/

        • contains device protected data from work profile (user 10)

  • APKs

    • User-installed apps

      • /data/app/
        • contains all user installed app apks (including apps installed to work profile)

    • Pre-installed system apps

      • /system/app/
      • /system/priv-app/
        • both contain the system apps apks

Parts of this information is taken from the Neo Backup FAQ (read there for more details), so credits to them.

Main point here is to understand that Android separates apps into software itself (.apk) and its data (stored in different places, as outlined in the list above).

Yes, that could work. System settings should also be included within the app data. Heres how that could be achieved (no guarantees, havent tried it myself to that extent, please make sure you HAVE BACKED UP ALL YOUR DATA FIRST):

  • Make sure you have the same device, same /e/OS version installed and your device is rooted the same way as before
  • Install all user apps you have installed before in the exact same version, but don’t open them just yet
  • Setup restic in Termux (as described in the initial post of this thread) and restore your previous backups from your restic repository with the following commands:
  • Restore your internal storage (without root): restic -r <YOUR_RESTIC_REPOSITORY> restore <SNAPSHOT>:/storage/emulated/0 --target /storage/emulated/0 --verbose
  • Restore your app data (with root):
    • sudo restic -r <YOUR_RESTIC_REPOSITORY> restore <SNAPSHOT>:/data/user/0 --target /data/user/0 --verbose
    • sudo restic -r <YOUR_RESTIC_REPOSITORY> restore <SNAPSHOT>:/data/user_de/0 --target /data/user_de/0 --verbose
  • Restart your phone
  • Check each app and settings if data has been restored

Personally, I would be doing this even more carefully: I would only restore app data on a per-app basis for the user apps, leave system apps untouched and adjust the system settings manually. Internal storage should be fine by just restoring all data as a whole with the command above.

Earlier in this post I pointed out the places where data is stored on Android, so it should be clear by now that this method with restic is actual capable to backup all data of your choice. You just need root and pass the correct path to restic.

A backup solution (no matter if it be SeedVault, SwiftBackup, or this one) is only of value if it is able to restore correctly. So I would recommend to just do a simple test-run for yourself: Assume you lost the data of a certain app. Now try to recover that data out of your backup.

I personally think it is always better to be on the safe side, so I dont trust a single backup solution, either. I use the data export settings of every app that offers it, I point those exports to external storage and sooner or later this data will also be backed up all together with my usual restic backup of the whole internal storage. Additionally, I regularly use Syncthing to sync such files from my phone to other devices (which then also get backed up automatically), so that’s another safe copy.

Of course, one can always write down all custom made settings for system and apps, but thats a time-consuming task. Doing screenshots could speed that up, but you still have to adjust your documentation each time there is a UI change. All in all, it all depends on the time you want to invest. Once you get a understanding of the whole process how to backup your data without Google, you will get used to it, you can even automate it. It is also possible to achieve this with this method, but I don’t have time to get into that right now. If you are interested just look up Termux job scheduling.

2 Likes