[HOWTO] End-to-end Encrypted Full Backup and Restore Solution for /e/OS using Termux and restic

outsidethebox · September 23, 2023, 4:12pm

In the wake of the upcoming /e/OS T release, I thought it would be handy to share a new Android backup solution I recently discovered and which is still fairly unknown, I guess

About this backup solution

We make use of restic here, a standalone command-line backup tool which generally works on all major platforms (Linux, BSD, Mac, Windows WSL, and Android)
restic is free Open Source software and backups are end-to-end encrypted per default
it allows you to store your encrypted backups both locally or on a remote server (in my guide below I will show you how to backup to sd card, own server, and Murena Cloud / Nextcloud, but there are even more storage types)
backups are de-duplicated, incremental and compressed, which saves you a lot of time, space, and data usage
as restic is a independent tool this backup solution does NOT depend on adb, adb backup, adb root, root debugging or custom shell scripts
it allows you to backup all the data on your device if you have root, so you don’t have to deal with backups being incomplete as in the case of SeedVault when certain apps have set a “do not backup” flag
restoring data works in two ways: specifying a certain path on Android to restore to or you mount a backup snapshot on your desktop file system and just copy files out of there (there is also a open source restic file explorer software to simplify this, see Restic Browser in honorable mentions below)

Downsides of this backup solution

Without root you are only able to backup internal storage (of personal and work profiles) and sd card
We have to utilize the command line (Termux), if you are fine with that, this guide should be pretty straight foward for you
restic consumes more RAM the bigger your backup repository gets (I guess we are talking about TBs here, my FP4 has 8 GB RAM, and worked perfectly fine so far)
<Your feedback is needed here>

I successfully tested this method on a FP4 with /e/OS R (Android 11) and S (Android 12).

Please note that the app data directories given in the examples here could differ depending on your specific device. The given sd card and internal storage paths should match accross most devices, I hope.

For more details on the usage of restic please refer to the restic documentation. I won’t go into topics like tags, backup rotation, key management, etc. here.

Honorable mentions

This guide uses restic natively via command line. If you prefer to use a GUI, there is a unofficial experimental Restic Backup Android App on F-Droid (I won’t cover this app here as it is quite limited as of now and therefore doesn’t offer the same capabilites on which we rely in this guide)
Restic Browser provides you with a GUI to browse and restore your restic backup repositories on your Windows/Linux/MacOS desktop machine
For those ppl who prefer a similar encrypted FLOSS backup solution: Instead of restic you can also use BorgBackup via Termux. It can be installed using the command: pkg install borgbackup

Prerequisites

Termux installed (install via F-Droid)
Grant Termux file access permissions in app settings (else you will run into permission denied errors when running restic)
For full backups: root (e.g. Magisk), Termux needs to be able to run as Superuser when running restic with sudo

1 Initial Setup: Install restic via Termux

Update packages

apt update
apt upgrade

→ If there are any prompts just hit ENTER all the time

Update all termux packages

pkg upgrade

Install sudo (ROOT ONLY)

pkg install tsu

Install restic

pkg install restic

Check if restic was installed successfully

restic version

restic 0.16.0 compiled with go1.20.6 on android/arm64

2 Backing up your data

2.1 OPTION 1: Backup to sd card

Replace “/storage/XXXX-XXXX/” with your sd card path

Create backups folder on sd card

mkdir /storage/XXXX-XXXX/backups/

Initialize restic backup repository

This step is only needed once at first setup

restic init -r /storage/XXXX-XXXX/backups/

enter password for new repository:
enter password again:
created restic repository 36fbfa9fb8 at /storage/XXXX-XXXX/backups/

Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost.

Backup internal storage (without root) (default/personal profile)

restic -r /storage/XXXX-XXXX/backups backup /storage/emulated/0/ --verbose

Backup app data (with root) (default/personal profile)

sudo restic -r /storage/XXXX-XXXX/backups backup /data/data/ /data/user/0/ /data/user_de/0/ --verbose

Show all backups

restic -r /storage/XXXX-XXXX/backups snapshots

Restore backup (latest backup)

restic -r /storage/XXXX-XXXX/backups restore --target /storage/emulated/0/my-restore-directory latest

Browse your backups / Mount backup repository to file system (for example on your desktop machine)

In order to use this restic needs also to be installed on your desktop machine

mkdir /mnt/restic
restic -r /your/sdcard/path/backups mount /mnt/restic

If you prefer a GUI for browsing your backups on your desktop machine: Use Restic Browser.

2.2 OPTION 2: Backup to Murena Cloud / your own Nextcloud via WebDAV

restic doesn’t support backing up to Murena Cloud / Nextcloud instances (WebDAV) out of the box, it needs rclone for that

Update packages

apt update
apt upgrade

→ If there are any prompts just hit ENTER all the time

Update all termux packages

pkg upgrade

Install rclone

pkg install rclone

Configure rclone

rclone config

Create new remote

2023/09/23 11:00:04 NOTICE: Config file “/data/data/com.termux/files/home/.config/rclone/rclone.conf” not found - using defaults
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q>

→ n

Specify name of your backup destination

name>

→ MurenaWebDAV / NextcloudWebDAV (or whatever name you like)

Specify type of your backup server

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.

→ Choose number corresponding to WebDAV

Specify backup-server URL

Option url.
URL of http host to connect to.
E.g. https://example.com.
Enter a value.
url>

→ Log into Murena Cloud / Nextcloud, navigate to Files → open menu → Files settings → copy WebDAV path from there (alternatively, use the URL below and replace USERNAME with your username)

→ https://murena.io/remote.php/dav/files/USERNAME/

Choose WebDAV Type

Option vendor.
Name of the WebDAV site/service/software you are using.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / Fastmail Files
\ (fastmail)
2 / Nextcloud
\ (nextcloud)
3 / Owncloud
\ (owncloud)
4 / Sharepoint Online, authenticated by Microsoft account
\ (sharepoint)
5 / Sharepoint with NTLM authentication, usually self-hosted or on-premises
\ (sharepoint-ntlm)
6 / Other site/service or software
\ (other)
vendor>

→ Choose number corresponding to Nextcloud

Specify Murena Cloud / Nextcloud username

Option user.
User name.
In case NTLM authentication is used, the username should be in the format ‘Domain\User’.
Enter a value. Press Enter to leave empty.
user>

→ Enter your Murena Cloud / Nextcloud USERNAME

Specify Murena Cloud / Nextcloud password

For security reasons it is recommended to use a dedicated App password

→ Log into Murena Cloud / Nextcloud, navigate to Settings → Security → scroll down to Devices & sessions → Fill in a app name of your liking, e.g. “restic-Android” → Create new app password → copy to clipboard

Option pass.
Password.
Choose an alternative below. Press Enter for the default (n).
y) Yes, type in my own password
g) Generate random password
n) No, leave this optional password blank (default)
y/g/n>

→ y

Enter the password:
password:
Confirm the password:
password:

→ Enter your Murena Cloud / Nextcloud PASSWORD

Skip Bearer token process

Option bearer_token.
Bearer token instead of user/pass (e.g. a Macaroon).
Enter a value. Press Enter to leave empty.
bearer_token>

→ ENTER (to skip)

Save configuration

Edit advanced config?
y) Yes
n) No (default)
y/n>

→ n

Confirm your configuration

Configuration complete.
Options:

type: webdav

url: https://murena.io/remote.php/dav/files/USERNAME/

vendor: nextcloud

user: USERNAME

pass: *** ENCRYPTED ***
Keep this “MurenaWebDAV” remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote

→ y

Your added cloud server will be listed as current remote

Current remotes:

Name Type
==== ====
MurenaWebDAV webdav

Finally, you can quit rclone

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q>

→ q

Test WebDAV connection to cloud via rclone

Don’t miss the colon “:” after the name)

rclone lsd MurenaWebDAV:

→ Should display files and folders you have inside your Murena Cloud / Nextcloud:

2023/09/23 11:19:58 NOTICE: Chunks temporary upload directory: https://murena.io/remote.php/dav/uploads/USERNAME/
-1 2023-09-09 10:33:55 -1 Documents
-1 2023-09-09 10:33:54 -1 Images
-1 2023-09-09 10:33:56 -1 Templates

Initialize restic backup repository

This step is only needed once at first setup

restic -r rclone:MurenaWebDAV:/backups init

Backup internal storage (without root) (default/personal profile)

restic -r rclone:MurenaWebDAV:/backups backup /storage/emulated/0/ --verbose

Backup app data (with root) (default/personal profile)

sudo restic -r rclone:MurenaWebDAV:/backups backup /data/data/ /data/user/0/ /data/user_de/0/ --verbose

Show all backups

restic -r rclone:MurenaWebDAV:/backups snapshots

Restore backup (latest backup)

restic -r rclone:MurenaWebDAV:/backups restore --target /storage/emulated/0/my-restore-directory latest

Browse your backups / Mount backup repository to file system (for example on your desktop machine)

In order to use this restic needs also to be installed on your desktop machine

mkdir /mnt/restic
restic -r rclone:MurenaWebDAV:/backups mount /mnt/restic

If you prefer a GUI for browsing your backups on your desktop machine: Use Restic Browser.

2.3 OPTION 3: Backup to own server via SFTP/SSH

needs restic and OpenSSH to be installed on both sides (Android & server)

Install OpenSSH

pkg install openssh

Create a dedicated user for restic on your server

Replace HOST and USER below with the ones of your server

Initialize restic backup repository

This step is only needed once at first setup

restic -r sftp:USER@HOST:backups init

On first connection you will be prompted once to verify the key fingerprint of your server

The authenticity of host ‘XXX.XXX.X.XX (XXX.XXX.X.XX)’ can’t be established. ED25519 key fingerprint is SHA256:XXXXXXXXXX…
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

→ yes

Backup internal storage (without root) (default/personal profile)

restic -r sftp:USER@HOST:backups backup /storage/emulated/0/ --verbose

Backup app data (with root) (default/personal profile)

sudo restic -r sftp:USER@HOST:backups backup /data/data/ /data/user/0/ /data/user_de/0/ --verbose

Show all backups

restic -r sftp:USER@HOST:backups snapshots

Restore backup (latest backup)

restic -r sftp:USER@HOST:backups restore --target /storage/emulated/0/my-restore-directory latest

Browse your backups / Mount backup repository to file system (for example on your desktop machine)

In order to use this restic needs also to be installed on your desktop machine

mkdir /mnt/restic
restic -r sftp:USER@HOST:backups mount /mnt/restic

If you prefer a GUI for browsing your backups on your desktop machine: Use Restic Browser.

3 Advanced usage

Backup work profile (without root)

Install Termux in work profile or clone to work profile using Shelter/Island/Insular

Run following commands from Termux in work profile:

Backup internal storage of work profile to sd card

restic -r /storage/XXXX-XXXX/backups backup /storage/emulated/10/ --verbose

Backup internal storage of work profile to Murena Cloud / Nextcloud

To make this work first repeat rclone configuration steps from guide above with Termux in work profile

restic -r rclone:MurenaWebDAV:/backups backup /storage/emulated/10/ --verbose

Backup internal storage of work profile to server via SFTP/SSH

restic -r sftp:USER@HOST:backups backup /storage/emulated/10/ --verbose

Backup work profile (with root)

No need to install Termux to work profile, just use sudo and specify work profile path (/data/media/10/)

Backup internal storage of work profile to sd card (root)

sudo restic -r /storage/XXXX-XXXX/backups backup /data/media/10/ --verbose

Backup internal storage of work profile to Murena Cloud / Nextcloud (root)

to make this work first repeat rclone configuration steps from guide above using: sudo rclone config

sudo restic -r rclone:MurenaWebDAV:/backups backup /data/media/10/ --verbose

Backup internal storage of work profile to SSH server (root)

sudo restic -r sftp:USER@HOST:backups backup /data/media/10/ --verbose

Ingo_FP_Angel · September 23, 2023, 5:02pm

That’s really interesting and I just tried it (not on /e/ though), but on my non-rooted phone I get a “permission denied” in termix if I try /storage/emulated/0/. As a consequence, the backup doesn’t really contain anything.

Edit: my bad, I hadn’t granted “file access” to termux yet and it didn’t ask, so I enabled in the the app settings.

outsidethebox · September 23, 2023, 5:21pm

Thx, I just added that bit to the guide, so that we hopefully got that permission requirement covered now

Thornskade · November 11, 2023, 6:08am

I’m stuck at the step for initializing the backup repository. It says:
Fatal: create key in repository at /storage/F01A5CD41A5C9984/System-Backups/RealmeGT/restic-backup failed: sync /storage/F01A5CD41A5C9984/System-Backups/RealmeGT/restic-backup/keys/3525ec52c31e3a12622d289628508ab9ebc4dfbdfefb0bfdbfaeacbdee609aa9-tmp-2782534080: operation not permitted

I have no idea why, I’m rooted, I used tsu, and I granted Termux all permissions

outsidethebox · November 19, 2023, 10:36pm

Without having some background info this is kinda hard to solve.

On what Android, Magisk, Termux, and restic version are you?
Did you enable Termux as Superuser in Magisk?

From what I can see you try to backup on SD card. If you are rooted anyway, does it work if you run the initial commands with sudo?:

# Create backups folder on sd card
sudo mkdir -p /storage/F01A5CD41A5C9984/System-Backups/RealmeGT/restic-backup/

# Initialize restic backup repository (NOTE: This step is only needed once at first setup)
sudo restic init -r /storage/F01A5CD41A5C9984/System-Backups/RealmeGT/restic-backup/

It could be on certain devices that the path to SD card differs from the given /storage/ example. To check if you can access your SD card from Termux, just try to run ls -l /path/to/your/sdcard/ there. If Termux returns the list of the content on your SD card you should be also able to initalize a restic backup repository there. Of course you should also make sure that the backup folder exists first and then initialize the restic repo in it.

Thornskade · November 22, 2023, 2:17am

Hi, many thanks for your reply

Android 13, RMX2202_13.1.0.1630(EX01)
Magisk 26.3
Termux 0.118.0
Restic: 0.16.2
Termux has Superuser access

I don’t know if it’s important, but this is not an SD card but an external hard drive connected via the OTG port.

Adding sudo to the commands makes no difference. Although Magisk notifies me Termux has been given superuser permissions, it still fails with the same error. ls -l works normally and lists everything. I have full access to everything on the drive from my phone and can edit and delete stuff as I please, both in Termux and in managers like Root Explorer. But it always fails with the same error message

outsidethebox · January 16, 2024, 8:47pm

Sorry for the late follow-up but it seems like that the source of your problem is indeed the fact that you are using your external hard drive connected via OTG. While researching that topic I stumbled over a lot of ppl having issues with Termux and USB storage.

In the Termux Wiki on internal and external storage it is stated that Termux will only write data into the app-private folder of external storage, e.g. /storage/F01A5CD41A5C9984/Android/data/com.termux. Maybe this will work but I’m not that confident because if you are rooted it shouldn’t matter which folder you choose:

# Create backups folder on sd card
sudo mkdir -p /storage/F01A5CD41A5C9984/Android/data/com.termux/restic-backup/

# Initialize restic backup repository (NOTE: This step is only needed once at first setup)
sudo restic init -r /storage/F01A5CD41A5C9984/Android/data/com.termux/restic-backup/

I tried to test it myself with a USB storage drive but couldn’t even figure out how to access it from within Termux because the mounted USB drive wouldn’t show up there, even though it is accessable via file manager. So for now I cant provide a definite solution to make this backup method work on connected USB storage drives, sorry.

Thornskade · January 17, 2024, 8:27am

Thank you very much for your time, research and effort, it’s really appreciated.

Is storing the backup on the phone itself the only solution, then? It strikes me as a bit counter-intuitive as that means you can only use half the storage space you otherwise could if you want to back it up

aibd · January 17, 2024, 8:40am

My experience (not this exact method) is that one would require an OTG USB drive rather than a standard HD to communicate with Android.

outsidethebox · January 17, 2024, 2:43pm

If you use a sd card (via sd card slot) instead of USB OTG it should work fine. If sd card doesnt work for you there are the two other methods: cloud via WebDAV (see 2.2 OPTION 2) and SSH (see 2.2 OPTION 3). I personally use the SSH method to backup to my local NextcloudPi NAS (which has a USB-SSD drive connected to it) and to a rented storage box at Hetzner which offers restic support. This way I achive the 3-2-1 backup rule: haveing 3 copies of data (phone, local NAS, external storage box) on 2 different media with 1 copy being off-site

Thornskade · January 17, 2024, 2:55pm

If it works with SSH I might try that, thank you