[HOWTO] Create your custom OTA server

About

This guide should help setting up and troubleshooting your own OTA server (tested just with PIE). Afaik it should work with all Android versions since Nougat but I had no chance testing any other version yet.

Setup

Docker (recommended)

For the most easiest setup follow the instructions here: LineageOTA repo and use Docker.
When you have that part done go on with the “OTA setup” topic here.

Alternative: LXC / LXD setup (if you do NOT want to use docker)

I don’t like docker but that’s just my personal preference. I had set up the whole thing in a lxc container instead. If you are not familiar with that or just like docker simply go on as in the repository README linked above.

The other reason is that I want to setup the OTA server for multiple devices and ROMs (not just /e/) and so I needed to understand first what all the requirements are and how that stuff actually works together (yea it was a little journey ; ) )

The process is like that: look up in the Dockerfile what packages you need to install etc.
The following assumes that you have lxc + lxd installed and initiated) and might be not up2date. But again: re-check the Dockerfile for any updates before blindly following here.

LXC container install

lxc launch ubuntu:18.04 <how-you-want-to-name-it> (e.g. ota-server)
lxc exec ota-server -- apt-get -y install zlib1g-dev composer php7.2 apache2 git ntp php-apcu php-zip --no-install-recommends

LXC container setup

jump into the container and setup the base:

lxc exec ota-server -- bash

a2enmod apcu
a2enmod php7.2
a2enmod mod_headers
a2enmod rewrite

echo -e 'ServerLimit 1024\nMaxClients 1024' >> /etc/apache2/conf-available/perf.conf
a2enconf perf
echo 'apc.ttl=7200' >> /etc/php/7.2/apache2/conf.d/10-opcache.ini

LXC OTA server install

lxc exec ota-server -- bash (if you left the container only ofc)

cd /var/www/html/
git clone https://gitlab.e.foundation/e/os/LineageOTA.git e-os
cd e-os
composer install --optimize-autoloader --no-interaction --no-progress

special patch (dirty but it gets the work done) for php 7.2 compat (thx for that @andrelam !):

cd /var/www/html/e-os/pie
wget https://raw.githubusercontent.com/picomatic/e_ota_patch/master/builds.patch
patch -p1 < builds.patch && rm builds.patch

finalize:

chmod -R 0775 /var/www/html
chown -R www-data:www-data /var/www/html

LXC storage sharing between host/container

sharing storage between (so you can easily make files available within the container. The following will map the local (i.e. the hosts) path defined in to the path in the container as defined in . I want to map my host path to /var/www/html/… to just have new updates magically available later in the OTA server.

lxc config device add ota-server <name-of-the-share> disk source=/your/host/path path=/var/www/html/e-os/pie/builds/full

OTA setup

The following is an example for a better understanding how the things playing together.

server path

/var/www/html/e-os/pie/builds/full/h815/<eos>.zip (also containing md5/sha256)
or for docker
/var/www/html/builds/full/h815/<eos>.zip (also containing md5/sha256)

apache rewrite (/etc/apache2/sites-enabled/000-default.conf)

<Location /e-os/pie>
        RewriteEngine On
        RewriteCond %{REQUEST_FILENAME} !-f
        RewriteCond %{REQUEST_FILENAME} !-d
        RewriteRule ^(.*)$ index.php [QSA,L]
</Location>

OTA URL (set at build time / can always be overwritten by setprop / build.prop):

When using a community/custom build the OTA server gets set within your device tree by the property lineage.updater.uri or when using Nougat or older: cm.updater.uri instead.

Format:

lineage.updater.uri="https://<servername>/e-os/pie/api"

This gets translated automatically by the above script to: https://<servername>/e-os/pie/api/v1/{device}/{type}/{incr}

{device} is a macro which will be automatically replaced by ro.product.device/ro.build.product
{type} is a macro which will be automatically replaced by ro.lineage.releasetype
{incr} is a macro which will be automatically replaced by ro.build.version.incremental

There is no need to hard-code, i.e. replacing these macros.

You might wonder about the URI above as it does not match your physical path within your container?!

e.g. that is the local path:
/var/www/html/e-os/pie/builds/full/h815/<my>.zip
or for docker
/var/www/html/builds/full/h815/<my>.zip

but the OTA URL points to (when the macros got translated):
https://<servername>/e-os/pie/api/v1/h815/unofficial/eng.jenkin.20200720.173054 (as you can see there is no “unofficial/eng.jenkin.20200720.173054” in the local path)

The reason behind this is that the OTA server will handle the request and constructs the path + what your current version is different. So that can be a little confusing first as the proper paths are crucial important ofc.

As the topic said the URI can be set either during build process (as done by the docker build script of /e/) or at any time (build as a system prop override or directly in your ROM) .
Especially for testing it can be useful to override the OTA URI from build:

Nougat or earlier:
setprop cm.updater.uri "https://<servername>/e-os/pie/api/v1/{device}/{type}/{incr}"

Oreo or later:
setprop lineage.updater.uri "https://<servername>/e-os/pie/api/v1/{device}/{type}/{incr}"

That change taken effect immediately and will not survive a reboot though (for that just set it in system/build.prop accordingly).

TEST

The first quick test is cloning the testing repo: UnitTest repo to your local machine (not within the container). Then modify the tests/lineage.js to match your server and URI and see how it goes.

If that one went fine you can test with your running Android like that:

adb shell (or adb root when enabled)
su 
setprop lineage.updater.uri "https://<your-servername>/e-os/pie/api/v1/{device}/{type}/{incr}" 

Troubleshooting / Good things to know

General

  1. the OTA updates information get cached, so if you do not see something changed: the first thing to do is restarting the webserver (e.g. systemctl restart apache2)
  2. to see what’s going on on server-site you should check the apache log files within your container and the LineageOTA.log within the root of the OTA installation (also within your container)
  3. to see what’s going on on client-site (i.e. Android):
    adb logcat |grep -vi weather | egrep -i "update|<yourservernamehere>|Utils|HttpURLConnectionClient"

Adventurous

Lets say you wanna play a bit to see how a new update would look like.
you can place a file with “.prop” extension (must be named like the zip just + “.prop”) and copy over the build.prop from your current running Android and modify it to your needs…

If you place a file like: /var/www/html/e-os/pie/builds/full/h815/<my>.zip.prop (or for docker: /var/www/html/builds/full/h815/<my>.zip.prop) it gets parsed instead of the build.prop within the zip. That way you can make e.g. an old zip shown up as “new” in the Updater, or when you changed the channel/RELEASE_TYPE then it can be simply overwritten here to match again.

so it took me a while but here is whats needed to show a version as “new update” in the Updater (taken from the android sources, OTA server sources and own testings ofc):

  • the utc timestamp must be newer then the current one (ro.build.date.utc)
  • the display_version must be in the same format between new update and current set ( ro.lineage.display.version, ro.lineage.version)
  • in any cases the version must be in digit only format: \d.\d (e.g. 1.0 but not 1.BETA). yea I had started like that… and it was a PITA to find out the reason for that :stuck_out_tongue:
  • the channel aka RELEASE_TYPE must match between ROM and new zip (ro.lineage.releasetype)
  • the channel aka RELEASE_TYPE needs to be part of the web path only (so it is not the actual physical path)

Locking down the OTA server

ATTENTION: whatever you do ensure you test it afterwards - after restarting apache.

file permissions /var/www (aka web-root)

the webserver do not have write access to anything else then a log (handled some steps later). In order to find files writeable by the user www-data:

find /var/www ! -type l \( -perm /o=w -o -perm /g=w -group www-data \)

when you have verified that these are fine to change fix it directly on-the-fly:

find /var/www ! -type l \( -perm /o=w -o -perm /g=w -group www-data \) -exec chmod u-w,g-w,o-w {} \;

file permissions OTA server

make all files unreadable by www-data which we do not want to expose publicly:

cd /var/www/html/<subdir-if-used>
chmod 000 composer.json composer.lock README.md LineageOTA.iml

ensure that the OTA server can write his log:

cd /var/www/html/<subdir-if-used>
touch LineageOTA.log
chown root:www-data LineageOTA.log
chmod 620 LineageOTA.log

/etc/apache2/apache2.conf

replace the whole <Directory /var/www/> object with:

<Directory /var/www/>
        # remove Indexes as there is no need to list them
        Options FollowSymLinks
        # if you want to allow users browsing your OTA server use that one instead
        #Options Indexes FollowSymLinks

        AllowOverride None
        Require all granted

        # enforce HTTP/1.1 (1.0 has security issues)
        RewriteEngine On
        RewriteCond %{THE_REQUEST} !HTTP/1.1$
        RewriteRule .* - [F]

        # only allow methods we really need.
        # Default methods: OPTIONS, GET, HEAD, POST, PUT, DELETE, TRACE, CONNECT (HTTP 1.1 protocol)
        <LimitExcept GET POST HEAD OPTIONS>
                deny from all
        </LimitExcept>
</Directory>

security.conf

in /etc/apache2/conf-available/security.conf set:

ServerTokens Prod
ServerSignature Off
TraceEnable Off
Header set X-Frame-Options: "sameorigin"
FileETag None

activate the config:

a2enconf security
systemctl restart apache2

After doing the above changes you might need to adapt the index.php as the global param HTTP_SERVER might not be available anymore which causing wrong download URI’s. That means the client can connect, see’s an update but cannot download as the URI misses the servername/port.
You can easily test that with the UnitTest repo and look briefly at the generated download URI and try to access that in a browser (it should begin downloading when open it).

Open index.php of the OTA service and add the following right before $app = new CmOta();:

$_SERVER['HTTP_HOST'] = '<FQDN or IP of your server>:<optional a non default port>';

Example:

[....]
$_SERVER['HTTP_HOST'] = 'ota.example.com';   <-- obviously that is just an example ;)
$app = new CmOta($logger);
$app
->setConfig( 'basePath', $protocol.$_SERVER['HTTP_HOST'].dirname($_SERVER['SCRIPT_NAME']) )
->run();

Verify!

Of course these changes might impact things so ensure you verify with the UnitTest repo if you can still access and see your ROMs. It is also a good idea placing a new ROM there and see if it gets printed correctly (so no caches involved nowhere).
Finally doing an OTA upgrade itself is your best test of course.

Even though the above are a good start there is more you can do like running your container itself unprivileged etc but that is something not covered here.

Finally

I hope that guide was useful and helping others trying to sort out how to setup an OTA server - especially when it comes to understanding and when not using docker as the base :grin:

8 Likes