Build /e/, docker, write a new local_manifest

smu44 · August 12, 2021, 6:58am

0.18 Q build for a3xelte succeeded

Command used (no custom manifest ) :

docker pull registry.gitlab.e.foundation/e/os/docker-lineage-cicd:community
docker run \
-v "/srv/src:/srv/src" \
-v "/srv/zips:/srv/zips" \
-v "/srv/logs:/srv/logs" \
-v "/srv/ccache:/srv/ccache" \
-v "/srv/mirror:/srv/mirror" \
-e "BRANCH_NAME=v0.18-q" \
-e "DEVICE_LIST=a3xelte" \
-e "REPO=https://gitlab.e.foundation/e/os/releases.git" \
-e "ANDROID_JACK_VM_ARGS=-Dfile.encoding=UTF-8 -XX:+TieredCompilation -Xmx20G" \
registry.gitlab.e.foundation/e/os/docker-lineage-cicd:community

Build disk is 440GB big, but it was a typo : I wanted to use a 400GB disk
If you want me to try again with a 400GB please let me know !

Also, do you want me to upload the ROM file somewhere for sharing ?

Logs here : https://ecloud.global/s/gwC28Dp39AZsnGp?path=%2Fa3xelte_Q_0.18

Some sizes after build (and it’s automatic cleaning) :

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb        433G  317G   94G  78% /srv

root@ubuntu:/srv# du -sxBG src/Q/*
1G      src/Q/android
0G      src/Q/Android.bp
1G      src/Q/art
1G      src/Q/bionic
1G      src/Q/bootable
0G      src/Q/bootstrap.bash
1G      src/Q/build
2G      src/Q/cts
1G      src/Q/dalvik
1G      src/Q/developers
1G      src/Q/development
1G      src/Q/device
9G      src/Q/external
2G      src/Q/frameworks
1G      src/Q/hardware
1G      src/Q/kernel
1G      src/Q/libcore
1G      src/Q/libnativehelper
1G      src/Q/lineage
1G      src/Q/lineage-sdk
1G      src/Q/Makefile
1G      src/Q/out
1G      src/Q/packages
1G      src/Q/pdk
1G      src/Q/platform_testing
33G     src/Q/prebuilts
1G      src/Q/sdk
1G      src/Q/system
1G      src/Q/test
1G      src/Q/toolchain
2G      src/Q/tools
43G     src/Q/vendor

root@ubuntu:/srv# du -sxBG src/Q/.repo/project-objects/*
1G      src/Q/.repo/project-objects/device
14G     src/Q/.repo/project-objects/e
1G      src/Q/.repo/project-objects/kernel
6G      src/Q/.repo/project-objects/LineageOS
158G    src/Q/.repo/project-objects/platform
28G     src/Q/.repo/project-objects/TheMuppets
9G      src/Q/.repo/project-objects/The-Muppets
1G      src/Q/.repo/project-objects/toolchain

root@ubuntu:/srv# du -sxBG src/Q/.repo/project-objects/platform/*
1G      src/Q/.repo/project-objects/platform/build
3G      src/Q/.repo/project-objects/platform/cts.git
1G      src/Q/.repo/project-objects/platform/dalvik.git
1G      src/Q/.repo/project-objects/platform/developers
16G     src/Q/.repo/project-objects/platform/external
2G      src/Q/.repo/project-objects/platform/frameworks
1G      src/Q/.repo/project-objects/platform/hardware
1G      src/Q/.repo/project-objects/platform/libnativehelper.git
1G      src/Q/.repo/project-objects/platform/packages
1G      src/Q/.repo/project-objects/platform/pdk.git
125G    src/Q/.repo/project-objects/platform/prebuilts
1G      src/Q/.repo/project-objects/platform/sdk.git
1G      src/Q/.repo/project-objects/platform/system
1G      src/Q/.repo/project-objects/platform/test
12G     src/Q/.repo/project-objects/platform/tools

root@ubuntu:/srv# du -sxBG src/Q/.repo/project-objects/platform/prebuilts/*
1G      src/Q/.repo/project-objects/platform/prebuilts/abi-dumps
15G     src/Q/.repo/project-objects/platform/prebuilts/android-emulator.git
1G      src/Q/.repo/project-objects/platform/prebuilts/asuite.git
1G      src/Q/.repo/project-objects/platform/prebuilts/bundletool.git
1G      src/Q/.repo/project-objects/platform/prebuilts/checkcolor.git
1G      src/Q/.repo/project-objects/platform/prebuilts/checkstyle.git
31G     src/Q/.repo/project-objects/platform/prebuilts/clang
2G      src/Q/.repo/project-objects/platform/prebuilts/clang-tools.git
1G      src/Q/.repo/project-objects/platform/prebuilts/devtools.git
1G      src/Q/.repo/project-objects/platform/prebuilts/fuchsia_sdk.git
2G      src/Q/.repo/project-objects/platform/prebuilts/gcc
1G      src/Q/.repo/project-objects/platform/prebuilts/gdb
4G      src/Q/.repo/project-objects/platform/prebuilts/go
6G      src/Q/.repo/project-objects/platform/prebuilts/gradle-plugin.git
3G      src/Q/.repo/project-objects/platform/prebuilts/jdk
1G      src/Q/.repo/project-objects/platform/prebuilts/ktlint.git
1G      src/Q/.repo/project-objects/platform/prebuilts/manifest-merger.git
2G      src/Q/.repo/project-objects/platform/prebuilts/maven_repo
4G      src/Q/.repo/project-objects/platform/prebuilts/misc.git
6G      src/Q/.repo/project-objects/platform/prebuilts/ndk.git
1G      src/Q/.repo/project-objects/platform/prebuilts/python
7G      src/Q/.repo/project-objects/platform/prebuilts/qemu-kernel.git
12G     src/Q/.repo/project-objects/platform/prebuilts/sdk.git
37G     src/Q/.repo/project-objects/platform/prebuilts/tools.git
1G      src/Q/.repo/project-objects/platform/prebuilts/vndk

aibd · August 12, 2021, 5:18pm

That is great! It was so good reading your logs with positive content that on a whim I ran you exact command into my root filesystem. It ran fine and I got the start of a more productive log. So I have formatted my partition again, super-cleaned docker, and started again.

Of interest your repo-log specifically includes
Downloading manifest from https://gitlab.e.foundation/e/os/releases.git
So do both of mine today! - this never appeared in any of my fail logs.

Of course I would be delighted to test-drive your ROM if you have an easy point to upload it to.

smu44 · August 12, 2021, 5:39pm

This is good news
Fingers crossed !

Here you are : Jaguar Network - Share
MD5 : 05412f6737e627e74b6ffcbaaa03e6df

piero · August 12, 2021, 7:02pm

très élégant ce jaguar…

aibd · August 12, 2021, 7:36pm

e_a3xelte-userdebug 10 QQ3A.200805.001 eng.root.20210812.020744 UNOFFICIAL-release,test-keys

Runs very well. Even Magic Earth found me straight away, tested very briefly in Bliss and Lawnchair. Great.

You are a star. Thank you.

smu44 · August 13, 2021, 6:07am

Please be aware that 0.18 /e/ is still in beta testing phase

aibd · August 19, 2021, 8:41pm

My thanks, Sylvain, for your help and support.

As you guessed, it seems my 373G partition is just too small. If there is a way to limit the size of the download in Docker with a local_manifest, it has eluded me.

During the final hour of the session is seen at 15:38 hours

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda4       373G  373G     0 100% /srv

A search for the last files to sync, shows this file downloaded at 16:12 hours.

/srv/src/Q/.repo/project-objects/platform/prebuilts/clang/host/linux-x86.git/objects/pack:
-r--r--r-- 1 root root 4244258257 Aug 17 16:12 tmp_pack_OR8ZKm

While this shows that the “build folders” start to be created in Q at 16.15 hours.

ls -alt /srv/src/Q
drwxr-xr-x 90 root root 4096 Aug 17 16:22 external
drwxr-xr-x  8 root root 4096 Aug 17 16:19 device
drwxr-xr-x 20 root root 4096 Aug 17 16:18 development
drwxr-xr-x  3 root root 4096 Aug 17 16:16 developers
drwxr-xr-x 10 root root 4096 Aug 17 16:16 dalvik
drwxr-xr-x  3 root root 4096 Aug 17 16:15 cts
lrwxrwxrwx  1 root root   19 Aug 17 16:15 Android.bp -> build/soong/root.bp
lrwxrwxrwx  1 root root   26 Aug 17 16:15 bootstrap.bash -> build/soong/bootstrap.bash
drwxr-xr-x  6 root root 4096 Aug 17 16:15 build
-r--r--r--  1 root root   92 Aug 17 16:15 Makefile
drwxr-xr-x  3 root root 4096 Aug 17 16:15 bootable
drwxr-xr-x 13 root root 4096 Aug 17 16:15 bionic
drwxr-xr-x 31 root root 4096 Aug 17 16:15 art
drwxr-xr-x  4 root root 4096 Aug 17 16:15 android

But Q folders are not fully populated:

/srv/src/Q$ du -sBG *
1G      android
0G      Android.bp
1G      art
1G      bionic
1G      bootable
0G      bootstrap.bash
1G      build
1G      cts
1G      dalvik
1G      developers
1G      development
1G      device
3G      external
1G      Makefile

Docker fails with Exit Code 1. No specific out of space error or final time stamp appears in the repo-log.

One assumes the sync completed ok as the repo-log shows 3,020 lines saying “Updating files”, speculation, populating the above directory, Q/

then, speculation, the “out of space error” is 44 lines similar to:

error: unable to write file examples/apache-httpd/corpus_http2 2511e2df990fb53afa7d65fd0d5438d6.00003b8b.honggfuzz.cov

which even seems to end “out of breath” with

error: unable to write file examples/apache-httpd/corpus_http2/952e916795345b2e386ba354a6a93306.0001f1bb.honggfuzz.cov
error: unable to write file examples/apache-httpd/corpus_http2/953b268a0acd5479f833925e0b039241.0000c26e.honggfuzz.cov
error: unable to write file examples/apache-httpd/corpus_http2/95926929821cc8a6dd0660d477e8a348.00000

I think I will try a non-docker build GitHub - kikislater/builde: Script to build /e/ next!

smu44 · August 20, 2021, 6:22am

You’re welcome

Your guessing seems correct to me : you’re out of disk space.

Some program may output a clear error message, some other won’t …
“unable to write file” looks like one of them !

I don’t think that trying to build out of Docker will solve the problem : Docker is only using a few GB (comparing to src), more likely located outside of your build filesystem.

This being said, 373GB of source tree is way too much !
I don’t know any way to reduce it’s size, it would imply to know if some repo/branches are useless …
Did you purge your source tree before download ? If you have some previous versions of some repo, they may have been kept, aside the new ones.

I will try to build for 0.18 Q a3xelte after reducing my storage to 373GB, we’ll see what happen.
But not before this evening.
I’ll try to record some sizes during the whole operation …

aibd · August 20, 2021, 7:34am

Yes, I formatted the partition (ext4). I also chose to clear /overlay2/ thoroughly. That is very good of you to offer to try to reproduce my problem, thanks but no hurry!

smu44 · August 21, 2021, 3:35pm

Hi !

Got some news …

I reduced my build filesystem to 369GB (sorry it’s a SAN disk, very difficult to get precise size because of allocation schemes …).

And my build failed, but not like yours
Repo sync was OK, and build failed with a classic “No space left on device”.

I’ve put a “du” in cron (every 5 minutes), but like a dumbass I forgot to output timestamp, so it’s unusable

If you wish to compare I can upload the repo log or retry the whole build, with timestamps this time !

aibd · August 21, 2021, 4:50pm

Thank you so much for the time spent on this!

I think we established one needs more than 370G to build /e/ with docker.

This is my feeling from the numbers I collected, I omitted any speculation about the way that ccache is accommodated, and how one estimates an “overhead” for building.

I will now be looking for a more space solution!

Thank you again!

smu44 · August 22, 2021, 5:41am

Seems like we have no choice here …

About CCACHE : you can manage it’s size by using the variable CCACHE_SIZE, i.e. by adding

-e “CCACHE_SIZE=10G” \

to your Docker command (default is 50G).
Remember, you can list variables with docker inspect <imageid> command
But decreasing CCACHE size will slow down the build process …

I’ll retry the build with a filesystem size around 400GB.
Why would a guy take the hard road when there is an easy way ?
So I attached a 512GB volume, improved the du followup with timestamps, and launched a new build.
This way, we’ll know how much space it will need !

smu44 · August 22, 2021, 2:32pm

Build done

The most disk space used was :

Aug 22 13:51:02 344G /srv
Aug 22 13:55:09 353G /srv
Aug 22 14:00:05 303G /srv

Corresponding entries in log (Docker is 2 hours behind my local clock) :

2021-08-22 11:54:52 - common.py - INFO : using prebuilt recovery.img from BOOTABLE_IMAGES…
2021-08-22 11:54:52 - common.py - INFO : Running: “brotli --quality=6 --output=/srv/src/Q/out/soong/.temp/tmpw7Htqs/system.new.dat.br /srv/src/Q/out/soong/.temp/tmpw7Htqs/system.new.dat”
2021-08-22 11:57:07 - common.py - INFO :
2021-08-22 11:57:09 - common.py - INFO : using prebuilt boot.img from BOOTABLE_IMAGES…

So, the most usage was at the end of the build process, when the system image was assembled.
That make sense

NB 1 : my CCACHE setting was the default 50G, after build the size of ccache directory is 11G …
NB 2 : the du command ran every 5 minutes, so real max size can be a little higher …

tcecyk · August 22, 2021, 6:51pm

I’m following along your disk space shortages to be prepared for my own build.

I want to link to [HOWTO] How much space do I need to build for /e/? as it already speaks of needing 390G combined, android official says 400G and quotes some numbers on machine size vs build time

smu44 · August 23, 2021, 6:06am

Please be aware that it was a 0.18 Q build for a3xelte; sizes may change for another device !
And obviously, the space needed increases with the versions of Android …

aibd · September 20, 2021, 8:29am

4 weeks later…

The larger hard drive did not provide the solution directly, but it did allow me to work on building a LineageOS ROM with breakfast and brunch side by side with Docker!

I eventually produced a Lineage ROM that ran nicely. For /e/ I followed the same method and overnight this ROM was delivered 0.19-q-20210919-UNOFFICIAL-a3xelte and it has just booted successfully.

What changed?

I read the repo script and in my imagination I thought my chaotic results could possibly result from repo being able to flip-flop between python2 and python 3.

I added this symlink

sudo ln -s /usr/bin/python3.8 /usr/bin/python

and most of the chaotic traces in my Lineage build disappeared and I could quite easily (during 3 weeks) resolve the issues.

Now that I search the errors I received I find that this Android page mentions the python2 python3 weakness and was updated only 5 days ago!

~~I will firm up these numbers but~~ I have a complete Lineage build occupying about 154G and ‘lean’ /e/ of 118G. (Detailed below)

If I went back to Docker I would check out

Dockers first reported error

A new version of repo (2.15) is available.
You should upgrade soon

As @Smu44 pointed out in post #15 that change would have to be made within docker; and

does Docker offer the possibility to use Python2 inappropriately?

Edit. An up to date repo command does exist as mentioned in post #15. It may/might be used during the build, but during repo init and repo sync, the “pre existing” repo will be used.

smu44 · September 20, 2021, 8:56am

Nice work !

The repo warning is to be ignored, as it is working fine.
However, one could build an updated Docker image, using the Docker file in there : https://gitlab.e.foundation/e/os/docker-lineage-cicd

The Docker build script manages the Python 2-3 switch, please see lines #200 & #303 : https://gitlab.e.foundation/e/os/docker-lineage-cicd/-/blob/master/src/build.sh

aibd · September 20, 2021, 9:10am

I did only see this warning in my above Android link, this morning:

Warning: Python 2 support was sunset on Jan 1, 2020 as detailed in Sunsetting Python 2. All major Linux distributions are deprecating support for the Python 2 package. Google strongly recommends that you migrate all your scripts over to Python 3.

Perhaps in my euphoria, I jumped to the wrong conclusion - lol.

Edit. My symlink of python to python3.8 seemed contrary (the late night I did it) knowing the previously_understood method included the use of python2 and python3.

[Edit to clarify]. The result that surprised me was that the Lineage build immediately resolved [having implemented this change] while the docker /e/ build [running its own unaltered system] continued with a repo sync which would not finish - which in my case seems to be the thing that breaks Docker on this machine.

smu44 · September 23, 2021, 6:49am

That’s the weird problem
Unfortunately, it’s difficult to know if it’s related to the Docker image, your host system, …
As you could download LOS source trees, your network may be out of cause.

You may want to try this method …

aibd · September 23, 2021, 8:25am

Hi @smu44 thanks for your reply, I think also part responsible that my machine (Intel i3 @ 2.40GHz, nproc=4, 8G RAM + 12G Swap) is probably at the bottom end of capable!

Another weird thing is the space used. I have 3 builds on two hard drives, here measuring with ncdu.

Docker /e/ unsuccessful

ncdu 1.13 /srv
372.1 GiB [##########] /src
2.3 MiB [ ] /logs
20.0 KiB [ ] /local_manifests
…Total disk usage: 372.1 GiB Apparent size: 370.9 GiB Items: 356359

Successful /e/ 0.19 UNOFFICIAL, complete with zip, but after any ‘clean’ made with

repo init -u https://gitlab.e.foundation/e/os/android.git -b v1-q --depth 1 --no-clone-bundle
repo sync -j 1 -c --fail-fast --no-clone-bundle, breakfast, brunch

ncdu 1.13 /media/iain/ebuild
50.6 GiB [##########] /out
32.9 GiB [###### ] /prebuilts
18.0 GiB [### ] /.repo
8.0 GiB [# ] /external
1.9 GiB [ ] /frameworks
…Total disk usage: 118.5 GiB Apparent size: 118.0 GiB Items: 1415020

Successful LineageOS 17.1, complete with zip, but after any ‘clean’ made with [no options] repo init, repo sync, breakfast, brunch

ncdu 1.13 /media/iain/android/lineage --------
54.6 GiB [##########] /.repo
47.3 GiB [######## ] /out
32.2 GiB [##### ] /prebuilts
8.0 GiB [# ] /external
3.5 GiB [ ] /vendor
…Total disk usage: 154.6 GiB Apparent size: 153.9 GiB Items: 1792990

Yes, that link is very nicely worded and has formed part of my education