Build /e/, docker, write a new local_manifest

Sorry, I forgot to mention : it was an “out of space” error.

Next tries : Pie build, Q build with minimum apps.
That won’t solve the problem, but as I couldn’t figure what’s changed (after some hours digging into source code …), it may help to understand.

Unfortunately, I’ll be running short on free time for some days on …

1 Like

I relaunched the build on Mon Aug 2 17:47:27 UTC 2021 and stopped it again arbitarilly on Tue Aug 3 15:46:02 UTC 2021. This is a direct follow on from my post #13, 2 August

Summary

The “e-partition” has grown to 275 G. This seems to me to be a potential blocker for a build to ever occur on a 373G partition. The growth of e/src/ is shown by a diff below (relative to the previous report.)

:$ df -h
/dev/sda4       373G  275G   79G  78% /rump

Launch command

:/rump/srv$ docker run \
-v "/rump/srv/e/src:/srv/src" \
-v "/rump/srv/e/zips:/srv/zips" \
-v "/rump/srv/e/logs:/srv/logs" \
-v "/rump/srv/e/ccache:/srv/ccache" \
-e "BRANCH_NAME=v0.17-q" \
-e "DEVICE_LIST=a3xelte" \
-e "REPO=https://gitlab.e.foundation/e/os/releases.git" \
registry.gitlab.e.foundation:5000/e/os/docker-lineage-cicd:community \

Build Trace

:~$ cd /rump/srv/
:/rump/srv$ docker run \
> -v "/rump/srv/e/src:/srv/src" \
> -v "/rump/srv/e/zips:/srv/zips" \
> -v "/rump/srv/e/logs:/srv/logs" \
> -v "/rump/srv/e/ccache:/srv/ccache" \
> -e "BRANCH_NAME=v0.17-q" \
> -e "DEVICE_LIST=a3xelte" \
> -e "REPO=https://gitlab.e.foundation/e/os/releases.git" \
> registry.gitlab.e.foundation:5000/e/os/docker-lineage-cicd:community \
> 
Set cache size limit to 50.0 GB
>> [Mon Aug  2 17:47:19 UTC 2021] Branch:  v0.17-q
>> [Mon Aug  2 17:47:19 UTC 2021] Devices: a3xelte,
>> [Mon Aug  2 17:47:19 UTC 2021] (Re)initializing branch repository
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 28177    0 28177    0     0  58096      0 --:--:-- --:--:-- --:--:-- 57977
[{"name":"v0.18-pie","message":"","target":"ec45b3ee099c65859217b8e764c90caad194fb54",{ - snip - }"protected":true}]

Branch name v0.17-q is a tag on e/os/releases, prefix with refs/tags/ for 'repo init'
>> [Mon Aug  2 17:47:26 UTC 2021] Copying '/srv/local_manifests/*.xml' to '.repo/local_manifests/'
>> [Mon Aug  2 17:47:27 UTC 2021] Syncing branch repository

This log is looking more like the ones I was seeing in my build run I commented on in post#4 July27 and post #6 31 July

/rump/srv/e/logs/repo-20210802.log

Stop docker
The process was stopped arbitarily after a further 22 hours syncing. I had now downloaded 275G no build has started.

:~$ sudo du -sxh /var/lib/docker/overlay2/
1.8G    /var/lib/docker/overlay2/

Inspect build and sizes

No build. No folder, no log, in zips/

This is a diff against the commands I ran last time, post #13, 2 August

Seems to show that the source tree is still growing in a logical way but why is it so big?

 :/rump/srv/e$ du -sBG src/ *
-208G +275G    src/
-206G +273G    project-objects

 :/rump/srv/e/src/Q/.repo/project-objects$ du -sBG *
-16G  +29G     e
-159G +203G    platform
-19G  +29G     TheMuppets
 
 :/rump/srv/e/src/Q/.repo/project-objects/platform$ du -sBG *

-122G +161G    prebuilts
-11G  +16G     tools
 
 :/rump/srv/e/src/Q/.repo/project-objects/platform/prebuilts$ du -sBG *
-11G   +15G     android-emulator.git
-19G   +28G     clang
-12G   +18G     gradle-plugin.git
-12G   +16G     jdk
-13G   +19G     qemu-kernel.git
-13G   +16G     sdk.git
-15G   +20G     tools.git
1 Like

In your repo.log there is a lot of errors involving a .lock file …
It usually happen when a repo sync has been aborted when running.
Also, a lot somehow network-related errors in pull (maybe due to the .lock errors, don’t know).
I’ll try to cleanup (rm -rf) platform/prebuilts/ndk

I also agree with your observations on the increase in size of the source tree.
Unfortunately I don’t know why this happens either :frowning:

Link to my logs : https://ecloud.global/s/gwC28Dp39AZsnGp (sorry files too big for hastebin).

1 Like

Thanks for the feedback on the log file.

I have been very uncertain about all the negatives you highlight. I have a cheapskate internet connection. NetworkManager always manages to reconnect - speculation, docker appears to run seamlessly, I cannot tell if some of these errors start from an interrupt.

Unfortunately, apart of the size problem, these errors will prevent the build to start.

I’ll advise (never tested) to remove manually .lock files, launch again Docker command.
And repeat this process until sync is complete and build is starting …

I can test something for you : fire up a CCX51 and try a build with it.
It will only cost me the price of a good beer, and we could see what gives a build with enough space and a reliable Internet link :wink:

I had been wondering about such an idea, but didn’t want to get into more unknowns before I had more of a grip on things! :slight_smile:

I will definitely try your proposed undo of locks, will be much later today.

1 Like

Building for a3xelte, v0.18 Pie succeeded, so the problem is related to Q builds …

Some details :


root@ubuntu:/srv# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb        344G  274G   53G  84% /srv
root@ubuntu:/srv# du -sxBG src/PIE/.repo/project-objects/*
1G      src/PIE/.repo/project-objects/device
11G     src/PIE/.repo/project-objects/e
1G      src/PIE/.repo/project-objects/kernel
5G      src/PIE/.repo/project-objects/LineageOS
152G    src/PIE/.repo/project-objects/platform
23G     src/PIE/.repo/project-objects/TheMuppets
3G      src/PIE/.repo/project-objects/The-Muppets
1G      src/PIE/.repo/project-objects/toolchain
2 Likes

Hi @smu44, well done! :sunny: Thank you so much for following through to this point, and for giving an idea of the space requirements. I also want to thank you for the many other clues and information you provided along the way. I will need to work out the best way to provide myself with a bigger working environment, and decide if I want to preserve my enormous, Q, sources. :slight_smile:

1 Like

Big thanks to those of you that do this work of building and maintaining. I wish I had your knowledge. Much of what you do is behind the scenes where little public recognition takes place. THANK YOU!

4 Likes

That was a good call, in that I got a clean repo log in my next sync run - but still no build started.

I stopped when the remaining available space was 54G in case I may be able to force a build and force the production of an error message!

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda4       373G  301G   54G  85% /rump

I will post the current state (relative to earlier stopping points) of used space for completeness.

:/rump/srv/e$ du -sBG src/ *
208G (275G) +301G    src/

:/rump/srv/e/src/Q/.repo$ du -sBG *
206G (273G) +299G    project-objects

:/rump/srv/e/src/Q/.repo/project-objects$ du -sBG *
16G (29G) +37G     e
159G (203G) +215G  platform
19G (29G) +35G     TheMuppets

:/rump/srv/e/src/Q/.repo/project-objects/platform$ du -sBG *
122G (161G ) +170G prebuilts
11G (16G) +19G     tools

:/rump/srv/e/src/Q/.repo/project-objects/platform/prebuilts$ du -sBG *
11G (15G) +16G     android-emulator.git
19G (28G) +30G     clang
12G (18G) +18G     gradle-plugin.git
12G (16G) +16G     jdk
13G (13G)  +15G    ndk.git
13G (19G ) +19G    qemu-kernel.git
13G (16G) +16G     sdk.git
15G (20G) +25G     tools.git

Also for completeness, the initial error, not mentioned in my OP, seems to amount to

Don’t have a product spec for: ‘lineage_a3xelte’
Do you have the right repo manifest?

If your repo sync is OK and you don’t have a custom manifest, it shouldn’t happen :frowning:
What gives a v0.17.1 build ?

I will definitively set-up a Hetzner temp build instance when I’ll have some free time, then give you access to.
This way, we could get a kind of reference !
Also, I’ll expand my personal building server, but it will take more time …

Exactly, and I installed a v0.17.1 nightly, very recently to compare it with /e/!

I had been thinking, perhaps I should do either a build from first principles or a lineage build first!

Now that I turn to the lineage/devices/a3xelte page (I know it is written from a template) I am quite surprised to find it includes extract proprietary blobs.

However, it does not explain how you have managed a v0.18 Pie build!

For /e/, I think that proprietary sources are stored here and here.
So I don’t understand why you have the missing product spec :frowning:
As I remember, I didn’t have such error with Q builds, only missing disk space …

For the Pie build, I simply ran the Docker command on a freshly formatted partition, no custom manifest or fancy stuff.

Of course you are correct, I am just doing my annoying thing of reflecting out loud, lol

I suspect that /e/ has taken this device “in house” in some way (recently?) - the device has had one problem or another for a long time with /e/.

In addition there might be a recent (related/unrelated) issue.

Not so sure here …
Remember : I had the same “out of space” problem when trying to build 0.18 Q for chiron.
It’s a device I’ve built for many times before, without problem.

1 Like

Bearing in mind the date we guess the problem started say Saturday 24th July I was trying to look for issues connected with default - latest build changing from v0.17-q to v0.18-q, as a recent (unrelated) issue affecting out of space. I think we may be looking for two culprits!

I / we have been given some clues here: https://gitlab.e.foundation/e/backlog/-/issues/3539#note_108871.

I am drawn to researching in what way this manifest might or might not work.

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
  <project name="LineageOS/android_device_samsung_a3xelte" path="device/samsung/a3xelte" remote="github" />
  <project name="LineageOS/android_device_samsung_universal7580-common" path="device/samsung/universal7580-common" remote="github" />
  <project name="LineageOS/android_hardware_samsung" path="hardware/samsung" remote="github" />
  <project name="LineageOS/android_hardware_samsung_slsi_exynos" path="hardware/samsung_slsi/exynos" remote="github" />
  <project name="LineageOS/android_hardware_samsung_slsi_exynos5" path="hardware/samsung_slsi/exynos5" remote="github" />
  <project name="LineageOS/android_hardware_samsung_slsi_exynos7580" path="hardware/samsung_slsi/exynos7580" remote="github" />
  <project name="LineageOS/android_hardware_samsung_slsi_openmax" path="hardware/samsung_slsi/openmax" remote="github" />
  <project name="LineageOS/android_kernel_samsung_universal7580" path="kernel/samsung/universal7580" remote="github" />
</manifest>

As far as I know, all Lineage official sources (along with some others) are downloaded automatically by the Docker script.

I succeeded to build Pie a3xelte without any custom manifest.
Once I find some time to expand my building VM storage, I’ll try to build Q the same way, and keep you posted.

1 Like

0.18 Q build for a3xelte succeeded :slight_smile:

Command used (no custom manifest :wink: ) :

docker pull registry.gitlab.e.foundation/e/os/docker-lineage-cicd:community
docker run \
-v "/srv/src:/srv/src" \
-v "/srv/zips:/srv/zips" \
-v "/srv/logs:/srv/logs" \
-v "/srv/ccache:/srv/ccache" \
-v "/srv/mirror:/srv/mirror" \
-e "BRANCH_NAME=v0.18-q" \
-e "DEVICE_LIST=a3xelte" \
-e "REPO=https://gitlab.e.foundation/e/os/releases.git" \
-e "ANDROID_JACK_VM_ARGS=-Dfile.encoding=UTF-8 -XX:+TieredCompilation -Xmx20G" \
registry.gitlab.e.foundation/e/os/docker-lineage-cicd:community

Build disk is 440GB big, but it was a typo : I wanted to use a 400GB disk :frowning:
If you want me to try again with a 400GB please let me know !

Also, do you want me to upload the ROM file somewhere for sharing ?

Logs here : https://ecloud.global/s/gwC28Dp39AZsnGp?path=%2Fa3xelte_Q_0.18

Some sizes after build (and it’s automatic cleaning) :

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb        433G  317G   94G  78% /srv
root@ubuntu:/srv# du -sxBG src/Q/*
1G      src/Q/android
0G      src/Q/Android.bp
1G      src/Q/art
1G      src/Q/bionic
1G      src/Q/bootable
0G      src/Q/bootstrap.bash
1G      src/Q/build
2G      src/Q/cts
1G      src/Q/dalvik
1G      src/Q/developers
1G      src/Q/development
1G      src/Q/device
9G      src/Q/external
2G      src/Q/frameworks
1G      src/Q/hardware
1G      src/Q/kernel
1G      src/Q/libcore
1G      src/Q/libnativehelper
1G      src/Q/lineage
1G      src/Q/lineage-sdk
1G      src/Q/Makefile
1G      src/Q/out
1G      src/Q/packages
1G      src/Q/pdk
1G      src/Q/platform_testing
33G     src/Q/prebuilts
1G      src/Q/sdk
1G      src/Q/system
1G      src/Q/test
1G      src/Q/toolchain
2G      src/Q/tools
43G     src/Q/vendor
root@ubuntu:/srv# du -sxBG src/Q/.repo/project-objects/*
1G      src/Q/.repo/project-objects/device
14G     src/Q/.repo/project-objects/e
1G      src/Q/.repo/project-objects/kernel
6G      src/Q/.repo/project-objects/LineageOS
158G    src/Q/.repo/project-objects/platform
28G     src/Q/.repo/project-objects/TheMuppets
9G      src/Q/.repo/project-objects/The-Muppets
1G      src/Q/.repo/project-objects/toolchain
root@ubuntu:/srv# du -sxBG src/Q/.repo/project-objects/platform/*
1G      src/Q/.repo/project-objects/platform/build
3G      src/Q/.repo/project-objects/platform/cts.git
1G      src/Q/.repo/project-objects/platform/dalvik.git
1G      src/Q/.repo/project-objects/platform/developers
16G     src/Q/.repo/project-objects/platform/external
2G      src/Q/.repo/project-objects/platform/frameworks
1G      src/Q/.repo/project-objects/platform/hardware
1G      src/Q/.repo/project-objects/platform/libnativehelper.git
1G      src/Q/.repo/project-objects/platform/packages
1G      src/Q/.repo/project-objects/platform/pdk.git
125G    src/Q/.repo/project-objects/platform/prebuilts
1G      src/Q/.repo/project-objects/platform/sdk.git
1G      src/Q/.repo/project-objects/platform/system
1G      src/Q/.repo/project-objects/platform/test
12G     src/Q/.repo/project-objects/platform/tools
root@ubuntu:/srv# du -sxBG src/Q/.repo/project-objects/platform/prebuilts/*
1G      src/Q/.repo/project-objects/platform/prebuilts/abi-dumps
15G     src/Q/.repo/project-objects/platform/prebuilts/android-emulator.git
1G      src/Q/.repo/project-objects/platform/prebuilts/asuite.git
1G      src/Q/.repo/project-objects/platform/prebuilts/bundletool.git
1G      src/Q/.repo/project-objects/platform/prebuilts/checkcolor.git
1G      src/Q/.repo/project-objects/platform/prebuilts/checkstyle.git
31G     src/Q/.repo/project-objects/platform/prebuilts/clang
2G      src/Q/.repo/project-objects/platform/prebuilts/clang-tools.git
1G      src/Q/.repo/project-objects/platform/prebuilts/devtools.git
1G      src/Q/.repo/project-objects/platform/prebuilts/fuchsia_sdk.git
2G      src/Q/.repo/project-objects/platform/prebuilts/gcc
1G      src/Q/.repo/project-objects/platform/prebuilts/gdb
4G      src/Q/.repo/project-objects/platform/prebuilts/go
6G      src/Q/.repo/project-objects/platform/prebuilts/gradle-plugin.git
3G      src/Q/.repo/project-objects/platform/prebuilts/jdk
1G      src/Q/.repo/project-objects/platform/prebuilts/ktlint.git
1G      src/Q/.repo/project-objects/platform/prebuilts/manifest-merger.git
2G      src/Q/.repo/project-objects/platform/prebuilts/maven_repo
4G      src/Q/.repo/project-objects/platform/prebuilts/misc.git
6G      src/Q/.repo/project-objects/platform/prebuilts/ndk.git
1G      src/Q/.repo/project-objects/platform/prebuilts/python
7G      src/Q/.repo/project-objects/platform/prebuilts/qemu-kernel.git
12G     src/Q/.repo/project-objects/platform/prebuilts/sdk.git
37G     src/Q/.repo/project-objects/platform/prebuilts/tools.git
1G      src/Q/.repo/project-objects/platform/prebuilts/vndk
1 Like

That is great! It was so good reading your logs with positive content that on a whim I ran you exact command into my root filesystem. It ran fine and I got the start of a more productive log. So I have formatted my partition again, super-cleaned docker, and started again.

Of interest your repo-log specifically includes
Downloading manifest from https://gitlab.e.foundation/e/os/releases.git
So do both of mine today! - this never appeared in any of my fail logs.

Of course I would be delighted to test-drive your ROM if you have an easy point to upload it to.