Build Error in SplitInstallService with PendingIntent.FLAG_MUTABLE

james.miller · September 22, 2022, 4:58pm

I have been building /e/ for several months now. My last successful build was on the 14th, and for the last couple of days it has failed at 98% complete. Naturally it would have to be failing almost at the end! The error is below.

FAILED: out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlin/SplitInstallService.jar
rm -rf "out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc/classes" "out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc/srcJars" "out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc-build.xml" "out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc/empty" && mkdir -p "out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc/classes" "out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc/srcJars" "out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc/empty" && out/soong/host/linux-x86/bin/zipsync -d out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc/srcJars -l out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc/srcJars/list -f "*.java" out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/gen/android/R.srcjar && build/soong/scripts/gen-kotlin-build-file.sh -classpath out/soong/.intermediates/libcore/mmodules/core_platform_api/core.platform.api.stubs/android_common/turbine-combined/core.platform.api.stubs.jar:out/soong/.intermediates/libcore/core-lambda-stubs/android_common/turbine-combined/core-lambda-stubs.jar:out/soong/.intermediates/frameworks/base/ext/android_common/turbine-combined/ext.jar:out/soong/.intermediates/frameworks/base/framework/android_common/turbine-combined/framework.jar:out/soong/.intermediates/packages/services/SplitInstallService/lib/splitinstall-lib/android_common/turbine-combined/splitinstall-lib.jar:out/soong/.intermediates/external/kotlinc/kotlin-stdlib/android_common/combined/kotlin-stdlib.jar:out/soong/.intermediates/external/kotlinc/kotlin-stdlib-jdk7/android_common/combined/kotlin-stdlib-jdk7.jar:out/soong/.intermediates/external/kotlinc/kotlin-stdlib-jdk8/android_common/combined/kotlin-stdlib-jdk8.jar "packages__services__SplitInstallService__service__android_common__SplitInstallService" out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc/classes out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlin/SplitInstallService.jar.rsp out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc/srcJars/list > out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc-build.xml &&external/kotlinc/bin/kotlinc -J-Xmx2048M -no-stdlib -no-jdk -jvm-target 1.8 -Xbuild-file=out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc-build.xml -kotlin-home out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc/empty && out/soong/host/linux-x86/bin/soong_zip -jar -o out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlin/SplitInstallService.jar -C out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc/classes -D out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc/classes && rm -rf "out/soong/.intermediates/packages/services/SplitInstallService/service/SplitInstallService/android_common/kotlinc/srcJars"
packages/services/SplitInstallService/service/java/foundation/e/splitinstall/service/SplitInstallBinder.kt:55:76: error: unresolved reference: FLAG_MUTABLE
            val flags = PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_MUTABLE
                                                                           ^
[ 98% 115843/117231] //packages/apps/Seedvault:Seedvault kotlinc [common]
warning: some JAR files in the classpath have the Kotlin Runtime library bundled into them. This may cause difficult to debug problems if there's a different version of the Kotlin Runtime library in the classpath. Consider removing these libraries from the classpath
out/soong/.intermediates/frameworks/base/packages/SettingsLib/SettingsLib/android_common/turbine-combined/SettingsLib.jar: warning: library has Kotlin runtime bundled into it
[ 98% 115844/117231] //frameworks/base/packages/SystemUI:SystemUI r8 [common]
Invalid descriptor (deserialized from Kotlin @Metadata): (LLandroid/animation/Animator;;)L;
Invalid descriptor (deserialized from Kotlin @Metadata): (LLandroid/animation/Animator;;)L;
[ 98% 115845/117231] Target Java: out/target/common/obj/APPS/Dialer_intermediates/classes-full-debug.jar
Note: Generating a Provider for com.android.dialer.glidephotomanager.impl.GlidePhotoManagerImpl. Prefer to run the dagger processor over that class instead.
Note: [1] Wrote GeneratedAppGlideModule with: []
09:15:02 ninja failed with: exit status 1

#### failed to build some targets (04:26:39 (hh:mm:ss)) ####

For more information I am building for the Pixel 4a_5g (bramble).

From the looks of it it appears to be related to this recent commit but I don’t understand why FLAG_MUTABLE would be an unresolved reference?
https://gitlab.e.foundation/e/os/android_packages_services_SplitInstallService/-/commit/0aa0e744c8075389e58173f1f7f419da8ebd9a3b

If this is related to this commit or not, how do I resolve this?

Regain your privacy! Adopt /e/ the unGoogled mobile OS and online services phone

aibd · September 22, 2022, 6:57pm

Are you using docker and did you already rerun the process?

Metalava is mentioned, and this happens just before at 97%, the usual stopping point due to out of memory “Java memory heap”. This is a contained space where Java can run freely, the memory heap error is seen when the heap is too small, even though all memory appears running nicely.

Just a guess but if your machine is running through all available threads on your machine it can try to work through a critical error, and fail to stop at the first error.

When you rerun from the start, it will take some time verifying everything from the start but it will only slow down when it finds its errors.

The log mentions difficult to debug, so I don’t think I would resort to unpatching, until I had rerun twice.

Also perhaps check the log for errors around 97% that might have been overlooked by the build process…

james.miller · September 22, 2022, 7:58pm

Thanks for the reply!

I am using the docker scripts, but running them directly instead of through the docker image. As shown here but using the most recent docker scripts from e’s gitlab.

I dont see anything but a few warnings at 96%-97%. My machine has 32GB of ram so that out to be enough… I have ran this about 3 times over the last 2ish days - all stopping in the same spot with the same error. It was running fine for multiple builds last week so I think my environment is setup correctly.

For now I have rolled back that recent change and am running a build again. Not sure if that will do anything or not but I am still learning!

aibd · September 22, 2022, 8:25pm

Sure 32G is a lot more than I manage with at 8G +16G swap!

Another thing that matters is the cores / threads on the machine.
If I run nproc then n=4
This is confirmed by cat /proc/cpuinfo n=number of cores=4.

It might be worth checking if your scripts by default run using the -j option set at -jn or perhaps -jn-1

I have found the best compromise n/2 (for symmetry!) so I specify -j2.

All I am doing is talking about the easy things first!

james.miller · September 22, 2022, 8:49pm

In the one script mentioned above I have added these lines to get it to work. Which if I understand right I am limiting it to 4GB? For most of the build I dont see it using much, but there are times that my ram usage goes way up and I have not figured out how that correlates with these settings as it uses way more than 4GB at that point.

# Provide a default JACK configuration in order to avoid out-of-memory issues
export ANDROID_JACK_VM_ARGS="-Dfile.encoding=UTF-8 -XX:+TieredCompilation -Xmx4G"
export _JAVA_OPTIONS="-Xmx4G"

As far as cpu goes I am running an i7-6700k so 4 core, 4 thread. It uses all 8 threads at 100% for the build process…

I see in the main build script on line 123 that the repo sync command is using all cores, but the actual build process does as well when I look at cpu usage. Not sure where in this build.sh you were referring to using the -j option? Somewhere in the lines 240-273?

aibd · September 22, 2022, 9:28pm

line 126
repo_out=$(repo sync -c -j$(nproc --all) --force-sync 2>&1 > /dev/null)

line 199
if ! repo sync -c -j$(nproc --all) --force-sync; then

I think --all is too high (edit, for my machine). It will be fine if the machine is strong enough, so just guessing. Try n-1 or n/2 and see what happens.

Is 8 the number you see when you run nproc ?

If you start a rerun, are you finding it quite fast?

Do you get to a point with repo sync where you see that the sync is reported successful?

I do not like (as docker run seems to do by default on my machine) to run the ongoing build without seeing that.

I just stick -j2 on every repo command! If it doesn’t like it it will complain!

When I use brunch I am afraid I don’t quite remember where I put it – sometimes it complains so I do another either brunch bacon -j2 or brunch -j2 bacon or maybe brunch -j 2 bacon. What ever works.

I tend to rely on repo help rather than my memory.

I happen to use: export _JAVA_OPTIONS="-Xmx9G"

Just by watching the monitor (conky). If it is too large Java will overwhelm the whole process, if it is too low Java will run out of memory. On my machine is seems significant how to optimise swap – it is less volatile, so needs to be carrying a little below half load already when Java spikes strike at 97%.

Edit.
I just noticed your script uses --force by default. I do not do that till I know what I want to fix.

In fact I do the opposite and --fail-fast

I work from [HOWTO] build /e/ the (full) classic way (no docker, no scripts, just 'make')
with its
README

Another edit.

Is your script aware of

After completing a repo sync, you will have to run repo forall -c ‘git lfs pull’

So you have to have git lfs installed already and patchelf. I didn’t mention this because if there was a fault here it would have shown up long before at maybe 80%. This was actually the first thought that jumped into my mind with that error.

https://www.atlassian.com/git/tutorials/git-lfs#installing-git-lfs

james.miller · September 22, 2022, 11:01pm

Both of these are just for syncing, but the error I ran into the during the build process right?

Yes 8 is what I see.

I don’t actually. I find it takes about 4.5 hours every time to build and I dont understand why? But I also don’t know how long it took the very first time to build so I dont know what is “normal” so to speak. I have cache enabled… I have wondered if it has something to do with this script cleaning house after every build? Below is the final lines of every build.


>> [Tue Sep 20 01:13:54 PM PDT 2022] Failed build for bramble
>> [Tue Sep 20 01:13:54 PM PDT 2022] Finishing build for bramble
>> [Tue Sep 20 01:13:54 PM PDT 2022] Cleaning source dir for device bramble
13:14:28 Entire build directory removed.

#### build completed successfully (34 seconds) ####

After seeing the above I think I definitely need to bump my settings up!

I just ran across this earlier today and was thinking I need to try this out. I looked through it quick but there were several things that I didnt understand so I need to go back and digest it/try it out.

I was not aware of this. I just pulled this script from e’s gitlab and have been using it trying to learn along the way. I did learn about git lfs the hard way (all the prebuilts were not syncing) and got that installed. It does look like that is in the build.sh script though.

  if [ "$sync_successful" = true ]; then
    repo forall -c 'git lfs pull'
  fi

And SUCCESS!! The build succeeded after changing that value as outlined in the commit I had mentioned in the first post. (FLAG_MUTABLE changed to FLAG_IMMUTABLE)

aibd · September 22, 2022, 11:55pm

Well done, you were right from the start, but it is how we all learn, by making and correcting errors.

Correct as your result shows. You searched the error and pinpointed the correct solution. (I was protecting against the build starting before full repo sync had finished successfully)

I have wondered if it has something to do with this script cleaning house after every build?

I would say that is good practice to prevent unknown errors from building up. The build directory is very small.

$ ncdu build/
    7.4 MiB [##########] /make                                                  
    4.9 MiB [######    ] /soong
    2.2 MiB [###       ] /kati
  992.0 KiB [#         ] /bluepri

and some small files.

I have read that Android 10 does not use Jack any more. I don’t know how docker deploys it. I like to see it in the log so I put 'export _JAVA_OPTIONS=-Xmx9g` in ~/.bashrc as well. Not saying I am right but I don’t know how to deploy Jack!

It does look like that is in the build.sh script though.

Good, so no problems – enjoy your build!

Re length of time I am used to 9 - 13 hours, but that is just the build. My clock time as I break everything down to --fail-fast in the repo sync will depend how often I come back and look at it!
After repo init I do
repo sync -j2 -c --fail-fast --no-clone-bundle perhaps I would run --force-sync in a second run if I know what the problem is, and maybe only force that item. This gives focus on the error.

repo forall -c 'git lfs pull'

I like to see that all confirmed complete before I start the build.

Then when I run
source build/envsetup.sh and
breakfast bacon
I get to see exactly what I am going to be building.

If (for fun) you just repo sync tomorrow in a clean environment, you can watch your monitor and see how efficiently you feel the cores are running at say -j4 and -j6 -j7. Of course when you repeat a build, there will be no doubts, so then --all would not be wrong. Especially as it has always worked for you and this issue was totally new.

The -j is just about the number of jobs you throw at the CPU at once. If you set -j4 , all the cores will work their best to get those 4 jobs done and move on to the next quicker without tripping over each other and possibly going back to make good what the other core missed because it wasn’t there for him in time! (If you will forgive the poor analogy)

james.miller · September 23, 2022, 10:24pm

OK that is good to know. I was worried it was clearing the cache and thereby extending build times.

Interesting about Jack. Those exports are included in the beginning script to export those variables. I am playing around with changing those values so will see how that goes, if it affects anything!

OK wow I guess my build times are pretty short then! For me the repo sync only takes a few minutes, then it starts the build process and that is what takes the time.

I did learn this morning that I can modify the number of cores used during the build process by using the following command before. I ran it with 0-6 leaving one thread open and continued to use my pc, although I didnt notice any difference with when I didnt run this. I have been amazed at how responsive the system is even when the cpu is pegged at 100% with the build process.
taskset -c 0-3 bash

https://www.reddit.com/r/LineageOS/comments/elhze3/number_of_threads_used_in_brunch_j_x_option/

Thanks again for the help, I really appreciate it!

aibd · September 25, 2022, 5:54am

Thanks for that … I later realised why I could not remember this detail … using the vendor-e method which I linked in Post #6 I can add

export EOS_MAX_THREADS=2

to vendorsetup.sh … then I find the envsetup.sh now reports the build will happen in -j2 … also with that method I can use mkeos for the build instead of brunch if I want.

(I guess docker users would be able to pass an environment variable do the same job.)