I see the compilation has a few hardcoded Xmx (java max heap space) values and some that are (or should be?) inherited from java/config/config.go. You can tweak them.
Seems to me like config.JavaCmd brings no Xms/Xmx config with itself at that version of Android 10 (later versions are refactored) at https://cs.android.com/android/platform/superproject/+/android-10.0.0_r41:build/soong/java/droiddoc.go;l=76 … so you can give it some upper bound yourself
build/soong/java/droiddoc.go line 76, change from red → to green
- `${config.JavaCmd} -jar ${config.MetalavaJar} -encoding UTF-8 -source $javaVersion @$out.rsp @$srcJarDir/list ` +
+ `${config.JavaCmd} -Xmx1024m -jar ${config.MetalavaJar} -encoding UTF-8 -source $javaVersion @$out.rsp @$srcJarDir/list ` +
(if you’d supply the Xmx param to javac instead of java it’d be -J-Xmx1024m, but I see no javac invocation at the command that is failing for you)