List items
Items from the current list are shown below.
Newpipe
15 Mar 2025 : Day 6 #
Yesterday we looked at the code needed to expose the Java internals of NewPipe Extractor to C++ when using GraalVM in a way that would allow us to pass richer datatypes and structures between them.
Today we're taking another slight detour, this time to find out whether it's possible to build the Java library using the Sailfish SDK, rather than on the phone itself. You'll recall we looked at building the library on-phone on Day 4.
Before getting in to things I'm going to update my SDK. I want the absolute latest tooling and targets to maximise the chances of success.
Even though I've set a default target, when entering scratchbox2 manually like this, we also need to specify the target explicitly.
So instead I'm going to output it on my laptop with a JVM installed, outside of scratchbox2. Here are the relevant parts of the help text output:
With the compressed class space flags added we get some slightly more nuanced output. For example if we use a memory value that's too low we get a segmentation fault:
Increasing the memory to nearly a gigabyte leaves us with similar output:
Maybe as we continue our journey a new path will open up, maybe someone out there has an idea for something to try (in which case, please do let me know!), or maybe I'll muster up the courage to try to better understand and fix the underlying issue. But for now, it means reverting to our fallback of executing GraalVM on the phone.
That's it for today. Tomorrow we move into new territory: attempting to get NewPipe Extractor built using GraalVM!
Today we're taking another slight detour, this time to find out whether it's possible to build the Java library using the Sailfish SDK, rather than on the phone itself. You'll recall we looked at building the library on-phone on Day 4.
Before getting in to things I'm going to update my SDK. I want the absolute latest tooling and targets to maximise the chances of success.
$ ./SDKMaintenanceTool --silentUpdate -v IFW Version: 3.2.3, built with Qt 5.15.14. Build date: Aug 21 2024 Installer Framework SHA1: 7699eb32 [0] Language: en-GB [...] [1231] Install size: 3 components [...] [144072] 100% ... [...] [239236] Sync completed\n [245881] Target 'SailfishOS-5.0.0.55EA-aarch64' set up\n [245899] Done [...] [246115] Stopping the build engine… (this may take some time) [248920] Components updated successfully.Great! That got all of the updates. Let's see what we have.
$ sfdk tools target list sfdk: [I] Starting the build engine… SailfishOS-3.4.0.24-aarch64 sdk-provided SailfishOS-3.4.0.24-armv7hl sdk-provided SailfishOS-3.4.0.24-i486 sdk-provided SailfishOS-4.6.0.13-aarch64 sdk-provided,latest SailfishOS-4.6.0.13-armv7hl sdk-provided,latest SailfishOS-4.6.0.13-i486 sdk-provided,latest SailfishOS-5.0.0.55EA-aarch64 sdk-provided,early-accessI'll be using the new SailfishOS-5.0.0.55EA-aarch64 target, so the next step is to configure the SDK to use this by default.
$ sfdk config --global target=SailfishOS-5.0.0.55EA-aarch64 $ sfdk config # ---- command scope --------- # <clear> # ---- session scope --------- # <clear> # ---- global scope --------- target = SailfishOS-5.0.0.55EA-aarch64 output-prefix = ~/RPMS device = kolbeNext I need to install the GraalVM tooling inside the SDK. There are two layers to the SDK: first of all you enter the tooling using the sfdk command, following which you enter the scratchbox2 target using the sb2 command.
Even though I've set a default target, when entering scratchbox2 manually like this, we also need to specify the target explicitly.
$ sfdk engine exec $ sb2 -t SailfishOS-5.0.0.55EA-aarch64 $ ls compile.sh flatbuffers.sh mvnw pom.xml src $ mkdir graalvm $ pushd graalvm/ graalvm sailing-the-flood-to-java/java-part $ curl -O https://download.oracle.com/graalvm/23/latest/ graalvm-jdk-23_linux-aarch64_bin.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 334M 100 334M 0 0 4245k 0 0:01:20 0:01:20 --:--:-- 4457k $ tar -xf graalvm-jdk-23_linux-aarch64_bin.tar.gz $ GRAAL=${PWD}/graalvm-jdk-23.0.2+7.1 $ mkdir m2 $ MAVENLOCAL=${PWD}/m2 $ popd sailing-the-flood-to-java/java-partThat's the tooling set up, now let's try building as if we were running the command on a phone:
$ GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./mvnw -Dmaven.repo.local=$MAVENLOCAL \ clean install -Pnative Error occurred during initialization of VM Could not reserve enough space for 8118272KB object heapThis isn't totally unexpected: the compiler hit a memory limit reserving memory for the job. The problem here isn't Maven, out build tool, but rather the Java Virtual Machine that it's attempting to spawn. So for testing purposes we can jump straight to that and see if we have better luck if we call it directly.
$ GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL graalvm/graalvm-jdk-23.0.2+7.1/bin/java \ --help Error occurred during initialization of VM Could not reserve enough space for 8118272KB object heapThe same error. Thankfully there are some ways to control the amount of memory that the JVM will try to claim. Unfortunately we can't output the help text using java --help inside scratchbox2 (the SDK engine target) itself because apparently the JVM has to get up and running before it'll even print out the help.
So instead I'm going to output it on my laptop with a JVM installed, outside of scratchbox2. Here are the relevant parts of the help text output:
$ java --help-extra | grep size -Xmn<size> sets the initial and maximum size (in bytes) of the heap -Xms<size> set initial Java heap size -Xmx<size> set maximum Java heap size -Xss<size> set java thread stack sizeOn top of these we also need to disable Compressed Class Space using the -XX:-UseCompressedClassPointers and -XX:+UseCompressedOops flags. These aren't listed in the Java man pages; nor are they listed in either the --help or --help-extra output. But without these we simply get an Out-Of-Memory error:
$ JVMMEM=512m _JAVA_OPTIONS="-Xmn${JVMMEM} -Xms${JVMMEM} -Xmx${JVMMEM} \ -Xss${JVMMEM}" GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./mvnw \ -Dmaven.repo.local=$MAVENLOCAL clean install -Pnative Picked up _JAVA_OPTIONS: -Xmn512m -Xms512m -Xmx512m -Xss512m Error occurred during initialization of VM Could not allocate compressed class space: 1073741824 bytesNotice here that I'm setting the flags using _JAVA_OPTIONS. That's so that they get automatically picked up by calls to use the JVM even though the JVM commands are being called by maven. Notice also that we get some output stating that the arguments have been picked up, so we know this approach is working.
With the compressed class space flags added we get some slightly more nuanced output. For example if we use a memory value that's too low we get a segmentation fault:
$ JVMMEM=128m _JAVA_OPTIONS="-Xmn${JVMMEM} -Xms${JVMMEM} -Xmx${JVMMEM} \ -Xss${JVMMEM} -XX:-UseCompressedClassPointers -XX:-UseCompressedOops" \ GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./mvnw -Dmaven.repo.local=$MAVENLOCAL \ clean install -Pnative Picked up _JAVA_OPTIONS: -Xmn128m -Xms128m -Xmx128m -Xss128m -XX:-UseCompressedClassPointers -XX:-UseCompressedOops Segmentation fault (core dumped)Raise it higher, to around 768 MiB and it looks like the JVM is managing to get further through its initialisation sequence:
$ JVMMEM=768m _JAVA_OPTIONS="-Xmn${JVMMEM} -Xms${JVMMEM} -Xmx${JVMMEM} \ -Xss${JVMMEM} -XX:-UseCompressedClassPointers -XX:-UseCompressedOops" \ GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./mvnw -Dmaven.repo.local=$MAVENLOCAL \ clean install -Pnative Picked up _JAVA_OPTIONS: -Xmn768m -Xms768m -Xmx768m -Xss768m -XX:-UseCompressedClassPointers -XX:-UseCompressedOops [0.773s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 786432k, guardsize: 0k, detached. [0.782s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Finalizer" Error occurred during initialization of VM java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached at java.lang.Thread.start0(java.base/Native Method) at java.lang.Thread.start(java.base/Thread.java:1518) at java.lang.ref.Finalizer.startFinalizerThread(java.base/ Finalizer.java:190) at java.lang.ref.Reference$1.startThreads(java.base/Reference.java:319) at java.lang.System.initPhase1(java.base/System.java:2214)This error seems to happen when the JVM fails to start threads, but I'm not able to fix this by increasing the thread limit using ulimit -u. So I suspect this may actually be memory related as well.
Increasing the memory to nearly a gigabyte leaves us with similar output:
$ JVMMEM=1020m _JAVA_OPTIONS="-Xmn${JVMMEM} -Xms${JVMMEM} -Xmx${JVMMEM} \ -Xss${JVMMEM} -XX:-UseCompressedClassPointers -XX:-UseCompressedOops" \ GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./mvnw -Dmaven.repo.local=$MAVENLOCAL \ clean install -Pnative Picked up _JAVA_OPTIONS: -Xmn1020m -Xms1020m -Xmx1020m -Xss1020m -XX:-UseCompressedClassPointers -XX:-UseCompressedOops [0.769s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1044480k, guardsize: 0k, detached. [0.777s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Reference Handler" Error occurred during initialization of VM java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached at java.lang.Thread.start0(java.base/Native Method) at java.lang.Thread.start(java.base/Thread.java:1518) at java.lang.ref.Reference.startReferenceHandlerThread(java.base/ Reference.java:306) at java.lang.ref.Reference$1.startThreads(java.base/Reference.java:318) at java.lang.System.initPhase1(java.base/System.java:2214)But as the configuration reaches a gigabyte, the original error returns:
$ JVMMEM=1021m _JAVA_OPTIONS="-Xmn${JVMMEM} -Xms${JVMMEM} -Xmx${JVMMEM} \ -Xss${JVMMEM} -XX:-UseCompressedClassPointers -XX:-UseCompressedOops" \ GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./mvnw -Dmaven.repo.local=$MAVENLOCAL \ clean install -Pnative Picked up _JAVA_OPTIONS: -Xmn1021m -Xms1021m -Xmx1021m -Xss1021m -XX:-UseCompressedClassPointers -XX:-UseCompressedOops Error occurred during initialization of VM Could not reserve enough space for 1046528KB object heapI also tried performing similar actions using java directly rather than going via Maven, but with very similar results:
$ JVMMEM=1020m _JAVA_OPTIONS="-Xmn${JVMMEM} -Xms${JVMMEM} -Xmx${JVMMEM} \ -Xss${JVMMEM} -XX:-UseCompressedClassPointers -XX:-UseCompressedOops" \ GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL \ graalvm/graalvm-jdk-23.0.2+7.1/bin/java --help Picked up _JAVA_OPTIONS: -Xmn1020m -Xms1020m -Xmx1020m -Xss1020m -XX:-UseCompressedClassPointers -XX:-UseCompressedOops [0.796s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1044480k, guardsize: 0k, detached. [0.805s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Finalizer" Error occurred during initialization of VM java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached at java.lang.Thread.start0(java.base/Native Method) at java.lang.Thread.start(java.base/Thread.java:1518) at java.lang.ref.Finalizer.startFinalizerThread(java.base/ Finalizer.java:190) at java.lang.ref.Reference$1.startThreads(java.base/Reference.java:319) at java.lang.System.initPhase1(java.base/System.java:2214)This is all rather sad. When using scratchbox2 the JVM is being executed inside a QEMU container. It's possible there's a 32 bit/64 bit problem here, or that there's some deeper incompatibility (it has been known for things to simply fail when run within QEMU). I'd really hoped that this may have changed or been fixed since I last tried this back in 2022, but sadly that's not the case. What's more, I'm not convinced I have the skill or knowledge to fix it myself.
Maybe as we continue our journey a new path will open up, maybe someone out there has an idea for something to try (in which case, please do let me know!), or maybe I'll muster up the courage to try to better understand and fix the underlying issue. But for now, it means reverting to our fallback of executing GraalVM on the phone.
That's it for today. Tomorrow we move into new territory: attempting to get NewPipe Extractor built using GraalVM!
Comments
Uncover Fediverse comments