flypig.co.uk

Yesterday I started trying to get the NewPipe Extractor build to complete on my phone. Unfortunately I hit some issues with resources. I attempted to fix this by reducing the maximum number of allowed threads to one, but this caused the build to stall. So I then set a build running overnight allowing a maximum of two threads.

Unfortunately this resulted in my phone rebooting during the night. Presumably the phone became unresponsive and the Sailfish OS watchdog stepped in to trigger the reboot.

$ ./compile.sh 
transfering data
~/Documents/Development/projects/newpipe/NewPipeExtractor
    ~/Documents/Development/projects/newpipe
skipping non-regular file "graalvm/graalvm-jdk-23.0.2+7.1/bin/
    native-image"
skipping non-regular file "graalvm/graalvm-jdk-23.0.2+7.1/bin/
    native-image-configure"
~/Documents/Development/projects/newpipe
starting compilation
Starting a Gradle Daemon (subsequent builds will be faster)
> Task :timeago-parser:compileJava
[...]
===============================================================================
GraalVM Native Image: Generating 'appwrapper' (shared library)...
===============================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/
    BuildOutput.md
-------------------------------------------------------------------------------
[1/8] Initializing...                                          (33.0s @ 0.13GB)
[...]
-------------------------------------------------------------------------------
 1 experimental option(s) unlocked:
 - '-H:DeadlockWatchdogInterval' (origin(s): command line)
-------------------------------------------------------------------------------
Build resources:
 - 4.06GB of memory (75.6% of 5.38GB system memory, determined at start)
 - 2 thread(s) (25.0% of 8 available processor(s), set via '--parallelism=2')
[2/8] Performing analysis...  [*******]                       (408.5s @ 2.62GB)
   21,377 reachable types   (93.0% of   22,980 total)
   34,005 reachable fields  (53.3% of   63,823 total)
  116,082 reachable methods (69.4% of  167,276 total)
   11,524 runtime compiled methods ( 6.9% of  167,276 total)
    6,289 types,    70 fields, and   902 methods registered for reflection
       83 types,    61 fields, and   192 methods registered for JNI access
        4 native libraries: dl, pthread, rt, z
client_loop: send disconnect: Broken pipe

My phone isn't just a phone, or just a development tool. It's also my alarm clock! But thankfully it didn't prevent the alarms from triggering this morning to wake me up!

This leaves us in a bit of a quandary. There's no intermediate ground between one and two threads, so we'll have to stick to the two thread limit if we're going to make progress. We need something else to control.

It would also be useful to try to better understand the reason the build is triggering a reboot. My suspicion is that it's a memory issue: the build is using more memory than the available RAM. But it would be good to confirm this using some measurements.

Here's the status of my memory while the phone is idling:

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           5.4G        2.1G      964.9M       66.4M        2.4G        3.6G
Swap:       1024.0M      475.0M      549.0M

Increasing swap size might be a way to approach this, but in the first instance I'm just going to try compiling using various different flags to set different memory limits. At the same time I plan to capture data about memory usage to try to get an idea of whether the flags are actually making any difference, as well as to try to understand what sort of memory usage we're looking at.

I've also updated the gradle build command so it now includes the verbose flag. I'm hoping this will output not just progress but also the exact commands being used for java and the native-image calls. Here's what I'm using now:

ssh $COMPILEHOST "cd $COMPILEHOST_WORKSPACE; \
    GRAALVM_HOME=$COMPILEHOST_GRAAL JAVA_HOME=$COMPILEHOST_GRAAL \
    ./gradlew nativeCompile" --console verbose

And here are the additional flags in the graalvmNative section of my build.gradle build configuration. As you can see, I've added the verbose flag here as well, plus the Xmx and Xms flags for setting the maximum and initial memory allocations respectively.

$ git diff
diff --git a/appwrapper/build.gradle b/appwrapper/build.gradle
index 39a9503e..c0bd305e 100644
--- a/appwrapper/build.gradle
+++ b/appwrapper/build.gradle
@@ -35,6 +35,12 @@ graalvmNative {
         buildArgs.add('-H:+AddAllCharsets')
         // Enable network protocols
         buildArgs.add('--enable-url-protocols=http,https')
+        buildArgs.add('--parallelism=2')
+        buildArgs.add('-H:DeadlockWatchdogInterval=0')
+        buildArgs.add('-H:+UnlockExperimentalVMOptions')
+        buildArgs.add('--verbose')
+        buildArgs.add('-J-Xmx256m')
+        buildArgs.add('-J-Xms128m')
     }
 }

Back when I was working on Gecko I also had to capture memory usage and ended up using psrecord for this. It's a Python utility which can easily be installed in a virtual environment, and which can then be attached to a process to capture CPU and memory usage. Here's how I've installed it directly on my phone:

$ python3 -m venv venv
$ . ./venv/bin/activate
$ pip install --upgrade pip
$ pip install psrecord matplotlib
$ psrecord --plot mem-gradle.png --interval 0.2 --include-children <PID>

There are multiple ways to attach psrecord to a process, but I've decided to go for a manual approach. Once the gradle build has started I can find the process using ps and attach psrecord to this so as to capture resource usage. Because gradle will spawn a bunch of other tools I've added the include-children flag to the psrecord command:

$ ps aux | grep gradle
22624 defaultu /home/defaultuser/Documents/Development/newpipe/graalvm/
    graalvm-jdk-23.0.2+7.1/bin/java -Xmx64m -Xms64m 
    -Dorg.gradle.appname=gradlew -classpath /home/defaultuser/Documents/
    Development/newpipe/NewPipeExtractor/gradle/wrapper/gradle-wrapper.jar 
    org.gradle.wrapper.GradleWrapperMain nativeCompile --console verbose
22664 defaultu grep gradle
$ psrecord --plot mem-gradle.png --interval 0.2 --include-children 22624
Attaching to process 22624

For my first attempt I set the memory to a maximum of 256 MiB. This is very low, but at this level with two threads it should at least prevent my phone from crashing.

$ ./compile.sh 
[...]
-------------------------------------------------------------------------------
 1 experimental option(s) unlocked:
 - '-H:DeadlockWatchdogInterval' (origin(s): command line)
-------------------------------------------------------------------------------
Build resources:
 - 0.22GB of memory (4.1% of 5.38GB system memory, set via '-Xmx256m')
 - 2 thread(s) (25.0% of 8 available processor(s), set via '--parallelism=2')
Terminating due to java.lang.OutOfMemoryError: Java heap space
The Native Image build process ran out of memory.
Please make sure your build system has more memory available.

[...]
veimage.driver/com.oracle.svm.driver.NativeImage.performBuild
    (NativeImage.java:1847)
        at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.main
            (NativeImage.java:1829)
        at java.base@23.0.2/java.lang.invoke.LambdaForm$DMH/sa346b79c
            .invokeStaticInit(LambdaForm$DMH)

FAILURE: Build failed with an exception.
[...]

With these numbers the build fails pretty swiftly with the advice "Please make sure your build system has more memory available". The actual error message is:

veimage.driver/com.oracle.svm.driver.NativeImage.performBuild
    (NativeImage.java:1847)
        at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.main
            (NativeImage.java:1829)
        at java.base@23.0.2/java.lang.invoke.LambdaForm$DMH/sa346b79c
            .invokeStaticInit(LambdaForm$DMH)

Let's take a look at the graph generated by psrecord for memory usage.
 

As we can see the memory usage gradually increases through the 300 second run. It peaks at about 1280 MiB, which seems high considering we have two threads capped at 256 MiB, but not completely absurdly so. Presumably there's a bunch of other overhead that's consuming the remaining 768 MiB.

The memory drops down abruptly just before the 300 second mark. That's when the build process crashes.

At least now the phone isn't running out of memory, rather it's that the Java build process is running out of memory inside the constrained JVM.

So I'll have to try increasing the JVM memory and give it another go. Before I do that it's worth checking the command that was used, which is now visible in the output after the inclusion of the verbose flog.

The command executed is incredibly long so I've abridged it slightly, while attempting to retain the most relevant parts.

Executing [
HOME=/home/defaultuser \
PATH=/usr/local/bin:/bin:/usr/bin \
PWD=/home/defaultuser/Documents/Development/newpipe/NewPipeExtractor \
USE_NATIVE_IMAGE_JAVA_PLATFORM_MODULE_SYSTEM=true \
/home/defaultuser/Documents/Development/newpipe/graalvm/graalvm-jdk-23.0.2+7.1/
    bin/java \
-XX:+UseParallelGC \
-XX:+UnlockExperimentalVMOptions \
-XX:+EnableJVMCI \
-Dtruffle.TrustAllTruffleRuntimeProviders=true \
-Dtruffle.TruffleRuntime=com.oracle.truffle.api.impl.DefaultTruffleRuntime \
-Dgraalvm.ForcePolyglotInvalid=true \
-Dgraalvm.locatorDisabled=true \
[...]
-XX:+UseJVMCINativeLibrary \
-Xss10m \
-XX:MaxRAMPercentage=84.99999999999999 \
-XX:GCTimeRatio=9 \
-XX:+ExitOnOutOfMemoryError \
-Djava.awt.headless=true \
'-Dorg.graalvm.vendor=Oracle Corporation' \
-Dorg.graalvm.vendorurl=https://www.graalvm.org/ \
'-Dorg.graalvm.vendorversion=Oracle GraalVM 23.0.2+7.1' \
-Dorg.graalvm.version=24.1.2 \
-Dcom.oracle.graalvm.isaot=true \
-Djava.system.class.loader=com.oracle.svm.hosted.NativeImageSystemClassLoader \
-Xshare:off \
-Dtruffle.TruffleRuntime=com.oracle.svm.truffle.api.SubstrateTruffleRuntime \
-Dgraalvm.ForcePolyglotInvalid=false \
-Dgraalvm.ForcePolyglotInvalid=false \
-Djdk.reflect.useOldSerializableConstructor=true \
-Djdk.internal.lambda.disableEagerInitialization=true \
-Djdk.internal.lambda.eagerlyInitialize=false \
-Djava.lang.invoke.InnerClassLambdaMetafactory.initializeLambdas=false \
-Djava.lang.invoke.MethodHandle.DONT_INLINE_THRESHOLD=-1 \
-Djava.lang.invoke.MethodHandle.PROFILE_GWT=false \
-Xmx256m \
-Xms128m \
--add-modules=ALL-DEFAULT \
[...]
-H:+AddAllCharsets@user \
-H:EnableURLProtocols@user+api=http,https \
-H:NumberOfThreads@user+api=2 \
-H:DeadlockWatchdogInterval@user=0 \
-H:+UnlockExperimentalVMOptions@user \
-H:-UnlockExperimentalVMOptions@user \
[...]
]

There's nothing unexpected in any of that. In particular, we can see some of the flags related to memory use, all of which are as expected:

-Xss10m \
-XX:MaxRAMPercentage=84.99999999999999 \
-XX:GCTimeRatio=9 \
-XX:+ExitOnOutOfMemoryError \
[...]
-Xmx256m \
-Xms128m \

My plan now is to gradually increase the memory until something different happens, all the time capturing the outputs. I tried 384 MiB, 512 MiB and 768 MiB, all of which resulted in the build failing with the following error:

com.oracle.svm.driver.NativeImage$NativeImageError
        at org.graalvm.natiThe Native Image build process ran out of memory.
Please make sure your build system has more memory available.

At 1024 MiB the error changed, but was still clearly related to a lack of memory. The processes are running in parallel so sometimes the output arrives with different ordering.

Please make sure your build system has more memory available.
[...]
com.oracle.svm.driver.NativeImage$NativeImageError
        at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage
            .showError(NativeImage.java:2300)
        at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage
            .build(NativeImage.java:1897)
        at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage
            .performBuild(NativeImage.java:1847)
        at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage
            .main(NativeImage.java:1829)
        at java.base@23.0.2/java.lang.invoke.LambdaForm$DMH/sa346b79c
            .invokeStaticInit(LambdaForm$DMH)

Increasing the maximum allowable memory to 2048 MiB gives similar results again:

$ ./compile.sh
[...]
Build resources:
 - 1.78GB of memory (33.1% of 5.38GB system memory, set via '-Xmx2048m')
 - 2 thread(s) (25.0% of 8 available processor(s), set via '--parallelism=2')
Terminating due to java.lang.OutOfMemoryError: Java heap space
The Native Image build process ran out of memory.
Please make sure your build system has more memory available.
<==========---> 81% EXECUTING [34m 2s]
> :appwrapper:nativeCompile
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
com.oracle.svm.driver.NativeImage$NativeImageError
        at org.graalvm.nati

At this point I decided to try doubling the memory to 4096 MiB — a rather large jump — and this time the build didn't fail as such, but it did cause my phone to reboot again. So I guess we now know that, if there is a configuration that's going to work, it'll be with memory set somewhere between 2048 MiB and 4096 MiB.

Whether there's a sweet spot somewhere between those two values isn't clear. I'm also making the assumption that there's only minimal swap space available on my phone and it may turn out that I need to increase the swap space to get it to work.

Here are the memory graphs from the different runs. Bear in mind that the x-axis scales represent time and aren't all directly comparable. Similarly there are two y-axis scales. The one we're interested in is the memory usage shown along the right hand edge of each graph. Note that these don't all share the same scale either, so they're not directly comparable.
 

I've included the 256 MiB graph again for comparison. Since psrecord writes out the image on completion I don't have the graph for 4096 MiB because the phone rebooted before the image could be written.

All of these graphs show a similar pattern: CPU usage and memory both increase and then plateau. After a period of time the build fails and both drop down to zero. The maximum memory usage increases over time, in MiB to around 1280, 1500, 1560, 1800, 2000 and 2800 respectively. It's hard to correlate these directly to the maximum available JVM memory that we set, except to say that as the setting increases, so does the maximum memory used by the build. That is at least what we'd expect.

It's also notable that the last three graphs for 768 MiB, 1024 MiB and 2048 MiB show a slightly different curve from the earlier graphs. These seem to increase, plateau, but then start to increase again before reaching a second plateau. This may be because they're reaching later portions of the build process.

Finally, the time taken for each build doesn't seem to correspond to the amount of memory available. The build with 768 MiB took especially long for some reason. It's possible this is just down to how the compiler is attempting to shuffle memory around to compensate for the available memory being restricted.

Tomorrow I'll have to do a few more experiments and will need to examine these graphs in more detail to try to figure out how much memory we're likely to need in practice.

If you've been following on Mastodon you'll already know, but I'm happy to share here that Thigg has also been looking in to this, attempting to get the build to complete in a third environment: a Hetzner aarch64 VM:
 
It wouldn't have occurred to me to try this, but it would be an excellent way to circumvent both the challenges of getting the code to build on a phone and the fact it won't build in the Sailfish SDK.

Thigg and I were planning to have a call to discuss ways forwards with this but unfortunately it didn't happen, entirely my failure. I'm very much hoping that we can still have a conversation about it this week.

Yesterday I worked through and removed all final references to the Rhino JavaScript interpreter, so that we can use the GraalJS interpreter instead. Just as I was about to commit the code I had a change of heart: rather than putting all of the replaced functionality inside a single Utils class I decided to split it across replacement Context, Kit and ScriptRuntime classes instead.

This increases the number of files, but reduces the diff required from the existing code, so I think it ends up a lot cleaner and nicer.

Having committed my changes it's time to move to the next stage, which is attempting to build and run the NewPipe source for Sailfish OS. Since we already attempted to build it locally using the Sailfish SDK unsuccessfully on Day 6 I'm going to jump straight to building it directly on my phone today.

This means adjusting the compile.sh file we used to build sailing-the-flood-to-java earlier for the NewPipe codebase.

My first attempt looks like this:

#!/bin/bash
COMPILEHOST=defaultuser@172.28.172.1
COMPILEHOST_WORKSPACE=/home/defaultuser/Documents/Development/newpipe/
    NewPipeExtractor
COMPILEHOST_GRAAL=/home/defaultuser/Documents/Development/newpipe/graalvm/
    graalvm-jdk-23.0.2+7.1
COMPILEHOST_MAVENLOCAL=/home/defaultuser/Documents/Development/newpipe/graalvm/
    m2
echo "transfering data"
pushd NewPipeExtractor
rsync . $COMPILEHOST:$COMPILEHOST_WORKSPACE -r
popd
echo "starting compilation"
ssh $COMPILEHOST "cd $COMPILEHOST_WORKSPACE; 
    GRAALVM_HOME=$COMPILEHOST_GRAAL \
    JAVA_HOME=$COMPILEHOST_GRAAL ./gradlew tasks"

All this does is sync the NewPipeExtractor source directory, then call gradlew tasks inside it. The result is good: the various tasks are listed to the console, including a bunch of native build and run tasks. There are a couple of warnings during the rsync operation though:

skipping non-regular file "graalvm/graalvm-jdk-23.0.2+7.1/bin/
    native-image"
skipping non-regular file "graalvm/graalvm-jdk-23.0.2+7.1/bin/
    native-image-configure"

That's because these files are symlinks:

$ ls -lh graalvm/graalvm-jdk-23.0.2+7.1/bin/native-image
lrwxrwxrwx 1 flypig flypig 27 Jan  7 14:16 native-image ->
    ../lib/svm/bin/native-image
$ ls -lh graalvm/graalvm-jdk-23.0.2+7.1/bin/native-image-configure
lrwxrwxrwx 1 flypig flypig 37 Jan  7 14:16 native-image-configure ->
    ../lib/svm/bin/native-image-configure

I could fix those manually on my phone, but in practice I don't think they should matter because I've configured the build process to use the GraalVM toolchain from a different location (the location that was used for sailing-the-flood-to-java as it happens). In future, I think the right way to deal with this is to ensure the build tools get downloaded and unpacked directly on my phone during the build process. That'll be something for the future.

The next step then has to be to try to build the native binaries using nativeCompile. I've therefore updated the final line of the script so it looks like this:

ssh $COMPILEHOST "cd $COMPILEHOST_WORKSPACE; 
    GRAALVM_HOME=$COMPILEHOST_GRAAL \
    JAVA_HOME=$COMPILEHOST_GRAAL ./gradlew nativeCompile"

Let's try executing it to see what happens.

$ ./compile.sh
transfering data
~/Documents/Development/projects/newpipe/NewPipeExtractor
    ~/Documents/Development/projects/newpipe
skipping non-regular file "graalvm/graalvm-jdk-23.0.2+7.1/bin/
    native-image"
skipping non-regular file "graalvm/graalvm-jdk-23.0.2+7.1/bin/
    native-image-configure"
~/Documents/Development/projects/newpipe
starting compilation
[...]

As the build progresses I notice that my email client, calendar, Whisperfish and the Documents app all running on my phone have been killed due to lack of memory. This also happened when building sailing-the-fllod-to-java so this isn't unexpected. But it makes me wonder whether there will be enough resources for the much larger NewPipe build to get through in its entirety.

Perhaps unsurprisingly the build is also taking a lot longer now it's happening on my phone. I should time it and the build on my laptop to compare.

I'm currently travelling to work and it's taking so long in fact that the build is still running as my train pulls in to King's Cross, the final stop of my journey. Although I'm running in a gnu screen session, the session is running on my laptop, not on my phone. I can send my laptop to sleep but I doubt the SSH connection will survive the interruption.

So I've cancelled the build and will have to kick it off again later.

And as expected, as my train comes in to land I've had to close the connection and stop the build.

$ ./compile.sh
transfering data
~/Documents/Development/projects/newpipe/NewPipeExtractor
    ~/Documents/Development/projects/newpipe
skipping non-regular file "graalvm/graalvm-jdk-23.0.2+7.1/bin/
    native-image"
skipping non-regular file "graalvm/graalvm-jdk-23.0.2+7.1/bin/
    native-image-configure"
~/Documents/Development/projects/newpipe
starting compilation
Starting a Gradle Daemon (subsequent builds will be faster)
[...]
===============================================================================
GraalVM Native Image: Generating 'appwrapper' (shared library)...
===============================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual
    /native-image/BuildOutput.md
-------------------------------------------------------------------------------
[1/8] Initializing...                                          (31.9s @ 0.14GB)
[...]
-------------------------------------------------------------------------------
Build resources:
 - 4.06GB of memory (75.6% of 5.38GB system memory, determined at start)
 - 8 thread(s) (100.0% of 8 available processor(s), determined at start)
[2/8] Performing analysis...  [*******]                       (337.2s @ 2.77GB)
   21,380 reachable types   (93.0% of   22,982 total)
   34,005 reachable fields  (53.3% of   63,827 total)
  116,104 reachable methods (69.4% of  167,289 total)
   11,524 runtime compiled methods ( 6.9% of  167,289 total)
    6,290 types,    70 fields, and   905 methods registered for reflection
       83 types,    61 fields, and   192 methods registered for JNI access
        4 native libraries: dl, pthread, rt, z
^C

I attempted a further build during the day. It seemed to make progress until it caused my phone to become unresponsive and reboot. So I'll need a way to reduce the resource consumption of the build. Looking at the output from earlier, I see there are both memory and threads listed as resources being claimed.

I don't see any flags available for the native-image tool for constraining memory usage, but there is a flag for controlling the number of threads that are used. The flag in question is --parallelism, which you can pass with a number to restrict the maximum number of threads.

I'm therefore adding this to the configuration so that no more than one thread is used. This will potentially slow the build down by a factor of eight, which is a huge difference, but still better than a failed build and a rebooted phone.

$ git diff
diff --git a/appwrapper/build.gradle b/appwrapper/build.gradle
index 39a9503e..c3c19dc3 100644
--- a/appwrapper/build.gradle
+++ b/appwrapper/build.gradle
@@ -35,6 +35,7 @@ graalvmNative {
         buildArgs.add('-H:+AddAllCharsets')
         // Enable network protocols
         buildArgs.add('--enable-url-protocols=http,https')
+        buildArgs.add('--parallelism=1')
     }
 }
 

With this in place, I kick the build off again. But now there appears to be a different issue. Brace yourself for a rather long console transcript. Feel free to skip past this; I really just want to capture the errors for future reference.

$ ./compile.sh 
transfering data
~/Documents/Development/projects/newpipe/NewPipeExtractor
    ~/Documents/Development/projects/newpipe
skipping non-regular file "graalvm/graalvm-jdk-23.0.2+7.1/bin/
    native-image"
skipping non-regular file "graalvm/graalvm-jdk-23.0.2+7.1/bin/
    native-image-configure"
~/Documents/Development/projects/newpipe
starting compilation
Starting a Gradle Daemon (subsequent builds will be faster)
[...]
> Task :appwrapper:nativeCompile
[native-image-plugin] GraalVM Toolchain detection is disabled
[native-image-plugin] GraalVM location read from environment variable: 
    GRAALVM_HOME
[native-image-plugin] Native Image executable path:
    graalvm-jdk-23.0.2+7.1/lib/svm/bin/native-image
Loading classes is taking a long time. This can be caused by class- or 
    module-path entries that point to large directory structures.
Total processed entries: 17989, current entry:
    jar:file:///home/defaultuser/.gradle/caches/modules-2/files-2.1/
    com.google.code.findbugs/jsr305/3.0.2/
    25ea2e8b0c338a877313bd4672d3fe056ea78f0d/jsr305-3.0.2.jar!/META-INF/maven/
    com.google.code.findbugs/jsr305
[...]
=== Image generator watchdog detected no activity. This can be a sign of a
    deadlock during image building. Dumping all stack traces. Current time:
    Tue Feb 25 21:11:04 GMT 2025
"main" Id=1 in WAITING on 
    lock=java.util.stream.ForEachOps$ForEachTask@2cb0e449
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.park
        (LockSupport.java:371)
    at java.base@23.0.2/java.util.concurrent.ForkJoinTask.awaitDone
        (ForkJoinTask.java:441)
    at java.base@23.0.2/java.util.concurrent.ForkJoinTask.awaitDone
        (ForkJoinTask.java:496)
    at java.base@23.0.2/java.util.concurrent.ForkJoinTask.join
        (ForkJoinTask.java:662)
    at java.base@23.0.2/java.util.concurrent.ForkJoinTask.invoke
        (ForkJoinTask.java:677)
    at java.base@23.0.2/java.util.stream.ForEachOps$ForEachOp.evaluateParallel
        (ForEachOps.java:160)
    at java.base@23.0.2/java.util.stream.ForEachOps$ForEachOp$OfRef
        .evaluateParallel(ForEachOps.java:174)
    at java.base@23.0.2/java.util.stream.AbstractPipeline.evaluate
        (AbstractPipeline.java:264)
    at java.base@23.0.2/java.util.stream.ReferencePipeline.forEach
        (ReferencePipeline.java:636)
    at java.base@23.0.2/java.util.stream.ReferencePipeline$Head.forEach
        (ReferencePipeline.java:810)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted
        .NativeImageClassLoaderSupport$LoadClassHandler.run
        (NativeImageClassLoaderSupport.java:678)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted
        .NativeImageClassLoaderSupport.loadAllClasses
        (NativeImageClassLoaderSupport.java:243)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted
        .ImageClassLoader.loadAllClasses(ImageClassLoader.java:88)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted
        .NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:
    386)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted
        .NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:711)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted
        .NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:139)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted
        .NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:94)

"Reference Handler" Id=5 in RUNNABLE
    at java.base@23.0.2/java.lang.ref.Reference.waitForReferencePendingList
        (Native Method)
    at java.base@23.0.2/java.lang.ref.Reference.processPendingReferences
        (Reference.java:246)
    at java.base@23.0.2/java.lang.ref.Reference$ReferenceHandler.run
        (Reference.java:208)

"Finalizer" Id=6 in WAITING on lock=java.lang.ref
    .NativeReferenceQueue$Lock@28e6ff9f
    at java.base@23.0.2/java.lang.Object.wait0(Native Method)
    at java.base@23.0.2/java.lang.Object.wait(Object.java:378)
    at java.base@23.0.2/java.lang.Object.wait(Object.java:352)
    at java.base@23.0.2/java.lang.ref.NativeReferenceQueue.await
        (NativeReferenceQueue.java:48)
    at java.base@23.0.2/java.lang.ref.ReferenceQueue.remove0
        (ReferenceQueue.java:166)
    at java.base@23.0.2/java.lang.ref.NativeReferenceQueue.remove
        (NativeReferenceQueu$ ./compile.sh 
transfering data
~/Documents/Development/projects/newpipe/NewPipeExtractor
    ~/Documents/Development/projects/newpipe
skipping non-regular file "graalvm/graalvm-jdk-23.0.2+7.1/bin/
    native-image"
skipping non-regular file "graalvm/graalvm-jdk-23.0.2+7.1/bin/
    native-image-configure"
~/Documents/Development/projects/newpipe
starting compilation
Starting a Gradle Daemon (subsequent builds will be faster)
[...]
===============================================================================
GraalVM Native Image: Generating 'appwrapper' (shared library)...
===============================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/
    BuildOutput.md
-------------------------------------------------------------------------------
[1/8] Initializing...                                          (33.0s @ 0.13GB)
[...]
-------------------------------------------------------------------------------
 1 experimental option(s) unlocked:
 - '-H:DeadlockWatchdogInterval' (origin(s): command line)
-------------------------------------------------------------------------------
Build resources:
 - 4.06GB of memory (75.6% of 5.38GB system memory, determined at start)
 - 2 thread(s) (25.0% of 8 available processor(s), set via '--parallelism=2')
e.java:89)
    at java.base@23.0.2/java.lang.ref.Finalizer$FinalizerThread.run(
    Finalizer.java:173)

"Signal Dispatcher" Id=7 in RUNNABLE

"Common-Cleaner" Id=14 in TIMED_WAITING on 
    lock=java.util.concurrent.locks
    .AbstractQueuedSynchronizer$ConditionObject@4321868d
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos
        (LockSupport.java:269)
    at java.base@23.0.2/java.util.concurrent.locks
        .AbstractQueuedSynchronizer$ConditionObject.await
        (AbstractQueuedSynchronizer.java:1852)
    at java.base@23.0.2/java.lang.ref.ReferenceQueue.await
        (ReferenceQueue.java:79)
    at java.base@23.0.2/java.lang.ref.ReferenceQueue.remove0
        (ReferenceQueue.java:151)
    at java.base@23.0.2/java.lang.ref.ReferenceQueue.remove
        (ReferenceQueue.java:229)
    at java.base@23.0.2/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:140)
    at java.base@23.0.2/java.lang.Thread.runWith(Thread.java:1588)
    at java.base@23.0.2/java.lang.Thread.run(Thread.java:1575)
    at java.base@23.0.2/jdk.internal.misc.InnocuousThread.run
        (InnocuousThread.java:186)

"Notification Thread" Id=15 in RUNNABLE
[...]
=== Memory statistics (in MB):
=== Used heap size: 84
=== Free heap size: 53
=== Maximum heap size: 4162
=== Image generator watchdog is aborting image generation. To configure the
    watchdog, use the options -H:DeadlockWatchdogInterval=10 and
    -H:+DeadlockWatchdogExitOnTimeout

> Task :appwrapper:nativeCompile FAILED
Error: Image build request for 'appwrapper' (pid: 59178, path:
    /home/defaultuser/Documents/Development/newpipe/NewPipeExtractor/appwrapper
    /build/native/nativeCompile) failed with exit status 30

[Incubating] Problems report is available at:
    file:///home/defaultuser/Documents/Development/newpipe/NewPipeExtractor
    /build/reports/problems/problems-report.html

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':appwrapper:nativeCompile'.
> Process 'command '/home/defaultuser/Documents/Development/newpipe/graalvm
    /graalvm-jdk-23.0.2+7.1/bin/native-image''
    finished with non-zero exit value 30
[...]
BUILD FAILED in 11m 42s

The build has clearly failed here, but despite all those backtraces it doesn't appear to be due to an error, but rather is triggered by the watchdog mechanism. This is supposed to identify builds that have hung, killing the build process after a certain duration of unresponsiveness. Rather conveniently the error message also provides some advice for how to work around this:

watchdog is aborting image generation. To configure the watchdog, use the
    options -H:DeadlockWatchdogInterval=10 and -H:+DeadlockWatchdogExitOnTimeout

I'm not sure what a good timeout would be, so instead I'm going to disable it completely by adding -H:DeadlockWatchdogInterval=0 as a build parameter, like this:

$ git diff
diff --git a/appwrapper/build.gradle b/appwrapper/build.gradle
index 39a9503e..dbf0a12b 100644
--- a/appwrapper/build.gradle
+++ b/appwrapper/build.gradle
@@ -35,6 +35,8 @@ graalvmNative {
         buildArgs.add('-H:+AddAllCharsets')
         // Enable network protocols
         buildArgs.add('--enable-url-protocols=http,https')
+        buildArgs.add('--parallelism=1')
+        buildArgs.add('H:DeadlockWatchdogInterval=0')
     }
 }

With this set the build is now running and not being killed. But it's taking a very long time and appears to be stuck in a loop even before reaching the first stage:

$ ./compile.sh 
transfering data
~/Documents/Development/projects/newpipe/NewPipeExtractor ~/Documents/
    Development/projects/newpipe
skipping non-regular file "graalvm/graalvm-jdk-23.0.2+7.1/bin/
    native-image"
skipping non-regular file "graalvm/graalvm-jdk-23.0.2+7.1/bin/
    native-image-configure"
~/Documents/Development/projects/newpipe
starting compilation
Starting a Gradle Daemon (subsequent builds will be faster)
[...]
> Task :appwrapper:nativeCompile
[native-image-plugin] GraalVM Toolchain detection is disabled
[native-image-plugin] GraalVM location read from environment variable: 
    GRAALVM_HOME
[native-image-plugin] Native Image executable path:
    graalvm/graalvm-jdk-23.0.2+7.1/lib/svm/bin/native-image
Warning: The option '-H:DeadlockWatchdogInterval=0' is experimental and must be
    enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and
    either remove or unlock it. The build output lists all active experimental
    options, including where they come from and possible alternatives. If you
    think an experimental option should be considered as stable, please file an
    issue.
Loading classes is taking a long time. This can be caused by class- or
    module-path entries that point to large directory structures.
Total processed entries: 17989, current entry:
    jar:file:///home/defaultuser/.gradle/caches/modules-2/files-2.1/com.google
    .code.findbugs/jsr305/3.0.2/25ea2e8b0c338a877313bd4672d3fe056ea78f0d
    /jsr305-3.0.2.jar!/META-INF/maven/com.google.code.findbugs/jsr305
Total processed entries: 17989, current entry:
    jar:file:///home/defaultuser/.gradle/caches/modules-2/files-2.1/com.google
    .code.findbugs/jsr305/3.0.2/25ea2e8b0c338a877313bd4672d3fe056ea78f0d
    /jsr305-3.0.2.jar!/META-INF/maven/com.google.code.findbugs/jsr305
[...]
Total processed entries: 17989, current entry:
    jar:file:///home/defaultuser/.gradle/caches/modules-2/files-2.1/com.google
    .code.findbugs/jsr305/3.0.2/25ea2e8b0c338a877313bd4672d3fe056ea78f0d
    /jsr305-3.0.2.jar!/META-INF/maven/com.google.code.findbugs/jsr305
[...]

That same message is repeated 25 times and the "Total processed entries" never seems to increase beyond 17989. It appears to be the case that with one thread the process will never progress. I've therefore increased parallelism so that it now allows up to two threads. It's late, so I'm going to now leave it running for the night and see how far it's progressed by morning.

Tomorrow we'll find out the result. If it's failed I'll need to look more deeply into how to fix it. If it's succeeded then it'll be time to test the resulting library!

Yesterday turned out to be an unexpectedly productive day. I was able to get the NewPipe Extractor code built on Linux as a native executable, linked to some C code that was able to collect metadata and a video download URL from YouTube.

As part of this I had to convert some code that executed JavaScript functions using Mozilla's Rhino JavaScript engine to use Polyglot instead. Under the hood, this is making use of the GraalJS JavaScript interpreter.

But, although I was able to switch out Rhino for GraalJS in the places needed for our test application, there are other places where Rhino is still used, most notably in the TokenStream.java source. Rhino is much more tightly intertwined with this code, so replacing it is going to be considerably harder.

In fact, looking at the header, it even seems like some of this source was taken directly from the Rhino codebase:

/* Source: Mozilla Rhino, org.mozilla.javascript.Token
 *
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, You can obtain one at http://mozilla.org/MPL/2.0/.
 * */
class TokenStream {
[...]

Digging through the code, it appears this is used to extract function names from JavaScript files, needed in two places:

1. YoutubeSignatureUtils: used to get the signature de-obfuscation code of YouTube's base JavaScript file;
2.  

YoutubeThrottlingParameterUtils: used to get the throttling parameter de-obfuscation code of YouTube's base JavaScript file. These are small parts of the overall code, but it doesn't look like they should be ignored.

So what's the actual code that's being used here? There's a lot of code in the TokenStream class, but the actual Rhino code being used is somewhat more limited. There are a couple of static ints taken from the Context class:

1. Context.VERSION_ES6
2. Context.VERSION_1_7

These are just integers, so it'd be pretty easy to redefine them (although I must also look in to how they're being used and whether they need to be updated for use with GraalJS). There are also four methods that are made use of:

1. Kit.xDigitToInt()
2. Kit.codeBug()
3. ScriptRuntime.isJSLineTerminator()
4. ObjToIntMap.ObjToIntMap()
5. ObjToIntMap.intern()

All of the links here are to Rhino version 1.7.15, since that's the version defined in the build.gradle for NewPipe Extractor.

Looking at these five methods, it also looks like they can mostly be easily replaced, potentially just by pulling in the small portion of code directly in to the source file in the NewPipe Extractor code. Since the code is MPL licensed and the MPL copyright notice is already at the top of the file, doing this shouldn't be problematic from a licensing perspective either.

First, checking where the version integers are being used from Context I actually only see one case, which is this, from Lexer.java:

    /**
     * Create a new JavaScript lexer with the given source code
     *
     * @param js JavaScript code
     */
    public Lexer(final String js) {
        this(js, Context.VERSION_DEFAULT);
    }

The VERSION_DEFAULT integer is also defined in the Rhino source and simply takes the value 0. In that case, these values should be safe for us to define explicitly.

So I've done some work to recreate the missing code. I've added four new files, all in the extractor/src/main/java/org/schabi/newpipe/extractor/utils/jsextractor directory:

1. ObjToIntMap.java
2. UniqueTag.java
3. Utils.java

The first of these reintroduces the minimal convex hull of code needed for the use of ObjToIntMap in the TokenStream class. Not very much code, but still a little too much to copy out here in full. Similarly UniqueTag contains the minimal convex hull of code needed for the references to UniqueTag made in ObjToIntMap. That contents just looks like this:

public final class UniqueTag implements Serializable {
    private static final int ID_NULL_VALUE = 2;
    public static final UniqueTag NULL_VALUE = new UniqueTag(ID_NULL_VALUE);    
        

    private final int tagId;

    private UniqueTag(int tagId) {
        this.tagId = tagId;
    }
}

Finally the Utils class contains the fields and methods from Context, Kit and ScriptRuntime that are needed for TokenStream. Because these are now in a class called Utils I also had to make minor adjustments to the TokenStream code to use the new names.

It might end up cleaner if I split these out into three separate files again, but for now this small set of changes will do the trick.

To check that things compile now without the need to reference Rhino, I've also removed Rhino as a dependency from the build configuration.

$ git diff extractor/build.gradle
diff --git a/extractor/build.gradle b/extractor/build.gradle
index 16d34e00..e617e39f 100644
--- a/extractor/build.gradle
+++ b/extractor/build.gradle
@@ -30,7 +30,6 @@ dependencies {
     implementation 'org.jsoup:jsoup:1.17.2'
     implementation "com.google.code.findbugs:jsr305:$jsr305Version"
 
-    implementation 'org.mozilla:rhino:1.7.15'
     implementation("org.graalvm.polyglot:polyglot:$graalVMVersion")
     implementation("org.graalvm.polyglot:js:$graalVMVersion")

Happily the build completes successfully despite this. And the example still runs.

This seems like a good place to stop for today. We've stripped the code of references to Rhino and replaced it so that GraalJS is used instead. This puts us in a good position to try building and running the code on Sailfish OS tomorrow.

Sadly it wasn't possible to get GraalVM working correctly using scratchbox2 yesterday, which means I'll have to continue running builds on my phone in future. Maybe I'll figure out a better approach in future, but for now this will do the job. But for now, it's time to move on.

And it feels like we're going to start our adventure properly today as we attempt to get the NewPipe Extractor code working on GraalVM. I'm expecting multiple challenges. First, NewPipe Extractor is set up to be built using Gradle, whereas our existing GraalVM pipeline uses Maven. So there's going to be some work needed either to adjust the Gradle configuration or to switch over to using Maven. Second, GraalVM doesn't support the full feature-set of Java. For example, some of the reflection features aren't supported. I have no idea what features NewPipe Extractor relies on, but these could result in some bumps along the way. Finally NewPipe Extractor is a much larger codebase than the sailing-the-flood-to-java code. So we may well hit memory issues.

Those are the known unknowns. There will surely be unknown unknowns as well!

For the entirety of today I won't be working on-phone, I'll be working on my Linux laptop. Moving to the phone will be for a future post.

As I mentioned, the GraalVM build process we've been using up until now has used Maven. So for the first step today I'm going to attempt to integrate GraalVM's native build process into the Gradle build scripts used for NewPipe Extractor.

I've therefore added the following to the build.gradle files for extractor and appwrapper:

plugins {
    id 'checkstyle'
    id 'org.graalvm.buildtools.native' version '0.10.5'
}

The version number — 0.10.5 — I got from the GraalVM instructions for building native images using Gradle. The value there mirrors the latest version of the native build tools available from the source repository. This should match up with the version of GraalVM we'll be downloading, which is also the latest version:

$ mkdir graalvm
$ pushd graalvm/
~/dev/graalvm ~/dev
$ curl -O https://download.oracle.com/graalvm/23/latest/
    graalvm-jdk-23_linux-x64_bin.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  361M  100  361M    0     0  4133k      0  0:01:29  0:01:29 --:--:-- 3495k
$ tar --totals -xf graalvm-jdk-23_linux-x64_bin.tar.gz
Total bytes read: 835983360 (798MiB, 106MiB/s)
$ GRAAL=${PWD}/graalvm-jdk-23.0.2+7.1/
$ popd
~/dev

With the plugin added and the GraalVM tools available for use, we can now see there are a bunch of new "native" tasks available when running Gradle:

$ GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./gradlew tasks | grep native
collectReachabilityMetadata - Obtains native reachability metadata for the 
    runtime classpath configuration
nativeCompile - Compiles a native image for the main binary
nativeRun - Executes the main native binary
nativeTestCompile - Compiles a native image for the test binary
nativeTest - Executes the test native binary

That's all very encouraging. I'm always in favour of just trying things out so why don't we go ahead and just try building a native library and see what happens...

$ GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./gradlew nativeCompile

> Task :extractor:nativeCompile
[native-image-plugin] GraalVM Toolchain detection is disabled
[native-image-plugin] GraalVM location read from environment variable: 
    GRAALVM_HOME
[native-image-plugin] Native Image executable path: /home/flypig/dev/graalvm/
    graalvm-jdk-23.0.2+7.1/lib/svm/bin/native-image
===============================================================================
GraalVM Native Image: Generating 'extractor' (shared library)...
===============================================================================
[...]
[1/8] Initializing...                                           (4.8s @ 0.08GB)
[...]
[2/8] Performing analysis...  [******]                          (5.4s @ 0.25GB)
[...]
[3/8] Building universe...                                      (1.0s @ 0.28GB)
[4/8] Parsing methods...      [*]                               (1.7s @ 0.26GB)
[5/8] Inlining methods...     [***]                             (0.8s @ 0.30GB)
<===========--> 90% EXECUTING [16s]
> :extractor:nativeCompile
[...]
/usr/libexec/gcc/x86_64-linux-gnu/13/collect2 -plugin /usr/libexec/gcc/
    x86_64-linux-gnu/13/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/
    x86_64-linux-gnu/13/lto-wrapper -plugin-opt=-fresolution=/tmp/ccyIUKnP.res 
    -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s 
    -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc 
    -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr -m elf_x86_64 
    --hash-style=gnu --as-needed -shared -z relro -o ~/dev/extractor/build/
    native/nativeCompile/extractor.so -z noexecstack -z text /usr/lib/gcc/
    x86_64-linux-gnu/13/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/
    x86_64-linux-gnu/13/crtbeginS.o -L/tmp/SVM-11991437499249190113 -L~/dev/
    graalvm/graalvm-jdk-23.0.2+7.1/lib/static/linux-amd64/glibc -L~/dev/graalvm/
    graalvm-jdk-23.0.2+7.1/lib/svm/clibraries/linux-amd64/glibc -L~/dev/graalvm/
    graalvm-jdk-23.0.2+7.1/lib/svm/clibraries/linux-amd64 -L~/dev/graalvm/
    graalvm-jdk-23.0.2+7.1/lib/svm/clibraries -L/usr/lib/gcc/x86_64-linux-gnu/
    13 -L/usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu -L/usr/lib/
    gcc/x86_64-linux-gnu/13/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../
    lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/
    x86_64-linux-gnu/13/../../.. --gc-sections --version-script /tmp/
    SVM-11991437499249190113/exported_symbols.list extractor.o ~/dev/graalvm/
    graalvm-jdk-23.0.2+7.1/lib/svm/clibraries/linux-amd64/glibc/liblibchelper.a 
    ~/dev/graalvm/graalvm-jdk-23.0.2+7.1/lib/static/linux-amd64/glibc/libnet.a 
    ~/dev/graalvm/graalvm-jdk-23.0.2+7.1/lib/static/linux-amd64/glibc/libnio.a 
    ~/dev/graalvm/graalvm-jdk-23.0.2+7.1/lib/static/linux-amd64/glibc/libjava.a 
    ~/dev/graalvm/graalvm-jdk-23.0.2+7.1/lib/svm/clibraries/linux-amd64/glibc/
    libjvm.a -lz -ldl -lpthread -lrt -lgcc --push-state --as-needed -lgcc_s 
    --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/
    gcc/x86_64-linux-gnu/13/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/13/../../../
    x86_64-linux-gnu/crtn.o
/usr/bin/ld: cannot find -lz: No such file or directory
collect2: error: ld returned 1 exit status
[...]
BUILD FAILED in 34s
6 actionable tasks: 1 executed, 5 up-to-date

Okay, well, things got a fair way through but seem to have failed at the linker stage. The error says that the linker failed to resolve the -lz flag. This is usually a reference to the zlib compression library. So maybe I just need to install it?

Let's find out.

$ sudo apt install zlib1g-dev
$ $GRAAL JAVA_HOME=$GRAAL ./gradlew nativeCompile
[...]
[1/8] Initializing...                                           (4.6s @ 0.08GB)
[...]
[2/8] Performing analysis...  [******]                          (4.9s @ 0.27GB)
[...]
[3/8] Building universe...                                      (1.0s @ 0.29GB)
[4/8] Parsing methods...      [*]                               (1.5s @ 0.24GB)
[5/8] Inlining methods...     [***]                             (0.7s @ 0.31GB)
[6/8] Compiling methods...    [****]                           (13.3s @ 0.43GB)
[7/8] Laying out methods...   [*]                               (1.4s @ 0.45GB)
[8/8] Creating image...       [*]                               (1.0s @ 0.50GB)
[...]
Build artifacts:
 ~/dev/extractor/build/native/nativeCompile/extractor.so (shared_library)
 ~/dev/extractor/build/native/nativeCompile/graal_isolate.h (c_header)
 ~/dev/extractor/build/native/nativeCompile/graal_isolate_dynamic.h (c_header)
===============================================================================
Finished generating 'extractor' in 29.4s.
[native-image-plugin] Native Image written to:
    ~/dev/extractor/build/native/nativeCompile
[...]
BUILD SUCCESSFUL in 31s
6 actionable tasks: 1 executed, 5 up-to-date

Amazing! The thing actually just went ahead and built a native library. Honestly, I'm pretty astonished at how easy this looks to have been. But maybe my excitement is premature? Let's take a look at what actually got built.

$ ls -hl extractor/build/native/nativeCompile/
total 7.2M
-rwxr-xr-x 1 flypig flypig 7.1M Feb 23 12:24 extractor.so
-rw-r--r-- 1 flypig flypig 5.3K Feb 23 12:24 graal_isolate.h
-rw-r--r-- 1 flypig flypig 5.5K Feb 23 12:24 graal_isolate_dynamic.h
-rw-r--r-- 1 flypig flypig  33K Feb 23 12:15 
    svm_err_b_20250223T121513.601_pid1562.md
-rw-r--r-- 1 flypig flypig  33K Feb 23 12:22 
    svm_err_b_20250223T122232.035_pid1963.md
$ file extractor/build/native/nativeCompile/extractor.so 
extractor/build/native/nativeCompile/extractor.so: ELF 64-bit LSB shared
    object, x86-64, version 1 (SYSV), dynamically linked,
    BuildID[sha1]=21b90538c3a9a3f151a578c48b4469b6cb37ebed, stripped

That looks pretty good actually. The only thing I find a little concerning is that there's no extractor.h header file to build against. This is something I was expected. We'll return to that shortly, but at this point it's probably also a good idea to run the automated test suite to see whether things are actually still working or not.

$ GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./gradlew nativeTest   

> Task :extractor:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :extractor:test
[...]
YoutubeStreamExtractorDefaultTest$StreamSegmentsTestTagesschau > 
    testRelatedItems() FAILED
    org.opentest4j.AssertionFailedError: List of items is empty ==> expected: 
    <false> but was: <true>
        at app//org.schabi.newpipe.extractor.services.DefaultTests
            .defaultTestListOfItems(DefaultTests.java:35)
        at app//org.schabi.newpipe.extractor.services
            .DefaultStreamExtractorTest.testRelatedItems
            (DefaultStreamExtractorTest.java:244)
        at java.base@23.0.2/java.lang.reflect.Method.invoke(Method.java:580)
        at java.base@23.0.2/java.util.ArrayList.forEach(ArrayList.java:1597)
        at java.base@23.0.2/java.util.ArrayList.forEach(ArrayList.java:1597)

YoutubeStreamExtractorRelatedMixTest > testRelatedItems() FAILED
    org.opentest4j.AssertionFailedError: Unexpected normal playlist in related
        items ==> expected: not equal but was: <NORMAL>
        at app//org.junit.jupiter.api.AssertionFailureBuilder
            .build(AssertionFailureBuilder.java:152)
        at app//org.junit.jupiter.api.AssertionFailureBuilder
            .buildAndThrow(AssertionFailureBuilder.java:132)
        at app//org.junit.jupiter.api.AssertNotEquals
            .failEqual(AssertNotEquals.java:277)
        at app//org.junit.jupiter.api.AssertNotEquals
            .assertNotEquals(AssertNotEquals.java:263)
        at app//org.junit.jupiter.api.Assertions
            .assertNotEquals(Assertions.java:2832)
        at app//org.schabi.newpipe.extractor.services.youtube.stream
            .YoutubeStreamExtractorRelatedMixTest.lambda$testRelatedItems$0
            (YoutubeStreamExtractorRelatedMixTest.java:95)
        at java.base@23.0.2/java.util.ArrayList.forEach(ArrayList.java:1597)
        at app//org.schabi.newpipe.extractor.services.youtube.stream
            .YoutubeStreamExtractorRelatedMixTest.testRelatedItems
            (YoutubeStreamExtractorRelatedMixTest.java:95)

2264 tests completed, 2 failed, 99 skipped
[...]
BUILD FAILED in 7m 46s
8 actionable tasks: 4 executed, 4 up-to-date

Only two test failures. It would be nice if there were none of course, but when I ran the tests without the native build I got similar results, so this isn't unexpected.

This is just the extractor. If you cast your mind back to Day 4 you may recall I added an appwrapper directory to the repository that contained a small example application. This made use of the library by outputting some info about a YouTube video, along with a URL that allows the video to be downloaded.

Because I also added org.graalvm.buildtools.native as a plugin to the build.gradle for the appwrapper code, it means that this was also run through the native build tooling. Let's take a look at what we got from that.

$ pushd appwrapper/build/native/nativeCompile/
~/dev/appwrapper/build/native/nativeCompile ~/dev
$ gcc -I ./ -L ./ -Wl,-rpath ./ -o main main.c -l:appwrapper.so
$ ls -lh
total 182M
-rw-r--r-- 1 flypig flypig  201 Feb 23 19:10 appwrapper.h
-rwxr-xr-x 1 flypig flypig 182M Feb 23 19:10 appwrapper.so
-rw-r--r-- 1 flypig flypig  225 Feb 23 19:10 appwrapper_dynamic.h
-rw-r--r-- 1 flypig flypig 5.3K Feb 23 19:10 graal_isolate.h
-rw-r--r-- 1 flypig flypig 5.5K Feb 23 19:10 graal_isolate_dynamic.h
drwxr-xr-x 3 flypig flypig 4.0K Feb 23 19:01 resources
$ popd
~/dev

This looks a bit more encouraging: we have an appwrapper.h header file and, if we look inside it, we can see that it exposes a single callable function called run_main():

#include <graal_isolate.h>

#if defined(__cplusplus)
extern &quot;C&quot; {
#endif

int run_main(int argc, char** argv);

#if defined(__cplusplus)
}
#endif

That's encouraging because our appwrapper code does indeed include a main() method. Calling this method should execute our test application, which will be a great way to check whether things are working as expected or not.

In order to test this out, I've written a very simple wrapper application in C that will initialise the native Gradle library, call this method and then quit. Here's the code, which I've stored in the file appwrapper/src/main.c.

#include <stdio.h>
#include <stdlib.h>

#include &quot;appwrapper.h&quot;

int main(int argc, char **argv) {
    graal_isolate_t *isolate = NULL;
    graal_isolatethread_t *thread = NULL;
    
    if (graal_create_isolate(NULL, &isolate, &thread) != 0) {
        fprintf(stderr, &quot;initialization error\n&quot;);
        return 1;
    }
    
    int result = run_main(argc, argv);
    
    graal_tear_down_isolate(thread);
    return result;
}

Having created this file we can try to build and link it against the appwrapper.so and extractor.so dynamic libraries that contain our native-build Java code.

$ gcc -I appwrapper/build/native/nativeCompile/ \
    -L appwrapper/build/native/nativeCompile/ \
    -L extractor/build/native/nativeCompile/ \
    -Wl,-rpath appwrapper/build/native/nativeCompile/ \
    -Wl,-rpath extractor/build/native/nativeCompile/ \
    -o main appwrapper/src/main.c -l:appwrapper.so -l:extractor.so
$ ls -lh main 
-rwxr-xr-x 1 flypig flypig 16K Feb 23 22:06 main
$ file main 
main: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically
    linked, interpreter /lib64/ld-linux-x86-64.so.2,
    BuildID[sha1]=b279e9d824c33b2c1fac1dff7a883a4360ba4c22, for GNU/Linux 3.2.0,
    not stripped

Here we're calling gcc with our main.c code, but including the directories that contain the header and library files that were generated by the Java native build tools. The result is a 64-bit executable ELF file.

Since it's executable, we should try to execute it.

$ ./main
Initialising
Exception in thread "main" java.lang.ExceptionInInitializerError
        at okhttp3.OkHttpClient.<clinit>(OkHttpClient.java:127)
        at okhttp3.OkHttpClient$Builder.<init>(OkHttpClient.java:475)
        at uk.co.flypig.Main.main(Main.java:40)
        at java.base@23.0.2/java.lang.invoke.LambdaForm$DMH/sa346b79c
            .invokeStaticInit(LambdaForm$DMH)
Caused by: java.nio.charset.UnsupportedCharsetException: UTF-32BE
        at java.base@23.0.2/java.nio.charset.Charset.forName(Charset.java:559)
        at okhttp3.internal.Util.<clinit>(Util.java:75)
        ... 4 more

Well, this is interesting. The code is being executed. We can see that because the Initialising output is coming from our appwrapper code:

    public static void main(final String[] args) {
        System.out.println("Initialising");
[...]

But then there's an error, indicating that the OkHttpClient that we're using has failed to initialise due to an UnsupportedCharsetException error. This appears to be a known issue and, according to the bug report for it, the solution is to add the -H:+AddAllCharsets flag to the native image generator. In an attempt to fix this, I've added the following snippet to the gradle build configuration in appwrapper/build.gradle:

graalvmNative {
    binaries.all {
        // Avoid java.lang.ExceptionInInitializerError runtime error
        // See https://github.com/oracle/graal/issues/1294
        buildArgs.add('-H:+AddAllCharsets')
    }
}

Let's try rebuilding the libraries, recompiling the main executable and executing the resulting binary again.

$ GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./gradlew nativeCompile
[...]
$ pushd appwrapper/build/native/nativeCompile/
$ gcc -I ./ -L ./ -Wl,-rpath ./ -o main main.c -l:appwrapper.so
$ ./main
Initialising
Downloading video
URL: https://www.youtube.com/watch?v=xvFZjo5PgG0
Exception: org.schabi.newpipe.extractor.exceptions.ParsingException:
    Malformed url: https://www.youtube.com/watch?v=xvFZjo5PgG0
Completed

Well, the previous exception is gone and the code is getting a little further now, but it seems we have a new exception to deal with. Now we have a ParsingException failure, apparently due to a malformed URL. But it's not malformed. The good news is that there's only one place in the code where this error string appears:

$ grep -rIn "Malformed url" *
extractor/src/main/java/org/schabi/newpipe/extractor/utils/Utils.java:235:
    throw new ParsingException("Malformed url: " + url, e);

We can use this to our advantage by finding out what the underlying error message actually is, with a small adjustment to the code. Here's the change I've made:

$ git diff extractor/src/main/java/org/schabi/newpipe/extractor/utils/
    Utils.java 
diff --git a/Utils.java b/Utils.java
index c061ce30..815015ff 100644
--- a/extractor/src/main/java/org/schabi/newpipe/extractor/utils/Utils.java
+++ b/extractor/src/main/java/org/schabi/newpipe/extractor/utils/Utils.java
@@ -232,7 +232,7 @@ public final class Utils {
                 return message.substring("unknown protocol: ".length(
    ));
             }
 
-            throw new ParsingException("Malformed url: " + url, e);
+            throw new ParsingException(e.getMessage(), e);
         }
     }

After rebuilding and executing the code again, we now get a much more helpful and explanatory exception error message when running our test application:

Exception: org.schabi.newpipe.extractor.exceptions.ParsingException: Accessing a
    URL protocol that was not enabled. The URL protocol https is supported but
    not enabled by default. It must be enabled by adding the
    --enable-url-protocols=https option to the native-image command.

It seems that although they can be made available, GraalVM disables both HTTP and HTTPS protocols by default. The rationale for this is to keep the output binaries as small as possible by only enabling the things that are really needed.

Well, for our purposes we really are going to need that HTTPS protocol. It's not so clear whether we'll need HTTP, but to avoid us hitting that problem in the future I'm going to enable it as well. I've done that by adding the relevant flags to the configuration from earlier:

graalvmNative {
    binaries.all {
        // Avoid java.lang.ExceptionInInitializerError runtime error
        // See https://github.com/oracle/graal/issues/1294
        buildArgs.add('-H:+AddAllCharsets')
        // Enable network protocols
        buildArgs.add('--enable-url-protocols=http')
        buildArgs.add('--enable-url-protocols=https')
    }
}

Time to rebuild the Java and recompile the C code again.

$ GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./gradlew nativeCompile
[...]
$ gcc -I appwrapper/build/native/nativeCompile/ \
    -L appwrapper/build/native/nativeCompile/ \
    -L extractor/build/native/nativeCompile/ \
    -Wl,-rpath appwrapper/build/native/nativeCompile/ \
    -Wl,-rpath extractor/build/native/nativeCompile/ \
    -o main appwrapper/src/main.c -l:appwrapper.so -l:extractor.so

Feeling hopeful now, time to try out the executable.

$ ./main
Initialising
Downloading video
URL: https://www.youtube.com/watch?v=xvFZjo5PgG0
Video name: Rick Roll (Different link + no ads)
Uploader: Duran
Category: Entertainment
Likes: 167725
Views: 16915531
Exception in thread "main" java.lang.NoClassDefFoundError:
    Could not initialize class org.mozilla.javascript.VMBridge
        at org.mozilla.javascript.Context.exit(Context.java:482)
        at org.schabi.newpipe.extractor.utils.JavaScript.compileOrThrow
            (JavaScript.java:20)
        at org.schabi.newpipe.extractor.services.youtube.YoutubeSignatureUtils
            .getDeobfuscationCode(YoutubeSignatureUtils.java:88)
        at org.schabi.newpipe.extractor.services.youtube
            .YoutubeJavaScriptPlayerManager.deobfuscateSignature
            (YoutubeJavaScriptPlayerManager.java:145)
        at org.schabi.newpipe.extractor.services.youtube.extractors
            .YoutubeStreamExtractor.buildAndAddItagInfoToList
            (YoutubeStreamExtractor.java:1387)
        at org.schabi.newpipe.extractor.services.youtube.extractors
            .YoutubeStreamExtractor.lambda$getStreamsFromStreamingDataKey$15
            (YoutubeStreamExtractor.java:1360)
[...]
        at java.base@23.0.2/java.util.stream.ReferencePipeline
            .forEachOrdered(ReferencePipeline.java:641)
        at org.schabi.newpipe.extractor.services.youtube.extractors
            .YoutubeStreamExtractor.getItags(YoutubeStreamExtractor.java:1215)
        at org.schabi.newpipe.extractor.services.youtube.extractors
            .YoutubeStreamExtractor.getVideoStreams
            (YoutubeStreamExtractor.java:657)
        at uk.co.flypig.Main.main(Main.java:57)
        at java.base@23.0.2/java.lang.invoke.LambdaForm$DMH/sa346b79c
            .invokeStaticInit(LambdaForm$DMH)

Argh! Yet another exception. So far, all of these have been unknown unknowns. But at least we're making progress. Interestingly we can see that the execution is at least now getting a lot further than it did before. It's managed to extract some metadata about the video, namely the video name, category, likes and so on which have been correctly extracted. However, the all important URL for downloading a copy of the video isn't being output. We seem to be hitting a NoClassDefFoundError before that can happen.

The missing class is org.mozilla.javascript.VMBridge. In a previous life I did a decent amount of work with Java, but that was well over a decade ago and I'm not especially familiar with the Java landscape as it exists now. But a bit of digging around on the Web uncovers the fact that this is part of Mozilla's Rhino JavaScript library. Rhino is a JavaScript library written in Java. Despite the fact it's still being developed, it seems that GraalVM has chosen a different approach, relying instead on its own JavaScript interpreter, part of its Polyglot approach to interoperability which allows a whole host of languages to play nicely together.

Having now read up a bit more on this, it seems there are potentially two approaches here. I could try to get Rhino integrated with the build, or I could amend the code to use Polyglot instead. Since it looks like Rhino may not be supported at all with GraalVM, I've decided to give the latter a go.

As we can see if we follow the exception stacktrace above, the problematic code is in the JavaScript.java file, which is part of the NewPipe Extractor code. So I've made the following changes to it, which essentially replace the calls that use Rhino with calls to Polyglot instead.

$ git diff extractor/src/main/java/org/schabi/newpipe/extractor/utils/
    JavaScript.java
diff --git a/JavaScript.java b/JavaScript.java
index ab30ed80..ee0468a7 100644
--- a/extractor/src/main/java/org/schabi/newpipe/extractor/utils/JavaScript.java
+++ b/extractor/src/main/java/org/schabi/newpipe/extractor/utils/JavaScript.java
@@ -1,8 +1,8 @@
 package org.schabi.newpipe.extractor.utils;
 
-import org.mozilla.javascript.Context;
-import org.mozilla.javascript.Function;
-import org.mozilla.javascript.ScriptableObject;
+import org.graalvm.polyglot.Context;
+import org.graalvm.polyglot.Source;
+import org.graalvm.polyglot.Value;
 
 public final class JavaScript {
 
@@ -10,31 +10,28 @@ public final class JavaScript {
     }
 
     public static void compileOrThrow(final String function) {
+        Value value;
+        final Context context = Context.create();
         try {
-            final Context context = Context.enter();
-            context.setOptimizationLevel(-1);
-
-            // If it doesn't compile it throws an exception here
-            context.compileString(function, null, 1, null);
+              final Source source = Source.create("js", function);
+              value = context.parse(source);
         } finally {
-            Context.exit();
+            context.close(true);
         }
     }
 
     public static String run(final String function,
                              final String functionName,
                              final String... parameters) {
+        final Context context = Context.create();
         try {
-            final Context context = Context.enter();
-            context.setOptimizationLevel(-1);
-            final ScriptableObject scope = context.initSafeStandardObjects();
+            final Source source = Source.create("js", function);
+            final Value value = context.eval("js", function);
 
-            context.evaluateString(scope, function, functionName, 1, null);
-            final Function jsFunction = (Function) scope.get(functionName, 
    scope);
-            final Object result = jsFunction.call(context, scope, scope, 
    parameters);
+            Value result = value.execute((Object) parameters);
             return result.toString();
         } finally {
-            Context.exit();
+            context.close(true);
         }
     }
 

There are actually fewer changes here than I'd feared might be necessary. If this works, I think it'll make for a nice solution, given that GraalVM seems to favour Polyglot. So what happens when we rebuild and execute?

$ GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./gradlew nativeCompile
[...]
BUILD SUCCESSFUL in 5m 13s
10 actionable tasks: 6 executed, 4 up-to-date
$ gcc -I appwrapper/build/native/nativeCompile/ \
    -L appwrapper/build/native/nativeCompile/ \
    -L extractor/build/native/nativeCompile/ \
    -Wl,-rpath appwrapper/build/native/nativeCompile/ \
    -Wl,-rpath extractor/build/native/nativeCompile/ \
    -o main appwrapper/src/main.c -l:appwrapper.so -l:extractor.so
$ ./main
Initialising
Downloading video
URL: https://www.youtube.com/watch?v=xvFZjo5PgG0
Video name: Rick Roll (Different link + no ads)
Uploader: Duran
Category: Entertainment
Likes: 167747
Views: 16918500
Content: https://rr1---sn-1xopouxgoxu-aigl.googlevideo.com/videoplayback?expire
    =1740359515&ei=-3K7Z7avGr_ep-oPzcOw6QM&ip=62.3.65.133&id=o-AIiqvBpexPfeF_Ot
    ovTgn8l1hpOBffKxoHfspDOTcYrC&itag=18&source=youtube&requiressl=yes&xpc=EgVo
    2aDSNQ%3D%3D&met=1740337915%2C&mh=bl&mm=31%2C29&mn=sn-1xopouxgoxu-aigl%2Csn
    -aigl6ned&ms=au%2Crdu&mv=m&mvi=1&pl=22&rms=au%2Cau&initcwndbps=3486250&bui=
    AUWDL3x9n8km503f_C6UeJ-nmAK-f2JXxFo3iHX8HgXWUrBxXlsxrKr63R0LnSqXd7G4MsxTbK3
    rHtSa&spc=RjZbSTMxxHkVycQ394WOmaNPcAiMR28iW5oscKuzyUPoVk6lOQ&vprv=1&svpuc=1
    &mime=video%2Fmp4&rqh=1&cnr=14&ratebypass=yes&dur=7.685&lmt=170873886759751
    5&mt=1740337503&fvip=4&fexp=51326932&c=ANDROID&txp=4530434&sparams=expire%2
    Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cxpc%2Cbui%2Cspc%2Cvprv%2Csvpuc
    %2Cmime%2Crqh%2Ccnr%2Cratebypass%2Cdur%2Clmt&sig=AJfQdSswRgIhANW5FqNuItTPiG
    SvgBHqzpa0UGLBegd9wUYd8-yHjH49AiEAmf8RsXSBT_Z4EpLKY7Mx6APBVjdK80dKnSodnJHKd
    n4%3D&lsparams=met%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Crms%2Cinitcwndbps&
    lsig=AGluJ3MwRQIgajuyXNfNyrHYnh6kS5ZbH47Hj8MO_-3vRauDTqz8nKYCIQDIHsgn6W7JBb
    6GEo7Wr5qhLwk9Lx1FJW_r6UeENOWE-w%3D%3D&cpn=nQp7YGHcYfwig4NE
Completed

And there it is! The example application, compiled from C code, linked to the native Java code, is producing exactly the result it should be! This is a good place to stop for today, but before signing off, I'm going to spend just a little time reflecting on where we've got to and what still needs to be done.

There are a few takeaways. First is that building native binaries using Gradle turns out to be pretty straightforward, but with a few easy-to-hit gotchas, including the fact that HTTP and HTTPS requests are disabled by default.

The other takeaway is that we have to switch out the Rhino JavaScript interpreter for the JavaScript interpreter provided by Polyglot. We did this for the one place it was used for our test application, but a quick grep suggests it's used in several other places as well:

$ grep -rIn "org.mozilla.javascript" * --include="*.java"
extractor/src/main/java/org/schabi/newpipe/extractor/utils/jsextractor/
    TokenStream.java:3:import org.mozilla.javascript.Context;
extractor/src/main/java/org/schabi/newpipe/extractor/utils/jsextractor/
    TokenStream.java:4:import org.mozilla.javascript.Kit;
extractor/src/main/java/org/schabi/newpipe/extractor/utils/jsextractor/
    TokenStream.java:5:import org.mozilla.javascript.ObjToIntMap;
extractor/src/main/java/org/schabi/newpipe/extractor/utils/jsextractor/
    TokenStream.java:6:import org.mozilla.javascript.ScriptRuntime;
extractor/src/main/java/org/schabi/newpipe/extractor/utils/jsextractor/
    TokenStream.java:9:/* Source: Mozilla Rhino, org.mozilla.javascript.Token
extractor/src/main/java/org/schabi/newpipe/extractor/utils/jsextractor/
    Lexer.java:3:import org.mozilla.javascript.Context;

Nevertheless, when we run the test application it's providing relevant metadata associated with a YouTube video and offering up a URL from which we're able to download the video. I find this really encouraging, because it suggests that it should be possible to get things to work properly when the Extractor is built into a native binary for Sailfish OS.

Up until this point that wasn't clear; not it looks a lot more promising.

So where next? Tomorrow I'll need to review the Extractor code again and remove all references to Rhino. That means taking a look at the TokenStream.java code to see whether we can switch out Rhino for Polyglot. Once I've done that, I'll then switch back to Sailfish OS to see whether we can get all this working there.

Assuming we can, the next step after that will be figuring out how to expose the Extractor functionality in a form that we can use from our Sailfish OS app. That'll require a lot of thought and work, but at least by that point I'll no longer be scrabbling in the dark. By then, we'll know whether the result we want is possible or not.

Yesterday we looked at the code needed to expose the Java internals of NewPipe Extractor to C++ when using GraalVM in a way that would allow us to pass richer datatypes and structures between them.

Today we're taking another slight detour, this time to find out whether it's possible to build the Java library using the Sailfish SDK, rather than on the phone itself. You'll recall we looked at building the library on-phone on Day 4.

Before getting in to things I'm going to update my SDK. I want the absolute latest tooling and targets to maximise the chances of success.

$ ./SDKMaintenanceTool --silentUpdate -v
IFW Version: 3.2.3, built with Qt 5.15.14.
Build date: Aug 21 2024
Installer Framework SHA1: 7699eb32
[0] Language: en-GB
[...]
[1231] Install size: 3 components
[...]
[144072] 100% ...
[...]
[239236] Sync completed\n
[245881] Target 'SailfishOS-5.0.0.55EA-aarch64' set up\n
[245899] Done
[...]
[246115] Stopping the build engine… (this may take some time)
[248920] Components updated successfully.

Great! That got all of the updates. Let's see what we have.

$ sfdk tools target list
sfdk: [I] Starting the build engine…
SailfishOS-3.4.0.24-aarch64    sdk-provided             
SailfishOS-3.4.0.24-armv7hl    sdk-provided             
SailfishOS-3.4.0.24-i486       sdk-provided             
SailfishOS-4.6.0.13-aarch64    sdk-provided,latest      
SailfishOS-4.6.0.13-armv7hl    sdk-provided,latest      
SailfishOS-4.6.0.13-i486       sdk-provided,latest      
SailfishOS-5.0.0.55EA-aarch64  sdk-provided,early-access

I'll be using the new SailfishOS-5.0.0.55EA-aarch64 target, so the next step is to configure the SDK to use this by default.

$ sfdk config --global target=SailfishOS-5.0.0.55EA-aarch64
$ sfdk config
# ---- command scope ---------
# <clear>

# ---- session scope ---------
# <clear>

# ---- global scope ---------
target = SailfishOS-5.0.0.55EA-aarch64
output-prefix = ~/RPMS
device = kolbe

Next I need to install the GraalVM tooling inside the SDK. There are two layers to the SDK: first of all you enter the tooling using the sfdk command, following which you enter the scratchbox2 target using the sb2 command.

Even though I've set a default target, when entering scratchbox2 manually like this, we also need to specify the target explicitly.

$ sfdk engine exec
$ sb2 -t SailfishOS-5.0.0.55EA-aarch64
$ ls
compile.sh  flatbuffers.sh  mvnw  pom.xml  src
$ mkdir graalvm
$ pushd graalvm/
graalvm sailing-the-flood-to-java/java-part
$ curl -O https://download.oracle.com/graalvm/23/latest/
    graalvm-jdk-23_linux-aarch64_bin.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  334M  100  334M    0     0  4245k      0  0:01:20  0:01:20 --:--:-- 4457k
$ tar -xf graalvm-jdk-23_linux-aarch64_bin.tar.gz
$ GRAAL=${PWD}/graalvm-jdk-23.0.2+7.1
$ mkdir m2
$ MAVENLOCAL=${PWD}/m2
$ popd
sailing-the-flood-to-java/java-part

That's the tooling set up, now let's try building as if we were running the command on a phone:

$ GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./mvnw -Dmaven.repo.local=$MAVENLOCAL \
    clean install -Pnative
Error occurred during initialization of VM
Could not reserve enough space for 8118272KB object heap

This isn't totally unexpected: the compiler hit a memory limit reserving memory for the job. The problem here isn't Maven, out build tool, but rather the Java Virtual Machine that it's attempting to spawn. So for testing purposes we can jump straight to that and see if we have better luck if we call it directly.

$ GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL graalvm/graalvm-jdk-23.0.2+7.1/bin/java \
    --help
Error occurred during initialization of VM
Could not reserve enough space for 8118272KB object heap

The same error. Thankfully there are some ways to control the amount of memory that the JVM will try to claim. Unfortunately we can't output the help text using java --help inside scratchbox2 (the SDK engine target) itself because apparently the JVM has to get up and running before it'll even print out the help.

So instead I'm going to output it on my laptop with a JVM installed, outside of scratchbox2. Here are the relevant parts of the help text output:

$ java --help-extra | grep size
    -Xmn<size>        sets the initial and maximum size (in bytes) of the heap
    -Xms<size>        set initial Java heap size
    -Xmx<size>        set maximum Java heap size
    -Xss<size>        set java thread stack size

On top of these we also need to disable Compressed Class Space using the -XX:-UseCompressedClassPointers and -XX:+UseCompressedOops flags. These aren't listed in the Java man pages; nor are they listed in either the --help or --help-extra output. But without these we simply get an Out-Of-Memory error:

$ JVMMEM=512m _JAVA_OPTIONS="-Xmn${JVMMEM} -Xms${JVMMEM} -Xmx${JVMMEM} \
    -Xss${JVMMEM}" GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./mvnw \
    -Dmaven.repo.local=$MAVENLOCAL clean install -Pnative
Picked up _JAVA_OPTIONS: -Xmn512m -Xms512m -Xmx512m -Xss512m
Error occurred during initialization of VM
Could not allocate compressed class space: 1073741824 bytes

Notice here that I'm setting the flags using _JAVA_OPTIONS. That's so that they get automatically picked up by calls to use the JVM even though the JVM commands are being called by maven. Notice also that we get some output stating that the arguments have been picked up, so we know this approach is working.

With the compressed class space flags added we get some slightly more nuanced output. For example if we use a memory value that's too low we get a segmentation fault:

$ JVMMEM=128m _JAVA_OPTIONS="-Xmn${JVMMEM} -Xms${JVMMEM} -Xmx${JVMMEM} \
    -Xss${JVMMEM} -XX:-UseCompressedClassPointers -XX:-UseCompressedOops" \
    GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./mvnw -Dmaven.repo.local=$MAVENLOCAL \
    clean install -Pnative
Picked up _JAVA_OPTIONS: -Xmn128m -Xms128m -Xmx128m -Xss128m
    -XX:-UseCompressedClassPointers -XX:-UseCompressedOops
Segmentation fault (core dumped)

Raise it higher, to around 768 MiB and it looks like the JVM is managing to get further through its initialisation sequence:

$ JVMMEM=768m _JAVA_OPTIONS="-Xmn${JVMMEM} -Xms${JVMMEM} -Xmx${JVMMEM} \
    -Xss${JVMMEM} -XX:-UseCompressedClassPointers -XX:-UseCompressedOops" \
    GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./mvnw -Dmaven.repo.local=$MAVENLOCAL \
    clean install -Pnative
Picked up _JAVA_OPTIONS: -Xmn768m -Xms768m -Xmx768m -Xss768m
    -XX:-UseCompressedClassPointers -XX:-UseCompressedOops
[0.773s][warning][os,thread] Failed to start thread "Unknown thread"
    - pthread_create failed (EAGAIN) for attributes: stacksize: 786432k,
    guardsize: 0k, detached.
[0.782s][warning][os,thread] Failed to start the native thread for
    java.lang.Thread "Finalizer"
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create native thread:
    possibly out of memory or process/resource limits reached
        at java.lang.Thread.start0(java.base/Native Method)
        at java.lang.Thread.start(java.base/Thread.java:1518)
        at java.lang.ref.Finalizer.startFinalizerThread(java.base/
    Finalizer.java:190)
        at java.lang.ref.Reference$1.startThreads(java.base/Reference.java:319)
        at java.lang.System.initPhase1(java.base/System.java:2214)

This error seems to happen when the JVM fails to start threads, but I'm not able to fix this by increasing the thread limit using ulimit -u. So I suspect this may actually be memory related as well.

Increasing the memory to nearly a gigabyte leaves us with similar output:

$ JVMMEM=1020m _JAVA_OPTIONS="-Xmn${JVMMEM} -Xms${JVMMEM} -Xmx${JVMMEM} \
    -Xss${JVMMEM} -XX:-UseCompressedClassPointers -XX:-UseCompressedOops" \
    GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./mvnw -Dmaven.repo.local=$MAVENLOCAL \
    clean install -Pnative
Picked up _JAVA_OPTIONS: -Xmn1020m -Xms1020m -Xmx1020m -Xss1020m
    -XX:-UseCompressedClassPointers -XX:-UseCompressedOops
[0.769s][warning][os,thread] Failed to start thread "Unknown thread"
    - pthread_create failed (EAGAIN) for attributes: stacksize: 1044480k,
    guardsize: 0k, detached.
[0.777s][warning][os,thread] Failed to start the native thread for
    java.lang.Thread "Reference Handler"
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create native thread:
    possibly out of memory or process/resource limits reached
        at java.lang.Thread.start0(java.base/Native Method)
        at java.lang.Thread.start(java.base/Thread.java:1518)
        at java.lang.ref.Reference.startReferenceHandlerThread(java.base/
    Reference.java:306)
        at java.lang.ref.Reference$1.startThreads(java.base/Reference.java:318)
        at java.lang.System.initPhase1(java.base/System.java:2214)

But as the configuration reaches a gigabyte, the original error returns:

$ JVMMEM=1021m _JAVA_OPTIONS="-Xmn${JVMMEM} -Xms${JVMMEM} -Xmx${JVMMEM} \
    -Xss${JVMMEM} -XX:-UseCompressedClassPointers -XX:-UseCompressedOops" \
    GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL ./mvnw -Dmaven.repo.local=$MAVENLOCAL \
    clean install -Pnative
Picked up _JAVA_OPTIONS: -Xmn1021m -Xms1021m -Xmx1021m -Xss1021m
    -XX:-UseCompressedClassPointers -XX:-UseCompressedOops
Error occurred during initialization of VM
Could not reserve enough space for 1046528KB object heap

I also tried performing similar actions using java directly rather than going via Maven, but with very similar results:

$ JVMMEM=1020m _JAVA_OPTIONS="-Xmn${JVMMEM} -Xms${JVMMEM} -Xmx${JVMMEM} \
    -Xss${JVMMEM} -XX:-UseCompressedClassPointers -XX:-UseCompressedOops" \
    GRAALVM_HOME=$GRAAL JAVA_HOME=$GRAAL \
    graalvm/graalvm-jdk-23.0.2+7.1/bin/java --help
Picked up _JAVA_OPTIONS: -Xmn1020m -Xms1020m -Xmx1020m -Xss1020m
    -XX:-UseCompressedClassPointers -XX:-UseCompressedOops
[0.796s][warning][os,thread] Failed to start thread "Unknown thread"
    - pthread_create failed (EAGAIN) for attributes: stacksize: 1044480k,
    guardsize: 0k, detached.
[0.805s][warning][os,thread] Failed to start the native thread for
    java.lang.Thread "Finalizer"
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create native thread:
    possibly out of memory or process/resource limits reached
        at java.lang.Thread.start0(java.base/Native Method)
        at java.lang.Thread.start(java.base/Thread.java:1518)
        at java.lang.ref.Finalizer.startFinalizerThread(java.base/
    Finalizer.java:190)
        at java.lang.ref.Reference$1.startThreads(java.base/Reference.java:319)
        at java.lang.System.initPhase1(java.base/System.java:2214)

This is all rather sad. When using scratchbox2 the JVM is being executed inside a QEMU container. It's possible there's a 32 bit/64 bit problem here, or that there's some deeper incompatibility (it has been known for things to simply fail when run within QEMU). I'd really hoped that this may have changed or been fixed since I last tried this back in 2022, but sadly that's not the case. What's more, I'm not convinced I have the skill or knowledge to fix it myself.

Maybe as we continue our journey a new path will open up, maybe someone out there has an idea for something to try (in which case, please do let me know!), or maybe I'll muster up the courage to try to better understand and fix the underlying issue. But for now, it means reverting to our fallback of executing GraalVM on the phone.

That's it for today. Tomorrow we move into new territory: attempting to get NewPipe Extractor built using GraalVM!

Yesterday we got Thigg's sailing-the-flood-to-java Java code compiling into a native library for Sailfish OS. I can't claim any credit for this because Thigg did all of the hard work. It's nevertheless an important step towards getting a similar library built from the NewPipe Extractor code.

But we can't just replace the Java portion of sailing-the-flood-to-java with the NewPipe Extractor code and expect it to work, because Thigg also had to do some work to expose the interface in the Game.java implementation. The really interesting bit happens in two files: the GameInterface.java file and the Shared.java file.

We don't have to worry about the internals of GameInterface.java, but it does help to consider the interface, which looks like this (with apologies for abusing Java syntax):

public class GameInterface {

    @SneakyThrows
    public static byte[] startGame(String json);

    private static byte[] getGameState();

    @SneakyThrows
    public static byte[] flood(String json);
}

There are three methods. The first initialises the game. The input string is a blob of JSON containing a size entry and a numColors entry, for example like this:

{
    "size": 20,
    "numColors": 4
}

The code inside this method takes this JSON and populates an instance of the NewGameParams class:

public class NewGameParams {
    public int size;
    public int numColors;
}

JSON is being used as the interop mechanism because it can be managed as just a string and we can move strings between C++ and Java without too much concern for types. On the other hand, the JSON has its own type semantics which can be harnessed to share richer datatypes between the C++ and Java code.

A complex type, all bundled up into a simple string.

The method returns a byte array, which is also a very low level datatype, but which is interesting because Thigg has implemented this using FlatBuffers. The method returns the full game state, including the colour of each block on the game board. Since there can be a large number of blocks (e.g. a 256 by 256 board would be an array of 65536 values), this would make for a rather cumbersome JSON structure. FlatBuffers provide a more efficient interop mechanism, which is why Thigg chose it.

For NewPipe my guess is that we won't need to worry about performance because we'll mostly be passing relatively short strings backwards and forwards (e.g. URLs). If we were transfering multimedia (images, videos or audio) then JSON would be a terrible, terrible, terrible choice. But for short strings, it's going to be perfectly efficient (probably no less efficient than FlatBuffers in practice).

Nevertheless it's interesting to see the mechanisms for how Thigg achieved this. You can see the Flatbuffers calls in flatbuffers.sh, which looks like this:

flatc --cpp -o ../qt-part/src/ src/main/resources/gamestate.fbs --gen-object-api
flatc --java -o ./src/main/java/ src/main/resources/gamestate.fbs 
    --gen-object-api

The first call generates the files needed for the C++ code. The second call generates the files needed for the Java code. The input comes from gamestates.gbs which looks like this:

namespace Game.model;

table GameState {
  field:[short];
  won: bool;
  steps: [int];
  max_steps:int;
  width: int;
  height: int;
  scores: [Score];
}

table Score {
   player: int;
   steps: int;
   won: bool;
   combo_score: int;
}

root_type GameState;

This tells us that there are two datatypes that will be used in both the C++ and Java code and which may need to be transferred between the two. The GameState is what's returned by startGame().

After passing this file through the flatc tool these two times, we end up with the following files:

1. GameState.java: Java source file containing the GamePlay interface.
2. GameStateT.java: Java source file containing the GamePlayT table interface.
3. gamestate_generated.h: C++ source containing the GamePlay and GamePlayT interfaces.

On the Java side the implementation of startGame() in GameInterface.java populates (via getGameState()) an instance of the GameStateT class, then calls serialize() on it to turn it into a byte array, which is returned to the C++ code. Note that serialize() is created as part of the generated FlatBuffers code.

On the C++ side we need to look in the jgateway.cpp file to see what's going on. When the postMessage() method is called with a method name provided as a string, the Java code is called using JGateway__invoke(). This returns a byte array stored in the result parameter which can then be deserialised as follows:

    auto gamestate = Game::model::UnPackGameState(result);

The UnPackGameState() method comes from the generated code which returns a GameStateT instance which is then used to populate the QGameState object that's a member of JGateway.

So what we see is that when we pass parameters from C++ to JavaScript it's done as a JSON string. The data that's returned from Java to C++ is transferred as a FlatBuffer.

The good news is that we don't have to worry about the FlatBuffer part as we can pass our results back in JSON format too. We'll need to make some changes to the JGateway C++ code and to the Shared Java code. But actually the changes will likely be pretty minimal.

Tomorrow I'm going to have another go at getting the GraalVM build to work on the Sailfish SDK. It might work, or it might not. But it feels like it's worth a try.

Yesterday we built the NewPipe Extractor for desktop Linux. Our ultimate aim is to get it built for use with Sailfish OS so that the API can be called from C++ or QML code.

But we'll need to spend a bit of time building up to that, because I want to make use of a very specific approach to using Java code on Sailfish OS.

This was something I learnt several years ago from the esteemed Java and Sailfish OS crossover expert Thigg. Back in 2022 I wrote a piece for the Sailfish Community News on a project called Sailing-to-Coffee that Thigg created. Thigg built a pipeline for making use of GraalVM with Sailfish OS.

At the time I thought it was shear brilliance. As time has gone on my opinion hasn't changed and so I'm keen to make use of the technique with NewPipe as well.

I'll let you read the article from the Community News to learn about what's going on. What I wrote there is still valid today. Today, however, I'm going to download the code for Thigg's sailing-the-flood-to-java application, which neatly demonstrates the technique.

If you've been following along closely you'll also note that this approach was also more recently suggested by pherjung, having read about it on the Sailfish OS Forum.

I'm especially gratified that following that, Thigg also got in touch to offer to contribute. We've arranged to have a call to discuss this further and I'm expecting this to add rocket fuel to the project!

Getting the GraalVM pipeline set up is a bit of work, so it'll be helpful to get this successfully running before returning to the NewPipe Extractor code.

To do this I'm going to start by setting up Thigg's sailing-the-flood-to-java application, or at least the Java build part, so that it outputs a native library for use on Sailfish OS built from the Java code in the repository. I've got my own fork of the code and that's where I'm going to start:

git clone git@github.com:llewelld/sailing-the-flood-to-java.git
cd sailing-the-flood-to-java
git submodule update --init --recursive
cd java-part

Unfortunately some things have changed since Thigg set up this repository. In particular it seems Project Lombok now needs to be added as a dependency (maybe it always did, but at least I've found I can't now build it without). This can be added to the plugins section of the pom.xml file. All I've done is add the following inside the build/plugins element:

<plugin>
    <groupId>org.apache.maven.plugins</groupid>
    <artifactId>maven-compiler-plugin</artifactid>
    <version>3.6.1</version>
    <configuration>
        <annotationProcessorPaths>
            <path>
                <groupId>org.projectlombok</groupid>
                <artifactId>lombok</artifactid>
                <version>1.18.30</version>
            </path>
        </annotationprocessorpaths>
    </configuration>
</plugin>

I've also bumped up the Lombok version to 1.18.30:

         <dependency>
             <groupId>org.projectlombok</groupid>
             <artifactId>lombok</artifactid>
-            <version>1.18.24</version>
+            <version>1.18.30</version>
         </dependency>

For context, Lombok doesn't provide any user-facing functionality for the application, but it does make writing Java code easier. For example, rather than having to write getter and setter methods for each field you can just annotate a class with @Data. Similarly you can use @NoArgsConstructor to have Lombok create the constructor for you.

Here's some example code taken from the NewGameParams.java file that illustrates this:

@Data
@NoArgsConstructor
public class NewGameParams {
    public int size;
    public int numColors;
}

This is equivalent to having to write out the following code in full:

@Data
@NoArgsConstructor
public class NewGameParams {
    public int size;
    public int numColors;
    
    public int getSize() { return size; };
    public void setSize(int size) { this.size = size;};
    public int getNumColors() { return numColors; };
    public void setNumColors(int numColors) { this.numColors = numColors; }

    public GameStateT() {
        this.size = 0;
        this.numColors = 0;
    }
}

You can see why people like Lombok. But it does add an extra dependency, so it's inclusion isn't entirely pain free.

Apart from that we also need to update the GraalVM configuration. GraalVM is going to provide us with the real magic, so we need to get this right. In the original version of the pom.xml was the following snippet of XML:

    <dependency>
        <groupId>org.graalvm.sdk</groupid>
        <artifactId>graal-sdk</artifactid>
        <version>22.1.0.1</version>
    </dependency>

This sets up the GraalVM to be a build dependency. But it turns out GraalVM doesn't like this anymore, so I've completely removed it as a dependency. We do still need GraalVM of course, but it's already defined as a plugin for native builds in the same file and it turns out that's as much as we need now.

That sets up the build structure. The next step is to adjust the compile.sh script. This script is unusual for developing code on Sailfish OS because, rather than building things in the Sailfish SDK, it instead copies the code to an actual phone and builds it there.

This may sound a little crazy, but it also makes perfect sense. One of the key motivations for having the Sailfish SDK is that it mimics both the environment and the target architecture (the CPU type: x86, arm, aarch64, etc.) of the phone you want your code to run on.

Why not avoid all that by going directly to the phone and building it there?

Going direct to the phone has a number of benefits. Unless it happens to match the architecture of the system you're working on, the Sailfish SDK has to emulate the target architecture, which can be very slow. The SDK performs a number of clever tricks to try to speed things up, such as switching out emulated tools for native tools where it can, but it still has an impact. Going direct to the phone also relaxes some restrictions that might otherwise apply to the Java compiler, specifically around memory usage.

It's the memory usage restraint that's key for us here. I've tried to set GraalVM up for use within the Sailfish SDK in the past with only limited success. It's possible things have changed in the meantime, so with this project I'm planning to give this another go at some point.

But first things first, let's get a working build.

The next step is to set up GraalVM on a phone. Thankfully all that's needed for this is a directory to unpack the appropriate GraalVM JDK archive into.

$ ssh defaultuser@172.28.172.1
$ mkdir ~/Documents/Development/newpipe/graalvm
$ cd ~/Documents/Development/newpipe/graalvm
$ curl -O https://download.oracle.com/graalvm/23/latest/
    graalvm-jdk-23_linux-aarch64_bin.tar.gz
$ tar -xf graalvm-jdk-23_linux-aarch64_bin.tar.gz

I executed these commands on my phone. Note also that the root of the project folder we're using is /home/defaultuser/Documents/Development/newpipe/. We'll be needing this again in the near future.

The next step is to take a look at and edit the compile.sh shell script. We do this not on the phone but on the device (desktop, laptop,...) we're using to connect to our phone for development. In future I'll refer to this as the orchestrating device, because it's the device that's orchestrating the build.

Here's what I've got in the compile.sh file:

#!/bin/bash
COMPILEHOST=defaultuser@172.28.172.1
COMPILEHOST_WORKSPACE=/home/defaultuser/Documents/Development/newpipe/java-part
COMPILEHOST_GRAAL=/home/defaultuser/Documents/Development/newpipe/graalvm/
    graalvm-jdk-23.0.2+7.1
COMPILEHOST_MAVENLOCAL=/home/defaultuser/Documents/Development/newpipe/graalvm/
    m2
echo "transfering data"
rsync . $COMPILEHOST:$COMPILEHOST_WORKSPACE -r
echo "starting compilation"
ssh $COMPILEHOST "cd $COMPILEHOST_WORKSPACE; 
    GRAALVM_HOME=$COMPILEHOST_GRAAL \
    JAVA_HOME=$COMPILEHOST_GRAAL ./mvnw \
    -Dmaven.repo.local=$COMPILEHOST_MAVENLOCAL clean install -Pnative"
scp $COMPILEHOST:"$COMPILEHOST_WORKSPACE/target/*.h" ../qt-part/lib/
scp $COMPILEHOST:"$COMPILEHOST_WORKSPACE/target/javafloodjava.so" \
    ../qt-part/lib/libjavafloodjava.so

Let's go through this file step-by-step to explain what's going on and so — if you're following along — you can update it with values relevant to your set up.

First we set the COMPILEHOST environment variable. This should contain the endpoint string for logging in to your phone via SSH. Typically this includes a username and an IP address. As you can see, that's what I've got for my setup.

Second we set COMPILEHOST_WORKSPACE to the directory where we want the Java code to be copied over to. Earlier we created a directory /home/defaultuser/Documents/Development/newpipe/ and now we're using a subdirectory of this called java-part, because it's where the JAVA code is going to go.

Next we set the COMPILEHOST_GRAAL environment variable. This should point to the directory where we unpacked the GraalVM to. This contains all of the GraalVM code, as we can see:

$ ls /home/defaultuser/Documents/Development/newpipe/graalvm/
    graalvm-jdk-23.0.2+7.1
GRAALVM-README.md  conf     legal                                man
LICENSE.txt        include  lib                                  release
bin                jmods    license-information-user-manual.zip

The final environment variable we set up is COMPILEHOST_MAVENLOCAL. This is something I added myself because otherwise Maven (the build tool that we're going to use) will store all of the files it downloads to a hidden directory on the phone. I really dislike it when tools do this; it's problematic for many reasons, not least because there's a good chance we'll want to remove all these files later. I always like to know where things are getting stored.

Later on in this script there's a ./mvnw command that gets run on the phone and I've added the COMPILEHOST_MAVENLOCAL reference to this.

There's no need to change the rsync command. This copies the source directory over to the phone ready to be built there.

Although I've edited the proceeding SSH statement, this shouldn't need configuring any further.

At the end there's an scp command which is used to copy the resulting library file that's built off the phone and back on to the orchestrating device.

And that's it. All of the really hard work is going to be done by the mvnw command.

When we execute the script, it will SSH into the phone, build the library and then copy it back, like this:

$ ./compile.sh 
transfering data
starting compilation
[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------------< de.thigg:javafloodjava >-----------------------
[INFO] Building javafloodjava 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[...]

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running GameTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.169 sec
Running InterfaceTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[...]

===============================================================================
GraalVM Native Image: Generating 'javafloodjava' (shared library)...
===============================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/
    BuildOutput.md
-------------------------------------------------------------------------------
[1/8] Initializing...                                          (21.0s @ 0.09GB)
[...]
[2/8] Performing analysis...  [*****]                          (87.7s @ 0.45GB)
    4,354 reachable types   (74.2% of    5,871 total)
    5,622 reachable fields  (48.6% of   11,558 total)
   24,439 reachable methods (50.4% of   48,535 total)
    1,343 types,    14 fields, and   159 methods registered for reflection
       57 types,    57 fields, and    52 methods registered for JNI access
        4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                     (11.1s @ 0.51GB)
[4/8] Parsing methods...      [*****]                          (26.9s @ 0.58GB)
[5/8] Inlining methods...     [***]                             (9.4s @ 0.66GB)
[6/8] Compiling methods...    [**************]                (220.0s @ 1.09GB)
[7/8] Laying out methods...   [****]                           (13.8s @ 1.25GB)
[8/8] Creating image...       [***]                            (11.4s @ 0.39GB)
[...]
-------------------------------------------------------------------------------
   26.6s (6.4% of total time) in 2369 GCs | Peak RSS: 1.83GB | CPU load: 5.45
-------------------------------------------------------------------------------
Build artifacts:
 newpipe/java-part/target/graal_isolate.h (c_header)
 newpipe/java-part/target/graal_isolate_dynamic.h (c_header)
 newpipe/java-part/target/javafloodjava.h (c_header)
 newpipe/java-part/target/javafloodjava.so (shared_library)
 newpipe/java-part/target/javafloodjava_dynamic.h (c_header)
===============================================================================
Finished generating 'javafloodjava' in 6m 52s.
[...]
graal_isolate.h                                   100% 5426   262.1KB/s   00:00
graal_isolate_dynamic.h                           100% 5538   804.7KB/s   00:00
javafloodjava.h                                   100%  422    74.9KB/s   00:00
javafloodjava_dynamic.h                           100%  478    64.9KB/s   00:00
javafloodjava.so

As you can see, we're left with a juicy-looking javafloodjava.so native library. That's what the C++ code will end up calling.

My plan is to have a similar arrangement to get NewPipe Extractor to build to a native library on my phone. But before doing that we should first also explore how Thigg managed to get the Java methods exposed for use in this library. That's what we'll do tomorrow.

I also want to try once again to get this building on the Sailfish SDK. This would have some benefits, for example it might allow us to build using Chum/OBS. That'll be for a future entry as well.

One final thing I wanted to mention is that yesterday I received a helpful email from Michał Szczepaniak, author of microTube, a YouTube app for Sailfish OS.

Michał suggested that rather than build an entirely new app, an alternative would be to set up NewPipe Extractor as a backend for microTube. He highlighted in particular that he's already had to switch out the backend before, given the projects he's used to access YouTube have had a tendency to come and go. Much like NewPipe, this has left microTube with a good separation between the front-end and backend code.

I personally think this would be a great idea and I'm grateful to Michał for reaching out to me. As this NewPipe project goes on I think it will become clearer whether this is a good fit or not. For example, NewPipe supports multiple services, not just YouTube, so this may or may not fit well with the approach microTube uses. But I'm very much in favour of building up the existing ecosystem rather than fragmenting it, so this is an option I'll be taking a very serious look into.

Yesterday we cloned the NewPipe Extractor repository and built ourselves the library. We weren't able to do anything with the resulting extractor-v0.24.5.jar archive, but our plan is to rectify that today.

The good news is that someone else has already come up with a nice wrapper for making use of the library, called New Valve. The bad news is that this is written in Kotlin and we really want to stick to Java.

So today I'll have to spend a little time converting it to Java. There's actually not a lot of code involved, so this shouldn't be too arduous.

The key piece of code is the DownloaderImpl class. This uses OkHttp to create a simple HTTP Client which is provided to the NewPipe Extractor class and used for downloading data over HTTP.

Presumably NewPipe Extractor doesn't come with its own HTTP Client because different apps will have different ways of wanting to provide this functionality. This OkHttp wrapper looks like it'll be more than enough for our needs.

To get this all setup I've created a new directory appwrapper inside the repository which contains three files and, as is typically for Java projects, way too much directory depth.

$ tree appwrapper/
appwrapper/
├── build.gradle
└── src
    └── main
        └── java
            └── uk
                └── co
                    └── flypig
                        ├── DownloaderImpl.java
                        └── Main.java

6 directories, 3 files

In case you're not familiar with Java, convention is that you put your file in a namespace that's the inverse of a URL. In my case I'm using uk.co.flypig. Unlike for C++ you can only have one class per file; the filename has to match the classname and the folder hierarchy typically matches the namespace.

Hence the deep and largely empty directory hierarchy.

As you may have guessed the build.gradle contains the instructions for building the code. This file is actually surprisingly terse:

plugins {
    id 'checkstyle'
    id 'application'
}

apply plugin : "java"

application {
    mainClass = 'uk.co.flypig.Main'
}

checkstyle {
    getConfigDirectory().set(rootProject.file("checkstyle"))
    ignoreFailures = false
    showViolations = true
    toolVersion = checkstyleVersion
}

checkstyleTest {
    enabled = false // do not checkstyle test files
}

dependencies {
    implementation project(':timeago-parser')
    api project(':extractor')

    implementation "com.squareup.okhttp3:okhttp:3.12.13"
}

Most of this is — I think — self explanatory. The checkstyle sections are to allow the code style to be tested against the requirements for the project (so I could have left these sections out if I'd wanted). The dependencies section highlights that the code makes use of the two other projects in the repository, along with OkHttp. With these added gradle will deal with downloading and hooking in the dependencies automatically.

The only other thing to note is that the application section tells gradle how to execute the code.

I also had to make some small changes to the gradle configuration of the containing directory (the top level of the repository). First to add the appwrapper directory to build.gradle, like so:

diff --git a/build.gradle b/build.gradle
index 335960db..b57add8c 100644
--- a/build.gradle
+++ b/build.gradle
@@ -37,4 +37,5 @@ allprojects {
 dependencies {
     api project(':extractor')
     implementation project(':timeago-parser')
+    implementation project(':appwrapper')
 }

Second to add appwrapper to the settings.gradle like so:

diff --git a/settings.gradle b/settings.gradle
index 725f6505..5cdd3d7d 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -1,1 +1,1 @@
-include 'extractor', 'timeago-parser'
+include 'extractor', 'timeago-parser', 'appwrapper'

Let's now turn to the code itself. I'm not going to paste it all here, but I will link to it in the repository. First we have the DownloaderImpl class. This is a conversion of the Kotlin class of the same name that we saw earlier. It extends the Downloader interface from the Extractor by providing init() and execute() methods.

In short, the execute() method is called by the Extractor when it wants to download a file from somewhere. The method returns a Response object, also defined as part of the Extractor interface, which represents the data retrieved.

The OkHttp library uses a thread pool to parallelise HTTP requests, but it likes to keep this pool around in case there are any new connections to be made. In testing I found this caused the app to linger for a minute or more before quitting while the threadpool awaits deinitialise. I therefore also added a teardown() method which forcefully deactivates the threadpool. This gets called when the app quits so that it returns immediately, rather than after a prolonged delay.

This DownloaderImp class then gets used in our main() method, which is really just a test function so we know things are working correctly:

public static void main(final String[] args) {
    System.out.println("Initialising");
    final String url = args.length > 0
        ? args[0]
        : "https://www.youtube.com/watch?v=xvFZjo5PgG0";
    final OkHttpClient.Builder builder = new OkHttpClient.Builder();
    final DownloaderImpl downloader = DownloaderImpl.init(builder);
    NewPipe.init(downloader);

    try {
        System.out.println("Downloading video");
        System.out.println("URL: " + url);
        final StreamingService service = ServiceList.YouTube;
        final StreamExtractor extractor = service.getStreamExtractor(url);
        extractor.fetchPage();

        System.out.println("Video name: " + extractor.getName());
        System.out.println("Uploader: " + extractor.getUploaderName(
    ));
        System.out.println("Category: " + extractor.getCategory());
        System.out.println("Likes: " + extractor.getLikeCount());
        System.out.println("Views: " + extractor.getViewCount());

        final java.util.List<VideoStream> streams = extractor.getVideoStreams();
        for (final VideoStream stream : streams) {
            System.out.println(&quot;Content: &quot; + stream.getContent());
        }

    } catch (final Exception e) {
        System.out.println(&quot;Exception: &quot; + e);
    }

    try {
        downloader.teardown();
    } catch (final IOException e) {
        System.out.println(&quot;IOException: &quot; + e);
    }

    System.out.println(&quot;Completed&quot;);
}

All this code does is initialise the NewPipe extractor by giving it a DownloaderImpl instance. It then calls getStreamExtractor() on a StreamExtractor class tailored for YouTube. It passes a URL, which is the YouTube page of the video.

When I run this, I get the following output:

$ ./gradlew run

> Task :appwrapper:run
Initialising
Downloading video
URL: https://www.youtube.com/watch?v=xvFZjo5PgG0
Video name: Rick Roll (Different link + no ads)
Uploader: Duran
Category: Entertainment
Likes: 167361
Views: 16866220
Content: https://rr1---sn-uigxx03-aige.googlevideo.com/videoplayback?expire
    =1740101145&ei=uYG3Z_uiBM2hp-oP9ZiJmAo&ip=92.40.170.100
    &id=o-AKIf20e0Ns4q-76XEVbZUy_J03FkxjUEdnV-2JIlPcYb&itag=18&source=youtube
    &requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&met=1740079545%2C&mh=bl&mm=31%2C29
    &mn=sn-uigxx03-aige%2Csn-aigl6ned&ms=au%2Crdu&mv=m&mvi=1&pl=23&rms=au%2Cau
    &initcwndbps=1090000&bui=AUWDL3yKSAgQAEVY4tqV40xoTeMnyIkQ-FVPRnuzuYxBfqw
    -cOiBj0vT4a3RcTnvShBlnHPx5x0hdYtx&spc=RjZbSSYcKOnuHKrN6GGIqdfAdJXAv-1Oua6P
    _PR7IG324-f-zd5ZbgapDa0tKPiOzA&vprv=1&svpuc=1&mime=video%2Fmp4
    &ns=WgqxjXFJraikODL7BBUFz9QQ&rqh=1&cnr=14&ratebypass=yes&dur=7.685
    &lmt=1708738867597515&mt=1740079252&fvip=4&lmw=1&fexp=51326932&c=TVHTML5
    &sefc=1&txp=4530434&n=mORDIKBp9rP1UQ&sparams=expire%2Cei%2Cip%2Cid%2Citag
    %2Csource%2Crequiressl%2Cxpc%2Cbui%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Crqh
    %2Ccnr%2Cratebypass%2Cdur%2Clmt&lsparams=met%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi
    %2Cpl%2Crms%2Cinitcwndbps&lsig=AGluJ3MwRQIhAJLLbhJvIq_01ibr5MzHnVaFuAS9j7G5
    RVHy_678ktfCAiA7LFBnPnZrgMGJANRJbxJ6GLJfL5I8x666zKpAHtvUZw%3D%3D
    &sig=AJfQdSswRQIhAMocRMCvpKEaqMcKT6mnU9M0fvY4AYNKYh0EGqKwl7asAiBVvWZSmDfanF
    SbX6OBgpJMH8q9ozm36-xbHVqHSvdNbQ==&cpn=P8sgdlqH3YiDM4MD
Completed

[Incubating] Problems report is available at: file:///home/flypig/dev/build/
    reports/problems/problems-report.html

Deprecated Gradle features were used in this build, making it incompatible with 
    Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
    and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.12.1/userguide/
    command_line_interface.html#sec:command_line_warnings in the Gradle 
    documentation.

BUILD SUCCESSFUL in 8s
6 actionable tasks: 1 executed, 5 up-to-date

The Content URL that gets spat out at the end points to a version of the YouTube video that can be streamed directly. Neat! I wouldn't bother trying to use the link above though as it's only temporarily active and, by the time you read this, will likely no longer work. But if we wanted a fresh one, we could run this code to generate it again.

That feels like a success and a good place to finish off for the day. Tomorrow I'm going to look at something completely different: running Java code on Sailfish OS.

Yesterday we looked at the NewPipe app and briefly discussed the NewPipe Extractor. The app may be useful for considering some of the design decisions that the NewPipe team have made, but I don't expect it to be of much use from a code perspective.

The Extractor on the other hand... that's where the treasure is!

The Extractor is apparently already used in several "YouTube Download" tools: the sort of tool you give a URL and it downloads a video for you to view locally. But the Extractor is just a library, we need to build some scaffolding around it to make it useful.

So let's try that today. As I mentioned on the preamble, there's no Java virtual machine available from the official Sailfish OS repositories. So to begin with I'll be doing this all on my Linux laptop running Ubuntu 22.04.

First we clone the repository and move inside the local copy:

git clone git@github.com:llewelld/NewPipeExtractor.git
cd NewPipeExtractor
$ ls
LICENSE  README.md  build  build.gradle  checkstyle  extractor  gradle  gradlew
    gradlew.bat  jitpack.yml  settings.gradle  timeago-parser

Inside you can see it's made up of two sub-projects: extractor and timeago-parser. It's the former that we're really interested in. You can also see there's a gradle directory alongside a gradlew bootstrapping script.

NewPipe Extractor is set up to use Gradle for building. Gradle is commonly used for Android app development, but I've never seen it used for building things on Sailfish OS (I'm sure there are examples, but they must be pretty rare).

Nevertheless, everything is set up for us and a nice feature of Gradle is that you don't need to install Gradle to use it. The gradlew script will run out-of-the-box without the need for any additional infrastructure (the Ubuntu docker image I'm using seems to have come with a JDK already installed, so I'm just using that).

We can find out what tasks are available for us to use with the gradle build tool as follows. Here are the first few:

$ ./gradlew tasks
Starting a Gradle Daemon (subsequent builds will be faster)

> Task :tasks

------------------------------------------------------------
Tasks runnable from root project 'NewPipeExtractor'
------------------------------------------------------------

Application tasks
-----------------
run - Runs this project as a JVM application

Build tasks
-----------
assemble - Assembles the outputs of this project.
build - Assembles and tests this project.
[...]

The interesting one for us is assemble because this will build the code into a reusable jar archive. There's nothing special about jar archives, they're literally just ZIP archives that contain a MANIFEST.INF manifest file. But they're also how Java libraries get packaged up, so this is what we need.

Let's build ourselves the NewPipe Extractor library.

$ ./gradlew assemble

> Task :extractor:compileJava
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

BUILD SUCCESSFUL in 6s
7 actionable tasks: 7 executed

We can see the results of this in the extractor/build/libs/ directory:

$ ls extractor/build/libs/
extractor-v0.24.5-sources.jar  extractor-v0.24.5.jar
$ file extractor/build/libs/extractor-v0.24.5.jar
extractor/build/libs/extractor-v0.24.5.jar: Zip archive data, at least v1.0 to 
    extract, compression method=deflate

Okay, we've downloaded and built the library, now we want to use it.

I'm not planning to directly use the library on Sailfish OS like this. But as pointed out by pherjung and thigg — two long-time fellow Sailfish OS enthusiasts — there are clever ways around this, for example using GraalVM. I'll return to this in a future post, but for now, I'm just going to stick to using the library on Ubuntu while I get familiar with it.

However we can't really make use of this library without also writing some code to make use of it. That'll be for tomorrow.

Today we're moving on from yesterday's preamble to start work proper on getting NewPipe running on Sailfish OS.

Let's start by asking "What is NewPipe?" If you've used it, it'll need no explanation, for while it's a complex application both under-the-hood and from a user-interface perspective, its purpose is pretty straightforward.

NewPipe is a YouTube client. Except it's not exactly. It's also a SoundCloud client. And a media.ccc.de client. And a Bandcamp client. And a PeerTube client. Although I don't think it's actually a PeerTube client, more a client for accessing Framatube, which is a PeerTube instance.

The point is it allows access to various platforms for streaming and downloading audio and video. The backend, incidentally, allows plugins to be added for all sorts of sites, building on those I listed above. But as with the NewPipe app for Android, I'll be focusing on the five core services, starting with YouTube.

As Henry Hiles (QuadRadical) pointed out yesterday, there are other YouTube clients already available, including the rather nice Gtk4-based Pipeline app which is designed for use on Linux and even already known to work nicely on mobile Linux.

Until QuadRadical pointed it out I wasn't familiar with Pipeline, which is the main reason I chose NewPipe over it. But Pipeline is also quite different. It's written in Rust for a start, although that's absolutely not a reason to discount it. It's true that Rust isn't a language that lives comfortably on Sailfish OS, but there are examples of excellent apps for Sailfish OS written in Rust (vide Whisperfish) and with NewPipe being written in Java, it's in no better position to work on Sailfish OS. I'd say that Sailfish OS's support for Java is even poorer than its support for Rust.

But for full functionality Pipeline also relies on an external download utility, such as yt-dlp. That makes it less suitable for our purposes.

So Pipeline is also interesting and hopefully I'll get the chance to return to it. But since I'm already on this NewPipe track, I don't think Pipeline offers a strong enough reason to be diverted.

Let's look at the NewPipe Android app in more detail. Here are a few screenshots from the app, running under Android App Support on my Sailfish OS phone.
 

These screenshots show the four main tabs that are presented to the user when they open the app. These tabs are labelled Trending, What's New, Subscriptions and Bookmarked Playlists.

The Trending tab shows a list of audio tracks or videos that the service wants to show you for whatever reason. Presumably intended to represent media that's either popular or looks like it might become popular. But in practice I'm not sure what the algorithm is that's being used in the background.

Personally I have my YouTube history disabled, which means that when I visit the site on the Web I'm just presented with an empty page. But I know others like to be given suggestions for what to watch, in which case this tab will be a useful addition.

It's worth noting that this tab changes its name depending on which service is being used, but they're all variations on a theme: Trending, New and hot, Recent and Featured.

The What's New tab shows a chronological list of items for channels the user has explicitly subscribed to. When you visit a channel there's an overt Subscribe button you can select to subscribe to the channel's feed. The subscription is handled entirely locally — the fact is never transmitted to YouTube — and just means that the app will periodically check for new media released to the channel's feed. By default this is every five minutes, but this can be adjusted in the settings.

Subscriptions are much more useful for someone like me who's looking for a curated list of videos. And it all works nicely. Pulling the list down will trigger an update outside of the usual cadence.

Next is the Subscriptions tab. As you might imagine this shows all of the feeds that you've subscribed to. Just the feeds, not the content in the feeds. But you can of course just select one of them to drill down deeper and see the videos that have been released on the channel.

Finally we have the Bookmarked Playlists tab. I suppose technically the difference between a channel and a playlist is that anyone can create a playlist, whereas a channel feed is managed only by the channel owner. When you open a video you have the option to add it to one of your own playlists, or you can pull one of the playlists created by some other YouTube user. Either way they'll appear on this tab.

If you look carefully at the screenshots you'll also notice there's a little magnifying glass at the top right of the screen. Selecting this brings up a new page allowing you to search the entire service based on the text you enter. This is probably the most important functionality as far as I'm concerned. But all of these tabs are essentially aimed at allowing you to surface content in different ways.

So far we've concentrated mostly on YouTube, but as I mentioned above the app actually supports multiple services. They can be switched between, but only one service is active at a time.

A nice touch is that when you switch to a different service the colour of the user interface changes to reflect the fact, as you can see in the screenshots below.
 

NewPipe does a decent job of maintaining consistency across the five services, even though they offer quite different types of media and categorisations. Part of the reason this is possible is because apart from search and accessing the media itself, much of the functionality is being provided by the app on your phone, rather than the backend service.

That's a consequence of the fact NewPipe is a read-only app. It accesses the media from the various services, but it doesn't authenticate to them, so there's no way for it to store data with the service. If you're looking for privacy then that's a feature, not a bug.

Another consequence of this is that although only one service can be active at a time, content you've subscribed to from one service is available even when you've switched to another. All of the subscriptions and playlists get mixed together into a single list, rather than being separated out by service. I really like this: it allows all of the most relevant content to be merged together and accessed easily.

Alright, let's dig a bit deeper into the user interface and find out what happens when we actually try to play some audio or video. The following screenshots show some of this functionality, but in practice there's such a wealth of options and user interface flows that there's no way I can describe all of it here.
 

Here we first see the video player with user comments below. As already mentioned, this is a read-only app so there's no way to add new comments. On this page there are actually two further tabs to replace comments with a list of related videos, or with the metadata related to the video.

At this point the user interface has become quite cluttered and if this is going to make it to Sailfish OS I feel it'll need to be simplified. At some point I'll have to do some proper design work to figure this all out.

From this page you can also download the video or audio to your local device, as shown in the second screenshot. This is critical functionality as far as I'm concerned. I like to download audio and video in advance while I'm at home over Wifi. Not only is this faster and cheaper than using a mobile connection, it also avoids the annoying situation when the mobile signal drops off and your video is left "buffering". That's something that happens surprisingly frequently when travelling by train in the UK.

There's also a nice music player as shown in the third screenshot. The interface is similar to that used for videos, which also means it's a bit too cluttered for my liking. But there's no shortage of juicy functionality.

Finally the screenshot on the right shows the main menu for when YouTube is the selected service. The little down arrow next to the "YouTube" text is an indication that you can switch service by tapping on the service name there.

The other menu entries are pretty self-explanatory. It's also worth noting there are nine items in this menu, which is far too many to work for a Sailfish-style pull-down menu. However the first four items are duplicates of the tabs so could be skipped and the last three could realistically be combined together leave just three items. That's more in line with what you might typical expect of a Sailfish OS menu.

It looks like this might all be workable.

I must admit that I'd not used the app before Schabi promoted it to me at FOSDEM. But it works perfectly using Android App Support on Sailfish OS and is well worth giving a go. Nonetheless the objective of this project is to make the Android app redundant for users of Sailfish OS.

This is all useful background material, but it's not actually the app itself that's interesting for us. You see the NewPipe devs have thoughtfully split the app into two parts: the user interface and the backend functionality.

The code for the former can be found in the aptly named NewPipe repository. But from a user-interface perspective Android is so different from Sailfish OS that it's unlikely we'd be able to make much use of this code.

It's the code for the latter, found in the NewPipe Extractor repository, that I'm hoping we'll be able to make better use of. This provides a Java library for searching, streaming and downloading media from the various services. It also provides an API for accessing the metadata, comments and other data associated with it.

The plan is to hook this library up to a Sailfish OS user interface built using Qt. We'll need to perform some API-boundary transmutation magic in order to get the Java library communicating effectively with the C++ front-end code. That'll be a bit of a challenge, but I'm hoping the challenge will also help make this a bit more interesting.

We'll take a deeper look at the NewPipe Extractor codebase tomorrow.

In the run-up to FOSDEM it got super-busy. I was preparing a talk, helping organise the Linux on Mobile stand, arranging a Sailfish Birds-of-a-Feather event, plus also working with my colleagues at the Turing on our conference arrangements too. The build up of work started all the way back in August, in fact.

So from August to February my focus outside of work has almost exclusively been on that. It sounds crazy, all leading up to just two days in February, but that's how it was.

But as the event approached I started to think about what I might do after FOSDEM. You know, once I got all of that glorious free time back! One obvious possibility was to start looking at the Gecko upgrade from ESR 91 to ESR 102, so I made a passing comment on the forum suggesting that I might pick up the work again after FOSDEM.

Quite quickly after this I heard that work had already been started on it behind the scenes. This is great news: more than anything else I believe Sailfish OS needs an up-to-date browser and anything that brings this closer has to be a good thing.

Since FOSDEM there's even been some public progress. New ESR102 branches have appeared in the embedlite-components and sailfish-browser repositories, each containing fresh commits. Unfortunately the really important changes are going to be in the gecko-dev repository and, at time of writing this, there's currently no new ESR102 branch there.

I'd love to help support the efforts and I'd also love for the work to happen in the open, but until something appears there, there's unfortunately no useful way for me to contribute. And in truth, that may be for the best. Coordinating a multi-developer project is a job in itself and often harder than just making the changes as a single developer.

So by the time I was behind the stand at FOSDEM I already knew I'd probably be looking for some other project to occupy my time.

You never know what sort of people you'll meet at FOSDEM. Working on a stand is, I honestly believe, a particularly special privilege. Everybody comes to you and shares their stories. You get to meet such an amazing range of fascinating people. It's self-selecting but otherwise unfiltered. I can honestly say that I met such wonderful people at FOSDEM, both new faces and old friends.

One unexpected interaction happened when one of the NewPipe developers came to the stand. There were several of the NewPipe team at FOSDEM, including Fabio Giovanazzi (Stypox), Fynn Godau (fynngodau) and Christian Schabesberger (Schabi). I think it was Schabi I spoke to and we had a really interesting conversation.

I hadn't heard of NewPipe before. It's a media-streaming client that gives access to content from the likes of YouTube, SoundCloud, PeerTube, Bandcamp and others. Unlike other clients NewPipe eschews the official service access APIs in favour of scraping the Web content directly or using reverse-engineered internal APIs. This allows the app to offer more privacy-respecting access.

"I like the sound of that", I thought.

But NewPipe is an Android app, which explains why I'd not heard of it. That means it's written primarily in Java. Outside of Android App Support, Java and Sailfish OS don't make great bedfellows. There's no Java virtual machine installed by default or available from the official repositories for example, which also means it would be a challenge getting a Java-based application through the Jolla Store review process.

Back to FOSDEM and I took a NewPipe leaflet, leaving it idly on the table amongst the other paraphernalia. As the day went on I was a little surprised about how many people commented on it. "Oh yes; I use NewPipe!" I heard at least a couple of times. This piqued my interest even more.

So I wondered: wouldn't it be neat if there was a version of NewPipe for Sailfish OS?

It's true that Sailfish OS already has a nice YouTube app in the form of MicroTube from Michał Szczepaniak. I like MicroTube, but I think it would be neat to have NewPipe functionality available as well. Unlike NewPipe MicroTube uses ytdl-core — written in JavaScript — for providing the underlying functionality for YouTube access. There's nothing wrong with that, but I was also taken with the NewPipe team's commitment to privacy, open source and keeping the software working in the face of the underlying YouTube API being updated.

So I've decided to have a go at writing a YouTube app using NewPipe. I'd say this is certainly going to be a smaller and less impactful project compared to the Gecko work. But I think it could be interesting for a number of reasons.

First, I don't see a lot of regular blogging happening about Sailfish OS app development. Don't get me wrong: Sailfish OS has a thriving app developer community, but mostly it doesn't seem to get serialised in prose.

Second, this is a technically unusual project, mixing C++ with Java and attempting to pull Android code into a Sailfish OS context. I think this will make it interesting to document.

Third, it may not result in the greatest utility in all the world, but NewPipe is a popular app and I think it has some functionality which would be nice to bring to Sailfish OS.

So why not? As with my Gecko diaries I plan to write about this daily. I'm aiming for a simple app: search YouTube then download or stream a video. My diary will run up until at least the first release, but I don't plan for this to be an Odyssey. Hopefully more of a scenic stroll in the park.

Just to be clear, it's not my intention to somehow replace or compete with existing apps like Microtube. Maybe in fact I'll just end up making some pull requests to Microtube after all this. But I don't see a problem in there being multiple YouTube clients available for Sailfish OS.

Tomorrow I'll take a look at the NewPipe Android app, before moving on to the technology that underpins it.

I'll post announcements about these diary entries on my Mastodon feed and after recent changes to this site, any replies to my Mastodon posts will also appear as comments here.

If you'd like to follow my progress, I'll be posting here every day and would love it if you fancy joining me.

For the last 13 years I've been using Disqus to add comment support to my website. I was never very satisfied with this; it had many drawbacks:

1. The client-side JavaScript meant it didn't properly allow for progressive enhancement.
2. Being beholden to an external service made me uncomfortable.
3. Disqus has a dubious privacy record, including having violated the GDPR.
4. Embedding Disqus code introduced trackers onto the site.

I did my best to work around the last of these by dynamically loading the Disqus code only when the user actively selects to view the comments. I wrote about this back in April 2020 as part of a description of how to build a privacy-respecting websites. But it remained a suboptimal arrangement.

So I was really pleased when my erstwhile colleague Bastian Greshake Tzovaras wrote about how he's added a Fediverse feed as a comment section for his blog. Bastian was building on the work of Robert W. Gehl, who was himself building on the ideas of Carl Schwan.

And so it is that I now also have the option use Mastodon for comments.

The original usecase for this was for static sites where this approach makes perfect sense. My site's not static, but I've thought long and hard about comments over the years and have never been able to come up with a solution that I'm fully happy with.

Using Mastodon doesn't address all of the issues I described above in relation to Disqus: it's still relying on an external service and is still all performed client-side using JavaScript. At least now when JavaScript is disabled there's a link that goes straight to the conversation on Mastodon. So there is at least some sort of fallback.

This feels like the best option yet given that there's no user tracking involved. My thanks to everyone involved in coming up with and refined this nice solution.