flypig.co.uk

Yesterday we looked at the NewPipe app and briefly discussed the NewPipe Extractor. The app may be useful for considering some of the design decisions that the NewPipe team have made, but I don't expect it to be of much use from a code perspective.

The Extractor on the other hand... that's where the treasure is!

The Extractor is apparently already used in several "YouTube Download" tools: the sort of tool you give a URL and it downloads a video for you to view locally. But the Extractor is just a library, we need to build some scaffolding around it to make it useful.

So let's try that today. As I mentioned on the preamble, there's no Java virtual machine available from the official Sailfish OS repositories. So to begin with I'll be doing this all on my Linux laptop running Ubuntu 22.04.

First we clone the repository and move inside the local copy:

git clone git@github.com:llewelld/NewPipeExtractor.git
cd NewPipeExtractor
$ ls
LICENSE  README.md  build  build.gradle  checkstyle  extractor  gradle  gradlew
    gradlew.bat  jitpack.yml  settings.gradle  timeago-parser

Inside you can see it's made up of two sub-projects: extractor and timeago-parser. It's the former that we're really interested in. You can also see there's a gradle directory alongside a gradlew bootstrapping script.

NewPipe Extractor is set up to use Gradle for building. Gradle is commonly used for Android app development, but I've never seen it used for building things on Sailfish OS (I'm sure there are examples, but they must be pretty rare).

Nevertheless, everything is set up for us and a nice feature of Gradle is that you don't need to install Gradle to use it. The gradlew script will run out-of-the-box without the need for any additional infrastructure (the Ubuntu docker image I'm using seems to have come with a JDK already installed, so I'm just using that).

We can find out what tasks are available for us to use with the gradle build tool as follows. Here are the first few:

$ ./gradlew tasks
Starting a Gradle Daemon (subsequent builds will be faster)

> Task :tasks

------------------------------------------------------------
Tasks runnable from root project 'NewPipeExtractor'
------------------------------------------------------------

Application tasks
-----------------
run - Runs this project as a JVM application

Build tasks
-----------
assemble - Assembles the outputs of this project.
build - Assembles and tests this project.
[...]

The interesting one for us is assemble because this will build the code into a reusable jar archive. There's nothing special about jar archives, they're literally just ZIP archives that contain a MANIFEST.INF manifest file. But they're also how Java libraries get packaged up, so this is what we need.

Let's build ourselves the NewPipe Extractor library.

$ ./gradlew assemble

> Task :extractor:compileJava
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

BUILD SUCCESSFUL in 6s
7 actionable tasks: 7 executed

We can see the results of this in the extractor/build/libs/ directory:

$ ls extractor/build/libs/
extractor-v0.24.5-sources.jar  extractor-v0.24.5.jar
$ file extractor/build/libs/extractor-v0.24.5.jar
extractor/build/libs/extractor-v0.24.5.jar: Zip archive data, at least v1.0 to 
    extract, compression method=deflate

Okay, we've downloaded and built the library, now we want to use it.

I'm not planning to directly use the library on Sailfish OS like this. But as pointed out by pherjung and thigg — two long-time fellow Sailfish OS enthusiasts — there are clever ways around this, for example using GraalVM. I'll return to this in a future post, but for now, I'm just going to stick to using the library on Ubuntu while I get familiar with it.

However we can't really make use of this library without also writing some code to make use of it. That'll be for tomorrow.