flypig.co.uk

Yesterday we cloned the NewPipe Extractor repository and built ourselves the library. We weren't able to do anything with the resulting extractor-v0.24.5.jar archive, but our plan is to rectify that today.

The good news is that someone else has already come up with a nice wrapper for making use of the library, called New Valve. The bad news is that this is written in Kotlin and we really want to stick to Java.

So today I'll have to spend a little time converting it to Java. There's actually not a lot of code involved, so this shouldn't be too arduous.

The key piece of code is the DownloaderImpl class. This uses OkHttp to create a simple HTTP Client which is provided to the NewPipe Extractor class and used for downloading data over HTTP.

Presumably NewPipe Extractor doesn't come with its own HTTP Client because different apps will have different ways of wanting to provide this functionality. This OkHttp wrapper looks like it'll be more than enough for our needs.

To get this all setup I've created a new directory appwrapper inside the repository which contains three files and, as is typically for Java projects, way too much directory depth.

$ tree appwrapper/
appwrapper/
??? build.gradle
??? src
    ??? main
        ??? java
            ??? uk
                ??? co
                    ??? flypig
                        ??? DownloaderImpl.java
                        ??? Main.java

6 directories, 3 files

In case you're not familiar with Java, convention is that you put your file in a namespace that's the inverse of a URL. In my case I'm using uk.co.flypig. Unlike for C++ you can only have one class per file; the filename has to match the classname and the folder hierarchy typically matches the namespace.

Hence the deep and largely empty directory hierarchy.

As you may have guessed the build.gradle contains the instructions for building the code. This file is actually surprisingly terse:

plugins {
    id 'checkstyle'
    id 'application'
}

apply plugin : "java"

application {
    mainClass = 'uk.co.flypig.Main'
}

checkstyle {
    getConfigDirectory().set(rootProject.file("checkstyle"))
    ignoreFailures = false
    showViolations = true
    toolVersion = checkstyleVersion
}

checkstyleTest {
    enabled = false // do not checkstyle test files
}

dependencies {
    implementation project(':timeago-parser')
    api project(':extractor')

    implementation "com.squareup.okhttp3:okhttp:3.12.13"
}

Most of this is — I think — self explanatory. The checkstyle sections are to allow the code style to be tested against the requirements for the project (so I could have left these sections out if I'd wanted). The dependencies section highlights that the code makes use of the two other projects in the repository, along with OkHttp. With these added gradle will deal with downloading and hooking in the dependencies automatically.

The only other thing to note is that the application section tells gradle how to execute the code.

I also had to make some small changes to the gradle configuration of the containing directory (the top level of the repository). First to add the appwrapper directory to build.gradle, like so:

diff --git a/build.gradle b/build.gradle
index 335960db..b57add8c 100644
--- a/build.gradle
+++ b/build.gradle
@@ -37,4 +37,5 @@ allprojects {
 dependencies {
     api project(':extractor')
     implementation project(':timeago-parser')
+    implementation project(':appwrapper')
 }

Second to add appwrapper to the settings.gradle like so:

diff --git a/settings.gradle b/settings.gradle
index 725f6505..5cdd3d7d 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -1,1 +1,1 @@
-include 'extractor', 'timeago-parser'
+include 'extractor', 'timeago-parser', 'appwrapper'

Let's now turn to the code itself. I'm not going to paste it all here, but I will link to it in the repository. First we have the DownloaderImpl class. This is a conversion of the Kotlin class of the same name that we saw earlier. It extends the Downloader interface from the Extractor by providing init() and execute() methods.

In short, the execute() method is called by the Extractor when it wants to download a file from somewhere. The method returns a Response object, also defined as part of the Extractor interface, which represents the data retrieved.

The OkHttp library uses a thread pool to parallelise HTTP requests, but it likes to keep this pool around in case there are any new connections to be made. In testing I found this caused the app to linger for a minute or more before quitting while the threadpool awaits deinitialise. I therefore also added a teardown() method which forcefully deactivates the threadpool. This gets called when the app quits so that it returns immediately, rather than after a prolonged delay.

This DownloaderImp class then gets used in our main() method, which is really just a test function so we know things are working correctly:

public static void main(final String[] args) {
    System.out.println("Initialising");
    final String url = args.length > 0
        ? args[0]
        : "https://www.youtube.com/watch?v=xvFZjo5PgG0";
    final OkHttpClient.Builder builder = new OkHttpClient.Builder();
    final DownloaderImpl downloader = DownloaderImpl.init(builder);
    NewPipe.init(downloader);

    try {
        System.out.println("Downloading video");
        System.out.println("URL: " + url);
        final StreamingService service = ServiceList.YouTube;
        final StreamExtractor extractor = service.getStreamExtractor(url);
        extractor.fetchPage();

        System.out.println("Video name: " + extractor.getName());
        System.out.println("Uploader: " + extractor.getUploaderName(
    ));
        System.out.println("Category: " + extractor.getCategory());
        System.out.println("Likes: " + extractor.getLikeCount());
        System.out.println("Views: " + extractor.getViewCount());

        final java.util.List<VideoStream> streams = extractor.getVideoStreams();
        for (final VideoStream stream : streams) {
            System.out.println(&quot;Content: &quot; + stream.getContent());
        }

    } catch (final Exception e) {
        System.out.println(&quot;Exception: &quot; + e);
    }

    try {
        downloader.teardown();
    } catch (final IOException e) {
        System.out.println(&quot;IOException: &quot; + e);
    }

    System.out.println(&quot;Completed&quot;);
}

All this code does is initialise the NewPipe extractor by giving it a DownloaderImpl instance. It then calls getStreamExtractor() on a StreamExtractor class tailored for YouTube. It passes a URL, which is the YouTube page of the video.

When I run this, I get the following output:

$ ./gradlew run

> Task :appwrapper:run
Initialising
Downloading video
URL: https://www.youtube.com/watch?v=xvFZjo5PgG0
Video name: Rick Roll (Different link + no ads)
Uploader: Duran
Category: Entertainment
Likes: 167361
Views: 16866220
Content: https://rr1---sn-uigxx03-aige.googlevideo.com/videoplayback?expire
    =1740101145&ei=uYG3Z_uiBM2hp-oP9ZiJmAo&ip=92.40.170.100
    &id=o-AKIf20e0Ns4q-76XEVbZUy_J03FkxjUEdnV-2JIlPcYb&itag=18&source=youtube
    &requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&met=1740079545%2C&mh=bl&mm=31%2C29
    &mn=sn-uigxx03-aige%2Csn-aigl6ned&ms=au%2Crdu&mv=m&mvi=1&pl=23&rms=au%2Cau
    &initcwndbps=1090000&bui=AUWDL3yKSAgQAEVY4tqV40xoTeMnyIkQ-FVPRnuzuYxBfqw
    -cOiBj0vT4a3RcTnvShBlnHPx5x0hdYtx&spc=RjZbSSYcKOnuHKrN6GGIqdfAdJXAv-1Oua6P
    _PR7IG324-f-zd5ZbgapDa0tKPiOzA&vprv=1&svpuc=1&mime=video%2Fmp4
    &ns=WgqxjXFJraikODL7BBUFz9QQ&rqh=1&cnr=14&ratebypass=yes&dur=7.685
    &lmt=1708738867597515&mt=1740079252&fvip=4&lmw=1&fexp=51326932&c=TVHTML5
    &sefc=1&txp=4530434&n=mORDIKBp9rP1UQ&sparams=expire%2Cei%2Cip%2Cid%2Citag
    %2Csource%2Crequiressl%2Cxpc%2Cbui%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Crqh
    %2Ccnr%2Cratebypass%2Cdur%2Clmt&lsparams=met%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi
    %2Cpl%2Crms%2Cinitcwndbps&lsig=AGluJ3MwRQIhAJLLbhJvIq_01ibr5MzHnVaFuAS9j7G5
    RVHy_678ktfCAiA7LFBnPnZrgMGJANRJbxJ6GLJfL5I8x666zKpAHtvUZw%3D%3D
    &sig=AJfQdSswRQIhAMocRMCvpKEaqMcKT6mnU9M0fvY4AYNKYh0EGqKwl7asAiBVvWZSmDfanF
    SbX6OBgpJMH8q9ozm36-xbHVqHSvdNbQ==&cpn=P8sgdlqH3YiDM4MD
Completed

[Incubating] Problems report is available at: file:///home/flypig/dev/build/
    reports/problems/problems-report.html

Deprecated Gradle features were used in this build, making it incompatible with 
    Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
    and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.12.1/userguide/
    command_line_interface.html#sec:command_line_warnings in the Gradle 
    documentation.

BUILD SUCCESSFUL in 8s
6 actionable tasks: 1 executed, 5 up-to-date

The Content URL that gets spat out at the end points to a version of the YouTube video that can be streamed directly. Neat! I wouldn't bother trying to use the link above though as it's only temporarily active and, by the time you read this, will likely no longer work. But if we wanted a fresh one, we could run this code to generate it again.

That feels like a success and a good place to finish off for the day. Tomorrow I'm going to look at something completely different: running Java code on Sailfish OS.