flypig.co.uk

Over the last couple of days now I've been building versions of gecko and its related packages that support the LOAD_FLAGS_FROM_EXTERNAL flag. According to comments in the code, when this flag is set the engine considers the request to be the result of user-interaction, so I'm hoping this change will help ensure the >Sec-Fetch headers are given the correct values.

But of course not all page loads are triggered externally. There are other flags to identify other types of user interaction which I'll need to fix as well. But one thing at a time. First of all I need to find out if the changes I've made to the code are having any effect.

The changes affect sixteen packages and comprise 771 MiB of data in all (including the debug packages), which I'm now copying over to my device so I can install and test them. Just doing this can take quite a while!

[...]

I've installed them and run them and they don't fix the problem. But as soon as I ran them I remembered that I've not yet hooked up things on the QML side to make them work. I still have a bit more coding to do before we'll see any good results.

[...]

I've updated the QML so that the flag is now passed in for the routes I could find that relate to page loads being triggered externally. There are some potential gotchas here: am I catching all of the situations in which this can happen? Are some of the paths used by other routes as well, such as entering a URL at the address bar? Once I've confirmed that these changes are having an effect I'll need to go back and check these other things as well.

First things first. Let's find out whether the flag is being applied in the case where a URL is passed on the command line.

$ gdb sailfish-browser
[...]
b DeclarativeTabModel::newTab
Breakpoint 1 at 0x6faf8: DeclarativeTabModel::newTab. (2 locations)
(gdb) r https://www.duckduckgo.com
[...]
Thread 1 "sailfish-browse" hit Breakpoint 1, DeclarativeTabModel::newTab
    (this=0x55559d7b80, url=..., fromExternal=true)
    at ../history/declarativetabmodel.cpp:196
196         return newTab(url, 0, 0, false, fromExternal);
(gdb) p fromExternal
$1 = true
(gdb) b EmbedLiteViewChild::RecvLoadURL
Breakpoint 2 at 0x7fbcb105f0: file mobile/sailfishos/embedshared/
    EmbedLiteViewChild.cpp, line 482.
(gdb) n

Thread 8 "GeckoWorkerThre" hit Breakpoint 2, mozilla::embedlite::
    EmbedLiteViewChild::RecvLoadURL (this=0x7f88690d50, url=..., 
    aFromExternal=@0x7f9f3d3598: true)
    at mobile/sailfishos/embedshared/EmbedLiteViewChild.cpp:482
482     {
(gdb) n
483       LOGT("url:%s", NS_ConvertUTF16toUTF8(url).get());
(gdb) n
867     ${PROJECT}/gecko-dev/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h: No such file or directory.
(gdb) n
487       if (Preferences::GetBool("keyword.enabled", true)) {
(gdb) n
493       if (aFromExternal) {
(gdb) n
497       LoadURIOptions loadURIOptions;
(gdb) p aFromExternal
$2 = (const bool &) @0x7f9f3d3598: true
(gdb) n
498       loadURIOptions.mTriggeringPrincipal = nsContentUtils::GetSystemPrincipal();
(gdb) n
499       loadURIOptions.mLoadFlags = flags;
(gdb) p /x flags
$4 = 0x341000
(gdb) p /x nsIWebNavigation::LOAD_FLAGS_FROM_EXTERNAL
$5 = 0x1000
(gdb) p /x (flags & nsIWebNavigation::LOAD_FLAGS_FROM_EXTERNAL)
$6 = 0x1000

Great! The flag is being set in the user-interface and is successfully passing through from sailfish-browser to qtmozembed and on to the EmbedLite wrapper for gecko itself. Let's see if it makes it all the way to the request header methods.

(gdb) b SecFetch::AddSecFetchUser
Breakpoint 3 at 0x7fbba7b680: file dom/security/SecFetch.cpp, line 333.
(gdb) b GetLoadTriggeredFromExternal
Breakpoint 4 at 0x7fb9d3430c: GetLoadTriggeredFromExternal. (5 locations)
(gdb) c
Continuing.
[New LWP 8921]

Thread 8 "GeckoWorkerThre" hit Breakpoint 4, nsILoadInfo::
    GetLoadTriggeredFromExternal (this=0x7f88eabce0)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsILoadInfo.h:664
664     ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsILoadInfo.h: No such file or directory.
(gdb) n

Thread 8 "GeckoWorkerThre" hit Breakpoint 4, mozilla::net::LoadInfo::
    GetLoadTriggeredFromExternal (this=0x7f88eabce0, 
    aLoadTriggeredFromExternal=0x7f9f3d3150) at netwerk/base/LoadInfo.cpp:1478
1478      *aLoadTriggeredFromExternal = mLoadTriggeredFromExternal;
(gdb) p mLoadTriggeredFromExternal
$9 = true
(gdb) c
[...]
mozilla::dom::SecFetch::AddSecFetchUser
    (aHTTPChannel=aHTTPChannel@entry=0x7f88eabed0)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/DebugOnly.h:97
97      ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/DebugOnly.h:
        No such file or directory.
(gdb) 
350       nsAutoCString user("?1");
(gdb) 

The Sec-Fetch-User flag is now being set correctly. Hooray! It's clear from this that the LOAD_FLAGS_FROM_EXTERNAL flag is making it through all the way to the place where the flag values are set. So let's check what the actual values are using the debug output on the console:

[ Request details ------------------------------------------- ]
    Request: GET status: 200 OK
    URL: https://duckduckgo.com/
    [ Request headers --------------------------------------- ]
        Host : duckduckgo.com
        User-Agent : Mozilla/5.0 (X11; Linux aarch64; rv:91.0) Gecko/20100101 Firefox/91.0
        Accept : text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
        Accept-Language : en-GB,en;q=0.5
        Accept-Encoding : gzip, deflate, br
        Connection : keep-alive
        Upgrade-Insecure-Requests : 1
        Sec-Fetch-Dest : document
        Sec-Fetch-Mode : navigate
        Sec-Fetch-Site : none
        Sec-Fetch-User : ?1
        If-None-Match : "65a81e81-4858"

That looks like just the set of Sec-Fetch flags we wanted.

So it's disappointing to discover that the page is still not rendering. Why is could this be?

I set the user agent to the ESR 91 value I used previously:

  "duckduckgo.com": "Mozilla/5.0 (Mobile; rv:91.0) Gecko/91.0 Firefox/91.0"

Close the browser; clear out the cache; try again:

$ rm -rf ~/.local/share/org.sailfishos/browser/.mozilla/cache2/ \
    ~/.local/share/org.sailfishos/browser/.mozilla/startupCache/ \
    ~/.local/share/org.sailfishos/browser/.mozilla/cookies.sqlite 
$ sailfish-browser https://duckduckgo.com/

 
The last thing I see before I head to bed is the DuckDuckGo logo staring back at me from the screen. There is a glitch with search though. The search action works, in the sense that it takes me to the result page, but no results are shown. So there's still some more work to be done.

Nevertheless I'll sleep a lot easier tonight after this progress today.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.