flypig.co.uk

Unfortunately a clerical error yesterday (me forgetting to start it) left me with an incomplete build this morning. I kicked it off immediately, but as I write this in the evening it's still running.

So, while I have my patched, partial-build, version installed, and since the changes didn't add or remove any lines, a practical task for me to work on while the build continues is to debug the changes I made using gdb to check whether the changes are actually having any effect.

The drill is: fire up the phone with the patched version on; ssh into it; start gdb with sailfish-browser set as its executable; stick breakpoints on EmbedLiteViewChild::RecvLoadURL(), nsDocShellLoadState::CreateFromLoadURIOptions() and WebNavigationFlagsToFixupFlags(); run the executable; then check out various variables once the breakpoint is hit. The interesting variables are likely to be the following:

1. In RecvLoadURL() we should take a look at sAllowKeyWordURL to ensure it's set to true.
2. In RecvLoadURL() we should take a look at flags see which if its bits are set.
3. In CreateFromLoadURIOptions() we should take a look at loadFlags see which if its bits are set.
4. In WebNavigationFlagsToFixupFlags() we should take a look at fixupFlags see which if its bits are set.

So, here goes:

[...]
Thread 8 "GeckoWorkerThre" hit Breakpoint 1, mozilla::embedlite::
    EmbedLiteViewChild::RecvLoadURL (this=0x7f88ba4070, url=...)
    at ${PROJECT}/gecko-dev/mobile/sailfishos/embedshared/EmbedLiteViewChild.cpp:534
534     ${PROJECT}/gecko-dev/mobile/sailfishos/embedshared/EmbedLiteViewChild.cpp:
        No such file or directory.
(gdb) p sAllowKeyWordURL
$1 = <optimized out>
(gdb) n
[LWP 11437 exited]
535     in ${PROJECT}/gecko-dev/mobile/sailfishos/embedshared/EmbedLiteViewChild.cpp
(gdb) n
867     ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:
        No such file or directory.
(gdb) n
545     ${PROJECT}/gecko-dev/mobile/sailfishos/embedshared/EmbedLiteViewChild.cpp:
        No such file or directory.
(gdb) n
546     in ${PROJECT}/gecko-dev/mobile/sailfishos/embedshared/EmbedLiteViewChild.cpp
(gdb) p /x flags
$3 = 0x340000
(gdb) 

Hitting our first breakpoint we can't determined what sAllowKeyWordURL is, but we can determine its effects, which are to the flags variable. By the end of the method this is set to 0x340000 which is equivalent to having the following three bits set:

LOAD_FLAGS_ALLOW_THIRD_PARTY_FIXUP = 0x100000
LOAD_FLAGS_FIXUP_SCHEME_TYPOS = 0x200000
LOAD_FLAGS_DISALLOW_INHERIT_PRINCIPAL = 0x40000

We can check that this is correct by dropping briefly into the Python interpreter:

$ python3
>>> (0x100000 + 0x200000 + 0x40000) - 0x340000
0

The larger of these two bits wouldn't be set if sAllowKeyWordURL was set to false, so sAllowKeyWordURL must be set to true. That means that our partial build did work. That sounds like good news, but it's a bit of a disappointment for me because it means that the changes made so far haven't yet fixed the problem: there's more to do.

The values of the other variables which we wanted to look at will all now be determined by this, so arguably there's no need to check. But I'll continue stepping through the executable just to be sure, and because maybe it's interesting to see.

(gdb) c
Continuing.

Thread 8 "GeckoWorkerThre" hit Breakpoint 2, nsDocShellLoadState::
    CreateFromLoadURIOptions (aBrowsingContext=0x7f88b67bd0, aURI=...,
    aLoadURIOptions=..., aResult=aResult@entry=0x7f9f3db3c8)
    at ${PROJECT}/gecko-dev/docshell/base/nsDocShellLoadState.cpp:226
226     ${PROJECT}/gecko-dev/docshell/base/nsDocShellLoadState.cpp:
    No such file or directory.
(gdb) p /x aLoadURIOptions.mLoadFlags
$4 = 0x340000
(gdb) p loadFlags
$5 = <optimized out>
(gdb) c
Continuing.

That's our second breakpoint there, and as we can see the flags have safely made it through to aLoadURIOptions.mLoadFlags in CreateFromLoadURIOptions(). Let's continue onward.

Thread 8 "GeckoWorkerThre" hit Breakpoint 3, WebNavigationFlagsToFixupFlags
    (aNavigationFlags=3407872, aURIString=..., aURI=0x0)
    at ${PROJECT}/gecko-dev/docshell/base/nsDocShellLoadState.cpp:211
211     in ${PROJECT}/gecko-dev/docshell/base/nsDocShellLoadState.cpp
(gdb) p /x aNavigationFlags
$6 = 0x340000
(gdb) n
[W] unknown:0 - QConnmanEngine: Unable to translate the bearer type of the
    unknown connection type: ""
nsDocShellLoadState::CreateFromLoadURIOptions (aBrowsingContext=0x7f88b67bd0,
    aURI=..., aLoadURIOptions=..., aResult=aResult@entry=0x7f9f3db3c8)
    at ${PROJECT}/gecko-dev/docshell/base/nsDocShellLoadState.cpp:266
266     in ${PROJECT}/gecko-dev/docshell/base/nsDocShellLoadState.cpp
(gdb) 
270     in ${PROJECT}/gecko-dev/docshell/base/nsDocShellLoadState.cpp
(gdb) p /x fixupFlags
$14 = 0x9
(gdb) c
Continuing.

There's a bit to be disentangled here. This is our third breakpoint and we can see that the correct bits made it safely into aNavigationFlags inside WebNavigationFlagsToFixupFlags(). Stepping through the method, eventually the fixupFlags variable gets set to 0x9. That's also what we'd expect as it tallies with the following flags taken from nsIURIFixup.idl:

    /**
     * Allow the fixup to use a keyword lookup service to complete the URI.
     * The fixup object implementer should honour this flag and only perform
     * any lengthy keyword (or search) operation if it is set.
     */
    const unsigned long FIXUP_FLAG_ALLOW_KEYWORD_LOOKUP = 1;

    /*
     * Fix common scheme typos.
     */
    const unsigned long FIXUP_FLAG_FIX_SCHEME_TYPOS = 8;

So everything is as we expected. But continuing on from here we still get an error page showing in the browser with the text "The address isn't valid" rather than the online search results we were hoping for.

Unfortunately the partial build doesn't show the source code lines after all, which makes it challenging to step through effectively. I'd really like to step through the rest of the nsDocShellLoadState::CreateFromLoadURIOptions() method to see whether there are any more clues as to why the search provider isn't being contacted. To get proper debugging back I'm going to have to wait for the build to complete.

[...]

So finally the build completed and I can now step through properly! So brace yourself, this is going to be a bit of a long section showing me stepping through the CreateFromLoadURIOptions() method. If gdb debugging isn't your thing, not to worry, you can safely skip past it.

Thread 8 "GeckoWorkerThre" hit Breakpoint 1, nsDocShellLoadState::
    CreateFromLoadURIOptions (aBrowsingContext=0x7f88cba350, aURI=...,
    aLoadURIOptions=..., aResult=aResult@entry=0x7f9f3d33c8) at /usr/src/debug/
    xulrunner-qt5-91.9.1-1.aarch64/docshell/base/nsDocShellLoadState.cpp:226
226         const LoadURIOptions& aLoadURIOptions, nsDocShellLoadState** aResult) {
(gdb) n
227       uint32_t loadFlags = aLoadURIOptions.mLoadFlags;
(gdb)
233       nsCOMPtr<nsIURI> uri;
(gdb) p /x loadFlags
$1 = 0x340000
(gdb) n
234       nsCOMPtr<nsIInputStream> postData(aLoadURIOptions.mPostData);
(gdb) n
237       NS_ConvertUTF16toUTF8 uriString(aURI);
(gdb) n
239       uriString.Trim(" ");
(gdb) n
241       uriString.StripCRLF();
(gdb) n
[LWP 20950 exited]
188     ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsTStringRepr.h:
        No such file or directory.
(gdb) n
245       rv = NS_NewURI(getter_AddRefs(uri), uriString);
(gdb) n
359     ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:
        No such file or directory.
(gdb) n
30      ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsError.h:
        No such file or directory.
(gdb) n
252       } else if (!sURIFixup && !XRE_IsContentProcess()) {
(gdb) n
262       nsAutoString searchProvider, keyword;
(gdb) p fixup
$2 = true
(gdb) n
264       if (fixup) {
(gdb) n
266             WebNavigationFlagsToFixupFlags(uri, uriString, loadFlags);
(gdb) n
270         if (!(fixupFlags & nsIURIFixup::FIXUP_FLAG_ALLOW_KEYWORD_LOOKUP)) {
(gdb) n
274         if (aBrowsingContext->UsePrivateBrowsing()) {
(gdb) n
278         RefPtr<nsIInputStream> fixupStream;
(gdb) n
279         if (!XRE_IsContentProcess()) {
(gdb) n
280           nsCOMPtr<nsIURIFixupInfo> fixupInfo;
(gdb) n
281           sURIFixup->GetFixupURIInfo(uriString, fixupFlags,
(gdb) p fixupInfo                 
$3 = {<nsCOMPtr_base> = {mRawPtr = 0x0}, <No data fields>}
(gdb) n
1363    ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:
        No such file or directory.
(gdb) n
JavaScript error: resource://gre/modules/URIFixup.jsm, line 443:
    NS_ERROR_MALFORMED_URI: Couldn't build a valid uri
283           if (fixupInfo) {
(gdb) n
266             WebNavigationFlagsToFixupFlags(uri, uriString, loadFlags);
(gdb) n
304         if (fixupStream) {
(gdb) n
312       if (rv == NS_ERROR_MALFORMED_URI) {
(gdb) n
262       nsAutoString searchProvider, keyword;
(gdb) n
237       NS_ConvertUTF16toUTF8 uriString(aURI);
(gdb) n
234       nsCOMPtr<nsIInputStream> postData(aLoadURIOptions.mPostData);
(gdb) n
233       nsCOMPtr<nsIURI> uri;
(gdb) n
nsDocShell::LoadURI (this=0x7f88adf170, aURI=..., aLoadURIOptions=...) at
    /usr/src/debug/xulrunner-qt5-91.9.1-1.aarch64/docshell/base/nsDocShell.cpp:3256
3256      if (NS_ERROR_MALFORMED_URI == rv) {
(gdb) c                                       
Continuing.
[...]

Looking at this step through it's clear that the problem happens on line 281 where the fixupInfo object is returned. Or, more specifically, the fact that it's not returned.

To understand why we have to move back into JavaScript mode, in the URIFixup.jsm file. From the getFixupURIInfo() it looks like we'll descend into the tryKeywordFixupForURIInfo() method, which will then call then keywordToURI() method. That's where the info.engine value gets set:

    // Try falling back to the search service's default search engine
    // We must use an appropriate search engine depending on the private
    // context.
    let engine = isPrivateContext
      ? Services.search.defaultPrivateEngine
      : Services.search.defaultEngine;

That's the value which will eventually be returned in the fixupInfo.keywordProviderName parameter, as well as the equally important info.preferredURI parameter.

Why aren't these being set properly I wonder...

That'll be the question for tomorrow.

If you'd like to catch up on all the diary entries, they're all available on my Gecko-dev Diary page.