flypig.co.uk

Unfortunately I don't have as much time today as I've had the last few days to work on Gecko, but hopefully there's still scope to make some meaningful progress.

I'm still trying to track down the address bar issue that's preventing Web search from working. In theory, when you enter a phrase that's clearly not a URL (or begins with a question mark) the browser should delegate the action to your chosen search provider (as configured in the Settings).

Over the last couple of days we got to the point where a number of issues have been fixed (so no overt error messages) but during initialisation the list of search engines is empty. It should be full of stuff, so today I want to try to find out why.

So, I've added a load of debug dumps to various important files related to the OpenSearch capability: SearchService.jsm, SearchUtils.jsm and OpenSearchEngine.js. These will generate debug output in the log (output to the console) so that we can see what's happening when and in what sequence, as well as what's not happening.

From this we can clearly see that addOpenSearchEngine() is being called for each of the stored search providers:

SEARCH: getFixupURIInfo() fixupFlags: 8
CONSOLE message:
[JavaScript Error: "NS_ERROR_FILE_NOT_FOUND: "{file: "file:///usr/lib64/
    mozembedlite/components/EmbedLiteChromeManager.js" line: 213}]
observe@file:///usr/lib64/mozembedlite/components/EmbedLiteChromeManager.js:213:7
SEARCH: engine num: 0
SEARCH: addOpenSearchEngine(): file:///usr/lib64/mozembedlite/chrome/
        embedlite/content/bing.xml
SEARCH: stack start
addOpenSearchEngine@resource://gre/modules/SearchService.jsm:1869:10
observe@file:///usr/lib64/mozembedlite/components/EmbedLiteSearchEngine.js:56:29
SEARCH: stack end
SEARCH: _install: uri: file:///usr/lib64/mozembedlite/chrome/
        embedlite/content/bing.xml
SEARCH: addOpenSearchEngine(): file:///home/defaultuser/.local/share/
        org.sailfishos/browser/searchEngines/duckduckgo.com.xml
SEARCH: stack start
addOpenSearchEngine@resource://gre/modules/SearchService.jsm:1869:10
observe@file:///usr/lib64/mozembedlite/components/EmbedLiteSearchEngine.js:56:29
SEARCH: stack end
SEARCH: _install: uri: file:///home/defaultuser/.local/share/
        org.sailfishos/browser/searchEngines/duckduckgo.com.xml
SEARCH: addOpenSearchEngine(): file:///home/defaultuser/.local/share/
        org.sailfishos/browser/searchEngines/github.com.xml
SEARCH: stack start
addOpenSearchEngine@resource://gre/modules/SearchService.jsm:1869:10
observe@file:///usr/lib64/mozembedlite/components/EmbedLiteSearchEngine.js:56:29
SEARCH: stack end
SEARCH: _install: uri: file:///home/defaultuser/.local/share/
        org.sailfishos/browser/searchEngines/github.com.xml
SEARCH: addOpenSearchEngine(): file:///usr/lib64/mozembedlite/chrome/
        embedlite/content/google.xml
SEARCH: stack start
addOpenSearchEngine@resource://gre/modules/SearchService.jsm:1869:10
observe@file:///usr/lib64/mozembedlite/components/EmbedLiteSearchEngine.js:56:29
SEARCH: stack end
SEARCH: _install: uri: file:///usr/lib64/mozembedlite/chrome/
        embedlite/content/google.xml
SEARCH: addOpenSearchEngine(): file:///usr/lib64/mozembedlite/chrome/
        embedlite/content/yahoo.xml
SEARCH: stack start
addOpenSearchEngine@resource://gre/modules/SearchService.jsm:1869:10
observe@file:///usr/lib64/mozembedlite/components/EmbedLiteSearchEngine.js:56:29
SEARCH: stack end
SEARCH: _install: uri: file:///usr/lib64/mozembedlite/chrome/
        embedlite/content/yahoo.xml
SEARCH: addOpenSearchEngine(): file:///usr/lib64/mozembedlite/chrome/
        embedlite/content/yandex.xml
SEARCH: stack start
addOpenSearchEngine@resource://gre/modules/SearchService.jsm:1869:10
observe@file:///usr/lib64/mozembedlite/components/EmbedLiteSearchEngine.js:56:29
SEARCH: stack end
SEARCH: _install: uri: file:///usr/lib64/mozembedlite/chrome/
        embedlite/content/yandex.xml
JavaScript error: chrome://embedlite/content/embedhelper.js, line 259:
    TypeError: sessionHistory is null
CONSOLE message:
[JavaScript Error: "TypeError: sessionHistory is null"
    {file: "chrome://embedlite/content/embedhelper.js" line: 259}]
receiveMessage@chrome://embedlite/content/embedhelper.js:259:29

We can also see from the stack where it's being called from, but that's not really the interesting part. The interesting part is what's not being output. Calls to addOpenSearchEngine() should almost immediately call the following:

      errCode = await new Promise(resolve => {
        engine._install(engineURL, errorCode => {
          resolve(errorCode);
        });
      });

I've added a bunch of dump outputs to the _install() method like this:

  _install(uri, callback) {
    dump("SEARCH: _install: uri: " + uri + "\n");
    let loadURI = uri instanceof Ci.nsIURI ? uri : SearchUtils.makeURI(uri);
    if (!loadURI) {
      throw Components.Exception(
        loadURI,
        "Must have URI when calling _install!",
        Cr.NS_ERROR_UNEXPECTED
      );  
    }   
    if (!/^https?$/i.test(loadURI.scheme)) {
      throw Components.Exception(
        "Invalid URI passed to SearchEngine constructor",
        Cr.NS_ERROR_INVALID_ARG
      );  
    }

    logConsole.debug("_install: Downloading engine from: ", loadURI.spec);
    dump("SEARCH: _install: Downloading engine from:" + loadURI.spec + "\n");

So the first dump() inside the _install() method is generating output, but the second isn't. Why not? I'll put some extra dump output inside those conditionals to see if the exceptions are firing.

SEARCH: addOpenSearchEngine(): file:///usr/lib64/mozembedlite/chrome/
        embedlite/content/bing.xml
SEARCH: stack start
addOpenSearchEngine@resource://gre/modules/SearchService.jsm:1869:10
observe@file:///usr/lib64/mozembedlite/components/EmbedLiteSearchEngine.js:56:29
SEARCH: stack end
SEARCH: _install: uri: file:///usr/lib64/mozembedlite/chrome/
        embedlite/content/bing.xml
SEARCH: _install: Invalid URI passed to SearchEngine constructor

And now things are beginning to take shape.

The file itself exists:

$ file /usr/lib64/mozembedlite/chrome/embedlite/content/bing.xml
/usr/lib64/mozembedlite/chrome/embedlite/content/bing.xml:
    exported SGML document, ASCII text, with very long lines (1815)

But this bit of code is causing the failure here:

if (!/^https?$/i.test(loadURI.scheme)) {
  dump("SEARCH: _install: Invalid URI passed to SearchEngine constructor\n");
  throw Components.Exception(
    "Invalid URI passed to SearchEngine constructor",
    Cr.NS_ERROR_INVALID_ARG
  );
}

The sailfish-browser code is passing in a URL that's a local file here, using the file scheme, but it's performing a regex test so that it will only accept an https scheme to download the data. The obvious question that springs to mind is what changed between ESR 78 and ESR 91 to be causing this?

In ESR 78 there is no OpenSearchEngine.jsm file because it was renamed from SearchEngine.jsm. The two are close enough to be able to compare though. Here's the equivalent piece of code from ESR 78:

    switch (optionsURI.scheme) {
      case "https":
      case "http":
      case "ftp":
      case "data":
      case "file":
      case "resource":
      case "chrome":
        uri = optionsURI;
        break;
      default:
        throw Components.Exception(
          "Invalid URI passed to SearchEngine constructor",
          Cr.NS_ERROR_INVALID_ARG
        );
    }

That's quite a difference. Let's find out why this was changed.

Checking using git blame shows this was the most recent change to this line:

$ git log -1 3760df94f0b94
commit 3760df94f0b94d8aab5ed10c6177e8fad6345cc3
Author: Valentin Gosu <valentin.gosu@gmail.com>
Date:   Wed Apr 7 10:20:58 2021 +0000

    Bug 1692018 - Remove support for installing OpenSearch engines via ftp
    r=Standard8
    
    Differential Revision: https://phabricator.services.mozilla.com/D107790

With the actual change looking like this:

--- a/toolkit/components/search/OpenSearchEngine.jsm
+++ b/toolkit/components/search/OpenSearchEngine.jsm
@@ -91,7 +91,7 @@ class OpenSearchEngine extends SearchEngine {
         Cr.NS_ERROR_UNEXPECTED
       );
     }
-    if (!/^(?:https?|ftp)$/i.test(loadURI.scheme)) {
+    if (!/^https?$/i.test(loadURI.scheme)) {
       throw Components.Exception(
         "Invalid URI passed to SearchEngine constructor",
         Cr.NS_ERROR_INVALID_ARG

We'll need to go back a bit further than this. Here's the change that happened before:

$ git log -1 269a6fbae9928
commit 269a6fbae9928d4fef5ca4cc9ee2b112b2772191
Author: Mark Banner <standard8@mozilla.com>
Date:   Wed Feb 10 18:12:08 2021 +0000

    Bug 1690750 - Simplify OpenSearchEngine to only allow loading engines from
                  protocols where users can load them from. r=mak
    
    The urls where an OpenSearch engine can be loaded from are already limited
    in LinkHandlerChild. This is cleaning up and simplifying what the
    OpenSearchEngine allows, and as a result allows the load path handling to be
    greatly simplified.
    
    The test changes are due to no longer allowing chrome or file protocols. For
    future, we probably want to move away from OpenSearch for most of these, but
    the changes will make it easier to find the places to update.
    
    Differential Revision: https://phabricator.services.mozilla.com/D104010

This is a much more significant change, which includes the removal of the file scheme.

This is all useful stuff, even if it's not yet a solution to the problem. I'm going to have to leave it here for today, but will return to this tomorrow. The next thing will be to try letting the file protocol through the regex to see what happens. I'm not expecting that to be enough, and if not, we'll have to dig into the whole of the D104010 diff to find out what's changed and whether we'll need to revert it, or find some other solution.

The underlying issue here seems to be the phasing out of OpenSearch as a way of managing search providers. So we should probably look seriously at the Web Extension alternative as well. But that's for tomorrow.

If you'd like to catch up on all the diary entries, they're all available on my Gecko-dev Diary page.