flypig.co.uk

Getting back into the swing of things this morning. Yesterday we looked at the changes Raine made to get APZ (Asynchronous Pan/Zoom) working. Today I'm going to start hacking away at the privileged JavaScript layer.

We've covered Mozilla's Inter-Process Definition Language in previous diary entries. Gecko uses IPDL files which define an interface which is then used to automatically generate a virtual C++ class. You can identify IPDL files by their .ipdl extension.

Gecko uses a similar approach for communication between C++ and JavaScript based on the W3C's Web IDL (Web Interface Definition Language). Again, you write interface definition files, in this case using the .webidl extension. They look pretty similar to IPDL files. These files are also used to automatically generate C++ virtual classes that can then be implemented in the C++ code.

In addition to being accessible from C++, gecko also then exposes these classes to its JavaScript parser. Gecko provides a lot of functionality like this. Things like cookie management, permissions, stored logins, in-page search and much, much, more are exposed to the JavaScript layer in this way.

The result is that between the user interface and the C++ library there is also a layer of JavaScript. For security reasons not all JavaScript can access these IDL interfaces (you wouldn't want any random piece of JavaScript downloaded from the Web accessing your stored passwords!), but in this layer between the user interface and the library the JavaScript is privileged. That is, it has special access to these interfaces.

EmbedLite has its own set of JavaScript modules that live in this privileged layer. I call them shims because they fill a thin gap between the two layers on either side. Often they're just shepherding data from one side to the other, but you can fill them up with all sorts of interesting functionality. For example, the User Agent exceptions that the Sailfish Browser uses are managed by the ?? JavaScript module.

Right now, if you run ESR 91 with debugging output enabled, you'll see a lot of output, including errors, thrown out to the console. Many of these will be coming from this JavaScript layer.

These JavaScript errors aren't fatal: they won't crash the application. But they more than likely will cause the particular JavaScript module they're coming from to fail. This makes the overall browser more robust, but makes individual functions prone to silent failure.

Many of these errors are likely to be because one of the IDL files was changed between ESR 78 and ESR 91, so that the interfaces used by the JavaScript aren't what was expected by our EmbedLite layer. Let's take a look.

$ EMBED_CONSOLE=1 sailfish-browser
[D] unknown:0 - Using Wayland-EGL
library "libGLESv2_adreno.so" not found
library "eglSubDriverAndroid.so" not found
greHome from GRE_HOME:/usr/bin
libxul.so is not found, in /usr/bin/libxul.so
Created LOG for EmbedLiteTrace
[W] unknown:0 - Unable to open bookmarks
    "/home/defaultuser/.local/share/org.sailfishos/browser/bookmarks.json"
[D] onCompleted:105 - ViewPlaceholder requires a SilicaFlickable parent
Created LOG for EmbedLite
JSComp: EmbedLiteConsoleListener.js loaded
JSComp: ContentPermissionManager.js loaded
JSComp: EmbedLiteChromeManager.js loaded
JSComp: EmbedLiteErrorPageHandler.js loaded
JSComp: EmbedLiteFaviconService.js loaded
JavaScript error: resource://gre/modules/LoginRecipes.jsm, line 56:
    TypeError: Services.appinfo is undefined
JSComp: EmbedLiteOrientationChangeHandler.js loaded
JSComp: EmbedLiteSearchEngine.js loaded
JSComp: EmbedLiteSyncService.js loaded
EmbedLiteSyncService app-startup
JSComp: EmbedLiteWebrtcUI.js: loaded
JSComp: EmbedLiteWebrtcUI.js: got app-startup
JSComp: EmbedPrefService.js loaded
EmbedPrefService app-startup
JSComp: EmbedliteDownloadManager.js loaded
JSComp: LoginsHelper.js loaded
JSComp: PrivateDataManager.js loaded
JSComp: UserAgentOverrideHelper.js loaded
UserAgentOverrideHelper app-startup
JavaScript error: 
    file:///usr/lib64/mozembedlite/components/UserAgentOverrideHelper.js,
    line 110: TypeError: Services.appinfo is undefined
JavaScript error: resource://gre/modules/EnterprisePoliciesParent.jsm,
    line 500: TypeError: Services.appinfo is undefined
JavaScript error: resource://gre/modules/AddonManager.jsm, line 1479:
    NS_ERROR_NOT_INITIALIZED: AddonManager is not initialized
JavaScript error: resource://gre/modules/URLQueryStrippingListService.jsm,
    line 42: TypeError: Services.appinfo is undefined
CONSOLE message:
[JavaScript Error: "TypeError: Services.appinfo is undefined"
    {file: "resource://gre/modules/LoginRecipes.jsm" line: 56}]
LoginRecipesParent@resource://gre/modules/LoginRecipes.jsm:56:7
get recipeParentPromise@resource://gre/modules/LoginManagerParent.jsm:1429:24
@file:///usr/lib64/mozembedlite/components/EmbedLiteGlobalHelper.js:17:1

CONSOLE message:
[JavaScript Error: "TypeError: Services.appinfo is undefined"
    {file: "file:///usr/lib64/mozembedlite/components/UserAgentOverrideHelper.js"
    line: 110}]
ua_init@file:///usr/lib64/mozembedlite/components/UserAgentOverrideHelper.js:110:19
observe@file:///usr/lib64/mozembedlite/components/UserAgentOverrideHelper.js:43:19

CONSOLE message:
[JavaScript Error: "TypeError: Services.appinfo is undefined" {file:
    "resource://gre/modules/EnterprisePoliciesParent.jsm" line: 500}]
_getConfigurationFile@resource://gre/modules/EnterprisePoliciesParent.jsm:500:1
_readData@resource://gre/modules/EnterprisePoliciesParent.jsm:556:27
JSONPoliciesProvider@resource://gre/modules/EnterprisePoliciesParent.jsm:477:10
_chooseProvider@resource://gre/modules/EnterprisePoliciesParent.jsm:148:24
_initialize@resource://gre/modules/EnterprisePoliciesParent.jsm:118:25
BG_observe@resource://gre/modules/EnterprisePoliciesParent.jsm:285:14
[...]

One of the recurrent errors seems to relate to Services.appinfo, so that's likely to be a good place to start. We can see it near the start. Most of the EmbedLite modules are loading without error, but then we hit this:

JavaScript error: resource://gre/modules/LoginRecipes.jsm, line 56:
    TypeError: Services.appinfo is undefined

This LoginRecipes.jsm file is part of the password manager, found in the gecko-dev sources ./toolkit/components/passwordmgr/LoginRecipes.jsm. Here's the bit of code generating the error:

/**
 * Create an instance of the object to manage recipes in the parent process.
 * Consumers should wait until {@link initializationPromise} resolves before
 * calling methods on the object.
 *
 * @constructor
 * @param {String} [aOptions.defaults=null] the URI to load the recipes from.
 *                                          If it's null, nothing is loaded.
 *
 */
function LoginRecipesParent(aOptions = { defaults: null }) {
  if (Services.appinfo.processType != Ci.nsIXULRuntime.PROCESS_TYPE_DEFAULT) {
    throw new Error(
      "LoginRecipesParent should only be used from the main process"
    );
  }
  this._defaults = aOptions.defaults;
  this.reset();
}

Now this is part of the main gecko source, so the problem probably isn't coming from here but rather from whatever is creating Services.appinfo. Earlier in the file we can see this line:

const { Services } = ChromeUtils.import("resource://gre/modules/Services.jsm");

The Services.jsm file referred to there is pretty simple:

/* This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, You can obtain one at http://mozilla.org/MPL/2.0/. */

var EXPORTED_SYMBOLS = ["Services"];

var Services = Cu.createServicesCache();

I'm not sure what the problem is here, but let's narrow it down. Let's add the Services.appinfo parts of Services that we need directly into this file.

All of these JavaScript shims are stored in the /usr/lib64/xulrunner-qt5-91.9.1/omni.ja file that gets installed alongside the libxul.so library. This omni.ja file is actually just a zip file, so we can unpack it, edit the contents and then repack it again.

All of this happens on the device:

$ zypper install -y zip unzip
$ mkdir omni
$ cd omni/
$ cp /usr/lib64/xulrunner-qt5-91.9.1/omni.ja .
$ unzip omni.ja 
$ find . -iname "Services.jsm"
./modules/Services.jsm

I've edited the ./modules/Services.jsm files so it now looks like this:

/* This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, You can obtain one at http://mozilla.org/MPL/2.0/. */

var EXPORTED_SYMBOLS = ["Services"];

var Services = Cu.createServicesCache();

Services.appinfo = {}
Services.appinfo.processType = Ci.nsIXULRuntime.PROCESS_TYPE_DEFAULT

Essentially I've hacked in a return value for Services.appinfo.processType so that the check in LoginRecipes.jsm will pass. We can now package back up the omni.ja and put the result back in the right place again.

$ zip -qr9XD omni.ja *
$ devel-su mv omni.ja /usr/lib64/xulrunner-qt5-91.9.1/

Now when we run the browser the particular error generated by LoginRecipes.jsm is gone:

$ EMBED_CONSOLE=1 sailfish-browser
[D] unknown:0 - Using Wayland-EGL
library "libGLESv2_adreno.so" not found
library "eglSubDriverAndroid.so" not found
greHome from GRE_HOME:/usr/bin
libxul.so is not found, in /usr/bin/libxul.so
Created LOG for EmbedLiteTrace
[W] unknown:0 - Unable to open bookmarks
    "/home/defaultuser/.local/share/org.sailfishos/browser/bookmarks.json"
[D] onCompleted:105 - ViewPlaceholder requires a SilicaFlickable parent
Created LOG for EmbedLite
JSComp: EmbedLiteConsoleListener.js loaded
JSComp: ContentPermissionManager.js loaded
JSComp: EmbedLiteChromeManager.js loaded
JSComp: EmbedLiteErrorPageHandler.js loaded
JSComp: EmbedLiteFaviconService.js loaded
JavaScript error:
    file:///usr/lib64/mozembedlite/components/EmbedLiteGlobalHelper.js,
    line 108: TypeError: XPCOMUtils.generateNSGetFactory is not a function
JSComp: EmbedLiteOrientationChangeHandler.js loaded
JSComp: EmbedLiteSearchEngine.js loaded
JSComp: EmbedLiteSyncService.js loaded
EmbedLiteSyncService app-startup
JSComp: EmbedLiteWebrtcUI.js: loaded
JSComp: EmbedLiteWebrtcUI.js: got app-startup
JSComp: EmbedPrefService.js loaded
EmbedPrefService app-startup
JSComp: EmbedliteDownloadManager.js loaded
JSComp: LoginsHelper.js loaded
JSComp: PrivateDataManager.js loaded
JSComp: UserAgentOverrideHelper.js loaded
UserAgentOverrideHelper app-startup
JavaScript error:
    file:///usr/lib64/mozembedlite/components/UserAgentOverrideHelper.js,
    line 110: TypeError: Services.appinfo.version is undefined
JavaScript error: resource://gre/modules/EnterprisePoliciesParent.jsm,
    line 500: TypeError: Services.appinfo.name is undefined

We get a different error, and we still get errors related to other parts of Services.appinfo that are missing. That makes sense because we didn't hack in a return values for the other parts that are now generating errors.

What does this tell us? It suggests that the Service component is being created properly, but that the appinfo portion isn't implemented. The question now is: where should it be implemented?

That part I've not yet found. Finding it shouldn't be hard, but it's late and my brain has decided I've done enough for today, so discovering it will have to wait for tomorrow.

Before I forget, one thing I notice is that the call to Services.scriptloader.loadSubScript() in EmbedLiteConsoleListener.js doesn't seem to be triggering an error. If I can find where scriptLoader() is implemented that my well give me a clue about appinfo.

If you'd like to catch up on all the diary entries, they're all available on my Gecko-dev Diary page.