flypig.co.uk

Yesterday we set up our gecko-dev repository submodule to point to the ESR 91 code and updated the spec file with related parameters. We even tried building the project, although that didn't get very far.

So, now we're on to the work proper and have to try to address some of these errors. The very first error that comes up for me is the following:

This mach command requires /home/mersdk/.mozbuild/_virtualenvs/mach/bin/python, which wasn't found on the system!
Consider running 'mach bootstrap' or 'mach create-mach-environment' to create the mach virtualenvs, or set MACH_USE_SYSTEM_PYTHON to use the system Python installation over a virtualenv.
error: Bad exit status from /var/tmp/rpm-tmp.ho9Gok (%build)

This seems to be a change from the previous version used by Sailfish OS. Previously we set up python and its dependencies directly without use of a virtual environment. The Sailfish SDK builds inside a docker container with snapshots to ensure the main target doesn't get tainted by package installs, so using a virtual environment doesn't gain us much. Similarly on OBS, the build environment is recreated each build. But gecko wants to create one, so we may as well let it.

So, first we add python3-devel to the spec file as a build a dependency (it gets used during the build, but now when the browser is actually running). We already have python3 there, but the development package is needed for the virtual environment. Next, we just follow the advice in the error, by adding the following into the build section of the spec file:

./mach create-mach-environment

I've added it just before the main build call:

./mach create-mach-environment
./mach build -j$RPM_BUILD_NCPUS

Maybe this new addition could go in the prep section instead? For now, it works in the build section and running it afresh for each build doesn't seem to do any harm, other than add some additional time to the build. If there's ways to optimise this, we can come back to it later.

I notice that further up in the build section we have the following:

# hack for when not using virtualenv
ln -sf "%BUILD_DIR"/config.status $PWD/build/config.status

Now I'm wondering whether this should be removed. Again, keeping it doesn't seem to do any harm, but it may now be creating the link unnecessarily. This will be easy to check by just removing the line and building to see what happens. However the testing will be much easier and more robust when more of the build process is working. If I do the test now, problems evident later on in the build process might not get exposed.

So, let's stick a pin in this and return to it later.

I've created the virtual environment, so the build should get a bit further. Let's see.

sfdk build -d --with git_workaround

Note how I've dropped the -p flag. This flag is used to trigger the prepare step of the build process as defined in the spec file. We already successfully completed the prepare step and we've not made any changes to it, so we can skip it for future runs. In the future skipping the prepare step will be essential in order to avoid applying the patches on top of already patched code (which will result in errors). We don't have any patches right now, so we don't actually have to worry about this, but it's good practice anyway.

Now the build does get further... but not by much. We now have an error during the creation of the virtual environment.

Collecting glean_sdk==36.0.0
  Using cached glean-sdk-36.0.0.tar.gz (2.2 MB)
    ERROR: Command errored out with exit status 1:
     command: /home/mersdk/.mozbuild/_virtualenvs/mach/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xlikpl2b/glean-sdk_9db1f9a0519b4c968d1e22841ac4a8f5/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xlikpl2b/glean-sdk_9db1f9a0519b4c968d1e22841ac4a8f5/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-1xv51c_9
         cwd: /tmp/pip-install-xlikpl2b/glean-sdk_9db1f9a0519b4c968d1e22841ac4a8f5/
    Complete output (9 lines):
    Traceback (most recent call last):
      File "", line 1, in 
      File "/tmp/pip-install-xlikpl2b/glean-sdk_9db1f9a0519b4c968d1e22841ac4a8f5/setup.py", line 16, in 
        from setup import *
      File "/tmp/pip-install-xlikpl2b/glean-sdk_9db1f9a0519b4c968d1e22841ac4a8f5/glean-core/python/setup.py", line 47, in 
        FROM_TOP = PYTHON_ROOT.relative_to(Path.cwd())
      File "/home/flypig/Programs/sailfish-sdk/sailfish-sdk/mersdk/targets/SailfishOS-devel-aarch64.default/usr/lib64/python3.8/pathlib.py", line 908, in relative_to
        raise ValueError("{!r} does not start with {!r}"
    ValueError: '/tmp/pip-install-xlikpl2b/glean-sdk_9db1f9a0519b4c968d1e22841ac4a8f5/glean-core/python' does not start with '/tmp/sb2-mersdk-20230809-221116.wepDIc/tmp/pip-install-xlikpl2b/glean-sdk_9db1f9a0519b4c968d1e22841ac4a8f5'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Could not install glean_sdk, so telemetry will not be collected. Continuing.

As it happens this error looks pretty bad, but doesn't seem to be fatal. We're not worried about build telemetry at this stage (maybe not ever?) so can we get away with ignoring this? It certainly doesn't look like a priority to fix it right now as the build continues irrespective, but maybe this will come back to haunt us.

I'm going to stick a pin in this one too. Let's hope it won't cause problems later, but if it does it should become pretty clear. Even if it doesn't cause problems we should also figure out a way to avoid the error by configuring the build to not even attempt to install glean_sdk.

Hopefully you're already getting a feel for my modus operandi. The objective is to get the build to complete as quickly as possible and by whatever means necessary. This means not worrying too much about the details or aesthetics of what we're doing. All of this can be neatly ironed out later when things are building and when it's easier to work in parallel.

The next error is the following.

 0:12.85 checking for vcs source checkout... no
 0:12.97 ERROR: Cannot find project mobile/sailfishos

This looks serious, but the good news is that this is easily fixed by adding a symlink from mobile/sailfishos pointing to the ../../embedding/embedlite directory.

How do I know this? It's because this is actually the action of patch 0001 in the original project from the ESR78 build. The patch already works as-is, so I can just apply it to restore the symlink. After restoring the link the build continues.

It's worth noting that I'm not going to add the patch back in at this stage. Instead, I'll record it as a commit in my personal mirror of the upstream code (so strictly speaking it will no longer be a mirror any more). This is something we discussed on Day 1 in case you want a refresh on why I'm doing it this way.

After making this change, things are looking even better again, so this seems like a good place to pause for the day. The build gets further and then triggers an error, which I'll look in to tomorrow.

Don't forget you can read all the other developer diary entries on my Gecko-dev Diary page.