Gecko-dev Diary
Between August 2023 and September 2024 I upgrading the Sailfish OS browser from Gecko version ESR 78 to ESR 91, writing a daily blog as I went along. This page catalogues my progress, alongside the other browser-related topics I've looked in to since.
Latest code changes are in the gecko-dev sailfishos-esr91 branch.
There is an index of all posts in case you want to jump to a particular day.
Click the icon for the Gecko-dev Diary RSS feed.
Gecko
5 most recent items
28 Aug 2024 : Day 333 #
I'm really happy with the response I got from yesterday's "release" of the browser packages for testing. It's great to see users taking the trouble to install, test and post comments or issues about it. It's important to get all of these problems recorded, even if it's going to take me a while to work through them all. Thank you to everyone who's provided feedback.
Quite a few comments noted that I've only made packages available for aarch64 devices. That is the case I'm afraid and, right now, I'm not in an immediate position to release arm32 packges. I've not tested the build process for arm32 recently and I expect it to require a bit of work. I know how patient everyone has been and I know this will be disapointing to hear for some of you. I can only apologise and emphasise that I would dearly love to have the time to get this done as well. I will do, but I can't quite yet.
There have also been some unsuccessful attempts to run the binaries on native devices such as the PinePhone. It's a shame to hear it's not working there, but admitedly not a surprise. Native builds have slightly different rendering requirements and I don't have a device to test on, so this part of the update was performed in the dark. Once other things have been ironed out I'll return to this.
Finally, a quick word about timelines. At some point in the near future it's no longer going to make sense for me to continue writing these daily diary entries. I expect that to happen once all of the patches have been organised and I've finalised pull requests into the Jolla repositories. Although that's not far off, it looks like it may go beyond this weekend.
Next week I'll be attending a conference so will have to pause gecko development while I'm away. Given current progress I won't have everything done by then, so I'll have seven days of hiatus, but will return to daily blogging after that, on the 9th September.
With all of the responses to the release today I have lots still to work on. But today I'm moving in to "tidying-up" mode. What does that mean? Well, up until now for the browser development I've been adding all of my changes as commits to a long-lived branch called FIREFOX_ESR_91_9_X_RELBRANCH_patches in my own mirror of the upstream gecko repository. When I say "upstream" I actually mean Mozilla upstream, via the Sailfish OS mirror of the repository.
However, we don't want to have a completely new fork of the gecko engine for Sailfish OS. If we did it'd make it much harder to benefit from the ongoing development of gecko by Mozilla. Instead Jolla completely mirrors Mozilla's repository and applies all of its changes on top of the upstream code as patches.
Currently my changes are commits, not patches. So the "tidying-up" process is going to involve me converting them into patches.
There's a git command to do this: git format-patch. If it were as simple as just running this command I'd be laughing. But in practice I've not always been conscious of the need to create patches when committing changes. That was intentional: getting gecko working has been the priority. But it means there are far too many commits, many of which can and should be combined in order to turn them into a smaller set of patches.
Let's make some comparisons. The ESR 78 build of Gecko had 98 patches. We can see this by looking inside the rpm directory. The patches are numbered sequentially which makes them really easy to count. Here are the pathces from ESR 78:
Ideally with patches you'd avoid having one patch overlay a change made by an earlier patch. Not only is any such overlay strictly unecessary, it also makes patches harder to deal with because it means they can't be reordered so easily. There are situations where it can make sense, for example for the sake of ensuring patches capture atomic changes. But this is soemthing I also need to figure out: how to combined the comments in a way that minimises overlap.
Finally I also want to try to align my commits with the ESR 78 patches. For example, in many cases I was able to apply a patch from ESR 78 unmodified. In these cases the original patch and author should be retained. Even in cases where I had to make substantial manual changes, it's still important to keep the original author and description.
Most of this is administration, but important administration nonetheless.
So how to proceed? My plan is to make a separate branch FIREFOX_ESR_91_9_X_RELBRANCH_sfos that starts as a copy of the branch I've been doing all my work on. I can then safely rebase the new branch starting from when it diverged from FIREFOX_ESR_91_9_X_RELBRANCH. I'll do my best to combined patches that need combining, update their descriptions and authors to match the ESR 78 patches and finally reorder patches to align with ER 78 if that turns out to be straightforward.
I'll be left with a branch that contains all of the new consolidated commit. The HEAD of the branch should contain exactly the same content as the HEAD of my current development branch. Only the commits in between will have changed. Once I have that I can then use git format-patch to generate all of the patches from it.
I'll then create a new branch in the gecko-dev repository (which, take care, is different from the gecko-dev-mirror repository!), copy all of the patches into it and then check everything is good by performing a full, clean rebuild including the prepare step.
That sounds like a plan. Let's see how I get on.
First up I'll create a new branch:
If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Quite a few comments noted that I've only made packages available for aarch64 devices. That is the case I'm afraid and, right now, I'm not in an immediate position to release arm32 packges. I've not tested the build process for arm32 recently and I expect it to require a bit of work. I know how patient everyone has been and I know this will be disapointing to hear for some of you. I can only apologise and emphasise that I would dearly love to have the time to get this done as well. I will do, but I can't quite yet.
There have also been some unsuccessful attempts to run the binaries on native devices such as the PinePhone. It's a shame to hear it's not working there, but admitedly not a surprise. Native builds have slightly different rendering requirements and I don't have a device to test on, so this part of the update was performed in the dark. Once other things have been ironed out I'll return to this.
Finally, a quick word about timelines. At some point in the near future it's no longer going to make sense for me to continue writing these daily diary entries. I expect that to happen once all of the patches have been organised and I've finalised pull requests into the Jolla repositories. Although that's not far off, it looks like it may go beyond this weekend.
Next week I'll be attending a conference so will have to pause gecko development while I'm away. Given current progress I won't have everything done by then, so I'll have seven days of hiatus, but will return to daily blogging after that, on the 9th September.
With all of the responses to the release today I have lots still to work on. But today I'm moving in to "tidying-up" mode. What does that mean? Well, up until now for the browser development I've been adding all of my changes as commits to a long-lived branch called FIREFOX_ESR_91_9_X_RELBRANCH_patches in my own mirror of the upstream gecko repository. When I say "upstream" I actually mean Mozilla upstream, via the Sailfish OS mirror of the repository.
However, we don't want to have a completely new fork of the gecko engine for Sailfish OS. If we did it'd make it much harder to benefit from the ongoing development of gecko by Mozilla. Instead Jolla completely mirrors Mozilla's repository and applies all of its changes on top of the upstream code as patches.
Currently my changes are commits, not patches. So the "tidying-up" process is going to involve me converting them into patches.
There's a git command to do this: git format-patch. If it were as simple as just running this command I'd be laughing. But in practice I've not always been conscious of the need to create patches when committing changes. That was intentional: getting gecko working has been the priority. But it means there are far too many commits, many of which can and should be combined in order to turn them into a smaller set of patches.
Let's make some comparisons. The ESR 78 build of Gecko had 98 patches. We can see this by looking inside the rpm directory. The patches are numbered sequentially which makes them really easy to count. Here are the pathces from ESR 78:
$ push gecko-dev-project/gecko-dev/gecko-dev $ ls -1 *.patch | wc -l 98 $ popLet's compare that with the number of commits I've made on the FIREFOX_ESR_91_9_X_RELBRANCH_patches since I branched it from the upstream FIREFOX_ESR_91_9_X_RELBRANCH branch:
$ push gecko-dev-esr91/gecko-dev/gecko-dev
$ git log --oneline HEAD...$(git log -1 \
--pretty="format:%H" FIREFOX_ESR_91_9_X_RELBRANCH) | wc -l
126
$ pop
So that's 126 commits compared to 98 patches. The smaller the number the better and these two numbers are closer than I had feared. It's not necessary to get back to exactly the same number of patches. But I do think it should be close and, realistically, it ought to be less than 100.Ideally with patches you'd avoid having one patch overlay a change made by an earlier patch. Not only is any such overlay strictly unecessary, it also makes patches harder to deal with because it means they can't be reordered so easily. There are situations where it can make sense, for example for the sake of ensuring patches capture atomic changes. But this is soemthing I also need to figure out: how to combined the comments in a way that minimises overlap.
Finally I also want to try to align my commits with the ESR 78 patches. For example, in many cases I was able to apply a patch from ESR 78 unmodified. In these cases the original patch and author should be retained. Even in cases where I had to make substantial manual changes, it's still important to keep the original author and description.
Most of this is administration, but important administration nonetheless.
So how to proceed? My plan is to make a separate branch FIREFOX_ESR_91_9_X_RELBRANCH_sfos that starts as a copy of the branch I've been doing all my work on. I can then safely rebase the new branch starting from when it diverged from FIREFOX_ESR_91_9_X_RELBRANCH. I'll do my best to combined patches that need combining, update their descriptions and authors to match the ESR 78 patches and finally reorder patches to align with ER 78 if that turns out to be straightforward.
I'll be left with a branch that contains all of the new consolidated commit. The HEAD of the branch should contain exactly the same content as the HEAD of my current development branch. Only the commits in between will have changed. Once I have that I can then use git format-patch to generate all of the patches from it.
I'll then create a new branch in the gecko-dev repository (which, take care, is different from the gecko-dev-mirror repository!), copy all of the patches into it and then check everything is good by performing a full, clean rebuild including the prepare step.
That sounds like a plan. Let's see how I get on.
First up I'll create a new branch:
$ git checkout FIREFOX_ESR_91_9_X_RELBRANCH_patches Already on 'FIREFOX_ESR_91_9_X_RELBRANCH_patches' Your branch is up-to-date with 'origin/FIREFOX_ESR_91_9_X_RELBRANCH_patches'. $ git checkout -b FIREFOX_ESR_91_9_X_RELBRANCH_sfos Switched to a new branch 'FIREFOX_ESR_91_9_X_RELBRANCH_sfos'Now I need to start editing the commits, but in a way that doesn't change the final result. The key commit is 78d17b06b04f since that's the most recent commit before I started making changes. It means I can easily list all of the relevant commits using the following command:
$ git log --oneline HEAD...78d17b06b04f~I'm not going to list all 126 of the new commits here, but since I'll be starting at the oldest commits and working forwards, it makes sense for me to list some of the oldest:
6889dbfaabe5 Make it possible to extend CompositorBridgeParent 5a88215c32d6 Work around upstream membarrier changes 8a6d40752f22 Ensure PGIOChannel methods are included in Necko 0ca273a6a639 Rename nsIdleService to nsUserIdleService 44eb2761c0b7 Backport Embed MessageLoop contructor back (sha1 eb2dcea271970) 9ac31b7d5097 Disable MOC code generation for message_pump_qt 8c36088e5b72 Build mSize into fontlist::Face::InitData f83d943479b4 Patch glslopt to build on arm f64b6cf84dc6 Reduce optimisation when building swgl dc0c1d441db8 Drop debuginfo for Rust components 41f1be11db66 Fix font method signatures 073ba69f3fe4 Use QRegion::united() 15f3e54524f9 Remove NS_LITERAL_CSTRING macro usage f7abb2cf1c66 Remove nsDataHashtable.h include c77bb39fd6bc Ensure QWidget header can be found 6729baec49bc Register GfxInfo service with components.conf 2ab356e4e31a [PATCH] [sailfishos][components] Cleanup static components... 9d28582c540a [PATCH] [sailfishos][ipc] Whitelist sync messages of EmbedLite... ff27901dc3ed [PATCH] [sailfishos][compositor] Fix GLContextProvider defines 2da747df9cc7 Ensure PGIOChannel is correctly exposed 9bcea9c706cc Update lib.rs checksum aa9461129437 Revert cbindgen dependency version workaround 20d07143125c [PATCH] [sailfishos][qt] Provide checkbox/radio renderer for... cf33677bb09a [PATCH] [gecko][configure] Read rustc host from environment... 1e70489f8c76 Work around build version requirements 6cf2926be653 Adjust NSPR version 38be5c5c7302 [PATCH] Revert "Bug 1611386 - Drop support for..." 57b3e5b766e8 Add --enable-dconf option 1c02a359c368 Add --with-embedlite flag f4c2a96b3363 [PATCH] [sailfishos][qt] Bring back Qt layer. JB#50505 4a17000a7b9b [PATCH] [sailfishos][gecko] Add symlink to embedlite. JB#52893 78d17b06b04f (upstream/FIREFOX_ESR_91_9_X_RELBRANCH) Bug 1770137 - Part 2...There are several changes I'll need to make to these:
- Remove the [PATCH] prefix from the title. These are added by git when the patch is applied so will need to be removed again.
- Add the [sailfishos][gecko] prefix to the commit titles. This is standard for Sailfish OS and allows the changelog to be generated more easily.
- Update the commit author. Since I committed these myself they'll almost all have my name attached, but where these have come from the patches, I need to restore the original author details.
- Update the commit date. Similarly for commits that have come from existing patches I'll need to adjust the date to match the patch they came from.
- Squash patches that are semantically related or touch the same part of the code.
$ git rebase -i 78d17b06b04fThis will allow me to reorder the commits, mark them with squash in case I need to merge several together, or mark them with edit to update authors and dates. When performing the latter, the details can be updated using the following commands (with suitable adjustments for the author and date):
$ git commit --amend \
--author="Raine Makelainen <raine.makelainen@jolla.com>" \
--date="Tue, 26 Jan 2021 14:13:31 +0200"
$ git rebase --continue
Using various combinations of these commands this evening I've been able to work through the oldest 30 commits listed above and convert them into the following 15 commits:
3035e931bc43 Make it possible to extend CompositorBridgeParent 52a4830c18cc [sailfishos][gecko] Work around upstream membarrier changes 0822548619a4 [sailfishos][gecko] Backport Embed MessageLoop contructor back... b4d09871056a [sailfishos][gecko] Disable MOC code generation for message_pump_qt ecee0cc29d6e [sailfishos][gecko] Patch glslopt to build on arm 50b47afc55aa [sailfishos][gecko] Reduce Rust build requirements 2a9f943a2b71 [sailfishos][components] Cleanup static components definitions... 87d833da8e65 [sailfishos][ipc] Whitelist sync messages of EmbedLite. JB#50505 980fb1c93b47 [sailfishos][compositor] Fix GLContextProvider defines 31792345c721 [sailfishos][qt] Provide checkbox/radio renderer for Sailfish OS... 4db3eb103c70 [sailfishos][gecko] Read rustc host from environment. JB#53019... 762e87861ca5 [sailfishos][gecko] Fix build version requirements e74cf5140577 [sailfishos][gecko] Revert "Bug 1611386 - Drop support for..." 6bb1c608d070 [sailfishos][gecko] Fix embedlite building. JB#50505 26d106c8ba10 [sailfishos][qt] Bring back Qt layer. JB#50505 d9e39b342094 [sailfishos][gecko] Add symlink to embedlite. JB#52893 78d17b06b04f (upstream/FIREFOX_ESR_91_9_X_RELBRANCH) Bug 1770137 - Part 2...That means that the original 126 commits are now down to 112 commits as a result of me merging commtis together. It also means I now only have another 96 commits to go! But that's all I have time to work through today. It's a slow process, but all I need to do is continue working through the commits methodically.
If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comments
Uncover Disqus comments