flypig.co.uk

Way back last year in August 2023, before actually starting the process of upgrading the Gecko engine in Sailfish OS from ESR 78 to ESR 91, I wrote a preamble in which I set out my objectives and sketched a brief plan for how to achieve them. Although the work isn't entirely complete, after 339 days I consider the main bulk of my work on the project to be complete. We're now in the mopping up stage. That means it's a good time to look back at the process, find out what went well, what went badly, what I've learned from the experience and how I feel about things. If the preamble was the opening bracket, this retrospective can be considered its closing partner. Together they're the bookends encapsulating all the diary entries in between.

The Journey

Back when I started I hadn't quite appreciated how long this whole process was going to take. Although somewhere between half a year and a year seemed reasonable, the final 339 day tally is a little closer to the latter than I'd hoped. Moreover a year in theory feels much shorter than a year in practice. Adjusting for the fact I'm employed full-time to not do Gecko work, in practice I must have worked only around three hours a day during the week and twelve hours at the weekend. Two thirds of that time was spent coding and the other third writing up the diary entries. Given that the 339 days was made up of 244 weekdays and 48 weekends, I can be a bit more precise about how much time I actually spent on it.

$ time gecko-dev

real    48w 3d 0h 0m 0.000s
code    5w 1d 12h 0m 0.000s
diary   2w 4d 4h 0m 0.000s

Let's convert that into work time. This is interesting because practically speaking this is the "Full Time Equivalent" (FTE) or the amount of person-hours needed to complete the project from a commercial perspective. Typically the work would of course be distributed between multiple people to speed up project implementation, so the real time would be shorter.

$ time --work gecko-dev

real    67w 4d 0h 0m 0.000s
code    23w 1d 2h 0m 0.000s
diary   11w 3d 1h 0m 0.000s

Let's consider now how those days were partitioned into tasks. The following diagram shows the linear sequence of how I spent each day of work. This oversimplifies things a little given I didn't always complete tasks sequentially, but is pretty close to reality.
 
On day 149 I gave a presentation of this work at FOSDEM'24, including an earlier version of this diagram. I thought I was about half way through the work at that stage, but this turned out to be an underestimate, as is clear from this diagram. I was in fact only 11/25 of the way through.

The longest task of 87 days involved getting the WebView render pipeline working. In comparison getting the first successful build to complete took only 45 days. Both of these were quite dark and gloomy times. Without a working build it's impossible to debug or test the code, whilst without a working renderer nothing else can be effectively tested. Both periods felt like dark tunnels that took an age to emerge from.

Following these in terms of length of task were PDF Printing at 28 days, the WebGL renderer at 25 days and the Sec-Fetch-* headers at 15 days. These were the only tasks that took more than 14 days, which is a bit of a cut-off point for me. After two weeks of writing daily about the tasks it becomes really hard to write more without sounding (and seeming) a bit lost and exhausted.

It was particularly frustrating how long it took to get the WebView rendering working, given the browser already worked nicely. It could have been worse though: I had a clear plan which involved gradually stripping out and adjusting pieces of code to align with the code in the ESR 78 implementation. This gradual convergence towards ESR 78 meant I knew the task was inevitably going to be time-limited, also allowing me to identify progress on a daily basis. As it turned out I had to do it twice: first removing code, then adding code back in again. But it did eventually get there.

Realising on Day 254, after all this, that I'd broken the WebGL rendering process was also a bit of a low point. By then I really wanted to move on from the render pipeline.

But eventually I did emerge from all of these tunnels and the joy of getting something to work is a crucial counterbalance to the frustration when it isn't. In retrospect the low points were all worth it for the sake of the enjoyment I also got out of it.

Apart from how things turned out in practice it's also interesting to compare how closely it matched my initial expectations. Returning to the preamble once again, it's clear I was expecting a long haul, but I also had experience to draw from my previous involvement in browser upgrades at Jolla:
 
I think it's fair to say that I did follow this approach, starting with getting things to compile and then focusing on the details afterwards. This led on to the following decision about the structure of the work:
  Looking back I did broadly follow this structure. I got the build to complete, then I applied the patches needed to get rendering working and only after that did I apply the other patches. I did diverge from this advice in one important respect. Rather than applying all of the remaining patches I actually only applied a minimal set required to get the render working.

In hindsight I think that was the right thing to do. But it also felt like a natural consequence of the situation I found myself in. Given the upstream code changes the patches I did apply needed quite a lot of work to get them to stick. That gave me the impression that many of the existing patches might turn out to be redundant, superseded by changes in the upstream code.

By applying only the patches that were necessary it give me the opportunity to potentially avoid patches which were no longer relevant in a more intentional way. Hopefully the patches I've ended up with are closer to the minimum required and have a slightly cleaner structure than would otherwise have been the case.

But practically speaking I think my original plan was a good one and, in retrospect, I followed it pretty closely.

Destination Gecko

Let's now consider where the journey took us. The point of all this work was to take the browser engine from ESR 78 to ESR 91. What does this give us?

Abstractly speaking, one of the most compelling reasons to want to upgrade is because websites routinely attempt to fingerprint browsers and serve different content depending on the result. This practice is as old as the hills, yet remains as common today as it is problematic. I understand that different browsers have different capabilities and that website creators will be blamed (unfairly) if a page renders poorly as a result of a user failing to keep their browser up-to-date. But you'd have thought at the very least browsers could test for features rather than versions.

When browsing using ESR 78 it's not uncommon for a site to chastise its own customers. Updating the engine on Sailfish OS is one way to reduce the chance of seeing these invectives, even if just changing the user agent string is often just as effective as a browser upgrade without any of the effort.

One of the worst offenders is Cloudflare, which routinely blocks the Sailfish browser from accessing sites on its content delivery network. Upgrading to ESR 91 seems to circumvent this in at least some cases.

But browser upgrades also bring genuine improvements as well. New features, improved stability, increased security and bug fixes. There have been a total of 45 point releases between the previous Sailfish OS engine of 78.15.0 and the upgraded version at 91.9.1. Each of these point releases has brought improvements, although not all will be relevant to Sailfish OS. Major releases (e.g. from 78 to 79) will typically include new features, stability improvements and security fixes, whereas point releases (e.g. 91.1.0 to 91.2.0) will often only include security and regression fixes.

Working through the Firefox changelogs the following are some of the obvious improvements that have a direct impact on the Sailfish browser:

1. Certificate performance improvements (80.0.1).
2. WebGL rendering improvements (80.0.1).
3. Support for viewing more filetypes (81.0).
4. Improved element rendering (81.0.1, 86.0.1).
5. Improved PDF export (81.0.1, 85.0.1, 90.0.2).
6. Increased startup and rendering speeds (82.0).
7. Fixes for WebSocket message duplication (82.0.2).
8. SpiderMonkey JavaScript performance improvements (83.0).
9. An HTTPS-Only mode option (83.0).
10. Improved shared memory performance (84.0).
11. Increased cookie and supercookie isolation (85.0, 86.0, 89.0, 90.0, 91.0).
12. Deprecation of WebRTC DTLS 1.0 (86.0).
13. Private browsing compatibility improvements (87.0).
14. Increased referrer privacy (87.0, 88.0).
15. Working hyperlinks in PDF export (90.0).
16. Removal of FTP support (90.0).
17. Improved user-action response times (91.0).
18. Fixes for microsoft.com certificate errors (91.4.1).
19. Many crash bug fixes (81.0.1, 82.0.1, 85.0.1).

If you look through the changelogs you'll also notice several references to enabling WebRender for various platforms at various times. These are the main changes that haven't been introduced to the Sailfish OS version since it would require a much bigger change to the rendering pipeline and the consequences of making this change are unclear at this point.

In addition to the above changes, there were 15 critical, 115 high severity, 68 medium severity and 30 low severity security fixes combined into these updates. The importance of these can be best understand with reference to Mozilla's security classification:

• Critical: Vulnerability can be used to run attacker code and install software, requiring no user interaction beyond normal browsing.
• High: Vulnerability can be used to gather sensitive data from sites in other windows or inject data or code into those sites, requiring no more than normal browsing actions.
• Moderate: Vulnerabilities that would otherwise be High or Critical except they only work in uncommon non-default configurations or require the user to perform complicated and/or unlikely steps.
• Low: Minor security vulnerabilities such as Denial of Service attacks, minor data leaks, or spoofs. (Undetectable spoofs of SSL indicia would have "High" impact because those are generally used to steal sensitive data intended for other sites.)

These are all great changes to have. Probably the most important for daily use on a phone are the efficiency and performance improvements. Based on feedback on the Sailfish OS forum, users also seem to be happy with the results in this respect, with many users claiming the browser feels faster and more responsive.

Whether this is actually the case is hard to say. My tests using various performance measurement tools don't suggest significant performance improvements. But I must admit to having the same feeling of improved responsiveness. I suspect that may be due to the upstream changes in version 91.0 that claim to have improved responsiveness for user-interactions by 10-20%. That would make a noticeable improvement for users in a way that may not show up in benchmarks. It's my suspicion that the page loading feedback that's used to drive the progress bar on Sailfish OS has also been improved, although I've not found any explicit changes that would do this.

What do all of these changes mean for the state of the code? The upgrade from ESR 78 to ESR 91 also, surprisingly for me, brought with it a larger codebase. Mozilla has been intentionally transitioning code from C++ to Rust, with the number of lines of Rust code increasing by 14%. But the number of lines of C++ code also increased by 3% and for the total combined C++, JavaScript and Rust code this increased by 7%. Plotting the lines of code categorised by language, these increases are clearly visible.
 
Although proportionally there's been a bigger increase in Rust code than C++, in absolute terms the increase in both is almost identical (380607 lines of Rust code added compared to 384080 lines of C++ code).

In the above diagram Docs refers to content that relates to documentation. Build refers to scripts used to manage the build pipeline. IDL refers to interface definition files.

It's worth pausing to consider the code needed to build the Gecko engine. Gecko has experienced several changes through its life accumulating a mixture technologies as it goes. As a result the build system is a strange combination of Build (the mozilla build system), Python, Make, ninja, GN and Cargo. At certain points the build system compiles Rust into native binaries that then become part of the build pipeline itself. This causes havoc for the scratchbox2 cross-platform build engine Sailfish OS uses. No small part of the work in getting gecko working for Sailfish OS involves taming these build systems.

Although the numbers for IDL shown in the graph are low compared to the other languages, I nevertheless wanted to include it because it's such a critical part of the way Gecko works. The combination of C++/Rust and JavaScript means that there needs to be a really solid way to expose native methods to JavaScript and JavaScript methods to native code. The type systems aren't equivalent and so this requires a careful arrangement. Gecko supports this using its Interface Definition Language. IDL files read a bit like C++ header files but are more generic. Any interface defined using IDL can be exposed both natively and to the JavaScript layer. It's critical glue that holds everything together.

The numbers shown in the graph are measured in millions of lines of code. They're big numbers, but it's worth bearing in mind that Gecko is a relative minnow when it comes to code size in the world of browsers. For comparison I ran the same code analysis on the Chromium source. I was pretty surprised by how large Chromium is compared to Gecko.
 
Chromium contains over four times the code: 154 981 674 lines of code for Chromium compared to a paltry 37 361 820 lines of code for Gecko ESR 91. It's also interesting to compare the range of technologies involved in the two projects. Chromium introduces TypeScript, Go, Java, Objective-C, Lua, AppleScript, TCL and WASM, although some of these will be target-specific.

Destination Sailfish

So far we've considered the differences between ESR 78 and ESR 91 in some detail, but none of this has touched on the actual changes needed to get the code to run on Sailfish OS.

As any Sailfish OS developer will be aware, Sailfish OS uses RPMs for packaging software, a technology that originated on Red Hat Linux as the Red Hat Package Management system. Work started on RPM in 1995, a good ten years before the initial release of git and two years before Netscape started work on Gecko. Back then it was commonplace for software to be provided in the form of a tarball and in some ways the RPM build process reflects this. Distribution-specific changes are provided as patches applied directly to the upstream source. These patches are all listed in the spec file which is passed to the rpm tool to perform the build. On Sailfish OS this is all hidden behind sfdk which is itself a wrapper for the scratchbox2 sb2 tool. It's a complex layered system with multiple abstractions.

The point is that even now on Sailfish OS packages that use upstream code can pull directly from the upstream repositories, rather than having to use Sailfish-specific implementations. Any Sailfish OS specific changes can then be applied onto this code in the form of patches. It's not a process I enjoy working with because patching is a lot messier and less flexible than working with commits in a repository. Even though it's possible to convert a patch list into a series of commits and back again this adds an extra step and contrains what actions can be performed at different times.

The benefit is that we always retain a very clean and clear distinction between the upstream code and the Sailfish OS specific changes, with the latter being encapsulated in the patches to be applied. We can use this separation to discover how the changes needed to get Gecko ESR 78 to work with Sailfish OS differ compared to those needed for ESR 91.
 
This figure shows only a very high-level view, but nevertheless tells a story. Note that unlike the previous figures the y-axis of this chart uses a logarithmic scale to account for the big differences in scale between different languages. This can make the values harder to read so, for clarity, here they are in tabular form.
 

Language	ESR 78 added	ESR 78 removed	ESR 91 added	ESR 91 removed
C++	22 726	606	22 476	631
Docs	510	2	508	12
Build	28 558	20 350	19 320	11 090
JavaScript	158	6	170	43
Rust	544	175	498	180
IDL	29	3	39	6

These numbers represent the actual code I've been working on for the last year. In general the number of lines added or removed has reduced as we've moved from ESR 78 to ESR 91. This is a good thing. The fewer changes made to the upstream code the better. In general the difference isn't huge, but it does exist. The total number of patches reduced from 98 to 84. The number of lines added to ESR 91 was 82% of the number of lines added to ESR 78. The number of lines removed from ESR 91 was only 57% of the number removed from ESR 91.

Interestingly, while there were fewer change made, the differences practically balance themselves out. Overall the patches to ESR 78 increased the code size by 31 383 lines compared to 31 049 lines for ESR 91. That's astonishingly similar.

These numbers don't quite capture all of the changes because they relate only to the gecko code. There were also changes needed in the other four components that make up the Sailfish browser stack, as well as to the EmbedLIte code (which is handled separately from gecko but ends up in the same xulrunner package). Let's briefly take a look at these other components.
 
The gecko renderer is by far the largest of the components. The qtmozembed component provides a QT wrapper around the renderer. The embedlite-components package adds the privileged JavaScript shims needed for Sailfish OS, largely replacing equivalent privileged JavaScript that would typically run in Firefox. The sailfish-components-webview component provides Qt components needed in order to support both the browser and WebView (for example the pop-up dialogues), but also provides the code needed to offer the rendering engine as a WebView component to other Qt apps. Finally the sailfish-browser component is the actual browser app you run when you open the browser on your phone.

Apart from the gecko renderer all of these are Sailfish-specific packages, so they don't have any "upstream" code. The Jolla repositories are the upstream repositories for these. Consequently there's no need to apply patches and we can work on the code directly. That means that when analysing changes for these we're just using the commits that take the code from ESR 78 versions to ESR 91 versions. Between them they accumulated 169 commits with the following additions and removals (these numbers also including the changes to the gecko source):
 

Language	Lines added	Lines removed
C++	23 456	1 281
Docs	508	12
Build	19 724	11 381
JavaScript	452	114
Rust	498	180
IDL	52	17
QML	14	14
Total	44 704	12 999

This table essentially captures the sum total of the changes needed to move from one version to the next. As you can see, the majority of the additions have been to C++ code. The build scripts saw rather a lot of churn. I'm very surprised to see more Rust additions than JavaScript additions. The QML code changed very little, which is perhaps to be expected given the external appearance, renderer aside, is almost identical. That was intentional: there's always scope to improve the Sailfish browser user interface, but my objective with this work was to get the renderer upgraded as quickly as possible. Changing the interface would have been a diversion.

Mental Health

I put a lot of myself into the Gecko upgrade. Working on it practically every day for a year, even if not full-time, required a level of commitment that I wouldn't typically give outside of my work hours. This is a personal perspective: the world is blessed with many people who commit far more for far less reward and who don't then feel the need to tell the world about it in a blog post.

Nevertheless, this was a big deal for me. I'm not a natural blogger so the prospect of writing about my coding on a daily basis was daunting at the outset. But it turned out to be surprisingly easy. Writing about specific tasks is very different from having to come up with inventive and interesting topics to write about on a daily basis.

Having to write daily diary entries undoubtedly helped keep me on track and working on the project every day. The need to have at least a few paragraphs to write about drove me to do the coding work.

There were a few occasions when I struggled with this. Typically on a Friday night after having spent two and a half hours on public transport returning from work. Having to then write up a diary entry in a tired state of semi-consciousness was not always ideal. But these cases were relatively rare.

There were also occasions — mostly in the middle of the work to get the various rendering pipelines working — when the work really got me down. Writing the diary entries made me very conscious of the progress I was — or in many cases wasn't — making. In the middle of the trough when it's really not clear whether it will be possible to come up with a solution, some of those occasions felt quite dark. If I hadn't been writing the diary I can imagine myself choosing to take a break and then having that break go on for several days.

But, and this is a big but, I was supported the entire way through by the amazing Sailfish community who responded to my posts on Mastodon and the forum, always encouraging and supportive. I'm not a social person and this was a bit of a shock for me. People out there in the Sailfish community and beyond really are the most encouraging and thoughtful people you could hope to interact with.

The amazing images and poetry from the likes of Thigg (thigg) and Leif-Jöran Olsson (ljo) are beautiful cases in point.
 
But there are so many people who helped and contributed in so many ways, I couldn't possibly mention everyone here. I apologise for not mentioning you all individually, but I'm really grateful.

Besides the community I also have to mention Joanna, my wife, who's sacrificed more than anyone else for the sake of me spending three hours each day and most of my weekends on gecko development. She carried me through this.

With all of this support, I found the experience surprisingly effortless. Perhaps the biggest challenge, as it turns out, was being able to find a suitable point to wind things down. Dropping off from posting diary entries every day and having a very clear purpose for my free time has been hard to manage in a measured way. It was too much of a cliff edge and, if I do this again, I think I'd want to look into ways to mitigate this. But I don't yet have a good solution: writing these diary entries doesn't lend itself to a tapered reduction of work.

Future Work

Future work for this project comes in two forms. There's the future work needed to achieve the (hopefully) near-term goal of getting the browser released to users as part of Sailfish OS. Then there's the longer term goal of what to do beyond that.

As I write this the current situation is that three out of five pull requests have been merged into Jolla's repositories. The remaining three have been through a couple of review rounds already. So the immediate task is to get them through review and merged in. This alone won't result in their release as part of Sailfish OS as they're currently being merged into bespoke ESR 91 branches. Jolla will need to merge these into the main branch before they can become part of any official Sailfish OS release.

It's nevertheless exciting to see that as part of the recent upgrade from Sailfish OS 4.6.0.13 to 4.6.0.15, several changes to libhybris were included that will support the move to ESR 91. As readers of my diary entries will know, there were several issues that caused the browser to crash or hang which were ultimately traced back to libhybris and which, looking at the changelog, will now be fixed. If ESR 91 does go out in some future Sailfish OS release, this will make the transition much smoother.

At present I've been building exclusively for aarch64. The build will need to be tested and potentially amended for armv7hl and i486 targets. On top of this, it appears that getting the browser to work on native platforms such as the emulator and the PinePhone, where there is no libhybris layer, will also require some additional work.

In the longer term, there are two, maybe three, objectives. The obvious next step after the release of ESR 91 would be to move to the next ESR release, which is 102.15.1. Checking the various release notes we can see that ESR 91.9.1 was released on 20 May 2022, whereas ESR 102.15.1 was released on 12 September 2023. That's a gap of around 16 months. So far the upgrade from ESR 78 has taken 13 months, so it looks like we may have an opportunity to catch up with Firefox ESR latest. In practice though it's usually around 12 months between ESR releases so some acceleration will be needed if we're to properly keep up. It's worth noting that the extended service releases have a much longer support cycle than other releases, which can lead to some overlap. For example both ESR 115.15.0 and ESR 128.2.0 were released on 3 September 2024.

Besides the obvious upgrade to the renderer engine it would also be great to add features to the browser. On the Sailfish OS Forum Niels (fingus) suggested supporting MPRIS for the video and audio controls of the browser. That's the sort of thing I'd love to add, but which would require some research and effort to investigate and implement. I'd also love to introduce support for reader mode, scrollbars and maybe even extensions. There's no shortage of interesting ideas for things to work on.

The third objective would be to properly support the WebRender compositor on Sailfish OS. It's not clear how much work this would involve, but it's potentially substantial. Integrating this with the Sailfish OS render pipeline could be quite a challenge.

Finally there's plenty of scope to make important improvements to the browser build process. Updating Rust, fixing the multi-process hang — which remains a significant barrier to reducing build times — and introducing a build cache would all help to make development easier.

Lessons Learned

The main outcome of this work for me has been the reaffirmation that the browser is a critical component of Sailfish OS. The better the browser the more usable Sailfish OS becomes as a daily driver. Make no mistake, the reason I wanted to do this work was for entirely selfish reasons: Sailfish OS is my mobile phone operating system of choice. I enjoy using it and I want it to remain relevant so that it continues to be supported. Upgrading the browser is my way of helping ensure this happens; it's my itch and I've been scratching it.

But I've learnt a whole lot more than this and not just from the process of development, but also from the experience of writing a daily diary about it. I'd like to think that the work has helped demonstrate the importance and benefit of open source, for users of course, but also for Jolla. Jolla invested heavily in ensuring the browser is open source. Not just in giving the code the right licence and making the source available, but also in documenting it, following an open development model and supporting the community in making it accessible. In no way was this a "free" browser upgrade for Jolla, but I hope it goes some small way to justifying this open source strategy. I'd also like to think the diary entries have demonstrated some of the benefits of being open about progress as well.

I've also learnt more than I'd like to admit about Brownian debugging. This is the process of performing a random walk, changing bits of the code en route, until it works. It may not be the most efficient debugging approach and it may be that an element of strategic direction improves matters, but as long as the problem space can be constrained I've found Brownian debugging can be unexpectedly effective. Given enough time and patience.

There's a follow-up to this, which is that it also demonstrates how much can be achieved without the benefit of understanding or insight, but relying on perseverance alone. I'm definitely more familiar with the gecko code than when I started, but the gaps in my knowledge remain prodigious. Armed only with my abilities in Brownian debugging and enough time to deploy them, I managed to make some progress.

I admit this wasn't my first involvement in upgrading the browser. While working at Jolla I contributed to the upgrade from ESR 60 to ESR 68, and then again from ESR 68 to ESR 78. But that was as part of a team with an incredible depth of knowledge of the browser and impressive software development skills. When I started this process I wasn't at all certain whether I'd be able to make any meaningful contribution to the next upgrade. I'm now much more confident that not only has this been possible, but that I'd be able to do it again.

It's been great to feel some purpose within the Sailfish OS community. I really enjoyed working for Jolla, not least because it felt worthwhile contributing to an operating system I love using, but also contributing to the community I felt a part of. Doing this work has served as a great way to continue feeling like I have something to contribute.

Writing the development diaries was, I hope, helpful in demonstrating that work was continuing on the browser: it hadn't been forgotten or left to decay. It gave me a lot more visibility than I would have got otherwise. Crucially though it made me realise that there are many, many, Sailfish OS developers putting in similar or greater levels of commitment, for ports and apps and bug checking, who may not have the same visibility because they're not writing a diary, but who nevertheless put in more work and deserve the same appreciation that I've felt privileged to have received from the community.

It's been a week since I last posted a diary entry. It turns out that not writing a diary entry everyday is almost as hard as writing one. If I'm not wandering aimlessly around the house trying to find a purpose, I find I'm sitting on the sofa quietly mumbling gdb backtrace nonsense to myself.

Thankfully there are still some tasks to do, addressing review comments and reviewing others' commits. I'll talk about these happenings today. But I also want to share some retrospective analysis of the project: what went well, what went badly and how the whole process felt as an experience. I'll be expanding on all that in a future post.

First up I want to highlight the really nice fix from from Vlad G (vlagged) that's fixed certificate overrides. When the browser accesses a site using HTTPS but receives a bad certificate, it'll present the user with an error page. This is to avoid the user mistakenly believing that the data being sent from the site, or that they're sending back to it, is being done so securely (i.e. using a channel that maintains integrity and confidentiality).

There are still some sites that don't support HTTPS which provide no more security than a site with a broken or bad certificate. But the latter is worse in that could give the user a false sense of security. If there's a failure it could be an attack. Exposing this to the user in an overt way is critical for them to make an informed decision about how to proceed.

That means that Firefox typically offers the option to view the site anyway, as long as the warning page has been prominently presented. It's hard to imagine anyone could miss the bad certificate page given the way it's presented on Sailfish OS.

Changes to the underlying API however mean that the ESR 91 upgrade broke the option to bypass the certificate check. With this broken code, selecting the option to "view the page anyway" caused the browser to return the same certificate warning page rather than actually going to the site. That's because the button is supposed to record a "certificate override" stating that the browser should ignore the bad certificate. But the method to perform this has changed parameters, so it was throwing up an error rather than recording the site details.

Here we can see the blame for the interface file where the new parameter was added by R. Martinho Fernandes:

$ git blame -c --date=short \
    security/manager/ssl/nsICertOverrideService.idl -L 84,113
[...]
345598abc1d6f (Masatoshi Kimura 2019-06-11)  void rememberValidityOverride(
                                                     in AUTF8String aHostName,
e368dc9c853cf (Ehsan Akhgari  2012-08-22)            in int32_t aPort,
7ea1bcd018a32 (R. Martinho Fernandes 2021-06-01)     in jsval aOriginAttributes,
61d16d4cd9588 (kaie@kuix.de   2007-10-03)            in nsIX509Cert aCert,
e368dc9c853cf (Ehsan Akhgari  2012-08-22)            in uint32_t aOverrideBits,
f2e2a270fb9a4 (kaie@kuix.de   2007-11-19)            in boolean aTemporary);

If we check out the commit 7ea1bcd018a32 that introduced the change, we can see it takes us to differential revision D91962:

$ git log -1 7ea1bcd018a32
commit 7ea1bcd018a32a8e98216cf9f675c49d582950ca
Author: R. Martinho Fernandes <bugs@rmf.io>
Date:   Tue Jun 1 06:55:07 2021 +0000

    Bug 1597600 - make certificate overrides depend on origin attributes 
    r=keeler,geckoview-reviewers,smaug,agi
    
    Differential Revision: https://phabricator.services.mozilla.com/D91962

This was noticed by Vlad who dug down into the code and found the source of the error, fixing it by adding the missing parameter. It's great detective work and a nice fix.

As I write this I've approved the change myself and hopefully it'll be picked up and reviewed by Jolla so that it can be merged swiftly in to the ESR 91 branch.

Besides this I also received really nice review feedback from sailors Raine (rainemak) and Mal (mlehtima) in relation to my gecko-dev pull request. This pull request is the one that actually performs the upgrade from ESR 78 to ESR 91. The upgrade takes the form of redirecting a submodule to a different upstream commit so is a super-simple change. The bigger part is the set of patches that then has to be applied on top of the upstream code to make it all work. This is all in the pull request as well.

Raine and Mal clearly spent some time going through the commits and I'm really grateful for their suggestions. The result was better preference handling and several commits being merged or removed entirely. On Mal's request I also rebased the code to bring in recent changes needed to support Jolla's updated SDK. The totality of these changes all make for a much cleaner commit history and impoved code.

This pull request, along with the pull requests for the sailfish-browser and sailfish-components-webview repositories, are now awaiting a second review from the Jolla team. As far as I'm aware I've completed the changes requested, but there may still be a few more review rounds to go before they get merged in.

The other change this week has been to the package tarballs available from my website. These are very much still testing packages and you should read all of the info in the installation instructions before you use them. Nevertheless these new packages should provide some nice improvements over the previous ones. The main changes are:

1. The user agent patch has now been applied, so problems with pages serving desktop rather than mobile variants should now be largely resolved.
2. Vlad's HTTPS fix is included so you can now view pages with broken TLS certificates if you so choose.

If you're still on ESR 78 you can just follow the installation instructions from before. If you already have the previous ESR 91 packages installed you can follow the same instructions but should skip steps 4 (copying the browser profile) and 7 (creating a symlink to the ESR 78 library directory).

You should, however, still do step 5 to clean out the old settings files. This won't clear cookies, tabs, history or anything like that, but it will reset any preferences you've changed using about:config. In summary, if you already have ESR 91 installed you should complete the following steps:

1. Download the tarball to your phone.

   $ curl -O https://www.flypig.co.uk/dnload/dnload/sailfishos/gecko/gecko-dev-esr91-release.tar.bz2
   

2. Unpack the tarball:

   $ tar -xvf gecko-dev-esr91-release.tar.bz2
   

3. Ensure the browser and any apps using the WebView are closed.
4. [Skip this step]
5. Remove settings that need to be updated for ESR 91:

   $ rm ~/.local/share/org.sailfishos/browser/.mozilla/ua-update.json
   $ rm -rf ~/.local/share/org.sailfishos/browser/.mozilla/startupCache
   $ rm ~/.local/share/org.sailfishos/browser/.mozilla/prefs.js
   $ rm ~/.local/share/org.sailfishos/browser/__PREFS_WRITTEN__
   

6. Install the ESR 91 packages:

   $ devel-su rpm -U --force \
       xulrunner-qt5-91.*.rpm \
       xulrunner-qt5-misc-91.*.rpm \
       qtmozembed-qt5-1.*.rpm \
       sailfish-components-webview-qt5-1.*.rpm \
       sailfish-components-webview-qt5-pickers-1.*.rpm \
       sailfish-components-webview-qt5-popups-1.*.rpm \
       embedlite-components-qt5-1.*.rpm \
       sailfish-browser-2.*.rpm \
       sailfish-browser-settings-2.*.rpm \
       mapplauncherd-booster-browser-0.*.rpm
   

7. [Skip this step]
8. Finally kill the browser booster:

   $ killall booster-browser
   

9. You're good to go!

As always, if you run in to trouble, drop me an email, create an issue on GitHub, or post a message in the Sailfish OS forum.

That summarises the changes this week. I'm still writing up my retrospective of the project so far and will post it once it's ready.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.

It's been just under a week since I posted my last diary entry. In the meantime I've been attending RSECon24 learning about assorted software engineering topics and meeting others in the research software engineering community. But you'll be pleased to hear that I did also find some time for gecko dev work as well.

Not as much as I would normally do, so in retrospect pausing my diary entries was the right thing to do. But it's been enough progress to justify my promised summary today.

Before we get into the details of what I've personally been working on I want to first talk about what others have been doing, as well as considering the future of these dev diaries.

Before heading to the conference I submitted five pull requests:

1. https://github.com/sailfishos/gecko-dev/pull/162
2. https://github.com/sailfishos/qtmozembed/pull/49
3. https://github.com/sailfishos/sailfish-browser/pull/1074
4. https://github.com/sailfishos/embedlite-components/pull/100
5. https://github.com/sailfishos/sailfish-components-webview/pull/169

These pull requests, alongside the test packages, marked an important step on the journey towards an official Sailfish OS ESR 91 browser release. I'm not for a second suggesting that these pull requests or packages represent final versions. But I'd like to think they opened the door for more development in preparation for that release.

And that seems to have been borne out by how things have been moving since then. There have been exciting developments pulling in input from across the Sailfish community and beyond, from both developers and users. I'd like to highlight just some of the things that have started happening recently and which give me great optimism for the future of both Sailfish OS browser development and the Sailfish OS platform more broadly.

Started by Frajo (krnlyng), but then contributed to by Mal (mlehtima) and Affe Nul (affenull2345), the hangs and crashes coming from the EGL rendering pipeline have been carefully ironed out in the libhybris layer. The solutions were completely unexpected as far as I was concerned and I'd never have been able to figure them out myself, so this was a source of much celebration and admiration from me.

There have also been other pull requests to address actual issues in the browser brought on by the upgrade. There are so many features in a browser it's very hard to test everything (an argument for good unit tests, but that's a real challenge given the discrete approach to upgrades required for Gecko on Sailfish OS). A great example of this is the pull request submitted by Vlad G (vlagged) to fix bad certificate overrides. It was easy for me to miss this functionality. Vlad has been a dedicated follower of the dev diaries and it's important not to underestimate how important contributions like this are. It's this kind of diligent, proactive and practical help that the browser really need to be a success.

Quite apart from the motivation it's given me over the last year, the discussion on the Sailfish forum has driven some important advances for the browser. There's been a healthy discussion about sites that have glitches or aren't working correctly, for example from Tone (tortoisedoc), Daniel (jauri.gagarin.II), Sebastian (smatkovi) and others.

I'm hoping that many of these issues will turn out to be related to user-agent strings and the fact that I'd missed off an important user-agent patch. This was flagged up through a discussion between Attah (attah) and Raine (rainemak). Now that the patch has been applied, thanks to their work, many of the pages will hopefully work more successfully.

Members of the Sailfish community have invested their time and energy into testing ESR 91 on a range of difference devices, including Pasi (Pasik2) testing on the Pinetab 2, Ladislav Hodas (lhodas) on the Gemini and Affe Nul (affenull2345) on the Samsung Galaxy A40. Matti Viljanen (direc85) deserves a special mention for getting complete rebuilds working for armv7hl and i486 versions of the packages, demonstrating that this can be achieved with minimal changes. From what I understand there are still issues to be worked out with the resulting packages, but this is work that I'd have struggled to do on my own.

Sticking with Matti, who I'm very pleased to see is now working for Jolla (congratulations!), he provided me with excellent advice on improving the build both in public and private communications, especially in relation to the Rust components. On the topic of Rust, I also received really useful suggestions from Tone (tortoisedoc) which also has the potential to improve Rust builds.

There was some useful discussion on the forum about the WebRender pipeline and its relationship to Progressive Web Apps, with Simon Schmeisser (simonschmeisser), Tone (tortoisedoc), Attah (attah) and olf (Olf0) all contributing. It was a worthwhile read covering important ground. These are the sorts of topics that will deserve even more discussion once we start thinking about ESR 102 (it will happen!).

I can't possibly mention everyone who's spent time encouraging me over the last year, but let me just say that it was gratifying to receive such kind remarks on the forum from so many users who tried out the packages, including Harry (hschmitt), hanswf, Helge, Patrick (pherjung), ric9k, 247, Tuomas Nurmi (Mazoon), Christian (p1rat), Maximilian (Maximilian1st), Edz, Uli (Fellfrosch), Mark Washeim (poetaster), Throwaway69, eson, Gabriele (gabrigubin), Felix (FelixWilke), Ricardo (monkeyisland) and KeksKopf (davidrasch), as well as many more on Mastodon.

And finally, the really hard work will now fall to the Jolla team. Raine (rainemak) and Mal (mlehtima) have already provided invaluable review comments. Alongside Frajo (krnlyng), Matti (direc85) and no doubt others, I'm hoping they'll be able to pick up some of this work to chisel off the sharper corners and sand down the rough edges. The result of Jolla's unparalleled engineering expertise will give a better result than I could ever hope to achieve.

An important thread that runs throughout these contributions is that I'd have struggled to do all this on my own. Focused on the code, I've not been spending my time trying to persuade others to contribute. Members of the community have therefore taken their own initiative to help and the browser has benefited enormously as a result.

This list turned out far longer than I was expecting and by necessity captures only some of the exciting developments that have been driven by the community recently. I'm very excited to see how things continue to progress.

But it also highlights the challenge that's now presented to me as I write this development diary. The work has become more distributed and this will inevitably dilute my contributions in a positive way. That's a great thing and my expectation is that I'll likely now spend less of my time directly working on the code and more of my time on the processes needed to get the browser released.

That means a daily development diary probably doesn't make much sense any more. Up until now the act of writing daily has been critical in driving me forwards, ensuring I don't lose focus and keeping me on track. That's no longer needed; the browser now has its own momentum.

But I do still want to maintain a record of progress. I hope to capture developments up until the browser is properly released, assuming it can get to that point. So my plan for this diary in the future is to write a weekly update on how things are progressing. If there's not much to write about, I'll write up my thoughts on a topic related to Gecko and Sailfish OS more generally. I'll publish them at the weekend because that's when the other pieces of my life are least frantic, giving me the opportunity to consolidate my thoughts from the week gone by. To reflect this change in pace I'll no longer count the days but rather give each post a proper title.

That's my plan. If it doesn't work out, I can always change it. Please do share your thoughts about it with me as I'm always grateful to receive them.

With all this in mind, let's return to those pull requests and talk about what's happened to them over the last week. I'm really happy that two of them have already been merged into the upstream repository. They're not on the main branch yet, which is still on ESR 78, but this is an important step towards that.

That leaves three remaining. Raine and Mal both added comments to these which left me with a bit of work to do to address their requests. Some of these changes were straightforward, like removing extraneous empty lines that I added during development and forgot to remove.

Changes like these have no impact on the code semantics and so wouldn't even necessarily justify a rebuild. But they still take a surprising length of time to address. If they're part of the mirrored gecko code the repository has to be reset, cleaned and all patches applied. Once the change has been made it needs to be squashed into the commit where it was added. Then the patches need to be regenerated. Only then can the changes be committed and pushed to the remote repository.

This is one of the reasons I'm not keen on working with patches. I find myself repeatedly having to clean the repository, apply the patches and then regenerate them before a fresh set of commits can be pushed. I find it surprisingly laborious. And while I fully understand the reasons for using them (the rpm build process has been tailored for this approach) that doesn't make me like them any more.

That's for the simplest of changes, but some of the changes are more substantial. For example Raine requested various changes to how preferences should be applied, all of which are very legitimate.

These changes centre around Sailfish OS specific preferences which differ depending on whether the browser or WebView is in use.

In these cases they can either be set to the default of the browser or set to the default of the WebView. Whichever way is chosen, some code is needed in either the WebView or the browser to ensure the value is correctly toggled from the default for the other.

My original code set the value to a default that worked for the browser, then flipped them in the WebView initialisation code. From what I understand, Raine was keen for me to do the opposite. That is, to set defaults that work for the WebView but are then flipped by the browser initialisation code. The request itself was spread over several or Raine's comments, but the following is perhaps the pithiest of the descriptions:
 
Consequently I updated the default values set in the gecko code, removed some code from the WebView and added some equivalent code to the browser like this:

    // Ensure the renderer is configured correctly
    webEngineSettings->setPreference(QStringLiteral(
                      "embedlite.compositor.external_gl_context"),
                      QVariant(true));
    webEngineSettings->setPreference(QStringLiteral(
                      
    "embedlite.compositor.request_external_gl_context_early"),
                      QVariant(true));

Thankfully these are also pretty simple changes, but they did need a bit of thought, not least because they touch on three separate packages. The gecko engine, the browser and the WebView code all needed updating, albeit each in only a small way.

Having made these changes I then merged them in to the existing branch. To keep things clean I've squashed the changes into relevant commits. But this did also mean spending a bit of time rebasing all of the commits in the pull request.

The resulting in changes to all three of the outstanding pull requests:

1. https://github.com/sailfishos/gecko-dev/pull/162
2. https://github.com/sailfishos/sailfish-browser/pull/1074
3. https://github.com/sailfishos/sailfish-components-webview/pull/169

I'm not entirely certain I've fully understood Raine's request, but hopefully this brings things a little closer. Having made these changes I let the build run overnight in my hotel room. In the morning a quick test on my phone showed no regressions, even after clearing out the profile directories.

Consequently I've pushed the changes upstream to the pull requests. My timing isn't ideal, right before the weekend, but I'll look forward to hearing more from the Jolla team in due course.

That's it for today. I'll return next week with a summary of changes that have taken place during the week. I'm also planning a retrospective to summarise the entire process of the last 339 days of development: what went well, what could have gone better and what the experience has been like for me.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.

As I write this it's the first day of Meteorological Autumn. It still feels very much like summer though (maybe because we won't reach Astronomical Autumn until later in the month). The sun is shining, there's a light breeze, crickets are chirping and there's a generally relaxed atmosphere in the air. But it's the changes in season that are always the most exciting time of year for me. I've enjoyed the summer, but I'm now looking forward to autumn and all of its mellow fruitfulness. Just thinking about it energises me.

But it also represents a shift in gear for this gecko diary as well. I'm hoping to get my pull requests submitted today. That doesn't mean the work is over, but it does mean that it's reached a stage where I feel it can be built upon (by both me and others). Moreover, as I mentioned previously I'll be taking a break this week while I'm attending a conference. So this will be my last diary entry until next weekend.

But that still means there's work to do today and I got up early this morning to check the build I started last night. And...? sadly it didn't complete. Most of the six patches I added last night seem to have built okay, but the startUpCache patch has caused errors. And a lot of errors at that. Although there are pages and pages of errors they all seem to be of one of two flavours.

First there are namespace errors

364:14.24 ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsILoadInfo.h: In member 
    function ‘bool nsILoadInfo::
    GetAllowListFutureDocumentsCreatedFromThisRedirectChain()’:
364:14.24 ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsILoadInfo.h:849:5: 
    error: reference to ‘mozilla’ is ambiguous
364:14.24   849 |     mozilla::DebugOnly<nsresult> rv = 
364:14.24   849 |     mozilla::DebugOnly rv = GetAllowListFutureDocumentsCreatedFromThisRedirectChain(&result);
364:14.24       |     ^~~~~~~

Next there are undefined type errors (in this case nsresult):

364:14.30 ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsILoadInfo.h:962:32: 
    error: expected primary-expression before ‘>’ token
364:14.30   962 |     mozilla::DebugOnly rv = GetIsMediaRequest(&result);
364:14.31       |                                ^

And then there are yet more undefined type errors like this one:

364:14.36 ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/Services.h:480:46:
     error: ‘XPCOMService_GetHistory’ was not declared in this scope; did you 
    mean ‘XPCOMService_GetGfxInfo’?
364:14.36   480 |   return already_AddRefed(XPCOMService_GetHistory());
364:14.36       |                                              ^~~~~~~~~~~~~~~~~~~~~~~
364:14.36       |                                              XPCOMService_GetGfxInfo

There are so many errors, but I'm hoping that it's a cascade: a single error that's triggering a whole cascade of follow-on errors, rather than a huge collection of unrelated errors. Time to dig in and find out.

If I compare the code from ESR 78 it looks pretty similar, but I am a little surprised to see that the newly added Logging.h include is placed inside the namespace qualifiers:

using namespace mozilla::Compression;

namespace mozilla {
namespace scache {

#include "mozilla/Logging.h"

Although that does tally with the changes to the ESR 78 code, it doesn't match with the other ESR 91 files the Logging.h header is being included in, such as the EmbedLite code and elsewhere, so I'm going to move it out of the namespace and try again.

This gives much better results: the majority of the errors are now gone and I'm just left with a couple of errors related to use of NS_LITERAL_STRING() and NS_LITERAL_CSTRING(). This has been a recurrent issue throughout this process since the gecko code now requires use of an _ns string suffix rather than one of the previous NS_LITERAL_STRING() macros.

Having replaced the two string literals with the annotations the code now compiles without error. But it means I have to do a complete build again, which means I have to regenerate the startupCache patch again. The commits have changed because the patch descriptions have changed as a result of being applied by the build process, but other than this I can use the same command to generate the patches as before:

$ git format-patch --start-number 79 -o ../rpm -N --no-signature --zero-commit 
    24f9ec2df3d3

The great news is that although this generates six patches, only the startUpCache patch is registering any changes, the others are all identical to the previous versions. This is a really good sign and exactly what I was hoping for.

So I've updated the patches, added the changes to the commits, cleaned the entire source tree of cruft and now it's time for another full rebuild.

I want at least one full rebuild to go through without errors before I create this mythical pull request I've been going on about. It's frankly turning out to be a little more laborious to get it to happen than I was expecting. But it'll get there.

[...]

Now finally after a full day of compilation, the final full build has completed. I've copied over and installed the packages and a quick check suggests things are working as expected. The pages are returning correctly as mobile pages. The email app doesn't crash when you switch back from the homescreen. The WebView seems to be working correctly.

The browser does still segfault on exit and, after a discussion with Frajo, it's possible that this could be fixable. But this isn't a new issue and while it'd be great to have it fixed, it could take a while to figure out. So that's going to have to be something for the future.

The task now, finally, is to submit the pull request. In fact, not just one pull request, but all five of them for the various packages that have needed changes needed for supporting the new engine. After a bit of playing around on GitHub, here they are:

1. https://github.com/sailfishos/gecko-dev/pull/162
2. https://github.com/sailfishos/qtmozembed/pull/49
3. https://github.com/sailfishos/sailfish-browser/pull/1074
4. https://github.com/sailfishos/embedlite-components/pull/100
5. https://github.com/sailfishos/sailfish-components-webview/pull/169

It really feels good to get those submitted. This is by no means the end of the process of course. There are at least four more tasks:

1. Respond to the comments on the pull requests so they can be merged in. There are already some comments I need to address. Until they're merged in, they could still change a lot and ultimately, it's Jolla that has to decide whether or not to accept them. So they have to fulfil Jolla's requirements.
2. Fix all the issues that have already come up in testing of the browser. It looks like many of these may be addressed by the user agent fix and the libhybris changes, but there are others. And I'm sure there will be many more besides.
3. Fix the segfault on browser shutdown. This isn't new, but with the latest libhybris changes getting a clean shutdown feels closer than ever. It'd be so great if that could be achieved so it's worth spending a bit of time on it.
4. Get the browser working on native devices. An important fix, mustn't forget this!

All of these are important steps. But right now my main objective is to take a short break. As I mentioned previously I'm attending RSECon24 this week and that'll leave me very little time to work on the gecko code anyway. But to be honest, I think I need some time to decompress; it'll do me good to think about something else for a change.

I'll return to this next weekend and I'll continue with fresh diary entries then. But right now, I plan to spend a bit of time breathing in and enjoying the fresh autumnal air.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.

Last night I kicked off the build to test the final arrangement of commits and patches that I hoped to convert into a pull request to the gecko-dev repository today. With the patches now stored in the gecko-dev rpm directory it means that the submodule mirror of upstream's gecko-dev source tree can be pulled in entirely unchanged and without any sailfish-specific commits. That's the whole point of the way the repository is structured: we pull in upstream's code without modification, apply our sailfish-specific patches to it and build the result.

This morning the build had finished and finished with a positive result. No glitches in the build process. But that doesn't mean it's built something that works.

But after transferring it over to my phone, installing the packages and running a few tests, I'm convinced that it's just the same as the previous version. So I conclude that I did successfully manage to reorganised the commits without affecting the final result.

At this point I thought I'd be ready to create the final pull request. But as I look through the patches I notice a few things that still need fixing. The first is to do with redundant commits. A couple of patches disable and then re-enable a flag, so they can be removed. There's also a commit that drops build optimisations to O1. I'd like to test it without this change so I've reverted this patch and will drop it entirely if it turns out not to be needed.

I also notice there are a few patches which aren't entirely required to get the browser working, but which do nevertheless provide important background functionality.

In fact, it was the discussion on the forum, particularly between Attah (attah) and Raine (rainemak) that alerted me to the fact that there are patches that haven't been applied but are needed, specifically related to user agent handling:
 

There are six patches like this that spring out as being likely needed, but which as yet aren't included in the build. Five of them apply without incident:

$ git am --3way \
    ../../../gecko-dev-project/gecko-dev/rpm/0059-sailfishos-gecko-Delete-startupCache-if-it-s-stale.patch
$ git am --3way \
    ../../../gecko-dev-project/gecko-dev/rpm/0067-sailfishos-gecko-Hardcode-loopback-address-for-profi.patch
$ git am --3way \
    ../../../gecko-dev-project/gecko-dev/rpm/0068-sailfishos-gecko-Start-using-user-agent-builder.-JB-.patch
$ git am --3way \
    ../../../gecko-dev-project/gecko-dev/rpm/0083-sailfishos-gecko-dev-Disallow-page-zooming-if-the-me.patch
$ git am --3way \
    ../../../gecko-dev-project/gecko-dev/rpm/0086-sailfishos-gecko-Add-preference-to-bypass-CORS-on-ns.patch

The final patch is messy: when I try to apply it, there are conflicts all the way down:

$ git am --3way \
    ../../../gecko-dev-project/gecko-dev/rpm/0061-sailfishos-locale-Get-12-24h-timeformat-setting-from.patch

In spite of this it turns out to be pretty simple to apply the changes manually, which I've now done. Since I did it as part of the git am process the correct description and attribution have already been automatically applied. So finally I can generate new patches from these commits and add them to the build configuration:

$ git format-patch \--start-number 79 -o ../rpm -N --no-signature --zero-commit bd026681f74d

I also received a really helpful message from Matti Viljanen (direc85) giving suggestions for improving the build configuration to force the Rust portions to build with just a single process. The importance of this is that running Rust with multiple processes results in sporadic hangs during the build. That's a real pain because after leaving the build running overnight you don't want to discover in the morning it got stuck after the first hour and didn't make progress beyond that. If the Rust parts can be locked to a single process, it would allow the non-Rust parts (primarily C++ code, which still makes up the majority of the codebase) to run using multiple processes. That'd give us a nice speed-up.

Unfortunately I wasn't able to get the changes to work correctly with my build engine. I suspect it's simply a case of me needing to reinstall the build engine to get a more recent version, but as this is quite a major task, it'll have to wait for now. I'm not as familiar as Matti is with the Rust toolchain in the Sailfish SDK, so I'm going to have to go back and discuss the details with him.

Finally, and this is really important, I received an email from Frajo concerning the libhybris changes. This also relates to the discussion on GitHub between Affe Null (affenull2345), Mal (mlehtima) and Frajo (krnlyng) in relation to getting the build running on Affe's Samsung Galaxy A40 port.

One of the outcomes of that discussion was to expose a further potential issue in libhybris. Once fixed Affe reported that the cover crash described in issue 1051 was no longer occurring. This cover crash has been a thorn in my side for several months now, so I'm really keen to know whether it does actually fix it or not.

Frajo provided me with some new libhybris rpm packages to test out that include these latest changes. Having installed them and tested the browser I find that... yes! The cover change no longer causes the browser to crash. This is good for having the cover working with the browser, but even more important is the fact it also resolves the issue with the email app and other WebView apps crashing when moving in and out of the app.

So I'm super-pleased with all this. I've sadly not had time to get involved in the topic in the forum over the last few days, but it's incredibly exciting to see all of the discussion and am thrilled at the effort people are putting in, not just into finding bugs, but also into resolving them.

Sadly I have to stop for the day today, but since I've made changes to the commits, especially to the optimisation of the build, I need to run a new build overnight from scratch before I can create my pull request. It'll add an extra day, but it's important to check everything is working correctly both at build and run-time after this change. Just one of the many reasons this has been such a slow process! Assuming the build passes, tomorrow morning I'll do a bit more testing of the changes and if all goes well I'll be able to create that pull request.

Tomorrow will be the last diary entry before I take a week off to attend RSECon24. I'm hoping I'll get the pull requests submitted by then, but either way there will still be more diary to write once I'm back in a week's time. So much to do!

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.