Gecko-dev Diary
Starting in August 2023 I'll be upgrading the Sailfish OS browser from Gecko version ESR 78 to ESR 91. This page catalogues my progress.
Latest code changes are in the gecko-dev sailfishos-esr91 branch.
There is an index of all posts in case you want to jump to a particular day.
Click the icon for the Gecko-dev Diary RSS feed.
Gecko
5 most recent items
8 Sep 2024 : Day 339 #
It's been just under a week since I posted my last diary entry. In the meantime I've been attending RSECon24 learning about assorted software engineering topics and meeting others in the research software engineering community. But you'll be pleased to hear that I did also find some time for gecko dev work as well.
Not as much as I would normally do, so in retrospect pausing my diary entries was the right thing to do. But it's been enough progress to justify my promised summary today.
Before we get into the details of what I've personally been working on I want to first talk about what others have been doing, as well as considering the future of these dev diaries.
Before heading to the conference I submitted five pull requests:
And that seems to have been borne out by how things have been moving since then. There have been exciting developments pulling in input from across the Sailfish community and beyond, from both developers and users. I'd like to highlight just some of the things that have started happening recently and which give me great optimism for the future of both Sailfish OS browser development and the Sailfish OS platform more broadly.
Started by Frajo (krnlyng), but then contributed to by Mal (mlehtima) and Affe Nul (affenull2345), the hangs and crashes coming from the EGL rendering pipeline have been carefully ironed out in the libhybris layer. The solutions were completely unexpected as far as I was concerned and I'd never have been able to figure them out myself, so this was a source of much celebration and admiration from me.
There have also been other pull requests to address actual issues in the browser brought on by the upgrade. There are so many features in a browser it's very hard to test everything (an argument for good unit tests, but that's a real challenge given the discrete approach to upgrades required for Gecko on Sailfish OS). A great example of this is the pull request submitted by Vlad G (vlagged) to fix bad certificate overrides. It was easy for me to miss this functionality. Vlad has been a dedicated follower of the dev diaries and it's important not to underestimate how important contributions like this are. It's this kind of diligent, proactive and practical help that the browser really need to be a success.
Quite apart from the motivation it's given me over the last year, the discussion on the Sailfish forum has driven some important advances for the browser. There's been a healthy discussion about sites that have glitches or aren't working correctly, for example from Tone (tortoisedoc), Daniel (jauri.gagarin.II), Sebastian (smatkovi) and others.
I'm hoping that many of these issues will turn out to be related to user-agent strings and the fact that I'd missed off an important user-agent patch. This was flagged up through a discussion between Attah (attah) and Raine (rainemak). Now that the patch has been applied, thanks to their work, many of the pages will hopefully work more successfully.
Members of the Sailfish community have invested their time and energy into testing ESR 91 on a range of difference devices, including Pasi (Pasik2) testing on the Pinetab 2, Ladislav Hodas (lhodas) on the Gemini and Affe Nul (affenull2345) on the Samsung Galaxy A40. Matti Viljanen (direc85) deserves a special mention for getting complete rebuilds working for armv7hl and i486 versions of the packages, demonstrating that this can be achieved with minimal changes. From what I understand there are still issues to be worked out with the resulting packages, but this is work that I'd have struggled to do on my own.
Sticking with Matti, who I'm very pleased to see is now working for Jolla (congratulations!), he provided me with excellent advice on improving the build both in public and private communications, especially in relation to the Rust components. On the topic of Rust, I also received really useful suggestions from Tone (tortoisedoc) which also has the potential to improve Rust builds.
There was some useful discussion on the forum about the WebRender pipeline and its relationship to Progressive Web Apps, with Simon Schmeisser (simonschmeisser), Tone (tortoisedoc), Attah (attah) and olf (Olf0) all contributing. It was a worthwhile read covering important ground. These are the sorts of topics that will deserve even more discussion once we start thinking about ESR 102 (it will happen!).
I can't possibly mention everyone who's spent time encouraging me over the last year, but let me just say that it was gratifying to receive such kind remarks on the forum from so many users who tried out the packages, including Harry (hschmitt), hanswf, Helge, Patrick (pherjung), ric9k, 247, Tuomas Nurmi (Mazoon), Christian (p1rat), Maximilian (Maximilian1st), Edz, Uli (Fellfrosch), Mark Washeim (poetaster), Throwaway69, eson, Gabriele (gabrigubin), Felix (FelixWilke), Ricardo (monkeyisland) and KeksKopf (davidrasch), as well as many more on Mastodon.
And finally, the really hard work will now fall to the Jolla team. Raine (rainemak) and Mal (mlehtima) have already provided invaluable review comments. Alongside Frajo (krnlyng), Matti (direc85) and no doubt others, I'm hoping they'll be able to pick up some of this work to chisel off the sharper corners and sand down the rough edges. The result of Jolla's unparalleled engineering expertise will give a better result than I could ever hope to achieve.
An important thread that runs throughout these contributions is that I'd have struggled to do all this on my own. Focused on the code, I've not been spending my time trying to persuade others to contribute. Members of the community have therefore taken their own initiative to help and the browser has benefited enormously as a result.
This list turned out far longer than I was expecting and by necessity captures only some of the exciting developments that have been driven by the community recently. I'm very excited to see how things continue to progress.
But it also highlights the challenge that's now presented to me as I write this development diary. The work has become more distributed and this will inevitably dilute my contributions in a positive way. That's a great thing and my expectation is that I'll likely now spend less of my time directly working on the code and more of my time on the processes needed to get the browser released.
That means a daily development diary probably doesn't make much sense any more. Up until now the act of writing daily has been critical in driving me forwards, ensuring I don't lose focus and keeping me on track. That's no longer needed; the browser now has its own momentum.
But I do still want to maintain a record of progress. I hope to capture developments up until the browser is properly released, assuming it can get to that point. So my plan for this diary in the future is to write a weekly update on how things are progressing. If there's not much to write about, I'll write up my thoughts on a topic related to Gecko and Sailfish OS more generally. I'll publish them at the weekend because that's when the other pieces of my life are least frantic, giving me the opportunity to consolidate my thoughts from the week gone by. To reflect this change in pace I'll no longer count the days but rather give each post a proper title.
That's my plan. If it doesn't work out, I can always change it. Please do share your thoughts about it with me as I'm always grateful to receive them.
With all this in mind, let's return to those pull requests and talk about what's happened to them over the last week. I'm really happy that two of them have already been merged into the upstream repository. They're not on the main branch yet, which is still on ESR 78, but this is an important step towards that.
That leaves three remaining. Raine and Mal both added comments to these which left me with a bit of work to do to address their requests. Some of these changes were straightforward, like removing extraneous empty lines that I added during development and forgot to remove.
Changes like these have no impact on the code semantics and so wouldn't even necessarily justify a rebuild. But they still take a surprising length of time to address. If they're part of the mirrored gecko code the repository has to be reset, cleaned and all patches applied. Once the change has been made it needs to be squashed into the commit where it was added. Then the patches need to be regenerated. Only then can the changes be committed and pushed to the remote repository.
This is one of the reasons I'm not keen on working with patches. I find myself repeatedly having to clean the repository, apply the patches and then regenerate them before a fresh set of commits can be pushed. I find it surprisingly laborious. And while I fully understand the reasons for using them (the rpm build process has been tailored for this approach) that doesn't make me like them any more.
That's for the simplest of changes, but some of the changes are more substantial. For example Raine requested various changes to how preferences should be applied, all of which are very legitimate.
These changes centre around Sailfish OS specific preferences which differ depending on whether the browser or WebView is in use.
In these cases they can either be set to the default of the browser or set to the default of the WebView. Whichever way is chosen, some code is needed in either the WebView or the browser to ensure the value is correctly toggled from the default for the other.
My original code set the value to a default that worked for the browser, then flipped them in the WebView initialisation code. From what I understand, Raine was keen for me to do the opposite. That is, to set defaults that work for the WebView but are then flipped by the browser initialisation code. The request itself was spread over several or Raine's comments, but the following is perhaps the pithiest of the descriptions:
Consequently I updated the default values set in the gecko code, removed some code from the WebView and added some equivalent code to the browser like this:
Having made these changes I then merged them in to the existing branch. To keep things clean I've squashed the changes into relevant commits. But this did also mean spending a bit of time rebasing all of the commits in the pull request.
The resulting in changes to all three of the outstanding pull requests:
Consequently I've pushed the changes upstream to the pull requests. My timing isn't ideal, right before the weekend, but I'll look forward to hearing more from the Jolla team in due course.
That's it for today. I'll return next week with a summary of changes that have taken place during the week. I'm also planning a retrospective to summarise the entire process of the last 339 days of development: what went well, what could have gone better and what the experience has been like for me.
If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Not as much as I would normally do, so in retrospect pausing my diary entries was the right thing to do. But it's been enough progress to justify my promised summary today.
Before we get into the details of what I've personally been working on I want to first talk about what others have been doing, as well as considering the future of these dev diaries.
Before heading to the conference I submitted five pull requests:
- https://github.com/sailfishos/gecko-dev/pull/162
- https://github.com/sailfishos/qtmozembed/pull/49
- https://github.com/sailfishos/sailfish-browser/pull/1074
- https://github.com/sailfishos/embedlite-components/pull/100
- https://github.com/sailfishos/sailfish-components-webview/pull/169
And that seems to have been borne out by how things have been moving since then. There have been exciting developments pulling in input from across the Sailfish community and beyond, from both developers and users. I'd like to highlight just some of the things that have started happening recently and which give me great optimism for the future of both Sailfish OS browser development and the Sailfish OS platform more broadly.
Started by Frajo (krnlyng), but then contributed to by Mal (mlehtima) and Affe Nul (affenull2345), the hangs and crashes coming from the EGL rendering pipeline have been carefully ironed out in the libhybris layer. The solutions were completely unexpected as far as I was concerned and I'd never have been able to figure them out myself, so this was a source of much celebration and admiration from me.
There have also been other pull requests to address actual issues in the browser brought on by the upgrade. There are so many features in a browser it's very hard to test everything (an argument for good unit tests, but that's a real challenge given the discrete approach to upgrades required for Gecko on Sailfish OS). A great example of this is the pull request submitted by Vlad G (vlagged) to fix bad certificate overrides. It was easy for me to miss this functionality. Vlad has been a dedicated follower of the dev diaries and it's important not to underestimate how important contributions like this are. It's this kind of diligent, proactive and practical help that the browser really need to be a success.
Quite apart from the motivation it's given me over the last year, the discussion on the Sailfish forum has driven some important advances for the browser. There's been a healthy discussion about sites that have glitches or aren't working correctly, for example from Tone (tortoisedoc), Daniel (jauri.gagarin.II), Sebastian (smatkovi) and others.
I'm hoping that many of these issues will turn out to be related to user-agent strings and the fact that I'd missed off an important user-agent patch. This was flagged up through a discussion between Attah (attah) and Raine (rainemak). Now that the patch has been applied, thanks to their work, many of the pages will hopefully work more successfully.
Members of the Sailfish community have invested their time and energy into testing ESR 91 on a range of difference devices, including Pasi (Pasik2) testing on the Pinetab 2, Ladislav Hodas (lhodas) on the Gemini and Affe Nul (affenull2345) on the Samsung Galaxy A40. Matti Viljanen (direc85) deserves a special mention for getting complete rebuilds working for armv7hl and i486 versions of the packages, demonstrating that this can be achieved with minimal changes. From what I understand there are still issues to be worked out with the resulting packages, but this is work that I'd have struggled to do on my own.
Sticking with Matti, who I'm very pleased to see is now working for Jolla (congratulations!), he provided me with excellent advice on improving the build both in public and private communications, especially in relation to the Rust components. On the topic of Rust, I also received really useful suggestions from Tone (tortoisedoc) which also has the potential to improve Rust builds.
There was some useful discussion on the forum about the WebRender pipeline and its relationship to Progressive Web Apps, with Simon Schmeisser (simonschmeisser), Tone (tortoisedoc), Attah (attah) and olf (Olf0) all contributing. It was a worthwhile read covering important ground. These are the sorts of topics that will deserve even more discussion once we start thinking about ESR 102 (it will happen!).
I can't possibly mention everyone who's spent time encouraging me over the last year, but let me just say that it was gratifying to receive such kind remarks on the forum from so many users who tried out the packages, including Harry (hschmitt), hanswf, Helge, Patrick (pherjung), ric9k, 247, Tuomas Nurmi (Mazoon), Christian (p1rat), Maximilian (Maximilian1st), Edz, Uli (Fellfrosch), Mark Washeim (poetaster), Throwaway69, eson, Gabriele (gabrigubin), Felix (FelixWilke), Ricardo (monkeyisland) and KeksKopf (davidrasch), as well as many more on Mastodon.
And finally, the really hard work will now fall to the Jolla team. Raine (rainemak) and Mal (mlehtima) have already provided invaluable review comments. Alongside Frajo (krnlyng), Matti (direc85) and no doubt others, I'm hoping they'll be able to pick up some of this work to chisel off the sharper corners and sand down the rough edges. The result of Jolla's unparalleled engineering expertise will give a better result than I could ever hope to achieve.
An important thread that runs throughout these contributions is that I'd have struggled to do all this on my own. Focused on the code, I've not been spending my time trying to persuade others to contribute. Members of the community have therefore taken their own initiative to help and the browser has benefited enormously as a result.
This list turned out far longer than I was expecting and by necessity captures only some of the exciting developments that have been driven by the community recently. I'm very excited to see how things continue to progress.
But it also highlights the challenge that's now presented to me as I write this development diary. The work has become more distributed and this will inevitably dilute my contributions in a positive way. That's a great thing and my expectation is that I'll likely now spend less of my time directly working on the code and more of my time on the processes needed to get the browser released.
That means a daily development diary probably doesn't make much sense any more. Up until now the act of writing daily has been critical in driving me forwards, ensuring I don't lose focus and keeping me on track. That's no longer needed; the browser now has its own momentum.
But I do still want to maintain a record of progress. I hope to capture developments up until the browser is properly released, assuming it can get to that point. So my plan for this diary in the future is to write a weekly update on how things are progressing. If there's not much to write about, I'll write up my thoughts on a topic related to Gecko and Sailfish OS more generally. I'll publish them at the weekend because that's when the other pieces of my life are least frantic, giving me the opportunity to consolidate my thoughts from the week gone by. To reflect this change in pace I'll no longer count the days but rather give each post a proper title.
That's my plan. If it doesn't work out, I can always change it. Please do share your thoughts about it with me as I'm always grateful to receive them.
With all this in mind, let's return to those pull requests and talk about what's happened to them over the last week. I'm really happy that two of them have already been merged into the upstream repository. They're not on the main branch yet, which is still on ESR 78, but this is an important step towards that.
That leaves three remaining. Raine and Mal both added comments to these which left me with a bit of work to do to address their requests. Some of these changes were straightforward, like removing extraneous empty lines that I added during development and forgot to remove.
Changes like these have no impact on the code semantics and so wouldn't even necessarily justify a rebuild. But they still take a surprising length of time to address. If they're part of the mirrored gecko code the repository has to be reset, cleaned and all patches applied. Once the change has been made it needs to be squashed into the commit where it was added. Then the patches need to be regenerated. Only then can the changes be committed and pushed to the remote repository.
This is one of the reasons I'm not keen on working with patches. I find myself repeatedly having to clean the repository, apply the patches and then regenerate them before a fresh set of commits can be pushed. I find it surprisingly laborious. And while I fully understand the reasons for using them (the rpm build process has been tailored for this approach) that doesn't make me like them any more.
That's for the simplest of changes, but some of the changes are more substantial. For example Raine requested various changes to how preferences should be applied, all of which are very legitimate.
These changes centre around Sailfish OS specific preferences which differ depending on whether the browser or WebView is in use.
In these cases they can either be set to the default of the browser or set to the default of the WebView. Whichever way is chosen, some code is needed in either the WebView or the browser to ensure the value is correctly toggled from the default for the other.
My original code set the value to a default that worked for the browser, then flipped them in the WebView initialisation code. From what I understand, Raine was keen for me to do the opposite. That is, to set defaults that work for the WebView but are then flipped by the browser initialisation code. The request itself was spread over several or Raine's comments, but the following is perhaps the pithiest of the descriptions:
I think embedding.js (embedlite omni.ja) should default to false and browser set it true.
Consequently I updated the default values set in the gecko code, removed some code from the WebView and added some equivalent code to the browser like this:
// Ensure the renderer is configured correctly
webEngineSettings->setPreference(QStringLiteral(
"embedlite.compositor.external_gl_context"),
QVariant(true));
webEngineSettings->setPreference(QStringLiteral(
"embedlite.compositor.request_external_gl_context_early"),
QVariant(true));
Thankfully these are also pretty simple changes, but they did need a bit of thought, not least because they touch on three separate packages. The gecko engine, the browser and the WebView code all needed updating, albeit each in only a small way.Having made these changes I then merged them in to the existing branch. To keep things clean I've squashed the changes into relevant commits. But this did also mean spending a bit of time rebasing all of the commits in the pull request.
The resulting in changes to all three of the outstanding pull requests:
- https://github.com/sailfishos/gecko-dev/pull/162
- https://github.com/sailfishos/sailfish-browser/pull/1074
- https://github.com/sailfishos/sailfish-components-webview/pull/169
Consequently I've pushed the changes upstream to the pull requests. My timing isn't ideal, right before the weekend, but I'll look forward to hearing more from the Jolla team in due course.
That's it for today. I'll return next week with a summary of changes that have taken place during the week. I'm also planning a retrospective to summarise the entire process of the last 339 days of development: what went well, what could have gone better and what the experience has been like for me.
If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comments
Uncover Disqus comments