Gecko-dev Diary
Starting in August 2023 I'll be upgrading the Sailfish OS browser from Gecko version ESR 78 to ESR 91. This page catalogues my progress.
Latest code changes are in the gecko-dev sailfishos-esr91 branch.
There is an index of all posts in case you want to jump to a particular day.
Click the icon for the Gecko-dev Diary RSS feed.
Gecko
5 most recent items
8 Sep 2024 : Day 339 #
It's been just under a week since I posted my last diary entry. In the meantime I've been attending RSECon24 learning about assorted software engineering topics and meeting others in the research software engineering community. But you'll be pleased to hear that I did also find some time for gecko dev work as well.
Not as much as I would normally do, so in retrospect pausing my diary entries was the right thing to do. But it's been enough progress to justify my promised summary today.
Before we get into the details of what I've personally been working on I want to first talk about what others have been doing, as well as considering the future of these dev diaries.
Before heading to the conference I submitted five pull requests:
And that seems to have been borne out by how things have been moving since then. There have been exciting developments pulling in input from across the Sailfish community and beyond, from both developers and users. I'd like to highlight just some of the things that have started happening recently and which give me great optimism for the future of both Sailfish OS browser development and the Sailfish OS platform more broadly.
Started by Frajo (krnlyng), but then contributed to by Mal (mlehtima) and Affe Nul (affenull2345), the hangs and crashes coming from the EGL rendering pipeline have been carefully ironed out in the libhybris layer. The solutions were completely unexpected as far as I was concerned and I'd never have been able to figure them out myself, so this was a source of much celebration and admiration from me.
There have also been other pull requests to address actual issues in the browser brought on by the upgrade. There are so many features in a browser it's very hard to test everything (an argument for good unit tests, but that's a real challenge given the discrete approach to upgrades required for Gecko on Sailfish OS). A great example of this is the pull request submitted by Vlad G (vlagged) to fix bad certificate overrides. It was easy for me to miss this functionality. Vlad has been a dedicated follower of the dev diaries and it's important not to underestimate how important contributions like this are. It's this kind of diligent, proactive and practical help that the browser really need to be a success.
Quite apart from the motivation it's given me over the last year, the discussion on the Sailfish forum has driven some important advances for the browser. There's been a healthy discussion about sites that have glitches or aren't working correctly, for example from Tone (tortoisedoc), Daniel (jauri.gagarin.II), Sebastian (smatkovi) and others.
I'm hoping that many of these issues will turn out to be related to user-agent strings and the fact that I'd missed off an important user-agent patch. This was flagged up through a discussion between Attah (attah) and Raine (rainemak). Now that the patch has been applied, thanks to their work, many of the pages will hopefully work more successfully.
Members of the Sailfish community have invested their time and energy into testing ESR 91 on a range of difference devices, including Pasi (Pasik2) testing on the Pinetab 2, Ladislav Hodas (lhodas) on the Gemini and Affe Nul (affenull2345) on the Samsung Galaxy A40. Matti Viljanen (direc85) deserves a special mention for getting complete rebuilds working for armv7hl and i486 versions of the packages, demonstrating that this can be achieved with minimal changes. From what I understand there are still issues to be worked out with the resulting packages, but this is work that I'd have struggled to do on my own.
Sticking with Matti, who I'm very pleased to see is now working for Jolla (congratulations!), he provided me with excellent advice on improving the build both in public and private communications, especially in relation to the Rust components. On the topic of Rust, I also received really useful suggestions from Tone (tortoisedoc) which also has the potential to improve Rust builds.
There was some useful discussion on the forum about the WebRender pipeline and its relationship to Progressive Web Apps, with Simon Schmeisser (simonschmeisser), Tone (tortoisedoc), Attah (attah) and olf (Olf0) all contributing. It was a worthwhile read covering important ground. These are the sorts of topics that will deserve even more discussion once we start thinking about ESR 102 (it will happen!).
I can't possibly mention everyone who's spent time encouraging me over the last year, but let me just say that it was gratifying to receive such kind remarks on the forum from so many users who tried out the packages, including Harry (hschmitt), hanswf, Helge, Patrick (pherjung), ric9k, 247, Tuomas Nurmi (Mazoon), Christian (p1rat), Maximilian (Maximilian1st), Edz, Uli (Fellfrosch), Mark Washeim (poetaster), Throwaway69, eson, Gabriele (gabrigubin), Felix (FelixWilke), Ricardo (monkeyisland) and KeksKopf (davidrasch), as well as many more on Mastodon.
And finally, the really hard work will now fall to the Jolla team. Raine (rainemak) and Mal (mlehtima) have already provided invaluable review comments. Alongside Frajo (krnlyng), Matti (direc85) and no doubt others, I'm hoping they'll be able to pick up some of this work to chisel off the sharper corners and sand down the rough edges. The result of Jolla's unparalleled engineering expertise will give a better result than I could ever hope to achieve.
An important thread that runs throughout these contributions is that I'd have struggled to do all this on my own. Focused on the code, I've not been spending my time trying to persuade others to contribute. Members of the community have therefore taken their own initiative to help and the browser has benefited enormously as a result.
This list turned out far longer than I was expecting and by necessity captures only some of the exciting developments that have been driven by the community recently. I'm very excited to see how things continue to progress.
But it also highlights the challenge that's now presented to me as I write this development diary. The work has become more distributed and this will inevitably dilute my contributions in a positive way. That's a great thing and my expectation is that I'll likely now spend less of my time directly working on the code and more of my time on the processes needed to get the browser released.
That means a daily development diary probably doesn't make much sense any more. Up until now the act of writing daily has been critical in driving me forwards, ensuring I don't lose focus and keeping me on track. That's no longer needed; the browser now has its own momentum.
But I do still want to maintain a record of progress. I hope to capture developments up until the browser is properly released, assuming it can get to that point. So my plan for this diary in the future is to write a weekly update on how things are progressing. If there's not much to write about, I'll write up my thoughts on a topic related to Gecko and Sailfish OS more generally. I'll publish them at the weekend because that's when the other pieces of my life are least frantic, giving me the opportunity to consolidate my thoughts from the week gone by. To reflect this change in pace I'll no longer count the days but rather give each post a proper title.
That's my plan. If it doesn't work out, I can always change it. Please do share your thoughts about it with me as I'm always grateful to receive them.
With all this in mind, let's return to those pull requests and talk about what's happened to them over the last week. I'm really happy that two of them have already been merged into the upstream repository. They're not on the main branch yet, which is still on ESR 78, but this is an important step towards that.
That leaves three remaining. Raine and Mal both added comments to these which left me with a bit of work to do to address their requests. Some of these changes were straightforward, like removing extraneous empty lines that I added during development and forgot to remove.
Changes like these have no impact on the code semantics and so wouldn't even necessarily justify a rebuild. But they still take a surprising length of time to address. If they're part of the mirrored gecko code the repository has to be reset, cleaned and all patches applied. Once the change has been made it needs to be squashed into the commit where it was added. Then the patches need to be regenerated. Only then can the changes be committed and pushed to the remote repository.
This is one of the reasons I'm not keen on working with patches. I find myself repeatedly having to clean the repository, apply the patches and then regenerate them before a fresh set of commits can be pushed. I find it surprisingly laborious. And while I fully understand the reasons for using them (the rpm build process has been tailored for this approach) that doesn't make me like them any more.
That's for the simplest of changes, but some of the changes are more substantial. For example Raine requested various changes to how preferences should be applied, all of which are very legitimate.
These changes centre around Sailfish OS specific preferences which differ depending on whether the browser or WebView is in use.
In these cases they can either be set to the default of the browser or set to the default of the WebView. Whichever way is chosen, some code is needed in either the WebView or the browser to ensure the value is correctly toggled from the default for the other.
My original code set the value to a default that worked for the browser, then flipped them in the WebView initialisation code. From what I understand, Raine was keen for me to do the opposite. That is, to set defaults that work for the WebView but are then flipped by the browser initialisation code. The request itself was spread over several or Raine's comments, but the following is perhaps the pithiest of the descriptions:
Consequently I updated the default values set in the gecko code, removed some code from the WebView and added some equivalent code to the browser like this:
Having made these changes I then merged them in to the existing branch. To keep things clean I've squashed the changes into relevant commits. But this did also mean spending a bit of time rebasing all of the commits in the pull request.
The resulting in changes to all three of the outstanding pull requests:
Consequently I've pushed the changes upstream to the pull requests. My timing isn't ideal, right before the weekend, but I'll look forward to hearing more from the Jolla team in due course.
That's it for today. I'll return next week with a summary of changes that have taken place during the week. I'm also planning a retrospective to summarise the entire process of the last 339 days of development: what went well, what could have gone better and what the experience has been like for me.
If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
Not as much as I would normally do, so in retrospect pausing my diary entries was the right thing to do. But it's been enough progress to justify my promised summary today.
Before we get into the details of what I've personally been working on I want to first talk about what others have been doing, as well as considering the future of these dev diaries.
Before heading to the conference I submitted five pull requests:
- https://github.com/sailfishos/gecko-dev/pull/162
- https://github.com/sailfishos/qtmozembed/pull/49
- https://github.com/sailfishos/sailfish-browser/pull/1074
- https://github.com/sailfishos/embedlite-components/pull/100
- https://github.com/sailfishos/sailfish-components-webview/pull/169
And that seems to have been borne out by how things have been moving since then. There have been exciting developments pulling in input from across the Sailfish community and beyond, from both developers and users. I'd like to highlight just some of the things that have started happening recently and which give me great optimism for the future of both Sailfish OS browser development and the Sailfish OS platform more broadly.
Started by Frajo (krnlyng), but then contributed to by Mal (mlehtima) and Affe Nul (affenull2345), the hangs and crashes coming from the EGL rendering pipeline have been carefully ironed out in the libhybris layer. The solutions were completely unexpected as far as I was concerned and I'd never have been able to figure them out myself, so this was a source of much celebration and admiration from me.
There have also been other pull requests to address actual issues in the browser brought on by the upgrade. There are so many features in a browser it's very hard to test everything (an argument for good unit tests, but that's a real challenge given the discrete approach to upgrades required for Gecko on Sailfish OS). A great example of this is the pull request submitted by Vlad G (vlagged) to fix bad certificate overrides. It was easy for me to miss this functionality. Vlad has been a dedicated follower of the dev diaries and it's important not to underestimate how important contributions like this are. It's this kind of diligent, proactive and practical help that the browser really need to be a success.
Quite apart from the motivation it's given me over the last year, the discussion on the Sailfish forum has driven some important advances for the browser. There's been a healthy discussion about sites that have glitches or aren't working correctly, for example from Tone (tortoisedoc), Daniel (jauri.gagarin.II), Sebastian (smatkovi) and others.
I'm hoping that many of these issues will turn out to be related to user-agent strings and the fact that I'd missed off an important user-agent patch. This was flagged up through a discussion between Attah (attah) and Raine (rainemak). Now that the patch has been applied, thanks to their work, many of the pages will hopefully work more successfully.
Members of the Sailfish community have invested their time and energy into testing ESR 91 on a range of difference devices, including Pasi (Pasik2) testing on the Pinetab 2, Ladislav Hodas (lhodas) on the Gemini and Affe Nul (affenull2345) on the Samsung Galaxy A40. Matti Viljanen (direc85) deserves a special mention for getting complete rebuilds working for armv7hl and i486 versions of the packages, demonstrating that this can be achieved with minimal changes. From what I understand there are still issues to be worked out with the resulting packages, but this is work that I'd have struggled to do on my own.
Sticking with Matti, who I'm very pleased to see is now working for Jolla (congratulations!), he provided me with excellent advice on improving the build both in public and private communications, especially in relation to the Rust components. On the topic of Rust, I also received really useful suggestions from Tone (tortoisedoc) which also has the potential to improve Rust builds.
There was some useful discussion on the forum about the WebRender pipeline and its relationship to Progressive Web Apps, with Simon Schmeisser (simonschmeisser), Tone (tortoisedoc), Attah (attah) and olf (Olf0) all contributing. It was a worthwhile read covering important ground. These are the sorts of topics that will deserve even more discussion once we start thinking about ESR 102 (it will happen!).
I can't possibly mention everyone who's spent time encouraging me over the last year, but let me just say that it was gratifying to receive such kind remarks on the forum from so many users who tried out the packages, including Harry (hschmitt), hanswf, Helge, Patrick (pherjung), ric9k, 247, Tuomas Nurmi (Mazoon), Christian (p1rat), Maximilian (Maximilian1st), Edz, Uli (Fellfrosch), Mark Washeim (poetaster), Throwaway69, eson, Gabriele (gabrigubin), Felix (FelixWilke), Ricardo (monkeyisland) and KeksKopf (davidrasch), as well as many more on Mastodon.
And finally, the really hard work will now fall to the Jolla team. Raine (rainemak) and Mal (mlehtima) have already provided invaluable review comments. Alongside Frajo (krnlyng), Matti (direc85) and no doubt others, I'm hoping they'll be able to pick up some of this work to chisel off the sharper corners and sand down the rough edges. The result of Jolla's unparalleled engineering expertise will give a better result than I could ever hope to achieve.
An important thread that runs throughout these contributions is that I'd have struggled to do all this on my own. Focused on the code, I've not been spending my time trying to persuade others to contribute. Members of the community have therefore taken their own initiative to help and the browser has benefited enormously as a result.
This list turned out far longer than I was expecting and by necessity captures only some of the exciting developments that have been driven by the community recently. I'm very excited to see how things continue to progress.
But it also highlights the challenge that's now presented to me as I write this development diary. The work has become more distributed and this will inevitably dilute my contributions in a positive way. That's a great thing and my expectation is that I'll likely now spend less of my time directly working on the code and more of my time on the processes needed to get the browser released.
That means a daily development diary probably doesn't make much sense any more. Up until now the act of writing daily has been critical in driving me forwards, ensuring I don't lose focus and keeping me on track. That's no longer needed; the browser now has its own momentum.
But I do still want to maintain a record of progress. I hope to capture developments up until the browser is properly released, assuming it can get to that point. So my plan for this diary in the future is to write a weekly update on how things are progressing. If there's not much to write about, I'll write up my thoughts on a topic related to Gecko and Sailfish OS more generally. I'll publish them at the weekend because that's when the other pieces of my life are least frantic, giving me the opportunity to consolidate my thoughts from the week gone by. To reflect this change in pace I'll no longer count the days but rather give each post a proper title.
That's my plan. If it doesn't work out, I can always change it. Please do share your thoughts about it with me as I'm always grateful to receive them.
With all this in mind, let's return to those pull requests and talk about what's happened to them over the last week. I'm really happy that two of them have already been merged into the upstream repository. They're not on the main branch yet, which is still on ESR 78, but this is an important step towards that.
That leaves three remaining. Raine and Mal both added comments to these which left me with a bit of work to do to address their requests. Some of these changes were straightforward, like removing extraneous empty lines that I added during development and forgot to remove.
Changes like these have no impact on the code semantics and so wouldn't even necessarily justify a rebuild. But they still take a surprising length of time to address. If they're part of the mirrored gecko code the repository has to be reset, cleaned and all patches applied. Once the change has been made it needs to be squashed into the commit where it was added. Then the patches need to be regenerated. Only then can the changes be committed and pushed to the remote repository.
This is one of the reasons I'm not keen on working with patches. I find myself repeatedly having to clean the repository, apply the patches and then regenerate them before a fresh set of commits can be pushed. I find it surprisingly laborious. And while I fully understand the reasons for using them (the rpm build process has been tailored for this approach) that doesn't make me like them any more.
That's for the simplest of changes, but some of the changes are more substantial. For example Raine requested various changes to how preferences should be applied, all of which are very legitimate.
These changes centre around Sailfish OS specific preferences which differ depending on whether the browser or WebView is in use.
In these cases they can either be set to the default of the browser or set to the default of the WebView. Whichever way is chosen, some code is needed in either the WebView or the browser to ensure the value is correctly toggled from the default for the other.
My original code set the value to a default that worked for the browser, then flipped them in the WebView initialisation code. From what I understand, Raine was keen for me to do the opposite. That is, to set defaults that work for the WebView but are then flipped by the browser initialisation code. The request itself was spread over several or Raine's comments, but the following is perhaps the pithiest of the descriptions:
I think embedding.js (embedlite omni.ja) should default to false and browser set it true.
Consequently I updated the default values set in the gecko code, removed some code from the WebView and added some equivalent code to the browser like this:
// Ensure the renderer is configured correctly
webEngineSettings->setPreference(QStringLiteral(
"embedlite.compositor.external_gl_context"),
QVariant(true));
webEngineSettings->setPreference(QStringLiteral(
"embedlite.compositor.request_external_gl_context_early"),
QVariant(true));
Thankfully these are also pretty simple changes, but they did need a bit of thought, not least because they touch on three separate packages. The gecko engine, the browser and the WebView code all needed updating, albeit each in only a small way.Having made these changes I then merged them in to the existing branch. To keep things clean I've squashed the changes into relevant commits. But this did also mean spending a bit of time rebasing all of the commits in the pull request.
The resulting in changes to all three of the outstanding pull requests:
- https://github.com/sailfishos/gecko-dev/pull/162
- https://github.com/sailfishos/sailfish-browser/pull/1074
- https://github.com/sailfishos/sailfish-components-webview/pull/169
Consequently I've pushed the changes upstream to the pull requests. My timing isn't ideal, right before the weekend, but I'll look forward to hearing more from the Jolla team in due course.
That's it for today. I'll return next week with a summary of changes that have taken place during the week. I'm also planning a retrospective to summarise the entire process of the last 339 days of development: what went well, what could have gone better and what the experience has been like for me.
If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
2 Sep 2024 : Day 338 #
As I write this it's the first day of Meteorological Autumn. It still feels very much like summer though (maybe because we won't reach Astronomical Autumn until later in the month). The sun is shining, there's a light breeze, crickets are chirping and there's a generally relaxed atmosphere in the air. But it's the changes in season that are always the most exciting time of year for me. I've enjoyed the summer, but I'm now looking forward to autumn and all of its mellow fruitfulness. Just thinking about it energises me.
But it also represents a shift in gear for this gecko diary as well. I'm hoping to get my pull requests submitted today. That doesn't mean the work is over, but it does mean that it's reached a stage where I feel it can be built upon (by both me and others). Moreover, as I mentioned previously I'll be taking a break this week while I'm attending a conference. So this will be my last diary entry until next weekend.
But that still means there's work to do today and I got up early this morning to check the build I started last night. And...? sadly it didn't complete. Most of the six patches I added last night seem to have built okay, but the startUpCache patch has caused errors. And a lot of errors at that. Although there are pages and pages of errors they all seem to be of one of two flavours.
First there are namespace errors
If I compare the code from ESR 78 it looks pretty similar, but I am a little surprised to see that the newly added Logging.h include is placed inside the namespace qualifiers:
This gives much better results: the majority of the errors are now gone and I'm just left with a couple of errors related to use of NS_LITERAL_STRING() and NS_LITERAL_CSTRING(). This has been a recurrent issue throughout this process since the gecko code now requires use of an _ns string suffix rather than one of the previous NS_LITERAL_STRING() macros.
Having replaced the two string literals with the annotations the code now compiles without error. But it means I have to do a complete build again, which means I have to regenerate the startupCache patch again. The commits have changed because the patch descriptions have changed as a result of being applied by the build process, but other than this I can use the same command to generate the patches as before:
So I've updated the patches, added the changes to the commits, cleaned the entire source tree of cruft and now it's time for another full rebuild.
I want at least one full rebuild to go through without errors before I create this mythical pull request I've been going on about. It's frankly turning out to be a little more laborious to get it to happen than I was expecting. But it'll get there.
[...]
Now finally after a full day of compilation, the final full build has completed. I've copied over and installed the packages and a quick check suggests things are working as expected. The pages are returning correctly as mobile pages. The email app doesn't crash when you switch back from the homescreen. The WebView seems to be working correctly.
The browser does still segfault on exit and, after a discussion with Frajo, it's possible that this could be fixable. But this isn't a new issue and while it'd be great to have it fixed, it could take a while to figure out. So that's going to have to be something for the future.
The task now, finally, is to submit the pull request. In fact, not just one pull request, but all five of them for the various packages that have needed changes needed for supporting the new engine. After a bit of playing around on GitHub, here they are:
I'll return to this next weekend and I'll continue with fresh diary entries then. But right now, I plan to spend a bit of time breathing in and enjoying the fresh autumnal air.
If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
But it also represents a shift in gear for this gecko diary as well. I'm hoping to get my pull requests submitted today. That doesn't mean the work is over, but it does mean that it's reached a stage where I feel it can be built upon (by both me and others). Moreover, as I mentioned previously I'll be taking a break this week while I'm attending a conference. So this will be my last diary entry until next weekend.
But that still means there's work to do today and I got up early this morning to check the build I started last night. And...? sadly it didn't complete. Most of the six patches I added last night seem to have built okay, but the startUpCache patch has caused errors. And a lot of errors at that. Although there are pages and pages of errors they all seem to be of one of two flavours.
First there are namespace errors
364:14.24 ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsILoadInfo.h: In member
function ‘bool nsILoadInfo::
GetAllowListFutureDocumentsCreatedFromThisRedirectChain()’:
364:14.24 ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsILoadInfo.h:849:5:
error: reference to ‘mozilla’ is ambiguous
364:14.24 849 | mozilla::DebugOnly<nsresult> rv =
364:14.24 849 | mozilla::DebugOnly rv = GetAllowListFutureDocumentsCreatedFromThisRedirectChain(&result);
364:14.24 | ^~~~~~~
Next there are undefined type errors (in this case nsresult):
364:14.30 ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsILoadInfo.h:962:32:
error: expected primary-expression before ‘>’ token
364:14.30 962 | mozilla::DebugOnly rv = GetIsMediaRequest(&result);
364:14.31 | ^
And then there are yet more undefined type errors like this one:
364:14.36 ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/Services.h:480:46:
error: ‘XPCOMService_GetHistory’ was not declared in this scope; did you
mean ‘XPCOMService_GetGfxInfo’?
364:14.36 480 | return already_AddRefed(XPCOMService_GetHistory());
364:14.36 | ^~~~~~~~~~~~~~~~~~~~~~~
364:14.36 | XPCOMService_GetGfxInfo
There are so many errors, but I'm hoping that it's a cascade: a single error that's triggering a whole cascade of follow-on errors, rather than a huge collection of unrelated errors. Time to dig in and find out.If I compare the code from ESR 78 it looks pretty similar, but I am a little surprised to see that the newly added Logging.h include is placed inside the namespace qualifiers:
using namespace mozilla::Compression;
namespace mozilla {
namespace scache {
#include "mozilla/Logging.h"
Although that does tally with the changes to the ESR 78 code, it doesn't match with the other ESR 91 files the Logging.h header is being included in, such as the EmbedLite code and elsewhere, so I'm going to move it out of the namespace and try again.This gives much better results: the majority of the errors are now gone and I'm just left with a couple of errors related to use of NS_LITERAL_STRING() and NS_LITERAL_CSTRING(). This has been a recurrent issue throughout this process since the gecko code now requires use of an _ns string suffix rather than one of the previous NS_LITERAL_STRING() macros.
Having replaced the two string literals with the annotations the code now compiles without error. But it means I have to do a complete build again, which means I have to regenerate the startupCache patch again. The commits have changed because the patch descriptions have changed as a result of being applied by the build process, but other than this I can use the same command to generate the patches as before:
$ git format-patch --start-number 79 -o ../rpm -N --no-signature --zero-commit
24f9ec2df3d3
The great news is that although this generates six patches, only the startUpCache patch is registering any changes, the others are all identical to the previous versions. This is a really good sign and exactly what I was hoping for.So I've updated the patches, added the changes to the commits, cleaned the entire source tree of cruft and now it's time for another full rebuild.
I want at least one full rebuild to go through without errors before I create this mythical pull request I've been going on about. It's frankly turning out to be a little more laborious to get it to happen than I was expecting. But it'll get there.
[...]
Now finally after a full day of compilation, the final full build has completed. I've copied over and installed the packages and a quick check suggests things are working as expected. The pages are returning correctly as mobile pages. The email app doesn't crash when you switch back from the homescreen. The WebView seems to be working correctly.
The browser does still segfault on exit and, after a discussion with Frajo, it's possible that this could be fixable. But this isn't a new issue and while it'd be great to have it fixed, it could take a while to figure out. So that's going to have to be something for the future.
The task now, finally, is to submit the pull request. In fact, not just one pull request, but all five of them for the various packages that have needed changes needed for supporting the new engine. After a bit of playing around on GitHub, here they are:
- https://github.com/sailfishos/gecko-dev/pull/162
- https://github.com/sailfishos/qtmozembed/pull/49
- https://github.com/sailfishos/sailfish-browser/pull/1074
- https://github.com/sailfishos/embedlite-components/pull/100
- https://github.com/sailfishos/sailfish-components-webview/pull/169
- Respond to the comments on the pull requests so they can be merged in. There are already some comments I need to address. Until they're merged in, they could still change a lot and ultimately, it's Jolla that has to decide whether or not to accept them. So they have to fulfil Jolla's requirements.
- Fix all the issues that have already come up in testing of the browser. It looks like many of these may be addressed by the user agent fix and the libhybris changes, but there are others. And I'm sure there will be many more besides.
- Fix the segfault on browser shutdown. This isn't new, but with the latest libhybris changes getting a clean shutdown feels closer than ever. It'd be so great if that could be achieved so it's worth spending a bit of time on it.
- Get the browser working on native devices. An important fix, mustn't forget this!
I'll return to this next weekend and I'll continue with fresh diary entries then. But right now, I plan to spend a bit of time breathing in and enjoying the fresh autumnal air.
If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
1 Sep 2024 : Day 337 #
Last night I kicked off the build to test the final arrangement of commits and patches that I hoped to convert into a pull request to the gecko-dev repository today. With the patches now stored in the gecko-dev rpm directory it means that the submodule mirror of upstream's gecko-dev source tree can be pulled in entirely unchanged and without any sailfish-specific commits. That's the whole point of the way the repository is structured: we pull in upstream's code without modification, apply our sailfish-specific patches to it and build the result.
This morning the build had finished and finished with a positive result. No glitches in the build process. But that doesn't mean it's built something that works.
But after transferring it over to my phone, installing the packages and running a few tests, I'm convinced that it's just the same as the previous version. So I conclude that I did successfully manage to reorganised the commits without affecting the final result.
At this point I thought I'd be ready to create the final pull request. But as I look through the patches I notice a few things that still need fixing. The first is to do with redundant commits. A couple of patches disable and then re-enable a flag, so they can be removed. There's also a commit that drops build optimisations to O1. I'd like to test it without this change so I've reverted this patch and will drop it entirely if it turns out not to be needed.
I also notice there are a few patches which aren't entirely required to get the browser working, but which do nevertheless provide important background functionality.
In fact, it was the discussion on the forum, particularly between Attah (attah) and Raine (rainemak) that alerted me to the fact that there are patches that haven't been applied but are needed, specifically related to user agent handling:
There are six patches like this that spring out as being likely needed, but which as yet aren't included in the build. Five of them apply without incident:
Unfortunately I wasn't able to get the changes to work correctly with my build engine. I suspect it's simply a case of me needing to reinstall the build engine to get a more recent version, but as this is quite a major task, it'll have to wait for now. I'm not as familiar as Matti is with the Rust toolchain in the Sailfish SDK, so I'm going to have to go back and discuss the details with him.
Finally, and this is really important, I received an email from Frajo concerning the libhybris changes. This also relates to the discussion on GitHub between Affe Null (affenull2345), Mal (mlehtima) and Frajo (krnlyng) in relation to getting the build running on Affe's Samsung Galaxy A40 port.
One of the outcomes of that discussion was to expose a further potential issue in libhybris. Once fixed Affe reported that the cover crash described in issue 1051 was no longer occurring. This cover crash has been a thorn in my side for several months now, so I'm really keen to know whether it does actually fix it or not.
Frajo provided me with some new libhybris rpm packages to test out that include these latest changes. Having installed them and tested the browser I find that... yes! The cover change no longer causes the browser to crash. This is good for having the cover working with the browser, but even more important is the fact it also resolves the issue with the email app and other WebView apps crashing when moving in and out of the app.
So I'm super-pleased with all this. I've sadly not had time to get involved in the topic in the forum over the last few days, but it's incredibly exciting to see all of the discussion and am thrilled at the effort people are putting in, not just into finding bugs, but also into resolving them.
Sadly I have to stop for the day today, but since I've made changes to the commits, especially to the optimisation of the build, I need to run a new build overnight from scratch before I can create my pull request. It'll add an extra day, but it's important to check everything is working correctly both at build and run-time after this change. Just one of the many reasons this has been such a slow process! Assuming the build passes, tomorrow morning I'll do a bit more testing of the changes and if all goes well I'll be able to create that pull request.
Tomorrow will be the last diary entry before I take a week off to attend RSECon24. I'm hoping I'll get the pull requests submitted by then, but either way there will still be more diary to write once I'm back in a week's time. So much to do!
If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
This morning the build had finished and finished with a positive result. No glitches in the build process. But that doesn't mean it's built something that works.
But after transferring it over to my phone, installing the packages and running a few tests, I'm convinced that it's just the same as the previous version. So I conclude that I did successfully manage to reorganised the commits without affecting the final result.
At this point I thought I'd be ready to create the final pull request. But as I look through the patches I notice a few things that still need fixing. The first is to do with redundant commits. A couple of patches disable and then re-enable a flag, so they can be removed. There's also a commit that drops build optimisations to O1. I'd like to test it without this change so I've reverted this patch and will drop it entirely if it turns out not to be needed.
I also notice there are a few patches which aren't entirely required to get the browser working, but which do nevertheless provide important background functionality.
In fact, it was the discussion on the forum, particularly between Attah (attah) and Raine (rainemak) that alerted me to the fact that there are patches that haven't been applied but are needed, specifically related to user agent handling:
attah: We also found that some of the UA handling looks to be missing a patch or two - so with those in, this may no longer be an issue.
[...]
rainemak: After we get the user-agent patch in place that should help in texture allocations (I think) and surely engine should start serving more mobile friendly pages.
[...]
rainemak: After we get the user-agent patch in place that should help in texture allocations (I think) and surely engine should start serving more mobile friendly pages.
There are six patches like this that spring out as being likely needed, but which as yet aren't included in the build. Five of them apply without incident:
$ git am --3way \
../../../gecko-dev-project/gecko-dev/rpm/0059-sailfishos-gecko-Delete-startupCache-if-it-s-stale.patch
$ git am --3way \
../../../gecko-dev-project/gecko-dev/rpm/0067-sailfishos-gecko-Hardcode-loopback-address-for-profi.patch
$ git am --3way \
../../../gecko-dev-project/gecko-dev/rpm/0068-sailfishos-gecko-Start-using-user-agent-builder.-JB-.patch
$ git am --3way \
../../../gecko-dev-project/gecko-dev/rpm/0083-sailfishos-gecko-dev-Disallow-page-zooming-if-the-me.patch
$ git am --3way \
../../../gecko-dev-project/gecko-dev/rpm/0086-sailfishos-gecko-Add-preference-to-bypass-CORS-on-ns.patch
The final patch is messy: when I try to apply it, there are conflicts all the way down:
$ git am --3way \
../../../gecko-dev-project/gecko-dev/rpm/0061-sailfishos-locale-Get-12-24h-timeformat-setting-from.patch
In spite of this it turns out to be pretty simple to apply the changes manually, which I've now done. Since I did it as part of the git am process the correct description and attribution have already been automatically applied. So finally I can generate new patches from these commits and add them to the build configuration:
$ git format-patch \--start-number 79 -o ../rpm -N --no-signature --zero-commit bd026681f74dI also received a really helpful message from Matti Viljanen (direc85) giving suggestions for improving the build configuration to force the Rust portions to build with just a single process. The importance of this is that running Rust with multiple processes results in sporadic hangs during the build. That's a real pain because after leaving the build running overnight you don't want to discover in the morning it got stuck after the first hour and didn't make progress beyond that. If the Rust parts can be locked to a single process, it would allow the non-Rust parts (primarily C++ code, which still makes up the majority of the codebase) to run using multiple processes. That'd give us a nice speed-up.
Unfortunately I wasn't able to get the changes to work correctly with my build engine. I suspect it's simply a case of me needing to reinstall the build engine to get a more recent version, but as this is quite a major task, it'll have to wait for now. I'm not as familiar as Matti is with the Rust toolchain in the Sailfish SDK, so I'm going to have to go back and discuss the details with him.
Finally, and this is really important, I received an email from Frajo concerning the libhybris changes. This also relates to the discussion on GitHub between Affe Null (affenull2345), Mal (mlehtima) and Frajo (krnlyng) in relation to getting the build running on Affe's Samsung Galaxy A40 port.
One of the outcomes of that discussion was to expose a further potential issue in libhybris. Once fixed Affe reported that the cover crash described in issue 1051 was no longer occurring. This cover crash has been a thorn in my side for several months now, so I'm really keen to know whether it does actually fix it or not.
Frajo provided me with some new libhybris rpm packages to test out that include these latest changes. Having installed them and tested the browser I find that... yes! The cover change no longer causes the browser to crash. This is good for having the cover working with the browser, but even more important is the fact it also resolves the issue with the email app and other WebView apps crashing when moving in and out of the app.
So I'm super-pleased with all this. I've sadly not had time to get involved in the topic in the forum over the last few days, but it's incredibly exciting to see all of the discussion and am thrilled at the effort people are putting in, not just into finding bugs, but also into resolving them.
Sadly I have to stop for the day today, but since I've made changes to the commits, especially to the optimisation of the build, I need to run a new build overnight from scratch before I can create my pull request. It'll add an extra day, but it's important to check everything is working correctly both at build and run-time after this change. Just one of the many reasons this has been such a slow process! Assuming the build passes, tomorrow morning I'll do a bit more testing of the changes and if all goes well I'll be able to create that pull request.
Tomorrow will be the last diary entry before I take a week off to attend RSECon24. I'm hoping I'll get the pull requests submitted by then, but either way there will still be more diary to write once I'm back in a week's time. So much to do!
If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
31 Aug 2024 : Day 336 #
After spending the last few days arranging commits I'm happy to have two rather exciting tasks to perform today. The first is to convert all of the commits into patches. Once I've done that I'll be able to do a complete build from beginning to end, including the "prepare" step that applies the patches. If that goes through I can create a pull request into Jolla's repositories. I'm not expecting this to be a final release version, but I'm hoping it will provide a suitably stable base to apply any other changes on top of.
The second thing I have to do is test out the new libhybris packages that Frajo has kindly built and provided me with. He actually sent them to me a couple of days back but I've been so wrapped up with the commits that I didn't yet have a chance to test them.
The new packages came about following a discussion on GitHub started by Affe Null (affenull2345) and then contributed to by Mal (mlehtima)) and Frajo (krnlyng). Affe is a secret porter, in that they've developed their own private Samsung Galaxy A40 port. Pretty cool! But Affe also hit an issue when testing ESR 91, in that it triggered the same Wayland EGL dynamic library loading crash in spite of the workaround being present in qtmozembed.
Some brilliant investigation by the three of them led to the tweaking of libhybris to avoid the crash. But, even more exciting for me, is that if I'm reading the discussion correctly this may also have fixed the "cover" crash that's been troubling me for so long.
As far as I'm concerned this cover crash is currently the most serious unresolved issue with the ESR 91 build. Not having a cover is fine, but sadly it doesn't just affect the browser: it affects WebView apps as well. That means that apps — like for example the email app — that change their covers also seem to crash if they've previously been displaying a WebView. The workaround I added to qtmozembed won't work for these apps because it has to be executed when the app starts up. That works for the browser and "pure" WebView apps, but apps like the browser don't initialise the WebView until much later on in their execution. A proper fix is needed for these.
The symptoms that result for the email app is that it'll crash if you move to the homescreen and then try to move back in to the app after having read an email. This is horrible of course and it absolutely needs to be fixed.
Now all these things may not be related, but if they are, then it would be great to have them fixed. And maybe, just maybe, these new libhybris packages will do the trick.
But first things first: patches. Well nearly patches at any rate. It turns out that in order to generate the patches I first need to sort out the commits in the gecko-dev repository. That's different to the gecko-dev-mirror repository. It's the latter that I need to generate patches for and all of the commits there have been arranged. The gecko-dev repository contains the gecko-dev-mirror repository as a submodule. The patches are stored in the gecko-dev repository where they're applied to the submodule at the start of a build.
But the new commits I've made in the gecko-dev repository are still in a bit of a messy state. For example, every few commits I've created a separate commit updating the submodule. That was necessary for me to do a rebuild, but I don't really want to keep them all in the tree. My plan is to merge the submodule updates into a single commit which, eventually, will be entirely removed.
Here are all of the submodule update commits:
My first attempt to consolidate all of these submodule commits into a single commit involved first trying to shift them all so that they are all consecutive. I used a git rebase -i to do this. Unfortunately this approach failed. Why? Because while all of these commits update the submodule, some of them do other things as well.
To resolve this I've determined to take any commit that both updates the submodule and makes changes into two commits: one to update the submodule, the other to apply the other changes.
My process for doing this is as follows:
Now I'm ready to do the test build.
That's fine, but it means unfortunately I won't have time to test out Frajo's packages today. It'll be something to look forward to in the morning instead.
If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
The second thing I have to do is test out the new libhybris packages that Frajo has kindly built and provided me with. He actually sent them to me a couple of days back but I've been so wrapped up with the commits that I didn't yet have a chance to test them.
The new packages came about following a discussion on GitHub started by Affe Null (affenull2345) and then contributed to by Mal (mlehtima)) and Frajo (krnlyng). Affe is a secret porter, in that they've developed their own private Samsung Galaxy A40 port. Pretty cool! But Affe also hit an issue when testing ESR 91, in that it triggered the same Wayland EGL dynamic library loading crash in spite of the workaround being present in qtmozembed.
Some brilliant investigation by the three of them led to the tweaking of libhybris to avoid the crash. But, even more exciting for me, is that if I'm reading the discussion correctly this may also have fixed the "cover" crash that's been troubling me for so long.
As far as I'm concerned this cover crash is currently the most serious unresolved issue with the ESR 91 build. Not having a cover is fine, but sadly it doesn't just affect the browser: it affects WebView apps as well. That means that apps — like for example the email app — that change their covers also seem to crash if they've previously been displaying a WebView. The workaround I added to qtmozembed won't work for these apps because it has to be executed when the app starts up. That works for the browser and "pure" WebView apps, but apps like the browser don't initialise the WebView until much later on in their execution. A proper fix is needed for these.
The symptoms that result for the email app is that it'll crash if you move to the homescreen and then try to move back in to the app after having read an email. This is horrible of course and it absolutely needs to be fixed.
Now all these things may not be related, but if they are, then it would be great to have them fixed. And maybe, just maybe, these new libhybris packages will do the trick.
But first things first: patches. Well nearly patches at any rate. It turns out that in order to generate the patches I first need to sort out the commits in the gecko-dev repository. That's different to the gecko-dev-mirror repository. It's the latter that I need to generate patches for and all of the commits there have been arranged. The gecko-dev repository contains the gecko-dev-mirror repository as a submodule. The patches are stored in the gecko-dev repository where they're applied to the submodule at the start of a build.
But the new commits I've made in the gecko-dev repository are still in a bit of a messy state. For example, every few commits I've created a separate commit updating the submodule. That was necessary for me to do a rebuild, but I don't really want to keep them all in the tree. My plan is to merge the submodule updates into a single commit which, eventually, will be entirely removed.
Here are all of the submodule update commits:
$ git log --oneline HEAD...9cb2ba8396d2~ -- gecko-dev/ f270bad5788e (HEAD -> sailfishos-esr91) Update gecko-dev mirror 5a68997c8450 Update gecko-dev mirror 515537b61413 Update gecko-dev mirror 08f9959dc235 Update gecko-dev mirror 14a2700021e5 Update gecko-dev mirror 67927c0423df Update gecko-dev mirror 2fea2e212b35 Update gecko-dev mirror 09260531aceb Update gecko-dev mirror d30cd18ee613 Update gecko-dev mirror 5b9c330998db Restore GLScreenBuffer and TextureImageEGL 96d960d6a455 Update gecko-dev mirror f803db211dfc Update gecko-dev mirror 720b4f6574e7 Convert AddBoolVarCache variables to static prefs 638ae4117394 Update gecko-dev mirror f659dbc79934 Update gecko-dev mirror 2df29a103c6f (tag: sailfishos/91.9.1+git1) Update gecko-dev mirror 9d155ca9394f Update gecko-dev mirror 98ac90c53a12 Update gecko-dev mirror 6ecb2bad664b Update gecko-dev mirror 46fc26b5752c Update gecko-dev mirror 4d365de6c475 Update gecko-dev mirror 72638c7a9872 Update gecko-dev mirror 04cd443a0a5f Update gecko-dev submodule 75c281796b78 Update gecko-dev mirror e4cea9f962a0 Rename nsIdleService to nsUserIdleService d53c33b45ca6 Update submodule to incorporate recent changes 12cef49647fa Update gecko-dev mirror 1603f42a923d Add freetype2 include directory 91c0e1a7eeee Update gecko-dev submoduleThere are quite a few of them. This list makes it look like they're all consecutive, but they're not, it's just that the commits in between these have been skipped.
My first attempt to consolidate all of these submodule commits into a single commit involved first trying to shift them all so that they are all consecutive. I used a git rebase -i to do this. Unfortunately this approach failed. Why? Because while all of these commits update the submodule, some of them do other things as well.
To resolve this I've determined to take any commit that both updates the submodule and makes changes into two commits: one to update the submodule, the other to apply the other changes.
My process for doing this is as follows:
- Execute git rebase -i 9cb2ba8396d2~. The commit always stays the same because it's the last commit before I made changes.
- Set the commit that needs splitting to edit.
- Execute git reset HEAD~ to reset the changes.
- Add the changes that don't relate to the submodule using git add.
- Commit these changes using git commit with the same commit message as the commit being edited.
- Add the submodule change using git add gecko-dev.
- Commit these changes using git commit -m "Update gecko-dev mirror".
- Continue the rebase with git rebase --continue.
- At this point it'll complain about a conflict, so reset the submodule with git reset <commit> -- gecko-dev where <commit> is the hash of the conflicting commit.
- Finally git rebase --continue.
git rebase -i 9cb2ba8396d2~After squashing all of the submodule commits together apart from the first and last I'm left with this rather ominous looking list of commits:
$ git log --oneline HEAD...9cb2ba8396d2~~ ab9cf7698875 (HEAD -> sailfishos-esr91) Update gecko-dev mirror 8725b48c1cf1 Set MOZ_USE_NATIVE_POPUP_WINDOWS as an environment variable 54b7a119188e Update patches 209beaa1221c Restore GLScreenBuffer and TextureImageEGL f366c03302b6 Ensure the Session History is initialised. f573fce46416 Support LOAD_FLAGS_FROM_EXTERNAL flag 5797043901a7 Introduce window hidden flag and expose isForPrinting through it bb9c40a96b42 Convert AddBoolVarCache variables to static prefs 9ffa340e2d21 Default keyword.enabled to true f8f4df84b0a3 Add appinfo annotations dffb82220d0a [sailfishos][embedlite] Fix AsyncPanZoomController fling state... 2b4c467f0d42 Disable patch "Fix 32-bit builds" 2db5567ddbcd Fix 32-bit builds b693fa1db02c Use system python executable bc78ac45646d Add Qt5Widgets build dependency 54c54a463e86 Help ensure the window is active a1f85adebda5 Set correct submodule path 3ce0728ceb34 Set compositor preferences correctly 5179dd071280 Build with only a single process 7967765674f7 Fix specfile versions 08230a27decd Hide SetIsActive() aActiveId parameter cbf00e47117f Add missing Fission methods to nsIXULRuntime 3bc37a85f5b7 Add missing GetChromeOuterWindowID() implementation b51363671814 Update WebBrowserChrome to align with upstream 293dbb9e8810 Move active flag handling explicitly to BrowsingContext a19eb4d211de Have the main thread double-check APZ's consumable state e12784b8e2f5 Apply refactoring from upstream 7dfb178cf4dd Reintroduce NS_APP_DISTRIBUTION_SEARCH_DIR_LIST 8ec4aa2fd581 Use mMessageListeners.GetOrInsertNew() 6dbb1073118a Switch APZEnentResult::mStatus for GetStatus() d3f9a43a17a1 Move mShouldSendWebProgressEventsToParent activation 1a575c80c4a9 Remove NS_LITERAL_CSTRING usage 4f3891a7a746 Switch CSSRect to ZoomTarget 1807b74962b0 Rename nsBaseHashtable::Put to InsertOrUpdate 0caddae65dc2 Add aActionId parameter to SetIsActive 1479b47c142c Add user interaction parameter to GoForward and GoBack b42962a118c6 Remove AffectPrivateSessionLifetime call 207faaaf245c Get outerWindowID from docShell rather than DOMWindowUtils 9f9aae313e1a Switch use of MOZ_MUST_USE for [[nodiscard]] a23bd2c76713 Specify the BrowsingContextGroup f68fadb53c6d Make `IMEState::Enabled` an enum class a853d8864c7b Remove plugin support from layout 4bb3ecf92186 Avoid unnecessary array copies in NotifyLayerTransforms and... 6de2af56afe2 Support the GetFrameUniformity API in content processes 12069f25adc4 Add flag to CompositorOptions to allow SW-WR on a per widget basis 9d2582254e30 Use a message router for primary child process channels add5a1974d00 Switch GLScreenBuffer for SwapChain cf26a8ce1fbd Add _WITH_DELETE_ON_EVENT_TARGET macros to nsISupportsImpl 2b0b1ef5c57a Remove all AddBoolVarCache usage cdb168d9f5a3 Remove use of NS_LITERAL_STRING d8b7845b563c Expose nsIThread to nsClipboard 05431db1eb64 Remove ChromeOuterWindowID from MessageManager override b9dec65db14a Switch use of MOZ_MUST_USE for [[nodiscard]] 7f64138b89e5 Switch from using nsDataHashtable to nsTHashMap 1594ad5e797a Rename nsIdleService to nsUserIdleService fec0b0d4789c Reduce optimisation to O1 3460285ab906 Add freetype2 include directory 75cb7986205c Update gecko-dev submodule 696c3801cdbd Update EmbedLite IPDL syntax 0eacc60a27ee Switch from disabling marionette to disabling webdriver 9e3dc13725a9 Update spec for ESR 91.9.0 902d36af8868 Enable dialog element 1ae9e714a3b6 (tag: sailfishos/78.15.1+git35) Merge pull request #153 from... 880b89ac0f8d [sailfishos][gecko] Backport support for ffmpeg 5.0. JB#60117...It's ominous, but all of the changes individually actually look quite sane now. So I think I'm going to say this is where things should be. So now it's time to generate the patches from the commits in gecko-dev-mirror. First off I need to actually generate all of the patches. I'll keep a backup of the ESR 78 patches as well, but I don't plan to commit them to the repository. During the last upgrade they were kept, so maybe I'll need to commit them too in due course. Nevertheless, generating the patches turns out to be pretty straightforward:
$ cd rpm $ mkdir esr78-patches $ mv *.patch esr78-patches/ $ cd ../gecko-dev/ $ git format-patch -o ../rpm -N --no-signature --zero-commit 78d17b06b04f $ git rebase -i 9cb2ba8396d2~The next step is to copy all of the patch filenames into the spec file so that the rpm builder knows what to apply. Then I can create a commit that contains all of the new patches:
$ cd ../rpm $ git add *.patch $ git commit $ cd ../gecko-dev $ git checkout FIREFOX_ESR_91_9_X_RELBRANCH $ cd .. $ git commit -m "Update gecko-dev mirror" $ git rebase -i 9cb2ba8396d2~ $ git clean -xdfNotice that I've also created a commit that resets the submodule to the head of FIREFOX_ESR_91_9_X_RELBRANCH. I've committed this change to the repository as well. The patches should bring FIREFOX_ESR_91_9_X_RELBRANCH all the way up to the same commit as the HEAD of FIREFOX_ESR_91_9_X_RELBRANCH_sfos.
Now I'm ready to do the test build.
sfdk build -d -p --with git_workaroundRather excitingly, all of the patches apply one on top of the other without any glitches.
[...]
+ unset DISPLAY
+ /bin/cat ${PROJECT}/rpm/
0001-sailfishos-gecko-Add-symlink-to-embedlite.-JB-52893.patch
+ patch -p1 -s --fuzz=2 --no-backup-if-mismatch
[546bceac6ff5] *sfdk-prepare* Add symlink to embedlite. JB#52893
1 file changed, 1 insertion(+)
+ /bin/cat ${PROJECT}/rpm/0002-sailfishos-qt-Bring-back-Qt-layer.-JB-50505.patch
+ patch -p1 -s --fuzz=2 --no-backup-if-mismatch
[4aa4f2c0af0c] *sfdk-prepare* Bring back Qt layer. JB#50505
109 files changed, 4780 insertions(+), 89 deletions(-)
[...]
+ /bin/cat ${PROJECT}/rpm/
0078-sailfishos-embedlite-egl-Fix-mesa-egl-display-and-bu.patch
+ patch -p1 -s --fuzz=2 --no-backup-if-mismatch
[70128c619c7e] *sfdk-prepare* Fix mesa egl display and buffer initialisation
1 file changed, 228 insertions(+), 7 deletions(-)
+ echo 'Target is aarch64-unknown-linux-gnu'
Target is aarch64-unknown-linux-gnu
[...]
Originally I'd planned to sort out the patches this morning, run the build during the day and then test out Frajo's new libhybris packages this evening. But unfortunately I didn't quite get everything done this morning, which means I couldn't run the build during the day, which means that the build will now have to run overnight tonight instead.
That's fine, but it means unfortunately I won't have time to test out Frajo's packages today. It'll be something to look forward to in the morning instead.
If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
30 Aug 2024 : Day 335 #
I'm back to corralling commits again this morning. Yesterday I got as far as commit 86 in the list. That means there are still 40 to go and I'd like to get them done before the end of the weekend.
The process is the same as yesterday. First, list all of the relevant commits:
And that's pretty much it! The good news is that following this cycle I've managed to complete the remaining 41 commits. Here's what I started with (I've left the last commit in from yesterday so it's clear how they hook together):
Before I turn these into patches there's a consistency check I want to do. As I mentioned a couple of days back, this process of rewording, re attributing, reordering and restructuring the commits should have left the final source tree in exactly the same state as before. Nothing should have changed apart from the git history.
But there's plenty scope for something to have gone wrong. For example deleting a line of the interactive rebase list can result in an entire commit being lost. If I did that accidentally, it'll affect the final result. So I need to figure out a way to perform a diff on the entire codebase: what I had previously vs. what I have now.
I don't really want to do a diff though, just a hash of the files should be sufficient. Here's what I've devised for this:
My plan is to run this on the current FIREFOX_ESR_91_9_X_RELBRANCH_sfos branch with all of my newly minted commits. I'll record the result and then switch to the FIREFOX_ESR_91_9_X_RELBRANCH_patches branch before running the command again. When this outputs a checksum I'll be able to compare the two in order to determine whether or not anything has changed. I've not cleaned the repository and I don't plan to, because although the checksum will pick up all of the untracked files as well, none of these should change when I switch branch, so these shouldn't affect whether or not the final results differ.
If even a single bit has changed between the two versions, it'll show up in the checksum. It'll be a miracle if both give the same result. Let's try it.
That's not a huge task, but it'll have to wait until tomorrow; I've done enough for today.
The comments and issues from the testing have come flooding in on the Sailfish OS Forum and that's great. What's more there's even been effort started to fix some of the remaining issues. This is really where I was hoping things would go and it bodes well for the future, so I'm very excited to see how everything progresses. I've not had a chance to join the discussion as I've just been concentrating on these patches for the last few days.
If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
The process is the same as yesterday. First, list all of the relevant commits:
$ git log --oneline HEAD...78d17b06b04f~Next, work through the logs and diffs a few commits at a time to figure out what needs doing to them. Do they need rewording? Do they need the attribution fixing? Do they need merging with some other commit? Once that's determined and I have a few lined up in my head, I can then perform an interactive rebase to perform all of these changes:
git rebase -i 78d17b06b04fThe commit 78d17b06b04f is set to the HEAD of ESR 91 before I started making changes, so this commit is always the base of everything I need to do. During the rebase I select the commits to edit, reword or squash and then manually perform appropriate actions to get everything into the correct state. For example, to set the attribution I might use this:
git commit --amend --author="<AUTHOR>" --date="<DATE>"Most of the commits need to be amended so that their titles include the topic prefix [sailfishos][gecko] as typically I left these out during development.
And that's pretty much it! The good news is that following this cycle I've managed to complete the remaining 41 commits. Here's what I started with (I've left the last commit in from yesterday so it's clear how they hook together):
5fdfd3eb0500 (HEAD) Set gfx.webrender.force-disabled to default to true 3c72121dd859 [sailfishos][embedlite][egl] Fix mesa egl display and buffer... 16f8accb9c47 Fix `unstable-name-collisions` warning by using fully qualified... 3aa670a3cc6a Fix build failure due to rust-lang/rust PR 99413 60d354ad440c Properly fill out videoframe buffer in GeckoCameraVideoDecoder edb9deacd049 Create GLLibraryEGL as a singleton in GLContextProviderEGL 5f814f20a01f Combined with patdh 0089 - Add a video decoder based on... 33133f24782c Ensure audio continues when screen is locked. Contributes to... 631eb9c9daaf Bug 1758948 [FFmpeg] Use AVFrame::pts instead of... 61617321d2dd Bug 1761471 [FFmpeg 5.0] Get frame color range and color space... ac7f5ffbb2fa Bug 1750760 Update audio and video decoders to ffmpeg 5.0... f5d4bf0de62e Bug 1750760 Open libavcodec.so.59 library and bind ffmpeg 5.0... 1e3291df5048 Bug 1750760 Create ffmpeg59 module for ffmpeg5.0 r=alwu 6d52cc7e9c32 Fix audio underruns for fullduplex mode. JB#55461 874b633f01f1 Add a video decoder based on gecko-camera. JB#56755 dbdf4a057c57 Fix video hardware accelaration not being used on first... d7912ed1ceec Force use of mobile video controls. JB#55484 OMP#JOLLA-371 c4e35d97756e Force recycling of gmp-droid instances. JB#51730 e72aff45bbf1 Prioritize GMP plugins over all others, and support decoding... f2f7a716ae1b Set embedlite.azpc.json.doubletap static pref to default to... 9859c4cc05f4 Make fullscreen enabling work as used to with pref ... b2e11c4108c0 Set security.external_protocol_requires_permission to default... 379f31eef47a Fix content action integration to work. Fixes JB#51235 95da0e408f53 Use libcontentaction for custom scheme uri handling. JB#47892 a9bc82c90f8e Update hash for mapped_hyph fad2d997cea7 [sailfishos][gecko] Add support for prefers-color-scheme 7caca291fbac [sailfishos][gecko] Allow LoginManagerPrompter to find its... b67d3cf6a2d3 Convert panic into early return in Hyphenator 891bd6933db7 Get ContentFrameMessageManager via nsIDocShellTreeOwner... 941e612af842 Drop AudioPlayback messages if no embedder element is defined 36cb2567f3c2 [sailfishos][webrtc] Implement video capture module. JB#53982 ffc93a057267 [sailfishos][webrtc] Implement video capture module. JB#53982 2e65e420780a Enable GMP for encoding/decoding. JB#53982 9a481753fd3b Regenerate moz.build files. c1b06c57549c Switch vp8_encoder_simulcast_proxy for encoder_simulcast_proxy 534ffad75341 Disable desktop sharing feature on SFOS. JB#53756 7bda060a012f [sailfishos][webrtc] Regenerate moz.build files. JB#53756 71ea0764f5ef [sailfishos][webrtc] Update GN build files for WebRTC. JB#53756 7ea10b57b465 Adapt build configuration for SailfishOS. JB#53756 1765dd0460a9 Restore GLScreenBuffer and TextureImageEGL d3ba4df29a32 (temp) Restore NotifyDidPaint event and timers f55057391ac0 Prevent errors from DownloadPrompterThese have reduced down by 10 to make 31 new commits to replace them. The new commits look like this:
24f9ec2df3d3 (HEAD) [sailfishos][embedlite][egl] Fix mesa egl display and... 7e5553ee4b22 [sailfishos][gecko] Fix `unstable-name-collisions` warning by... b08a73b0d2f1 [sailfishos][gecko] Fix build failure due to rust-lang/rust PR... 3705ec69aafa [sailfishos][gecko] Ensure audio continues when screen is... 5cc32e715fac [sailfishos][gecko] Bug 1758948 [FFmpeg] Use AVFrame::pts... 370917e54b8e [sailfishos][gecko] Bug 1761471 [FFmpeg 5.0] Get frame color... 56e2ee73233f [sailfishos][gecko] Bug 1750760 Update audio and video decoders... d4a651ccd7be [sailfishos][gecko] Bug 1750760 Open libavcodec.so.59 library... 65f0141c466a [sailfishos][gecko] Bug 1750760 Create ffmpeg59 module for... b4c938ac9240 [sailfishos][gecko] Fix audio underruns for fullduplex mode... 37bd01779d83 [sailfishos][gecko] Add a video decoder based on gecko-camera... 4c4f0626452d [sailfishos][gecko] Fix video hardware accelaration not being... 570ef20869fd [sailfishos][gecko] Force use of mobile video controls... 7a46a215d649 [sailfishos][gecko] Force recycling of gmp-droid instances... a5faa0c8aabe [sailfishos][gecko] Prioritize GMP plugins over all others, and... 22f531b9a58b [sailfishos][gecko] Make fullscreen enabling work as used to... 974010cd0fda [sailfishos][gecko] Fix content action integration to work... ea2a8a393a1f [sailfishos][gecko] Update hash for mapped_hyph a62d17ace5cb [sailfishos][gecko] Add support for prefers-color-scheme... 83477c046211 [sailfishos][gecko] Allow LoginManagerPrompter to find its... e7209e460314 [sailfishos][gecko] Convert panic into early return in Hyphenator d46228ac0fc2 [sailfishos][gecko] Get ContentFrameMessageManager via... cd66aada3508 [sailfishos][gecko] Drop AudioPlayback messages if no embedder... 18a71b573840 [sailfishos][webrtc] Regenerate moz.build files. JB#53756 4e7332543828 [sailfishos][webrtc] Implement video capture module. JB#53982 1a290ea1db22 [sailfishos][gecko] Enable GMP for encoding/decoding. JB#53982 f10b5c2cf0cf [sailfishos][gecko] Disable desktop sharing feature on SFOS... c961f8a04412 [sailfishos][webrtc] Update GN build files for WebRTC. JB#53756 d69d8be0fea4 [sailfishos][gecko] Adapt build configuration for SailfishOS... 36d30b7f7566 [sailfishos][gecko] Restore NotifyDidPaint event and timers b1406101475c [sailfishos][gecko] Prevent errors from DownloadPrompterIn addition to these I was also able to squash together all of the GLScreenBuffer changes into just a couple of commits. The end result is a total of 78 commits, down from the original 126:
$ git log --oneline HEAD...78d17b06b04f | wc -l 78That's pretty good going. That means there will be 78 patches in the rpm directory. This won't be all of them as I've intentionally skipped some patches that may yet need to be added. But this brings things to a pretty solid state and there's a reasonable chance that the total patch count at the end of all this will be less than 100.
Before I turn these into patches there's a consistency check I want to do. As I mentioned a couple of days back, this process of rewording, re attributing, reordering and restructuring the commits should have left the final source tree in exactly the same state as before. Nothing should have changed apart from the git history.
But there's plenty scope for something to have gone wrong. For example deleting a line of the interactive rebase list can result in an entire commit being lost. If I did that accidentally, it'll affect the final result. So I need to figure out a way to perform a diff on the entire codebase: what I had previously vs. what I have now.
I don't really want to do a diff though, just a hash of the files should be sufficient. Here's what I've devised for this:
$ find . -type f -exec sha256sum {} \; | sha256sum -
Short but sweet. This will cycle through all files in the directory and any subdirectories and take the SHA256 checksum of the file. The list of these checksums is then piped into the same SHA256 algorithm to give a final checksum that represents all files and filenames. It's like a super-simple single-layer Merkle tree.My plan is to run this on the current FIREFOX_ESR_91_9_X_RELBRANCH_sfos branch with all of my newly minted commits. I'll record the result and then switch to the FIREFOX_ESR_91_9_X_RELBRANCH_patches branch before running the command again. When this outputs a checksum I'll be able to compare the two in order to determine whether or not anything has changed. I've not cleaned the repository and I don't plan to, because although the checksum will pick up all of the untracked files as well, none of these should change when I switch branch, so these shouldn't affect whether or not the final results differ.
If even a single bit has changed between the two versions, it'll show up in the checksum. It'll be a miracle if both give the same result. Let's try it.
$ git checkout FIREFOX_ESR_91_9_X_RELBRANCH_sfos
Already on 'FIREFOX_ESR_91_9_X_RELBRANCH_sfos'
$ find . -type f -exec sha256sum {} \; | sha256sum -
2f8460ce00ef4304058421a444d1d5a08f58cc2f006cf009ee3fa6f7e5cc50b5 -
$ git checkout FIREFOX_ESR_91_9_X_RELBRANCH_patches
Switched to branch 'FIREFOX_ESR_91_9_X_RELBRANCH_patches'
Your branch is up-to-date with 'origin/FIREFOX_ESR_91_9_X_RELBRANCH_patches'.
$ find . -type f -exec sha256sum {} \; | sha256sum -
2f8460ce00ef4304058421a444d1d5a08f58cc2f006cf009ee3fa6f7e5cc50b5 -
And they're identical! Superb. That puts things in a good place. The next step will be to turn the 78 new commits into patches using git format-patch and commit the result to the gecko-dev repository in order to create a pull request.That's not a huge task, but it'll have to wait until tomorrow; I've done enough for today.
The comments and issues from the testing have come flooding in on the Sailfish OS Forum and that's great. What's more there's even been effort started to fix some of the remaining issues. This is really where I was hoping things would go and it bodes well for the future, so I'm very excited to see how everything progresses. I've not had a chance to join the discussion as I've just been concentrating on these patches for the last few days.
If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.