flypig.co.uk

Thinking overnight about the problem I was hitting up against yesterday, I've hatched some kind of plan. I spent yesterday circulating around EmbedLitePuppetWidget, which is the top level element but which, on ESR 91 at least, has no layer manager defined for it. From what I can tell, this isn't a situation which should ever arise. The top level element should create a layer manager if it's asked for one and doesn't otherwise already have one.

So today I want to find out where the EmbedLitePuppetWidget is being created. My plan is then to place a breakpoint on the mLayerManager container variable to see where it gets set on ESR 78. Once I have a backtrace for that, I should be in a better position to figure out why the same thing isn't happening on ESR 91.

First task for the day then: find out where the EmbedLitePuppetWidget is being created. And whether there's more than one of them!

As we progress, if you're reading this, I need to warn you that the entry today is going to be full of very long, impenetrable and not-very-interesting backtraces. For this task it's going to be really important for me to keep track of these, and while they make for terrible reading, they also make for crucial reference material. This diary is both reference material and reading material, and today it's going to be far more of the former than the latter. So apologies in advance.

My advice: skip past the backtraces. Thankfully, as a digital diary, both the cost of keeping the backtraces in and the effort of skipping past them is low.

Here's the breakpoint, with backtrace, of the first case of an EmbedLitePuppetWidget being constructed on ESR 78:

Thread 10 "GeckoWorkerThre" hit Breakpoint 1, mozilla::embedlite::
    EmbedLitePuppetWidget::EmbedLitePuppetWidget (this=0x7f80c09a90, 
    view=0x7f80be5eb8)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:47
47      EmbedLitePuppetWidget::EmbedLitePuppetWidget(EmbedLiteViewChildIface* 
    view)
(gdb) bt 4
#0  mozilla::embedlite::EmbedLitePuppetWidget::EmbedLitePuppetWidget (
    this=0x7f80c09a90, view=0x7f80be5eb8)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:47
#1  0x0000007fbca92f5c in mozilla::embedlite::EmbedLiteViewChild::
    InitGeckoWindow (this=0x7f80be5e80, parentId=0, parentBrowsingContext=0x0, 
    isPrivateWindow=<optimized out>, isDesktopMode=false)
    at obj-build-mer-qt-xr/dist/include/mozilla/cxxalloc.h:33
#2  0x0000007fbca84b84 in mozilla::detail::RunnableMethodArguments<unsigned int 
    const, mozilla::dom::BrowsingContext*, bool const, bool const>::
    applyImpl<mozilla::embedlite::EmbedLiteViewChild, void (mozilla::embedlite::
    EmbedLiteViewChild::*)(unsigned int, mozilla::dom::BrowsingContext*, bool, 
    bool), StoreCopyPassByConstLRef<unsigned int const>, 
    StoreRefPtrPassByPtr<mozilla::dom::BrowsingContext>, 
    StoreCopyPassByConstLRef<bool const>, StoreCopyPassByConstLRef<bool const>, 
    0ul, 1ul, 2ul, 3ul> (args=..., m=<optimized out>, o=<optimized out>)
    at xpcom/threads/nsThreadUtils.h:990
#3  mozilla::detail::RunnableMethodArguments<unsigned int const, mozilla::dom::
    BrowsingContext*, bool const, bool const>::apply<mozilla::embedlite::
    EmbedLiteViewChild, void (mozilla::embedlite::EmbedLiteViewChild::*)(
    unsigned int, mozilla::dom::BrowsingContext*, bool, bool)> (m=<optimized 
    out>, 
    o=<optimized out>, this=<optimized out>) at xpcom/threads/nsThreadUtils.h:
    1191
(More stack frames follow...)

It turns out this isn't the only instance though, there are two others as well. Here's the second being created:

Thread 10 "GeckoWorkerThre" hit Breakpoint 1, mozilla::embedlite::
    EmbedLitePuppetWidget::EmbedLitePuppetWidget (this=0x7f80c02fd0, view=0x0)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:47
47      EmbedLitePuppetWidget::EmbedLitePuppetWidget(EmbedLiteViewChildIface* 
    view)
(gdb) bt 23
#0  mozilla::embedlite::EmbedLitePuppetWidget::EmbedLitePuppetWidget (
    this=0x7f80c02fd0, view=0x0)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:47
#1  0x0000007fbca92664 in mozilla::embedlite::EmbedLitePuppetWidget::
    CreateChild (aForceUseIWidgetParent=<optimized out>, 
    aInitData=0x7fa69cfde0, 
    aRect=..., this=0x7f80c09a90) at obj-build-mer-qt-xr/dist/include/mozilla/
    cxxalloc.h:33
#2  mozilla::embedlite::EmbedLitePuppetWidget::CreateChild (this=0x7f80c09a90, 
    aRect=..., aInitData=0x7fa69cfde0, aForceUseIWidgetParent=<optimized out>)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:69
#3  0x0000007fbbdb03e8 in nsView::CreateWidgetForParent (this=0x7f8045dd00, 
    aParentWidget=0x7f80c09a90, aWidgetInitData=0x7fa69cfde0, 
    aWidgetInitData@entry=0x0, aEnableDragDrop=true, 
    aResetVisibility=aResetVisibility@entry=false)
    at view/nsView.cpp:574
#4  0x0000007fbbf50d1c in nsDocumentViewer::MakeWindow (
    this=this@entry=0x7f8045da10, aSize=..., 
    aContainerView=aContainerView@entry=0x0)
    at layout/base/nsDocumentViewer.cpp:2353
#5  0x0000007fbbf92438 in nsDocumentViewer::InitInternal (this=0x7f8045da10, 
    aParentWidget=<optimized out>, aState=0x0, aActor=0x0, aBounds=..., 
    aDoCreation=<optimized out>, aNeedMakeCX=<optimized out>, 
    aForceSetNewDocument=<optimized out>)
    at obj-build-mer-qt-xr/dist/include/mozilla/gfx/BaseSize.h:34
#6  0x0000007fbc7c4d38 in nsDocShell::SetupNewViewer (
    this=this@entry=0x7f80c0b160, aNewViewer=aNewViewer@entry=0x7f8045da10, 
    aWindowActor=aWindowActor@entry=0x0) at obj-build-mer-qt-xr/dist/include/
    nsCOMPtr.h:847
#7  0x0000007fbc7ce75c in nsDocShell::Embed (this=this@entry=0x7f80c0b160, 
    aContentViewer=0x7f8045da10, aWindowActor=aWindowActor@entry=0x0)
    at docshell/base/nsDocShell.cpp:5441
#8  0x0000007fbc7cec44 in nsDocShell::CreateAboutBlankContentViewer (
    this=this@entry=0x7f80c0b160, aPrincipal=aPrincipal@entry=0x0, 
    aStoragePrincipal=aStoragePrincipal@entry=0x0, aCSP=<optimized out>, 
    aBaseURI=0x0, aTryToSaveOldPresentation=<optimized out>, 
    aTryToSaveOldPresentation@entry=true, 
    aCheckPermitUnload=aCheckPermitUnload@entry=true, aActor=aActor@entry=0x0)
    at obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:847
#9  0x0000007fbc7cf070 in nsDocShell::EnsureContentViewer (
    this=this@entry=0x7f80c0b160)
    at obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:847
#10 0x0000007fbc7cff94 in nsDocShell::EnsureContentViewer (this=0x7f80c0b160)
    at docshell/base/nsDocShell.cpp:6264
#11 nsDocShell::GetDocument (this=0x7f80c0b160) at docshell/base/nsDocShell.cpp:
    3041
#12 0x0000007fba95dcf8 in nsPIDOMWindowOuter::MaybeCreateDoc (this=<optimized 
    out>)
    at dom/base/nsGlobalWindowOuter.cpp:7594
#13 0x0000007fba95e17c in non-virtual thunk to nsGlobalWindowOuter::WrapObject(
    JSContext*, JS::Handle<JSObject*>) ()
    at obj-build-mer-qt-xr/dist/include/js/HeapAPI.h:677
#14 0x0000007fba325c60 in XPCConvert::NativeInterface2JSObject (
    cx=cx@entry=0x7f80225e50, d=d@entry=..., aHelper=..., 
    iid=iid@entry=0x7fa69d0710, 
    allowNativeWrapper=allowNativeWrapper@entry=true, pErr=pErr@entry=0x0)
    at obj-build-mer-qt-xr/dist/include/js/RootingAPI.h:596
#15 0x0000007fba326684 in XPCConvert::NativeData2JS (cx=cx@entry=0x7f80225e50, 
    d=d@entry=..., s=s@entry=0x7fa69d0858, type=..., 
    iid=iid@entry=0x7fa69d0710, arrlen=<optimized out>, pErr=pErr@entry=0x0)
    at js/xpconnect/src/XPCConvert.cpp:351
#16 0x0000007fba342b48 in nsXPCWrappedJS::CallMethod (this=<optimized out>, 
    methodIndex=<optimized out>, info=0x7fbdfc0f68 <xpt::detail::
    sMethods+8800>, 
    nativeParams=0x7fa69d0858) at obj-build-mer-qt-xr/dist/include/js/
    RootingAPI.h:1279
#17 0x0000007fb9c13da4 in PrepareAndDispatch (self=0x7f8042ebc0, 
    methodIndex=<optimized out>, args=<optimized out>, gprData=0x7fa69d0920, 
    fprData=0x7fa69d08e0) at xpcom/reflect/xptcall/md/unix/
    xptcstubs_aarch64.cpp:183
#18 0x0000007fb9c140f4 in SharedStub ()
    at xpcom/reflect/xptcall/md/unix/xptcstubs_asm_aarch64.s:38
#19 0x0000007fb9ba4bc0 in nsObserverList::NotifyObservers (this=<optimized 
    out>, aSubject=aSubject@entry=0x7f80bcbf70, 
    aTopic=aTopic@entry=0x7fbe6a3008 &quot;embedliteviewcreated&quot;, 
    someData=someData@entry=0x0)
    at xpcom/ds/nsTArray.h:1182
#20 0x0000007fb9ba7ab4 in nsObserverService::NotifyObservers (
    this=0x7f800470e0, aSubject=0x7f80bcbf70, aTopic=0x7fbe6a3008 
    &quot;embedliteviewcreated&quot;, 
    aSomeData=0x0) at xpcom/ds/nsObserverService.cpp:288
#21 0x0000007fbca93460 in mozilla::embedlite::EmbedLiteViewChild::
    InitGeckoWindow (this=0x7f80be5e80, parentId=<optimized out>, 
    parentBrowsingContext=<optimized out>, isPrivateWindow=<optimized out>, 
    isDesktopMode=false)
    at obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:847
#22 0x0000007fbca84b84 in mozilla::detail::RunnableMethodArguments<unsigned int 
    const, mozilla::dom::BrowsingContext*, bool const, bool const>::
    applyImpl<mozilla::embedlite::EmbedLiteViewChild, void (mozilla::embedlite::
    EmbedLiteViewChild::*)(unsigned int, mozilla::dom::BrowsingContext*, bool, 
    bool), StoreCopyPassByConstLRef<unsigned int const>, 
    StoreRefPtrPassByPtr<mozilla::dom::BrowsingContext>, 
    StoreCopyPassByConstLRef<bool const>, StoreCopyPassByConstLRef<bool const>, 
    0ul, 1ul, 2ul, 3ul> (args=..., m=<optimized out>, o=<optimized out>)
    at xpcom/threads/nsThreadUtils.h:990
(More stack frames follow...)

And here's the third:

Thread 10 "GeckoWorkerThre" hit Breakpoint 1, mozilla::embedlite::
    EmbedLitePuppetWidget::EmbedLitePuppetWidget (this=0x7f81005400, view=0x0)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:47
47      EmbedLitePuppetWidget::EmbedLitePuppetWidget(EmbedLiteViewChildIface* 
    view)
(gdb) bt 18
#0  mozilla::embedlite::EmbedLitePuppetWidget::EmbedLitePuppetWidget (
    this=0x7f81005400, view=0x0)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:47
#1  0x0000007fbca92664 in mozilla::embedlite::EmbedLitePuppetWidget::
    CreateChild (aForceUseIWidgetParent=<optimized out>, 
    aInitData=0x7fa69d02a0, 
    aRect=..., this=0x7f80c09a90) at obj-build-mer-qt-xr/dist/include/mozilla/
    cxxalloc.h:33
#2  mozilla::embedlite::EmbedLitePuppetWidget::CreateChild (this=0x7f80c09a90, 
    aRect=..., aInitData=0x7fa69d02a0, aForceUseIWidgetParent=<optimized out>)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:69
#3  0x0000007fbbdb03e8 in nsView::CreateWidgetForParent (this=0x7f80ec1750, 
    aParentWidget=0x7f80c09a90, aWidgetInitData=0x7fa69d02a0, 
    aWidgetInitData@entry=0x0, aEnableDragDrop=true, 
    aResetVisibility=aResetVisibility@entry=false)
    at view/nsView.cpp:574
#4  0x0000007fbbf50d1c in nsDocumentViewer::MakeWindow (
    this=this@entry=0x7f80ee9300, aSize=..., 
    aContainerView=aContainerView@entry=0x0)
    at layout/base/nsDocumentViewer.cpp:2353
#5  0x0000007fbbf92438 in nsDocumentViewer::InitInternal (this=0x7f80ee9300, 
    aParentWidget=<optimized out>, aState=0x0, aActor=0x0, aBounds=..., 
    aDoCreation=<optimized out>, aNeedMakeCX=<optimized out>, 
    aForceSetNewDocument=<optimized out>)
    at obj-build-mer-qt-xr/dist/include/mozilla/gfx/BaseSize.h:34
#6  0x0000007fbc7c4d38 in nsDocShell::SetupNewViewer (
    this=this@entry=0x7f80c0b160, aNewViewer=aNewViewer@entry=0x7f80ee9300, 
    aWindowActor=aWindowActor@entry=0x0) at obj-build-mer-qt-xr/dist/include/
    nsCOMPtr.h:847
#7  0x0000007fbc7ce75c in nsDocShell::Embed (this=this@entry=0x7f80c0b160, 
    aContentViewer=0x7f80ee9300, aWindowActor=aWindowActor@entry=0x0)
    at docshell/base/nsDocShell.cpp:5441
#8  0x0000007fbc7de358 in nsDocShell::CreateContentViewer (this=0x7f80c0b160, 
    aContentType=..., aRequest=0x7f80e162a0, aContentHandler=<optimized out>)
    at docshell/base/nsDocShell.cpp:7662
#9  0x0000007fbc7dee80 in nsDSURIContentListener::DoContent (
    this=this@entry=0x7ea4109b30, aContentType=..., 
    aIsContentPreferred=aIsContentPreferred@entry=false, 
    aRequest=aRequest@entry=0x7f80e162a0, aContentHandler=0x7f80724250, 
    aAbortProcess=aAbortProcess@entry=0x7fa69d08a0)
    at docshell/base/nsDSURIContentListener.cpp:178
#10 0x0000007fba4b0350 in nsDocumentOpenInfo::TryContentListener (
    this=0x7f80724230, aListener=0x7ea4109b30, aChannel=0x7f80e162a0)
    at obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:1351
#11 0x0000007fba4b07c8 in nsDocumentOpenInfo::DispatchContent (
    this=this@entry=0x7f80724230, request=request@entry=0x7f80e162a0, 
    aCtxt=aCtxt@entry=0x0)
    at obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:847
#12 0x0000007fba4b1398 in nsDocumentOpenInfo::OnStartRequest (
    this=0x7f80724230, request=0x7f80e162a0)
    at uriloader/base/nsURILoader.cpp:190
#13 0x0000007fb9f26b84 in mozilla::net::DocumentLoadListener::<lambda(const 
    mozilla::net::DocumentLoadListener::OnStartRequestParams&)>::operator() (
    __closure=<optimized out>, __closure=<optimized out>, aParams=...)
    at obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:847
#14 mozilla::detail::VariantImplementation<unsigned char, 0, mozilla::net::
    DocumentLoadListener::OnStartRequestParams, mozilla::net::
    DocumentLoadListener::OnDataAvailableParams, mozilla::net::
    DocumentLoadListener::OnStopRequestParams, mozilla::net::
    DocumentLoadListener::OnAfterLastPartParams>::matchN<mozilla::
    Variant<mozilla::net::DocumentLoadListener::OnStartRequestParams, mozilla::
    net::DocumentLoadListener::OnDataAvailableParams, mozilla::net::
    DocumentLoadListener::OnStopRequestParams, mozilla::net::
    DocumentLoadListener::OnAfterLastPartParams>, mozilla::net::
    DocumentLoadListener::ResumeSuspendedChannel(nsIStreamListener*)::<lambda(
    const mozilla::net::DocumentLoadListener::OnStartRequestParams&)>, mozilla::
    net::DocumentLoadListener::ResumeSuspendedChannel(nsIStreamListener*)::
    <lambda(const mozilla::net::DocumentLoadListener::OnDataAvailableParams&)>, 
    mozilla::net::DocumentLoadListener::ResumeSuspendedChannel(
    nsIStreamListener*)::<lambda(const mozilla::net::DocumentLoadListener::
    OnStopRequestParams&)>, mozilla::net::DocumentLoadListener::
    ResumeSuspendedChannel(nsIStreamListener*)::<lambda(const mozilla::net::
    DocumentLoadListener::OnAfterLastPartParams&)> > (aMi=..., aV=...)
    at obj-build-mer-qt-xr/dist/include/mozilla/Variant.h:280
#15 mozilla::Variant<mozilla::net::DocumentLoadListener::OnStartRequestParams, 
    mozilla::net::DocumentLoadListener::OnDataAvailableParams, mozilla::net::
    DocumentLoadListener::OnStopRequestParams, mozilla::net::
    DocumentLoadListener::OnAfterLastPartParams>::match<mozilla::net::
    DocumentLoadListener::ResumeSuspendedChannel(nsIStreamListener*)::<lambda(
    const mozilla::net::DocumentLoadListener::OnStartRequestParams&)>, mozilla::
    net::DocumentLoadListener::ResumeSuspendedChannel(nsIStreamListener*)::
    <lambda(const mozilla::net::DocumentLoadListener::OnDataAvailableParams&)>, 
    mozilla::net::DocumentLoadListener::ResumeSuspendedChannel(
    nsIStreamListener*)::<lambda(const mozilla::net::DocumentLoadListener::
    OnStopRequestParams&)>, mozilla::net::DocumentLoadListener::
    ResumeSuspendedChannel(nsIStreamListener*)::<lambda(const mozilla::net::
    DocumentLoadListener::OnAfterLastPartParams&)> > (aM1=..., aM0=..., 
    this=0x7f80f22de8)
    at obj-build-mer-qt-xr/dist/include/mozilla/Variant.h:811
#16 mozilla::net::DocumentLoadListener::ResumeSuspendedChannel (
    this=0x7f80e31960, aListener=0x7f80724230)
    at netwerk/ipc/DocumentLoadListener.cpp:979
#17 0x0000007fb9f27144 in mozilla::net::ParentProcessDocumentChannel::
    OnRedirectVerifyCallback (this=0x7f807beaa0, aResult=nsresult::NS_OK)
    at obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:847
(More stack frames follow...)

On ESR 78 there are only these three instances created, but which one is the one we're interested in? To find out I've placed a breakpoint on EmbedLitePuppetWidget::GetLayerManager() and will proceed to press the button on screen that's causing us the trouble.

(gdb) b EmbedLitePuppetWidget::GetLayerManager
Breakpoint 2 at 0x7fbca92838: EmbedLitePuppetWidget::GetLayerManager. (2 
    locations)
(gdb) c
Continuing.

Thread 10 "GeckoWorkerThre" hit Breakpoint 2, mozilla::embedlite::
    EmbedLitePuppetWidget::GetLayerManager (this=0x7f81005400, 
    aShadowManager=0x0, 
    aBackendHint=mozilla::layers::LayersBackend::LAYERS_NONE, 
    aPersistence=nsIWidget::LAYER_MANAGER_CURRENT)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:374
374       if (!mLayerManager) {
(gdb) p mLayerManager
$1 = {mRawPtr = 0x7f8088ffc0}
(gdb) 

What we're interested in here is the value of this, which is equal to 0x7f81005400 on this run. Comparing that with the values for this from the previous constructor calls, we can see that it's the third of the three instances of EmbedLitePuppetWidget that we're interested in:

#0  mozilla::embedlite::EmbedLitePuppetWidget::EmbedLitePuppetWidget (
    this=0x7f81005400, view=0x0)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:47

Notice also that the mLayerManager member is set to a valid pointer here. Let's run a similar test on ESR 91. There are five EmbedLitePuppetWidget instances being created, although I'm not sure why there are two more for ESR 91 than for ESR 78. I'm just keeping the backtrace for the last one because it turns out that's the one we're interested in.

Thread 10 "GeckoWorkerThre" hit Breakpoint 1, mozilla::embedlite::
    EmbedLitePuppetWidget::EmbedLitePuppetWidget (this=0x7fb8ab8040, 
    view=0x7fb8a7fe78)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:47
47      EmbedLitePuppetWidget::EmbedLitePuppetWidget(EmbedLiteViewChildIface* 
    view)

Thread 10 "GeckoWorkerThre" hit Breakpoint 1, mozilla::embedlite::
    EmbedLitePuppetWidget::EmbedLitePuppetWidget (this=0x7fb8aaf500, view=0x0)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:47
47      EmbedLitePuppetWidget::EmbedLitePuppetWidget(EmbedLiteViewChildIface* 
    view)

Thread 10 "GeckoWorkerThre" hit Breakpoint 1, mozilla::embedlite::
    EmbedLitePuppetWidget::EmbedLitePuppetWidget (this=0x7fb8d88b70, view=0x0)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:47
47      EmbedLitePuppetWidget::EmbedLitePuppetWidget(EmbedLiteViewChildIface* 
    view)

Thread 10 "GeckoWorkerThre" hit Breakpoint 1, mozilla::embedlite::
    EmbedLitePuppetWidget::EmbedLitePuppetWidget (this=0x7fb8aaf500, view=0x0)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:47
47      EmbedLitePuppetWidget::EmbedLitePuppetWidget(EmbedLiteViewChildIface* 
    view)

Thread 10 "GeckoWorkerThre" hit Breakpoint 1, mozilla::embedlite::
    EmbedLitePuppetWidget::EmbedLitePuppetWidget (this=0x7fb86664a0, view=0x0)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:47
47      EmbedLitePuppetWidget::EmbedLitePuppetWidget(EmbedLiteViewChildIface* 
    view)
(gdb) bt
#0  mozilla::embedlite::EmbedLitePuppetWidget::EmbedLitePuppetWidget (
    this=0x7fb86664a0, view=0x0)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:47
#1  0x0000007ff4c4d828 in mozilla::embedlite::EmbedLitePuppetWidget::
    CreateChild (this=0x7fb8aaf500, aRect=..., aInitData=0x7fde7be0c0,
    aForceUseIWidgetParent=<optimized out>)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/cxxalloc.h:33
#2  0x0000007ff3e3e5c0 in nsView::CreateWidgetForPopup (
    this=this@entry=0x7fb919a840, 
    aWidgetInitData=aWidgetInitData@entry=0x7fde7be0c0,
    aParentWidget=aParentWidget@entry=0x0, 
    aEnableDragDrop=aEnableDragDrop@entry=true, 
    aResetVisibility=aResetVisibility@entry=true)
    at view/nsView.cpp:615
#3  0x0000007ff41a1718 in nsComboboxControlFrame::ShowList (
    this=this@entry=0x7fb9174118, aShowList=aShowList@entry=true)
    at layout/forms/nsComboboxControlFrame.cpp:335
#4  0x0000007ff41a180c in nsComboboxControlFrame::ShowDropDown (
    this=this@entry=0x7fb9174118, aDoDropDown=aDoDropDown@entry=true)
    at layout/forms/nsComboboxControlFrame.cpp:903
#5  0x0000007ff41a4018 in nsComboboxControlFrame::SetFocus (this=0x7fb9174118, 
    aOn=<optimized out>, aRepaint=<optimized out>)
    at layout/forms/nsComboboxControlFrame.cpp:266
#6  0x0000007ff35109f8 in nsGenericHTMLFormElement::PreHandleEvent (
    this=0x7fb8ccf370, aVisitor=...)
    at dom/html/nsGenericHTMLElement.cpp:1922
#7  0x0000007ff33c437c in mozilla::EventTargetChainItem::PreHandleEvent (
    this=0x7fb8c98e48, aVisitor=...)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:869
#8  0x0000007ff33dc394 in mozilla::EventDispatcher::Dispatch (
    aTarget=<optimized out>, aPresContext=0x7fb9112c60, 
    aEvent=aEvent@entry=0x7fde7be4c0,
    aDOMEvent=aDOMEvent@entry=0x0, aEventStatus=aEventStatus@entry=0x0, 
    aCallback=aCallback@entry=0x0, aTargets=aTargets@entry=0x0)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsTArray.h:413
#9  0x0000007ff28ab1f8 in FocusBlurEvent::Run (this=0x7fb8238890)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/RefPtr.h:289
#10 0x0000007ff26f71cc in nsContentUtils::AddScriptRunner (aRunnable=..., 
    aRunnable@entry=...)
    at dom/base/nsContentUtils.cpp:5763
#11 0x0000007ff26f725c in nsContentUtils::AddScriptRunner (
    aRunnable=aRunnable@entry=0x7fb8238890)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/AlreadyAddRefed.h:48
#12 0x0000007ff28c6700 in nsFocusManager::FireFocusOrBlurEvent (
    this=this@entry=0x7fb80eeae0, aEventMessage=aEventMessage@entry=mozilla::
    eFocus,
    aPresShell=aPresShell@entry=0x7fb9168750, 
    aTarget=aTarget@entry=0x7fb8ccf370, aWindowRaised=aWindowRaised@entry=false,
    aIsRefocus=aIsRefocus@entry=false, aRelatedTarget=aRelatedTarget@entry=0x0)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:435
#13 0x0000007ff28c6b14 in nsFocusManager::SendFocusOrBlurEvent (
    this=this@entry=0x7fb80eeae0, aEventMessage=aEventMessage@entry=mozilla::
    eFocus,
    aPresShell=aPresShell@entry=0x7fb9168750, aDocument=0x7fb8e62730, 
    aTarget=aTarget@entry=0x7fb8ccf370, aWindowRaised=aWindowRaised@entry=false,
    aIsRefocus=aIsRefocus@entry=false, aRelatedTarget=aRelatedTarget@entry=0x0)
    at dom/base/nsFocusManager.cpp:2782
#14 0x0000007ff28c95b8 in nsFocusManager::Focus (this=this@entry=0x7fb80eeae0, 
    aWindow=0x7fb8571990, aElement=aElement@entry=0x7fb8ccf370,
    aFlags=aFlags@entry=2101250, aIsNewDocument=<optimized out>, 
    aIsNewDocument@entry=false, aFocusChanged=aFocusChanged@entry=true,
    aWindowRaised=aWindowRaised@entry=false, 
    aAdjustWidget=aAdjustWidget@entry=true, aActionId=5, 
    aBlurredElementInfo=...)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/Maybe.h:443
#15 0x0000007ff28d3ce8 in nsFocusManager::SetFocusInner (
    this=this@entry=0x7fb80eeae0, aNewContent=aNewContent@entry=0x7fb8ccf370,
    aFlags=aFlags@entry=2101250, aFocusChanged=aFocusChanged@entry=true, 
    aAdjustWidget=aAdjustWidget@entry=true, aActionId=<optimized out>)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:859
#16 0x0000007ff28d403c in nsFocusManager::SetFocus (
    this=this@entry=0x7fb80eeae0, aElement=0x7fb8ccf370, aFlags=2101250)
    at dom/base/nsFocusManager.cpp:492
#17 0x0000007ff339a6a8 in mozilla::EventStateManager::PostHandleEvent (
    this=this@entry=0x7fb856e100, aPresContext=aPresContext@entry=0x7fb9112c60,
    aEvent=aEvent@entry=0x7fde7bed88, aTargetFrame=0x0, 
    aStatus=aStatus@entry=0x7fde7bed6c, 
    aOverrideClickTarget=aOverrideClickTarget@entry=0x0)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/dom/Element.h:2057
#18 0x0000007ff401e808 in mozilla::PresShell::EventHandler::DispatchEvent (
    this=this@entry=0x7fde7bebd8,
    aEventStateManager=aEventStateManager@entry=0x7fb856e100, 
    aEvent=aEvent@entry=0x7fde7bed88, aTouchIsNew=false,
    aEventStatus=aEventStatus@entry=0x7fde7bed6c, 
    aOverrideClickTarget=aOverrideClickTarget@entry=0x0)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/RefPtr.h:289
#19 0x0000007ff401f588 in mozilla::PresShell::EventHandler::
    HandleEventWithCurrentEventInfo (this=this@entry=0x7fde7bebd8,
    aEvent=aEvent@entry=0x7fde7bed88, 
    aEventStatus=aEventStatus@entry=0x7fde7bed6c, 
    aIsHandlingNativeEvent=aIsHandlingNativeEvent@entry=true,
    aOverrideClickTarget=0x0)
    at layout/base/PresShell.cpp:8177
#20 0x0000007ff4023dbc in mozilla::PresShell::EventHandler::
    HandleEventUsingCoordinates (this=this@entry=0x7fde7beca8,
    aFrameForPresShell=aFrameForPresShell@entry=0x7fb9172e70, 
    aGUIEvent=aGUIEvent@entry=0x7fde7bed88, 
    aEventStatus=aEventStatus@entry=0x7fde7bed6c,
    aDontRetargetEvents=aDontRetargetEvents@entry=false)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:859
#21 0x0000007ff4023fa0 in mozilla::PresShell::EventHandler::HandleEvent (
    this=this@entry=0x7fde7beca8,
    aFrameForPresShell=aFrameForPresShell@entry=0x7fb9172e70, 
    aGUIEvent=aGUIEvent@entry=0x7fde7bed88,
    aDontRetargetEvents=aDontRetargetEvents@entry=false, 
    aEventStatus=aEventStatus@entry=0x7fde7bed6c)
    at layout/base/PresShell.cpp:6898
#22 0x0000007ff40240ec in mozilla::PresShell::HandleEvent (this=0x7fb9168750, 
    aFrameForPresShell=0x7fb9172e70, aGUIEvent=aGUIEvent@entry=0x7fde7bed88,
    aDontRetargetEvents=aDontRetargetEvents@entry=false, 
    aEventStatus=aEventStatus@entry=0x7fde7bed6c)
    at layout/base/PresShell.cpp:6841
#23 0x0000007ff270373c in nsContentUtils::SendMouseEvent (
    aPresShell=aPresShell@entry=0x7fb9168750, aType=..., aX=aX@entry=105.25927,
    aY=aY@entry=132.481491, aButton=aButton@entry=0, 
    aButtons=aButtons@entry=-1, aClickCount=aClickCount@entry=1, 
    aModifiers=aModifiers@entry=0,
    aIgnoreRootScrollFrame=aIgnoreRootScrollFrame@entry=false, 
    aPressure=aPressure@entry=0, aInputSourceArg=aInputSourceArg@entry=5,
    aIdentifier=aIdentifier@entry=0, aToWindow=aToWindow@entry=true, 
    aPreventDefault=aPreventDefault@entry=0x7fde7bef57,
    aIsDOMEventSynthesized=<optimized out>, aIsDOMEventSynthesized@entry=true, 
    aIsWidgetEventSynthesized=aIsWidgetEventSynthesized@entry=false)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsView.h:268
#24 0x0000007ff2716980 in nsDOMWindowUtils::SendMouseEventCommon (
    this=this@entry=0x7ef0000ff0, aType=..., aX=aX@entry=105.25927, 
    aY=aY@entry=132.481491,
    aButton=aButton@entry=0, aClickCount=aClickCount@entry=1, 
    aModifiers=aModifiers@entry=0, 
    aIgnoreRootScrollFrame=aIgnoreRootScrollFrame@entry=false,
    aPressure=aPressure@entry=0, aInputSourceArg=aInputSourceArg@entry=5, 
    aPointerId=aPointerId@entry=0, aToWindow=aToWindow@entry=true,
    aPreventDefault=aPreventDefault@entry=0x0, 
    aIsDOMEventSynthesized=aIsDOMEventSynthesized@entry=true,
    aIsWidgetEventSynthesized=aIsWidgetEventSynthesized@entry=false, 
    aButtons=aButtons@entry=-1)
    at dom/base/nsDOMWindowUtils.cpp:732
#25 0x0000007ff2716ca0 in nsDOMWindowUtils::SendMouseEventToWindow (
    this=0x7ef0000ff0, aType=..., aX=105.25927, aY=132.481491, aButton=0, 
    aClickCount=1,
    aModifiers=0, aIgnoreRootScrollFrame=false, aPressure=0, aInputSourceArg=5, 
    aIsDOMEventSynthesized=<optimized out>, aIsWidgetEventSynthesized=false,
    aButtons=0, aIdentifier=<optimized out>, aOptionalArgCount=3 '\003')
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/ProfilerLabels.h:249
#26 0x0000007ff19ca220 in _NS_InvokeByIndex ()
    at xpcom/reflect/xptcall/md/unix/xptcinvoke_asm_aarch64.S:74
[...]
#47 0x0000007ff4c543b4 in mozilla::embedlite::EmbedLiteViewChild::
    RecvHandleSingleTap (this=0x7fb8a7fe40, aPoint=..., 
    aModifiers=@0x7fde7c0594: 0,
    aGuid=..., aInputBlockId=@0x7fde7c05a0: 1)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/RefPtr.h:313
#48 0x0000007ff1f06b6c in mozilla::embedlite::PEmbedLiteViewChild::
    OnMessageReceived (this=0x7fb8a7fe40, msg__=...) at PEmbedLiteViewChild.cpp:
    1718
#49 0x0000007ff1ef3844 in mozilla::embedlite::PEmbedLiteAppChild::
    OnMessageReceived (this=<optimized out>, msg__=...)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/ipc/ProtocolUtils.h:
    675
#50 0x0000007ff1ddfb20 in mozilla::ipc::MessageChannel::DispatchAsyncMessage (
    this=this@entry=0x7fb8b234f8, aProxy=aProxy@entry=0x7fb83a8700, aMsg=...)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/ipc/ProtocolUtils.h:
    675
#51 0x0000007ff1dee59c in mozilla::ipc::MessageChannel::DispatchMessage (
    this=this@entry=0x7fb8b234f8, aMsg=...)
    at ipc/glue/MessageChannel.cpp:2001
#52 0x0000007ff1def9f4 in mozilla::ipc::MessageChannel::RunMessage (
    this=0x7fb8b234f8, aTask=...)
    at ipc/glue/MessageChannel.cpp:1860
[...]
#76 0x0000007fef54889c in ?? () from /lib64/libc.so.6

From here I've placed a breakpoint on the PresShell::Paint() method so that we can catch the call to GetLayerManager() before the crash happens. Then I've pressed the button, waited for the breakpoint to hit, then switched to a breakpoint on the GetLayerManager() method. Here's what this all gives us:

Thread 10 "GeckoWorkerThre" hit Breakpoint 2, mozilla::embedlite::
    EmbedLitePuppetWidget::GetLayerManager (this=0x7fb86664a0, 
    aShadowManager=0x0,
    aBackendHint=mozilla::layers::LayersBackend::LAYERS_NONE, 
    aPersistence=nsIWidget::LAYER_MANAGER_CURRENT)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:371
371     {
(gdb) n
372       if (!mLayerManager) {
(gdb) p mLayerManager
$15 = {mRawPtr = 0x0}

There, as you can see, is the problematic null pointer of mLayerManager. And here's the backtrace for this call, just before the browser crashes:

(gdb) bt             
#0  mozilla::embedlite::EmbedLitePuppetWidget::GetLayerManager (
    this=0x7fb86664a0, aShadowManager=0x0, 
    aBackendHint=mozilla::layers::LayersBackend::LAYERS_NONE, 
    aPersistence=nsIWidget::LAYER_MANAGER_CURRENT)
    at mobile/sailfishos/embedshared/EmbedLitePuppetWidget.cpp:372
#1  0x0000007ff4c57cec in nsIWidget::GetLayerManager (this=0x7fb86664a0)
    at widget/nsIWidget.h:1303
#2  mozilla::embedlite::PuppetWidgetBase::Invalidate (this=0x7fb86664a0, 
    aRect=...)
    at mobile/sailfishos/embedshared/PuppetWidgetBase.cpp:261
#3  0x0000007ff4c56e70 in mozilla::embedlite::PuppetWidgetBase::Resize (
    this=0x7fb86664a0, aWidth=1.1986628770828247, aHeight=1.8786317110061646, 
    aRepaint=<optimized out>)
    at mobile/sailfishos/embedshared/PuppetWidgetBase.cpp:188
#4  0x0000007ff3e4a89c in nsBaseWidget::ResizeClient (this=0x7fb86664a0, 
    aSize=..., aRepaint=true)
    at widget/nsBaseWidget.cpp:1601
[...]
#17 0x0000007ff270373c in nsContentUtils::SendMouseEvent (
    aPresShell=aPresShell@entry=0x7fb9168750, aType=..., aX=aX@entry=105.25927, 
    aY=aY@entry=132.481491, aButton=aButton@entry=0, 
    aButtons=aButtons@entry=-1, aClickCount=aClickCount@entry=1, 
    aModifiers=aModifiers@entry=0, 
    aIgnoreRootScrollFrame=aIgnoreRootScrollFrame@entry=false, 
    aPressure=aPressure@entry=0, aInputSourceArg=aInputSourceArg@entry=5, 
    aIdentifier=aIdentifier@entry=0, aToWindow=aToWindow@entry=true, 
    aPreventDefault=aPreventDefault@entry=0x7fde7bef57, 
    aIsDOMEventSynthesized=<optimized out>, aIsDOMEventSynthesized@entry=true, 
    aIsWidgetEventSynthesized=aIsWidgetEventSynthesized@entry=false)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsView.h:268
#18 0x0000007ff2716980 in nsDOMWindowUtils::SendMouseEventCommon (
    this=this@entry=0x7ef0000ff0, aType=..., aX=aX@entry=105.25927, 
    aY=aY@entry=132.481491, 
    aButton=aButton@entry=0, aClickCount=aClickCount@entry=1, 
    aModifiers=aModifiers@entry=0, 
    aIgnoreRootScrollFrame=aIgnoreRootScrollFrame@entry=false, 
    aPressure=aPressure@entry=0, aInputSourceArg=aInputSourceArg@entry=5, 
    aPointerId=aPointerId@entry=0, aToWindow=aToWindow@entry=true, 
    aPreventDefault=aPreventDefault@entry=0x0, 
    aIsDOMEventSynthesized=aIsDOMEventSynthesized@entry=true, 
    aIsWidgetEventSynthesized=aIsWidgetEventSynthesized@entry=false, 
    aButtons=aButtons@entry=-1)
    at dom/base/nsDOMWindowUtils.cpp:732
#19 0x0000007ff2716ca0 in nsDOMWindowUtils::SendMouseEventToWindow (
    this=0x7ef0000ff0, aType=..., aX=105.25927, aY=132.481491, aButton=0, 
    aClickCount=1, 
    aModifiers=0, aIgnoreRootScrollFrame=false, aPressure=0, aInputSourceArg=5, 
    aIsDOMEventSynthesized=<optimized out>, aIsWidgetEventSynthesized=false, 
    aButtons=0, aIdentifier=<optimized out>, aOptionalArgCount=3 '\003')
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/ProfilerLabels.h:249
#20 0x0000007ff19ca220 in _NS_InvokeByIndex ()
    at xpcom/reflect/xptcall/md/unix/xptcinvoke_asm_aarch64.S:74
#21 0x0000007ff19ca5c8 in NS_InvokeByIndex (that=<optimized out>, 
    methodIndex=<optimized out>, paramCount=<optimized out>, params=<optimized 
    out>)
    at xpcom/reflect/xptcall/md/unix/xptcinvoke_aarch64.cpp:167
#22 0x0000007ff21a8584 in CallMethodHelper::Invoke (this=0x7fde7bf1e8)
    at js/xpconnect/src/XPCWrappedNative.cpp:1644
[...]
#41 0x0000007ff4c543b4 in mozilla::embedlite::EmbedLiteViewChild::
    RecvHandleSingleTap (this=0x7fb8a7fe40, aPoint=..., 
    aModifiers=@0x7fde7c0594: 0, 
    aGuid=..., aInputBlockId=@0x7fde7c05a0: 1)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/RefPtr.h:313
#42 0x0000007ff1f06b6c in mozilla::embedlite::PEmbedLiteViewChild::
    OnMessageReceived (this=0x7fb8a7fe40, msg__=...) at PEmbedLiteViewChild.cpp:
    1718
#43 0x0000007ff1ef3844 in mozilla::embedlite::PEmbedLiteAppChild::
    OnMessageReceived (this=<optimized out>, msg__=...)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/ipc/ProtocolUtils.h:
    675
#44 0x0000007ff1ddfb20 in mozilla::ipc::MessageChannel::DispatchAsyncMessage (
    this=this@entry=0x7fb8b234f8, aProxy=aProxy@entry=0x7fb83a8700, aMsg=...)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/ipc/ProtocolUtils.h:
    675
#45 0x0000007ff1dee59c in mozilla::ipc::MessageChannel::DispatchMessage (
    this=this@entry=0x7fb8b234f8, aMsg=...)
    at ipc/glue/MessageChannel.cpp:2001
#46 0x0000007ff1def9f4 in mozilla::ipc::MessageChannel::RunMessage (
    this=0x7fb8b234f8, aTask=...)
    at ipc/glue/MessageChannel.cpp:1860
[...]
#70 0x0000007fef54889c in ?? () from /lib64/libc.so.6
(gdb) 

By comparing the value of this at the head of this backtrace with the constructors from earlier, we can see it's the last of the instances of EmbedLitePuppetWidget that's created which is causing the problem and which has the value of mLayerManager set to null.

Step one is completed. Step two is to add a breakpoint to the memory location of mLayerManager associated with the third instance of EmbedLitePuppetWidget on ESR 78 and the fifth instance on ESR 91. Confused? I know I am. But I'm going to try to keep a clear head. My theory is that the ESR 78 breakpoint will hit, but the ESR 91 breakpoint won't.

Unfortunately, pulling together all of these breakpoints is time consuming and I've hit the limit of what I can spend on this today, so I'll have to pick this up again in the morning. This is taking longer than expected, but we're inching forwards. I think this one is going to be solvable.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.