flypig.co.uk

I was getting a little despondent trying to fix DuckDuckGo on ESR 91, but the tests I performed yesterday have invigorated me somewhat. It's now clear that there's a decent quantity of accesses being made when using ESR 91 and that wouldn't be happening unless at least some of the data was being interpreted as HTML.

But it's also clear that not everything is as it should be: on ESR 78 there are a mixture of SVG and PNG files being downloaded to provide the images on the page. In contrast, on ESR 91, there are no images being downloaded at all. There are two possible reasons for this I can think of:

1. DuckDuckGo is serving pages that don't contain any images. Seems unlikely, but nevertheless a possibility.
2. There are images but ESR 91 isn't turning them in to access requests. I have no idea why this might happen, yet it still feels the more likely of the two scenarios.

I think it would be useful to know how much overlap there is between the sets of files that are being downloaded. So I put together a command line monstrosity to compare them. Before giving the final command, let me break down what it's doing.

First, there are some extraneous accesses in the list that have nothing to do with DuckDuckGo but are a consequence of gecko collecting settings data after the profile was wiped; lines like this:

https://firefox.settings.services.mozilla.com/v1/buckets/monitor/collections/
    changes/records?collection=hijack-blocklists&bucket=main

So the command filters all of the lines that don't include duckduckgo.

Second, I noticed that some files appear to be the same but located at different URLs. For example, this can be found in the ERS 78 list:

https://duckduckgo.com/font/ProximaNova-ExtraBold-webfont.woff2

While this can be found in the ESR 91 list:

https://duckduckgo.com/static-assets/font/ProximaNova-Reg-webfont.woff2

These are both font files; they must surely be the same file, right? But they're located in different folders. So the command then strips the URLs down to the leafnames.

Third, it then sorts the results alphabetically so that any identical lines in both files will appear in the same order. If there are any matching lines, this will make any diff of the two much cleaner.

Finally the command then performs a side-by-side diff on the result. Here's all of that put together:

$ diff --side-by-side \
    <(sed -e 's/^.*\/\(.*\)/\1/g' <(grep "duckduckgo" log1.txt) | sort) \
    <(sed -e 's/^.*\/\(.*\)/\1/g' <(grep "duckduckgo" log2.txt) | sort)

When I came up with this approach I thought it would give amazing results, but in practice it's not as exciting as I was hoping for.

Let's concentrate on the mobile version of the page first. Here's the diff:

$ diff --side-by-side \
    <(sed -e 's/^.*\/\(.*\)/\1/g' <(grep "duckduckgo" ddg-urls-esr78-mobile.txt) | sort) \
    <(sed -e 's/^.*\/\(.*\)/\1/g' <(grep "duckduckgo" ddg-urls-esr91-mobile.txt) | sort)

1-monster-v2--no-animation.svg             | 18040-1287342b1f839f70.js
2-ghost-v2.svg                             | 38407-070351ade350c8e4.js
3-bathtub-v2--no-animation.svg             | 39337-cd8caeeff0afb1c4.js
4-alpinist-v2.svg                          | 41966-c9d76895b4f9358f.js
arrow.svg                                  | 55015-29fec414530c2cf6.js
b.9e45618547aaad15b744.js                  | 55672-19856920a309aea5.js
d.01ff355796b8725c8dad.js                  | 61754-29df12bb83d71c7b.js
duckduckgo85.js                            | 6a4833195509cc3d.css
h.2d6522d4f29f5b108aed.js                  | 703c9a9a057785a9.css
hi?7857271&b=firefox&ei=true&i=false&[...] | 81125-b74d1b6f4908497b.js
l.656ceb337d61e6c36064.js                  | 93432-ebd443fe69061b19.js
logo_homepage.alt.v109.svg                 | 94623-d5bfa67fc3bada59.js
logo_homepage.normal.v109.svg              | 95665-30dd494bea911abd.js
o.2988a52fdfb14b7eff16.css                 | a2a29f84956f2aac.css
p.f5b58579149e7488209f.js                  | _app-ce0b94ea69138577.js
post3.html                                 | _buildManifest.js
                                           > c89114cfe55133c4.css
                                           > ed8494aa71104fdc.css
                                           > f0b3f7da285c9dbd.css
                                           > framework-f8115f7fae64930e.js
                                           > home-34dda07336cb6ee1.js
                                           > main-17a05b704438cdd6.js
                                           > ProximaNova-Bold-webfont.woff2
ProximaNova-ExtraBold-webfont.woff2          ProximaNova-ExtraBold-webfont.woff2
                                           > ProximaNova-RegIt-webfont.woff2
ProximaNova-Reg-webfont.woff2                ProximaNova-Reg-webfont.woff2
ProximaNova-Sbold-webfont.woff2              ProximaNova-Sbold-webfont.woff2
s.b49dcfb5899df4f917ee.css                 | _ssgManifest.js
teaser-2@2x.png                            | webpack-7358ea7cdec0aecf.js
ti.b07012e30f6971ff71d3.js                 <
tl.3db2557c9f124f3ebf92.js                 <
u.a3c3a6d4d7bf9244744d.js                  <

Only three files are shared across the two collections. The remaining 22 and 27 files respectively are apparently different. I was honestly hoping for there to be more similarity.

For completeness let's do the same for the desktop collections:

$ diff --side-by-side \
    <(sed -e 's/^.*\/\(.*\)/\1/g' <(grep "duckduckgo" ddg-urls-esr78-desktop.txt) | sort) \
    <(sed -e 's/^.*\/\(.*\)/\1/g' <(grep "duckduckgo" ddg-urls-esr91-d.txt) | sort)

18040-1287342b1f839f70.js                    18040-1287342b1f839f70.js
38407-070351ade350c8e4.js                    38407-070351ade350c8e4.js
39337-cd8caeeff0afb1c4.js                    39337-cd8caeeff0afb1c4.js
41966-c9d76895b4f9358f.js                    41966-c9d76895b4f9358f.js
48292.8c8d6cb394d25a15.js                  <
55015-29fec414530c2cf6.js                    55015-29fec414530c2cf6.js
55672-19856920a309aea5.js                    55672-19856920a309aea5.js
61754-29df12bb83d71c7b.js                    61754-29df12bb83d71c7b.js
6a4833195509cc3d.css                         6a4833195509cc3d.css
703c9a9a057785a9.css                         703c9a9a057785a9.css
81125-b74d1b6f4908497b.js                    81125-b74d1b6f4908497b.js
93432-ebd443fe69061b19.js                    93432-ebd443fe69061b19.js
94623-d5bfa67fc3bada59.js                    94623-d5bfa67fc3bada59.js
95665-30dd494bea911abd.js                    95665-30dd494bea911abd.js
a2a29f84956f2aac.css                         a2a29f84956f2aac.css
add-firefox.f0890a6c.svg                   <
_app-ce0b94ea69138577.js                     _app-ce0b94ea69138577.js
app-protection-back-dark.png               <
app-protection-front-dark.png              <
app-protection-ios-dark.png                <
app-store.501fe17a.png                     <
atb_home_impression?9836955&b=firefox[...] <
_buildManifest.js                            _buildManifest.js
burn@2x.be0bd36d.png                       <
c89114cfe55133c4.css                         c89114cfe55133c4.css
chrome-lg.a4859fb2.png                     <
CNET-DARK.e3fd496e.png                     <
dark-mode@2x.3e150d01.png                  <
devices-dark.png                           <
ed8494aa71104fdc.css                         ed8494aa71104fdc.css
edge-lg.36af7682.png                       <
email-protection-back-dark.png             <
email-protection-front-light.png           <
email-protection-ios-dark.png              <
f0b3f7da285c9dbd.css                         f0b3f7da285c9dbd.css
firefox-lg.8efad702.png                    <
flame.1241f020.png                         <
flame@2x.40e1cfa0.png                      <
flame-narrow.70589b7c.png                  <
framework-f8115f7fae64930e.js                framework-f8115f7fae64930e.js
home-34dda07336cb6ee1.js                     home-34dda07336cb6ee1.js
legacy-homepage-btf-dark.png               <
legacy-homepage-btf-mobile-dark.png        <
macos.61889438.png                         <
main-17a05b704438cdd6.js                     main-17a05b704438cdd6.js
night@2x.4ca79636.png                      <
opera-lg.237c4418.png                      <
page_home_commonImpression?2448534&[...]   <
play-store.e5d5ed36.png                    <
ProximaNova-Bold-webfont.woff2               ProximaNova-Bold-webfont.woff2
ProximaNova-ExtraBold-webfont.woff2          ProximaNova-ExtraBold-webfont.woff2
ProximaNova-RegIt-webfont.woff2              ProximaNova-RegIt-webfont.woff2
ProximaNova-Reg-webfont.woff2                ProximaNova-Reg-webfont.woff2
ProximaNova-Sbold-webfont.woff2              ProximaNova-Sbold-webfont.woff2
safari-lg.8406694a.png                     <
search-protection-back-light.png           <
search-protection-front-dark.png           <
search-protection-ios-dark.png             <
set-as-default.d95c3465.svg                <
_ssgManifest.js                              _ssgManifest.js
UT-DARK-DEFAULT.6cd0020d.png               <
VERGE-DARK-DEFAULT.8850a2d2.png            <
webpack-7358ea7cdec0aecf.js                  webpack-7358ea7cdec0aecf.js
web-protection-back-dark.png               <
web-protection-front-dark.png              <
web-protection-ios-dark.png                <
widget-big@2x.a260ccf6.png                 <
widget-small@2x.07c865df.png               <
windows.477fa143.png                       <
WIRED-DARK-DEFAULT.b4d48a49.png            <

Here we see something more interesting. There are 30 files the same across the two collections with 43 and 0 files being different respectively. In other words, the ESR 91 collection is a subset of the ESR 78 collection.

There might be something in this, but initially I'm more interested in the mobile version of the site and there the overlap is far less. However, now that that I have the URLs, one thing I can do is download all of the files and try to use them to recreate the site on my own server. It's possible this will give better results than saving out the files from the desktop browser, so I'll be giving that a go tomorrow.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.