Personal Blog

View the blog index.

RSS feed Click the icon for the blog RSS feed.


5 most recent items

2 Jan 2022 : Reckoning and Renewal, Part II #
New year, new resolutions. At the start of last year I made some new year's resolutions, and the time has come to judge my actions against them.

Unlike 2020, my 2021 achievements aren't best measured in television series and computer games. Which isn't to say I didn't get through a lot of television (How to Get Away with Murder, Borgen, Capitani, Lupin, The Expanse, Witcher, Another Life, Cursed, all 172 episodes of Voyager) or computer games (Hades, the Banner Saga, Creaks, Dishonored Death to the Outsider, Titanfall 2, Trine 4). But unlike in 2020 these weren't the completion of multi-decade odysseys.

Instead, I spent much of my year focussing on work. A surprising amount of my free time was taken up writing the Sailfish OS community newsletter. I also spent a lot of time trying to learn Finnish, wracking up a 339 day streak on Duolingo and, in the second half of the year, attending intensive Finnish classes. Unfortunately my actual progress in being able to communicate doesn't match what my Duolingo stats imply.


I managed to eradicate some long-standing sores in my life. In September I finally released my website code as open source, thereby completing the first of my 2020 resolutions.

"1. Put this website code into a public git repository."

This had been on my to-do list for fifteen years, so I was pretty happy to get it finally resolved. Also by September I'd managed to transition my password approach from the Stanford to the Cambridge algorithm, releasing an updated PwdHash app in the process. This has been on my to-do list for five years. In December I managed to upgrade Constantia, my home server, to the latest versions of the software it runs (Linux, NextCloud, jitsi, etc.). Upgrades are always daunting in prospect and painful in practice, so I'm really happy to have this back under control. Finally Joanna and I have arranged to have our gas central heating converted to a heat pump. It's taken a lot of organising from us both, and it won't actually happen until later this year, but this is another long-standing issue I'm really happy we're going to get fixed.

The other positive from 2021 was that I managed to reduce my carbon footprint, and my waste output.

Those are the big positives from 2021 for me. It wasn't all positives of course. Joanna and I still haven't addressed our living-in-two-different-countries challenge. We really need to find a better solution. This was made worse by the travel restrictions this year, which really meant we spent much less time together than is right.

While I succeeded with my first resolution, I did much worse with my other two.

"2. Each week by the end of the weekend, spend at least an hour doing something calming that doesn't require a computer."

This was always going to be a challenge, and even though I spent ages trying to formulate this resolution into something measurable, it still didn't work out. My year was busy, sometimes stressful, and I didn't make progress in the non-computer calming things, such as reading, that I'd hoped to. I did at least do a lot of walking in the forest and by the lake, and even managed to do some work in the forest and by the lake. But I need to keep track of it better in the future.

"3. Complete the bisection analysis that Frajo and I started working on a year ago."

Again, this was a fail. I made some progress, but the work isn't even close to being complete. The fact I've not pushed my write-up to git is a good indication that it needs a lot more work.

So that was my 2021. What about 2022? Based on the above, what are my resolutions for the next year going to be?
  1. Maintain a digital record of all of my grocery purchases during the year. Since I already keep track of my waste output, I think it'd be interesting to compare this against my goods input
  2. Learn quantum programming, measured by completing the book I'm reading on the topic, and writing at least one quantum computer programme. The programme should do something.
  3. Complete my Curious Correspondence course with Joanna. We've already completed four out of the twelve tasks, so we have another eight to complete this year at least.
  4. Write up the bisection analysis work.
Plus keep my average waste down to below 200g per day on average, which is a reduction on my 2021 goal, but above my 2021 level of 114 g/day, in order to keep it achievable.

I'm dropping my aim to spend a measurable hour away from my computer each week in favour of these specific tasks instead. If I can manage the above, then I think I'll have achieved something similar.

Finally, despite my usual cynicism I was won over by this article about improving without trying. I think it's because the sentences are so short. Here are five I really buy into and will be test-driving this year.
  1. Sharpen your knives.
  2. Start a Saturday morning with some classical music – it sets the tone for a calm weekend.
  3. Be polite to rude strangers — it’s oddly thrilling.
  4. Learn the names of 10 trees.
  5. Politely decline invitations if you don’t want to go.
One thing I'm pleased to note is that none of my 2022 resolutions are intended to fix long-term sores, which I think means I must have tackled the most serious of them. In which case, I can now focus on preventing any new ones from forming.

Plenty of things to aim for in 2022. Let's see how this one goes. I predict 3% chance of success.
31 Dec 2021 : How lightly did I tread in 2021? #
I'm trying to make doing an environmental check-up an annual habit. I have so many bad habits, it feels like getting a good habit would make a nice change. So this is my attempt. Looking back is also a lot safer than making future commitments.

It turns out that 2021 was a good year for me environmentally, or that it at least looks that way on paper. Hemmed in by the pandemic and forced to reduce flying, it wasn't hard to do less this year. On top of that 2021 made my third year of collecting waste data, which — even unconsciously — has trained me into throwing less stuff away.

So let's start with my 2021 household carbon footprint. According to the Carbon Footprint Calculator, in 2021 Joanna and I contributed a combined total of 7.73 tonnes of CO2 to the atmosphere. That's a lot of CO2, but our output is at least following a downward trend. In 2019 we contributed 14.47 tonnes and in 2020 it was 8.50 tonnes. The following table summarises where all that gas came from.
Source Details for 2021 CO2 output 2019 (t) CO2 output 2020 (t) CO2 output 2021 (t)
Electricity 3 009 kWh 0.50 0.40 0.59
Natural gas 9 089 kWh 1.18 1.26 1.66
Flights 3 HEL-LHR, 4 TMP-STA 5.76 2.26 1.90
Car 3 219 km 1.45 0.39 0.39
Bus 168 km 0.00 0.01 0.02
National rail 676 km 0.08 0.01 0.02
International rail 513 km 0.02 0.01 0.00
Taxi 100 km 0.01 0.01 0.01
Food and drink   1.69 1.11 1.05
Pharmaceuticals   0.26 0.32 0.31
Clothing   0.03 0.06 0.06
Paper-based products   0.34 0.15 0.14
Computer usage   1.30 1.48 0.75
Electrical   0.12 0.29 0.19
Non-fuel car   0.00 0.10 0.00
Manufactured goods   0.50 0.03 0.03
Hotels, restaurants   0.51 0.16 0.15
Telecoms   0.15 0.05 0.04
Finance   0.24 0.24 0.22
Insurance   0.19 0.11 0.10
Education   0.05 0.00 0.04
Recreation   0.09 0.06 0.05
Total   14.47 8.50 7.73

The main reasons for the reduction compared to 2020 were fewer flights, and fewer computer purchases (I purchased precisely one less laptop than the one I purchased in 2020). Laptops, it turns out, are surprisingly carbon-intensive to make.

So those reductions are benefits, but I'm not sure they're benefits we'll be able to maintain over time. In early 2022 we've arranged to have a heat pump installed to replace our gas central heating. This is a big change, with the main aim to reduce that 9 089 kWh of natural gas usage in the table above. Gas is clean to burn, but as a non-renewable fossil fuel it's especially problematic, with no easily switchable environmentally-friendly alternative. Hopefully a heat pump will reduce our overall power usage, not just our non-renewable usage.

Our numbers equate to an average of 3.87 tonnes of CO2 per person in 2021. That compares favourably to the UK average of 5.4 tonnes, an EU average of 6.4 tonnes and a world average of 4.8 tonnes according to the World Bank.

How about waste output? My average waste output for 2021 was 114.69 g/day. You can see how this came about, and how it was split across different types of waste, in the graph below.
Daily waste data histocurve

This average is equivalent to a total waste output of 42 kg for the year. In theory everything except the General waste shown in the graph was recycled. The total is also a reduction on previous years, comparing to 57 kg of output in 2020 and 118 kg in 2019. These number are slightly lower than the actual amount. For example this year I've spent around six weeks in the UK, during which I'm not able to collect waste output data.

This all looks quite positive, but I'm becoming increasingly aware that waste output is a volatile metric. For example, if at some point I have to replace a piece of furniture, my waste output will go through the roof for the year. This does honestly motivate me to try to fix things rather than throw them away, but it's also a source of angst, knowing that it'll happen eventually.

According to eurostate, average per capita municipal waste output across the EU was 505 kg per person, with the average in Finland being slightly higher at 596 kg. Compared to this, my 42 kg of output looks pretty good. Still, I'm supposing that at least some of that 505 kg was made up of chairs.

So in summary I'm happy that Joanna's and my CO2 output was down on 2020, as was my waste output. We both trod a bit more lightly, even if it's not yet light enough. We've not quite reached that fully circular economy. The main driving factor for the reduction seems to have been the pandemic, so it will at least be interesting to see what happens next year.

12 Sep 2021 : Changing password approaches is really hard #
Last month I spent a couple of weeks with Joanna in a cottage next to a lake in the Finnish countryside. It was a time for reflection and an opportunity to re-evaluate my life choices. A chance happening raised the topics of passwords and phishing. It's a subject I used to be well-acquainted with, but which my work has drifted away from more recently.

Everyone needs a way to manage their passwords and ideally everyone should have a good way to manage their passwords. About a decade ago I started using PwdHash as my method. It has several advantages that are similar to those offered by the more-familiar database-backed password managers. For example it ensures you use a different password for every website; it guards against phishing attacks; it avoids the need to remember anything other than a master password; and it works across all devices (desktop, phone, browser, etc.). Because it generates passwords on-the-fly, it also has the benefit of not needing to store a database of passwords, neither in the cloud nor locally. Pretty neat. This last point makes it particularly attractive to me, since I'm generally uncomfortable relying on cloud services I don't host myself.

It has downsides too though. The main practical downside is that — because they're deterministically generated — the passwords can't be amended. This causes issues for systems that require regular password changes (thankfully less of an issue now than it was five years ago), or if you have to change your password on a specific site for some other reason (e.g. the site's passwords are compromised by an attacker). There are ways to work around this, but they're pretty awkward and user-unfriendly.

A further downside is that the password generation can be reversed, meaning that compromising a password for a single account could lead to compromise of the master password used to generate all of the individual site passwords, and hence to all of your accounts. This was such a major concern that my Cambridge colleague Graham Rymer and I investigated it back in 2016 and showed it to be a very real threat. We even managed to extract 79 PwdHash master passwords from three well-known compromised and publicly-leaked password databases (Stratfor, Rootkit and LinkedIn). In the paper we published on it we discussed ways to mitigate this threat, presenting our own improved alternative scheme. Other than technical improvements, the most crucial countermeasure we suggested was to use a really strong master password. This may seem obvious, but apparently it wasn't for many users of PwdHash up to that point.

Another significant problem with PwdHash is that changing to a different scheme at a later date is a tremendously painful experience. And that's exactly what I'm experiencing now.

The work Graham and I did convinced me I needed to change my approach. And what better approach than the one we recommended?

Yet it's been five years and I'm only just now starting to make the switch, which maybe tells you something about the effort involved. In theory I should have been able to switch over gradually, one site at a time, just updating my passwords on my next visit to each site. In practice the infrastructure wasn't there to allow me to do this easily enough. What I needed was a website and an app that would allow me to generate my old password, then seamlessly generate the new password without having to type in a new URL, open a new app or whatever. Basically, for something I'm going to have to perform hundreds of times, the process has to be as effortless as possible.

So this is what I've spent the last few weeks arranging. First off was the website. There are already sites for the original Stanford PwdHash and for our updated variant. But the thought of having to switch from one to the other across hundreds of sites was just too much. So I combined them into a single site that allows switching between the two with a single click. It's a really small, simple thing. but it's enough to make the difference between inertia holding me back and momentum pushing me forwards. I also made it easier to get the generated password onto the clipboard, allowing my web workflow to be fully optimised.

That deals with the web but what about apps on my phone? I've been using a PwdHash app on my phone for many years, written by Robert Gerlach. It's a really simple app, but all the more effective for it. The app only supported the original Stanford variant, so I've just spent my weekend updating it to support the new algorithm as well. Not only have I added support for the new algorithm, but following the advice from our paper of five years ago, I also added a password strength meter using the zxcvbn algorithm. Plus I also made a few other improvements to better suit my usual workflow.

So now the fun of changing all my passwords can finally begin. So far it's been more preparation than progress, but I have managed to convert the passwords for three sites, so there's no going back now. The whole experience has reaffirmed my empathy for everyone struggling with password management. There are plenty of good solutions out there for managing passwords of course, but frankly the fact there are so many options just adds to the complexity of making a good choice. Especially when it's the sort of decision you really want to get right first time. Or in my case, hopefully, second time.
29 Aug 2021 : New and improved waste data graphs #
I've just hit two full years of waste output data, which has given me a nice idea about how much waste I generate on a daily basis. Since I started back in August 2019 I've been updating a graph showing the results on my waste page. It's provided quite a fascinating picture. Not only has my waste output gone down over time, but it's also become more consistent.

I attribute this improvement squarely to the act of measuring my data each fortnight. The process has made me far more aware, not just about how much waste I produce, but also the sorts of products that generate more or less waste.

For example, glass is really heavy and it became clear quite early on that it was contributing significantly to the weight of waste I was producing. This motivated me to look into it more deeply, which ultimately resulted in me almost completely eradicating glass from my daily usage.

As a result of this and other changes, my daily usage has gone down from 322.80 g/day in 2019 to 154.98 g/day in 2020, and now in 2021 I'm currently averaging 123.34 g/day. Admittedly my average this year is likely to increase during the winter (and Christmas especially) but my aim is to keep it at least as low as my 2020 average.

One of the downsides to accumulating all this data is that the graphs I've been posting here have become increasingly hard to read. Placing all of the data onto a single graph has become unsustainable, so over the last week I've been updating my graph-generating scripts to make them more flexible. As a result, I'm now going to only show data for the current year on the main waste page. The data for previous years can still be viewed on the pages for 2019 and 2020, and I'll add new pages as the years tick forwards.

I've also created a new page showing the complete data set. These "all-data" graphs are plotted wider now, and while this makes it easier to read the individual entries, it also makes them impractically long and thin. The "fixed in time" preview below already gives an idea of the problem, but the graphs will only get wider, and the issue more accuate, over time. So they're really only going to be of interest for the masochistic.
Daily waste data histocurve snapshot 29/08/2021

While the full-data graph is interesting by virtue of its absurdity, splitting the graph up into annual chunks turns out to be the more interesting case. In particular, because I take readings when I take out the rubbish, these rarely actually fall on the first or last day of the year. So, how to split the readings across the year boundaries?

The solution I've came up with is to scale the readings at each end of the year in proportion to how much of the period falls into the year in question. For example, here are the actual readings I took over the 2020-2021 year boundary.

Date Paper Card Glass Metal Returnables Composts Plastic General
12/12/2020 57 515 0 0 0 449 107 322
14/3/2021 641 225 0 0 93 443 88 473

This covers an unusually long period of time because I was stuck in the UK for January, February and most of March due to Covid travel restrictions. But this is also convenient for making a more exaggerated example. So the period between 12th December and 14th March contains a total of 92 days. That splits into the two periods "12th December - 31st December" and "1st January - 14th March", which contain 20 and 72 days respectively. The proportion of time for each of these periods is therefore 20 / 92 = 21.74% that falls into 2020 and 72 / 92 = 78.26% that falls into 2021.

To manage the data split across the year, we therefore have to scale it appropriately. Each entry represents the end of a period, so the 12th December data falls entirely within 2020. The 14th March data represents the period that's split across both years. We can therefore scale this entry and turn it into two separate entries like this, scaling each of the data points based on the proportions calculated above.
Date Paper Card Glass Metal Returnables Composts Plastic General
31/12/2020 139.35 48.91 0 0 20.22 96.30 19.13 102.83
14/3/2021 501.65 176.09 0 0 72.78 346.70 68.87 370.17

To get the correct picture this has to be done at both ends of the year being plotted.

Managing the data this way makes some obvious assumptions which may not necessarily be true (it assumes I generate waste uniformly across the time period, which is obviously not the case). However it has several nice properties. The annual histograms get drawn in a way that broadly speaking matches up across the year boundary; and the annual averages also match up correctly. At least, it seems to me to be the most honest way to tackle the issue when apportioning the data across year boundaries.

Check back to my waste page over time to see how I'm getting on with keping my waste output down (or not), and whether I'm able to hit my 2021 target.
12 Jun 2021 : A BIOS reset caused my hard drive to disappear #
First of all a little context. My laptop is a Dell XPS 13 9300 (2020 edition), running Ubuntu 20.04 LTS and BIOS version 1.5.0 and with a 1Tb SSD hard drive. I use an encrypted LVM root partition with no Windows partition. I mention all of this in case someone else finds themselves in the same situation as me; maybe they'll find my story helpful.

Just before briefly heading outside yesterday evening (something I've not been doing very often recently) I suspended my laptop. It's something I've been doing every day for the last year without a hitch. When I got back later in the evening I found my laptop couldn't be awoken. Usually just hitting a random key on the keyboard is enough to pull it from its slumbers, but I ended up having to do a 15-second long-press on the power button.

What followed has been a very traumatic 16-hours-worth of frustration and worry. Rather than rebooting normally the laptop immediately dropped into the BIOS "SupportAssist Pre-Boot System Performance Check". It's never done this before so it felt like a bad sign. I left it to run through its checks, taking about 30 minutes to complete. All tests passed. "Phew", I thought.

I rebooted, and it dropped into GRUB. It usually skips GRUB so this was also a concerning sign. Eventually it halted on the Ubuntu pre-boot logo with the error "cryptsetup: Waiting for encrypted source device UUID=574e19ff-e89e-112b-ba3e-76e1-929f4473d12c...". I love the fact my drive is encrypted for security, I hate the worry that comes with knowing that corrupting certain parts can leave all my data unrecoverable.

Hitting a key dropped me to a busybox initram prompt with the following error:
Gave up waiting for root device. Common problems:
  - Boot args (cat /proc/cmdline)
    - Check rootdelay= (did the system wait long enough?)
    - Check root= (did the system wait for the right device?)
  - Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/mapper/ubuntu--vg-root does not exist. Dropping to a shell! 

I scoured the Web last night and into the early hours looking for a solution, and then again for several hours this morning. There's plenty of good advice out there about how to recover from failures involving a nonexistent /dev/mapper/ubuntu--vgroot. In fact I used to get this periodically on another laptop I use, fixed by running fsck from the inetramfs terminal. But this was different. According to the output from lsblk it had literally no hard drive present.

Checking the BIOS, things looked okay, the hard drive was still present and there were no devastating failures shown in the logs. The last firmware update was a couple of months ago. The only odd thing was the following line:
02/03/2021 00:00:12 Time-of-day not set - please run SETUP program.
BIOS System Logs

Well, the current date according to the BIOS was a somewhat inexplicable 02/03/2021 (US-style date, mm/dd/yyyy), even though the date today is 12/6/2021 (European-style dd/mm/yyyy). So this log entry seemed surely to be related to the strange drop into the system check last night. For some reason, something had reset my BIOS settings. Nevertheless I couldn't see anything particularly out of order in the BIOS settings apart from the odd date and time. Neither the BIOS nor the boot shell were offering a way to access the drive any better, so I decided to try gparted from an Ubuntu Live CD.

Luckily I have a second laptop so was able to burn an Ubuntu ISO to USB drive and boot up into a live Ubuntu session. Firing up gparted also showed a complete lack of hard drives present. This didn't make me feel any better. The system check had tested the drive, and it showed up in the BIOS, so I knew it must still be there somewhere, with or without my data.

I tried contacting Dell support. Even though the legal warranty on the device is for two years, my one year support warranty ran out just 36 days ago, so my options were to pay another £109, or wait until Monday when the non-paid support lines opened. I was fairly certain that on Monday they'd tell my I had to stump up for a support contract in order to get help. And that they'd then tell me I had to send back my laptop for investigation and repairs. I really didn't want to lose my laptop for three months on a roundtrip from Finland to the UK and back.

So I checked my server, noted that my last backup was only the day before, and resigned myself to reinstalling everything. Even this felt like a lost cause, given I had no drive to install to. But in a triumph of hope over experience I decided to try it anyway. Inside the live Ubuntu session I clicked on the button to run the install, only after a few steps to be presented with this:
Turn off RST

That looks bad, but it was actually the hint I needed. I dropped back to the BIOS, switched from Intel Rapid Storage Technology (RST) to Advanced Host Controller Interface (AHCI) and rebooted, expecting everything to get wiped in the process.

But lo, it booted fine, found the encrypted volume and requested my passphrase. After offering it up the boot continued and I'm now back in my laptop, no need to re-install everything, and more relieved than I've felt in a long time.

I've grown to rely very heavily on my laptop over the last year due to the lockdown preventing me from travelling. The prospect of losing access to it for potentially months was pretty soul crushing. So I'm hugely relieved.

Trying to make sense of what happened, it seems the BIOS just decided to reset some of its settings for no apparent reason. So as a note to future me, or for anyone else who may find themselves in a similar situation: it can happen, and the solution for me was to disable RST, and re-enable AHCI in the BIOS. When making the switch the BIOS presents an ominous warning that making this switch could prevent the device from booting or even result in wiping the contents. Thankfully that didn't happen to me, but be aware that if you try the same, it may happen to you.