List items
Items from the current list are shown below.
Blog
All items from June 2021
12 Jun 2021 : A BIOS reset caused my hard drive to disappear #
First of all a little context. My laptop is a Dell XPS 13 9300 (2020 edition), running Ubuntu 20.04 LTS and BIOS version 1.5.0 and with a 1Tb SSD hard drive. I use an encrypted LVM root partition with no Windows partition. I mention all of this in case someone else finds themselves in the same situation as me; maybe they'll find my story helpful.
Just before briefly heading outside yesterday evening (something I've not been doing very often recently) I suspended my laptop. It's something I've been doing every day for the last year without a hitch. When I got back later in the evening I found my laptop couldn't be awoken. Usually just hitting a random key on the keyboard is enough to pull it from its slumbers, but I ended up having to do a 15-second long-press on the power button.
What followed has been a very traumatic 16-hours-worth of frustration and worry. Rather than rebooting normally the laptop immediately dropped into the BIOS "SupportAssist Pre-Boot System Performance Check". It's never done this before so it felt like a bad sign. I left it to run through its checks, taking about 30 minutes to complete. All tests passed. "Phew", I thought.
I rebooted, and it dropped into GRUB. It usually skips GRUB so this was also a concerning sign. Eventually it halted on the Ubuntu pre-boot logo with the error "cryptsetup: Waiting for encrypted source device UUID=574e19ff-e89e-112b-ba3e-76e1-929f4473d12c...". I love the fact my drive is encrypted for security, I hate the worry that comes with knowing that corrupting certain parts can leave all my data unrecoverable.
Hitting a key dropped me to a busybox initram prompt with the following error:
I scoured the Web last night and into the early hours looking for a solution, and then again for several hours this morning. There's plenty of good advice out there about how to recover from failures involving a nonexistent /dev/mapper/ubuntu--vgroot. In fact I used to get this periodically on another laptop I use, fixed by running fsck from the inetramfs terminal. But this was different. According to the output from lsblk it had literally no hard drive present.
Checking the BIOS, things looked okay, the hard drive was still present and there were no devastating failures shown in the logs. The last firmware update was a couple of months ago. The only odd thing was the following line:
Well, the current date according to the BIOS was a somewhat inexplicable 02/03/2021 (US-style date, mm/dd/yyyy), even though the date today is 12/6/2021 (European-style dd/mm/yyyy). So this log entry seemed surely to be related to the strange drop into the system check last night. For some reason, something had reset my BIOS settings. Nevertheless I couldn't see anything particularly out of order in the BIOS settings apart from the odd date and time. Neither the BIOS nor the boot shell were offering a way to access the drive any better, so I decided to try gparted from an Ubuntu Live CD.
Luckily I have a second laptop so was able to burn an Ubuntu ISO to USB drive and boot up into a live Ubuntu session. Firing up gparted also showed a complete lack of hard drives present. This didn't make me feel any better. The system check had tested the drive, and it showed up in the BIOS, so I knew it must still be there somewhere, with or without my data.
I tried contacting Dell support. Even though the legal warranty on the device is for two years, my one year support warranty ran out just 36 days ago, so my options were to pay another £109, or wait until Monday when the non-paid support lines opened. I was fairly certain that on Monday they'd tell my I had to stump up for a support contract in order to get help. And that they'd then tell me I had to send back my laptop for investigation and repairs. I really didn't want to lose my laptop for three months on a roundtrip from Finland to the UK and back.
So I checked my server, noted that my last backup was only the day before, and resigned myself to reinstalling everything. Even this felt like a lost cause, given I had no drive to install to. But in a triumph of hope over experience I decided to try it anyway. Inside the live Ubuntu session I clicked on the button to run the install, only after a few steps to be presented with this:
That looks bad, but it was actually the hint I needed. I dropped back to the BIOS, switched from Intel Rapid Storage Technology (RST) to Advanced Host Controller Interface (AHCI) and rebooted, expecting everything to get wiped in the process.
But lo, it booted fine, found the encrypted volume and requested my passphrase. After offering it up the boot continued and I'm now back in my laptop, no need to re-install everything, and more relieved than I've felt in a long time.
I've grown to rely very heavily on my laptop over the last year due to the lockdown preventing me from travelling. The prospect of losing access to it for potentially months was pretty soul crushing. So I'm hugely relieved.
Trying to make sense of what happened, it seems the BIOS just decided to reset some of its settings for no apparent reason. So as a note to future me, or for anyone else who may find themselves in a similar situation: it can happen, and the solution for me was to disable RST, and re-enable AHCI in the BIOS. When making the switch the BIOS presents an ominous warning that making this switch could prevent the device from booting or even result in wiping the contents. Thankfully that didn't happen to me, but be aware that if you try the same, it may happen to you.
Comment
Just before briefly heading outside yesterday evening (something I've not been doing very often recently) I suspended my laptop. It's something I've been doing every day for the last year without a hitch. When I got back later in the evening I found my laptop couldn't be awoken. Usually just hitting a random key on the keyboard is enough to pull it from its slumbers, but I ended up having to do a 15-second long-press on the power button.
What followed has been a very traumatic 16-hours-worth of frustration and worry. Rather than rebooting normally the laptop immediately dropped into the BIOS "SupportAssist Pre-Boot System Performance Check". It's never done this before so it felt like a bad sign. I left it to run through its checks, taking about 30 minutes to complete. All tests passed. "Phew", I thought.
I rebooted, and it dropped into GRUB. It usually skips GRUB so this was also a concerning sign. Eventually it halted on the Ubuntu pre-boot logo with the error "cryptsetup: Waiting for encrypted source device UUID=574e19ff-e89e-112b-ba3e-76e1-929f4473d12c...". I love the fact my drive is encrypted for security, I hate the worry that comes with knowing that corrupting certain parts can leave all my data unrecoverable.
Hitting a key dropped me to a busybox initram prompt with the following error:
Gave up waiting for root device. Common problems: - Boot args (cat /proc/cmdline) - Check rootdelay= (did the system wait long enough?) - Check root= (did the system wait for the right device?) - Missing modules (cat /proc/modules; ls /dev) ALERT! /dev/mapper/ubuntu--vg-root does not exist. Dropping to a shell!
I scoured the Web last night and into the early hours looking for a solution, and then again for several hours this morning. There's plenty of good advice out there about how to recover from failures involving a nonexistent /dev/mapper/ubuntu--vgroot. In fact I used to get this periodically on another laptop I use, fixed by running fsck from the inetramfs terminal. But this was different. According to the output from lsblk it had literally no hard drive present.
Checking the BIOS, things looked okay, the hard drive was still present and there were no devastating failures shown in the logs. The last firmware update was a couple of months ago. The only odd thing was the following line:
02/03/2021 00:00:12 Time-of-day not set - please run SETUP program.
Well, the current date according to the BIOS was a somewhat inexplicable 02/03/2021 (US-style date, mm/dd/yyyy), even though the date today is 12/6/2021 (European-style dd/mm/yyyy). So this log entry seemed surely to be related to the strange drop into the system check last night. For some reason, something had reset my BIOS settings. Nevertheless I couldn't see anything particularly out of order in the BIOS settings apart from the odd date and time. Neither the BIOS nor the boot shell were offering a way to access the drive any better, so I decided to try gparted from an Ubuntu Live CD.
Luckily I have a second laptop so was able to burn an Ubuntu ISO to USB drive and boot up into a live Ubuntu session. Firing up gparted also showed a complete lack of hard drives present. This didn't make me feel any better. The system check had tested the drive, and it showed up in the BIOS, so I knew it must still be there somewhere, with or without my data.
I tried contacting Dell support. Even though the legal warranty on the device is for two years, my one year support warranty ran out just 36 days ago, so my options were to pay another £109, or wait until Monday when the non-paid support lines opened. I was fairly certain that on Monday they'd tell my I had to stump up for a support contract in order to get help. And that they'd then tell me I had to send back my laptop for investigation and repairs. I really didn't want to lose my laptop for three months on a roundtrip from Finland to the UK and back.
So I checked my server, noted that my last backup was only the day before, and resigned myself to reinstalling everything. Even this felt like a lost cause, given I had no drive to install to. But in a triumph of hope over experience I decided to try it anyway. Inside the live Ubuntu session I clicked on the button to run the install, only after a few steps to be presented with this:
That looks bad, but it was actually the hint I needed. I dropped back to the BIOS, switched from Intel Rapid Storage Technology (RST) to Advanced Host Controller Interface (AHCI) and rebooted, expecting everything to get wiped in the process.
But lo, it booted fine, found the encrypted volume and requested my passphrase. After offering it up the boot continued and I'm now back in my laptop, no need to re-install everything, and more relieved than I've felt in a long time.
I've grown to rely very heavily on my laptop over the last year due to the lockdown preventing me from travelling. The prospect of losing access to it for potentially months was pretty soul crushing. So I'm hugely relieved.
Trying to make sense of what happened, it seems the BIOS just decided to reset some of its settings for no apparent reason. So as a note to future me, or for anyone else who may find themselves in a similar situation: it can happen, and the solution for me was to disable RST, and re-enable AHCI in the BIOS. When making the switch the BIOS presents an ominous warning that making this switch could prevent the device from booting or even result in wiping the contents. Thankfully that didn't happen to me, but be aware that if you try the same, it may happen to you.