After reboot of Ubuntu-2110 on my Hetzner VPS I found myself in a no boot situation.Based on journalctl I had an issue starting a target for swap and boot partitions. It takes 1.5min and fails leaving me in emergency mode.
I checked UUIDs in /etc/fstab against lsblk, and they match.I tried to change UUID to /dev/md/0 but it didn't help either.
My fstab looks like that:
proc /proc proc defaults 0 0#/dev/md/0 none swap sw 0 0UUID=7b5a72f5-8576-4989-acc3-de6f15e14f46 none swap sw 0 0# /dev/md/1UUID=716fc135-1944-4959-9652-c5105c037ec9 /boot ext3 defaults 0 1# /dev/md/2UUID=5644440c-f53c-4a4c-998d-dd9777c2c879 / ext4 defaults 0 0My lsblk look like this
NAME UUIDloop0 1aee0e3e-0e1e-4686-a9a9-61e7152e0209nvme1n1├─nvme1n1p1 0648d0f8-4623-db7a-8fe6-48eee7572c5c│└─md0 7b5a72f5-8576-4989-acc3-de6f15e14f46├─nvme1n1p2 58e884a5-30cd-1afa-4b73-0f2f50f09009│└─md1 716fc135-1944-4959-9652-c5105c037ec9└─nvme1n1p3 586d3257-b7e1-9214-d4cc-6b5f1198c2bf└─md2 5644440c-f53c-4a4c-998d-dd9777c2c879nvme0n1├─nvme0n1p1 0648d0f8-4623-db7a-8fe6-48eee7572c5c│└─md0 7b5a72f5-8576-4989-acc3-de6f15e14f46├─nvme0n1p2 58e884a5-30cd-1afa-4b73-0f2f50f09009│└─md1 716fc135-1944-4959-9652-c5105c037ec9└─nvme0n1p3 586d3257-b7e1-9214-d4cc-6b5f1198c2bf└─md2 5644440c-f53c-4a4c-998d-dd9777c2c879I have no problem mounting for example /boot in emergency mode, and my / is even auto-mounted properly, yet if I would try to systemctl default it falls right back into emergency mode.
Some highlights from journal:
---This one may be related to what causes it (more of it later)Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-5.13.0-21-generic root=UUID=5644440c-f53c-4a4c-998d-dd9777c2c879 ro consoleblank=>Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: You have booted with nomodeset. This means your GPU drivers are DISABLEDMar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: Any video related functionality will be severely degraded, and you may not even be able to suspend the system properlyMar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: Unless you actually understand what nomodeset does, you should reboot without enabling it---Googling says that part should be harmlessMar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Probing EISA bus 0Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: EISA: Cannot allocate resource for mainboardMar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Cannot allocate resource for EISA slot 1Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Cannot allocate resource for EISA slot 2Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Cannot allocate resource for EISA slot 3Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Cannot allocate resource for EISA slot 4Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Cannot allocate resource for EISA slot 5Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Cannot allocate resource for EISA slot 6Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Cannot allocate resource for EISA slot 7Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Cannot allocate resource for EISA slot 8Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: EISA: Detected 0 cards---This part seems to be related to floppy only and harmless, but the last string bothers me. Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: floppy0: disk absent or changed during operationMar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: blk_update_request: I/O error, dev fd0, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: floppy0: disk absent or changed during operationMar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: blk_update_request: I/O error, dev fd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: Buffer I/O error on dev fd0, logical block 0, async page readMar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: EXT4-fs (md2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base systemd[1]: Inserted module 'autofs4'---And here is where it seems for me to failMar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: dev-disk-by\x2duuid-716fc135\x2d1944\x2d4959\x2d9652\x2dc5105c037ec9.device: Job dev-disk-by\x2duuid-716fc135\x2d1944\>Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: Timed out waiting for device /dev/disk/by-uuid/716fc135-1944-4959-9652-c5105c037ec9.Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: Dependency failed for File System Check on /dev/disk/by-uuid/716fc135-1944-4959-9652-c5105c037ec9.Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: Dependency failed for /boot.Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: Dependency failed for Local File Systems.Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: local-fs.target: Job local-fs.target/start failed with result 'dependency'.Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: local-fs.target: Triggering OnFailure= dependencies.Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: boot.mount: Job boot.mount/start failed with result 'dependency'.Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: systemd-fsck@dev-disk-by\x2duuid-716fc135\x2d1944\x2d4959\x2d9652\x2dc5105c037ec9.service: Job systemd-fsck@dev-disk-b>Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: dev-disk-by\x2duuid-716fc135\x2d1944\x2d4959\x2d9652\x2dc5105c037ec9.device: Job dev-disk-by\x2duuid-716fc135\x2d1944\>Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: dev-disk-by\x2duuid-7b5a72f5\x2d8576\x2d4989\x2dacc3\x2dde6f15e14f46.device: Job dev-disk-by\x2duuid-7b5a72f5\x2d8576\>Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: Timed out waiting for device /dev/disk/by-uuid/7b5a72f5-8576-4989-acc3-de6f15e14f46.Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: Dependency failed for /dev/disk/by-uuid/7b5a72f5-8576-4989-acc3-de6f15e14f46.Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: Dependency failed for Swap.Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: swap.target: Job swap.target/start failed with result 'dependency'.Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: dev-disk-by\x2duuid-7b5a72f5\x2d8576\x2d4989\x2dacc3\x2dde6f15e14f46.swap: Job dev-disk-by\x2duuid-7b5a72f5\x2d8576\x2>Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: dev-disk-by\x2duuid-7b5a72f5\x2d8576\x2d4989\x2dacc3\x2dde6f15e14f46.device: Job dev-disk-by\x2duuid-7b5a72f5\x2d8576\>As for what probably caused it all to happen I was trying to run a headless server for dominions5, so I installed it, tried to run, it missed some libs, so without second thought I just installed a couple of packets namely libglu1 and libsdl2-dev, and only after that I realized that I'm trying to start the server without propper option, and without them, I just trying to run a graphical binary, and I don't really need those libs on VDS. It may be the cause, but I'm not sure, something else may be the cause as well the server had a long uptime.
I removed those packets, and it didn't help, I also switched default systemd target from graphical.target to multi-user.target but it didn't change much.
I also checked SMART and all disks are fine.
So If anyone have any Ideas about it - I would be very grateful.