linuxmemes

Not fixed but there is an Arch problem that is and will always be the bane of mi existence.

For some reason when I click with the trackpad buttons the touchpad gets frozen for like a second (it's like they are recognised by the system as keyboard buttons, I have enabled that option to temporarily disable it when using keyboard).

I've checked for hours and days the libinput documentation and some synaptics libraries, even legacy ones. It is to this day the only problem that has lead me to reinstall my system but the problem remains.

It's not even like I have some niche setup, I mean, surely there must be thousands of Arch users running with a ThinkPad X1 Carbon Gen 7, and surely not every single one of them must be running it like this, right?

It has come to a point where I just gave up and got used to my system as is, but I'm sure I would be running fanfare if some day I am able to fix it.

Using Linux on a GTX660 without proprietary drivers. I never managed to succeed. Desktop would always freeze. Never again.

idk how i would define difficult, but the thing i probably put the most time into figuring out thus far is LXC containers.

Or LXC, if you like not using redundant acronyms. Those containers are good shit, weird shit, but good shit nonetheless.

Are you including back in the day when we had to use windows device drivers via ndiswrappers?

I've managed to remove a critical library once but did manage to extract it from an RPM on another machine and manually install it. That was good enough to get me to the point where I could yum reinstall.

Pre-linux we had an HP workstation where the disc drive died and of course we had no backups. I managed to frankenstein the disc by connecting the platters on the broken disc to the circuit board of a working disc. This worked and I was able to back up the disk and reload on to a new drive.

And then we bought an 8mm tape drive for backups and I had to port some drivers to HP-UX to get it to work. But we had awesome backups after that!

I managed a CentOS system where someone accidentally deleted everything from /usr, so no lib64, and no bin. I didn't have a way to get proper files at the time, so I hooked the drive up to my Arch system, made sure glibc matched, and copied yum and other tools from Arch.

Booted the system, reinstalled a whole lot of yum packages, and... the thing still worked.

That's almost equivalent to a reinstall, though. As a broke college student, I had a laptop with a loose drive, that would fall out very easily. I set it up to load a few crucial things into a ramdisk at boot, so that I could browse the web and take notes even if the drive was disconnected, and it would still load images and things. I could pull the cover off and push the drive back in place to save files, but doing that every time I had class got really tiring, so I wanted it to run a little like a live system.

More than a decade ago a user came into #ubuntu-server on Freenode (now libera.chat ) and said that they had accidentally run "rm -rf /* something*" in a root shell.

Note the errant space that made that a fatal mistake. I don't remember how far it actually got in deleting files, but all of /bin/ /sbin/ and /usr/ were gone.

He had 1 active ssh connection, and couldn't start another one.

It was a server that was "in production", was thousands of miles away from him, and which had no possibility for IPMI / remote hands.

Everyone (but me) in the channel said that he was just SoL and should just give up.

I stayed up most of the night helping him. I like challenges and I like helping people.

This was in the sysv-init (maybe upstart) days, and so a decent number of shell scripts were running, and using basic *nix commands.

We recovered the bash binary by running something along the lines of

bash_binary_contents="$( </proc/self/exe)"
printf "%s" > /tmp/bash

(If you can access "lsof" then "sudo lsof | grep deleted" will show you any files that are open, but also "deleted". You may be surprised at how many there are!)

But bash needed too many shared libraries to make that practical.

Somehow we were able to recover curl and chmod, after which I had him download busybox-static. From there we downloaded an Ubuntu LiveCD iso, loop mounted it, loop mounted the squashfs image inside the iso, and copied all of /bin/ , /sbin/ , /etc , and so on from there onto his root FS.

Then we re-installed missing packages, fixed up /etc/ (a lot of important daemons, including the one that was production critical, kept their configuration files open, and so we were able to use lsof to find the magic symlinks to them in /proc/$pid/fd/ and just cp them back into /etc/.

We were able to restart openssh-server, log in again, and I don't remember if we were brave enough to test rebooting.

But we fucking did it!

I am certainly getting a lot of details wrong from memory. It's all somewhere at irclogs.ubuntu.com though. My nick was / is Jordan_U.

I tried to find it once, and failed.

I used to main Gentoo.

Breaking the install was more of a guarantee.

I once removed most of X by trying to remove Gnome dependencies and it lead to an interesting couple of hours but I did have a working system when I was done.

There were countless dependency bugs and broken systems but at least I learned how to use the Gentoo Forum and also a lot of how Linux works.

I kind of want to give it another go.

I did a partial system upgrade when installing nginx without upgrading the rest of my Arch system. One of the things it upgraded was libssl.

Turns out systemd depends on that.

Turns out programs won't start at all if one of their shared libraries is missing.

Turns out that if you write init=bash in the kernel command line, not even Ethernet connections work if systemd isn't running.

I had to boot off archiso, chroot into my / partition, and run the system upgrade from there.

A Gentoo upgrade package list with over 100 packages and conflicts all over the place. Then do it again when the list grows to the same size in a few months.

This is why I don't use Gentoo anymore.

Some programs still relying on python2 when the operating system has long since upgraded to python3.

Not really an issue per se, I just had to switch those apps over to using the flatpak version which would have it installed as needed. (I'm looking at you GIMP)

Rescuing home partition from ZFS, actually that doesn't really count since I did have to reinstall (was no longer booting), but recovering the Home partition from ZFS and to the other ext4 drive was much harder than it should've been and that's why I would never recommend people use ZFS.

Nvidia driver fucking X in the arse without lube.

Installing a hadoop cluster across 5 machines. I wouldn't say I fixed it, but I made it so it wouldn't collapse until long after I'd left that company.

Not Linux, but Solaris, back in the day.

We had a system with a mirrored boot disk. One of the disks failed. And we were unable to boot from the other, because the boot device in OBP (~BIOS) pointed to a device-specific partitIon. When we manually booted from the live device, it was lacking the boot sector code, and wouldn't boot. When we booted from CDROM, the partitions wouldn't mount because the virtual device mapping pointed to the dead drive.

This was a gas futures trading system, and rebuild wasn't an option. Restoring from backup woyld have lost four hours of trades, which would be an extreme last resort.

A coworker and I spent all night on the box. We had a whiteboard covered with every stage of the boot sequence broken down, and every redirection we needed to (a) boot and (b) repair the system. The issue started mid-afternoon, and we finally got it back up by around 6:30 am.

Fast data transmission via TCP over a lossy link.

Nothing. I've fixed nothing.

windows update kept downloading these bloated "updates" that included brand new software that I didn't want or use, broke my settings, added a bunch of spyware, adware and other shit and slowed down my system

installing linux fixed that instantly and permanently

Removed the libc by hand, and restored the system to a usable state without turning it off and putting the file back on the FS from external source.

Hmm I have come up with a bunch of neat solutions over the years. Where to start?

One time I broke the sudoers file on a distro without a root account, thoroughly locking myself out. I used docker -v /:/chroot to get myself root access to my root filesystem where I fixed the sudoers file. Protip always use visudo

I feel seen here, I was building a Ubuntu server and messed up the firewall settings not being able to get an internet connection, hours of trying to get back to where I was I gave up and plan to just start from scratch next time.

Is there a way of taking system snapshots with Linux?