===== Troubleshooting =====

==== Removing old kernels leads to broken symlinks ====

=== Description ===

apt-get autoremove leads to a broken symlink which requires a reload of grub

<code>
apt-get autoremove
...
The link /vmlinuz.old is a damaged link
Removing symbolic link vmlinuz.old
 you may need to re-run your boot loader[grub]
The link /initrd.img.old is a damaged link
Removing symbolic link initrd.img.old
 you may need to re-run your boot loader[grub]
</code>

=== Reason === 

Broken symlinks

=== Fix === 

Run "update-grub"
<code>
update-grub
 Generating grub configuration file ...
 Found linux image: /boot/vmlinuz-3.13.0-157-generic
 Found initrd image: /boot/initrd.img-3.13.0-157-generic
 Found linux image: /boot/vmlinuz-3.13.0-153-generic
 Found initrd image: /boot/initrd.img-3.13.0-153-generic
 Found memtest86+ image: /boot/memtest86+.elf
 Found memtest86+ image: /boot/memtest86+.bin
 done
</code>

\\
\\

==== Linux starts in emergency mode - faulty logical volume (xfs) ====

=== Description ===

{{:linux:general:linuxemergencymode.png|}}

After entering your root password and opening "journalctl -xb" you find red entries, something like

<code>
kernel: [299102] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
kernel: [299102] ffff880308ed2000: c7 00 00 00 48 89 5c 24 08 48 89 74 24 10 57 48  ....H.\$.H.t$.WH
kernel: [299103] ffff880308ed2010: 83 ec 30 48 8d 54 24 50 e8 03 38 c9 ff 85 c0 0f  ..0H.T$P..8.....
kernel: [299104] ffff880308ed2020: 88 97 00 00 00 48 8b 5c 24 50 48 8d 54 24 58 48  .....H.\$PH.T$XH
kernel: [299104] ffff880308ed2030: 8b cb e8 e1 25 d2 ff 85 c0 78 74 48 8b 7c 24 58  ....%....xtH.|$X
kernel: [299114] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x66/0xc0 [xfs], xfs_inode block 0x13bfa0
kernel: [299115] XFS (dm-2): Unmount and run xfs_repair
kernel: [299115] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
kernel: [299116] ffff880308ed2000: c7 00 00 00 48 89 5c 24 08 48 89 74 24 10 57 48  ....H.\$.H.t$.WH
kernel: [299116] ffff880308ed2010: 83 ec 30 48 8d 54 24 50 e8 03 38 c9 ff 85 c0 0f  ..0H.T$P..8.....
kernel: [299117] ffff880308ed2020: 88 97 00 00 00 48 8b 5c 24 50 48 8d 54 24 58 48  .....H.\$PH.T$XH
kernel: [299117] ffff880308ed2030: 8b cb e8 e1 25 d2 ff 85 c0 78 74 48 8b 7c 24 58  ....%....xtH.|$X
kernel: [299189] XFS (dm-2): metadata I/O error: block 0x13bfa0 ("xfs_trans_read_buf_map") error 117 numblks 16
kernel: [299195] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.
kernel: [299196] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3519 of file ../fs/xfs/xfs_inode.c.  Return address = 0xffffffffa02d4192
kernel: [299200] XFS (dm-2): Corruption of in-memory data detected.  Shutting down filesystem
kernel: [299200] XFS (dm-2): Please umount the filesystem and rectify the problem(s)
kernel: [372139] XFS (dm-2): xfs_log_force: error -5 returned
</code>

=== Reason === 
The server might have KVM installed which messed something up...\\
[[https://serverfault.com/questions/897842/corruption-of-in-memory-data-detected-where-does-the-issue-lie|Serverfault]]
\\

=== Fix === 
Check "journalctl -xb" to find out which LV is corrupted\\
Get the right LV for "dm-X" <code>dmsetup info /dev/dm-2</code>then have a look at your disks 
<code>df -h</code> and mounts <code>mount</code>the LV should not be mounted.\\

Try to repair the filesystem
<code>xfs_repair /dev/mapper/VG02-LVdata</code>
If this fails because of the journal log, try resetting the log
<code>xfs_repair -L /dev/mapper/VG02-LVdata</code>
If completed, rerun the first xfs_repair (without -L) again.\\
If successful, try to mount the device
<code>mount -a</code>
and check your filesystem
<code>
df -h
ll /data
</code>

Finally restart your system and pray...

\\
\\
==== ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' ====

=== Description ===

You can connect to your MySQL with 127.0.0.1 but not with localhost

<code>
mysql -h127.0.0.1 -uroot -p
#Welcome to the MySQL monitor...

mysql -hlocalhost -uroot -p
#ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'
</code>

=== Reason === 

MySQL will try to connect to the unix socket if you tell it to connect to localhost. If you tell it to connect to 127.0.0.1 you are forcing it to connect to the network socket.

=== Fix === 

Verify if the socket is really your problem. 

<code>
mysql --print-defaults

mysql would have been started with the following arguments:
--port=3306 --socket=/var/run/mysqld/mysqld.sock
</code>
<code>
mysqld --print-defaults

mysqld would have been started with the following arguments:
--user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306...
</code>

Compare the defaults for "socket", both should be the same. If you get a different socket for your client, try to connect to your database by using the same as specified for mysqld
<code>mysql --socket=/var/run/mysqld/mysqld.sock -hlocalhost -uroot -p</code>

If that worked, check if you have specified a socket for both, the client and the daemon in your my.cnf
<code>
...
[client]
port            = 3306
socket          = /var/run/mysqld/mysqld.sock
...
[mysqld]
port            = 3306
socket          = /var/run/mysqld/mysqld.sock
...
</code>

If this is the case, check if your my.cnf is in one of the following folders. The default options are read from these files in the given order:
<code>
/etc/mysql/my.cnf 
/etc/my.cnf 
~/.my.cnf       
</code>

If you dont have your my.cnf in one of the folders, create a symlink at one of these locations referencing your config file
<code>
example:
ln -s /usr/local/mysql/etc/my.cnf /etc/my.cnf
</code>

The connection should work now. 
\\
\\

==== Bug: soft lockup in messages ====

=== Description ===

You can find multiple "Bug: soft lockup" entries in /var/log/messages or journalctl

<code>
May 25 07:23:59 XXXXXXX kernel: [13445315.881356] BUG: soft lockup - CPU#16 stuck for 23s! [yyyyyyy:81602]
</code>

=== Reason === 

>A 'soft lockup' is defined as a bug that causes the kernel to loop in kernel mode for more than 20 seconds without giving other tasks a chance to run. The watchdog daemon will send an non-maskable interrupt (NMI) to all CPUs in the system who, in turn, print the stack traces of their currently running tasks. 
-SUSE KB [[https://www.suse.com/support/kb/doc/?id=7017652|7017652]]

=== Fix === 

__Solution 1:__

Restart your system and/ or decrease your CPU load.

__Solution 2:__

Increase the time (default 10) before soft lockups are fired.

<code bash >echo 20 > /proc/sys/kernel/watchdog_thresh</code> 
or
<code bash>
echo "kernel.watchdog_thresh=20" > /etc/sysctl.d/99-watchdog_thresh.conf

sysctl -p  /etc/sysctl.d/99-watchdog_thresh.conf
</code>
\\
\\


==== systemctl runs in timeout ====

=== Description ===

In this example, installation of docker-ce with the following command doesnt work
<code>
curl -sSL https://get.docker.com | sh

# Executing docker install script, commit: f45d7c11389849ff46a6b4d94e0dd1ffebca32c1
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | apt-key add -qq - >/dev/null
+ sh -c echo "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable" > /etc/apt/sources.list.d/docker.list
+ sh -c apt-get update -qq >/dev/null
+ [ -n  ]
+ sh -c apt-get install -y -qq --no-install-recommends docker-ce >/dev/null

Broadcast message from systemd-journald@lunetikk (Wed 2019-10-23 00:22:12 CEST):

systemd[1]: Caught <SEGV>, dumped core as pid 26368.


Broadcast message from systemd-journald@lunetikk (Wed 2019-10-23 00:22:12 CEST):

systemd[1]: Freezing execution.

E: Sub-process /usr/bin/dpkg returned an error code (1)
</code>

Rerun "apt-get install docker-ce" shows the following
<code>
apt-get install docker-ce
Reading package lists... Done
Building dependency tree
Reading state information... Done
docker-ce is already the newest version (5:19.03.4~3-0~ubuntu-xenial).
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Setting up docker-ce (5:19.03.4~3-0~ubuntu-xenial) ...
Failed to execute operation: Connection timed out
Failed to execute operation: Connection timed out
Failed to retrieve unit state: Connection timed out
Failed to start docker.service: Connection timed out
See system logs and 'systemctl status docker.service' for details.
invoke-rc.d: initscript docker, action "start" failed.
Failed to get properties: Connection timed out
dpkg: error processing package docker-ce (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 docker-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)
</code>

You cant reconfigure
<code>
dpkg-reconfigure docker-ce
/usr/sbin/dpkg-reconfigure: docker-ce is broken or not fully installed
</code>

Listing the units for "systemctl status" runs in timeout
<code>
systemctl status docke<TAB>
Failed to list unit files: Connection timed out
Failed to list units: Connection timed out
Failed to list unit files: Connection timed out
</code>

=== Reason ===  
\\

In my case, my disk was "inconsistent". Reboot got me stuck in busybox. \\

{{:linux:general:pasted:20191023-005512.png}}\\

=== Fix ===  
\\

I was able to run "fsck.ext4 /dev/vda2" to fix the orphaned inodes

{{:linux:general:pasted:20191023-005629.png}}

{{:linux:general:pasted:20191023-005648.png}}

Reboot after this got me back onto my system and "systemctl" was working again.

\\