Lunetikk's IT Wiki

After entering your root password and opening “journalctl -xb” you find red entries, something like

kernel: [299102] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
kernel: [299102] ffff880308ed2000: c7 00 00 00 48 89 5c 24 08 48 89 74 24 10 57 48  ....H.\$.H.t$.WH
kernel: [299103] ffff880308ed2010: 83 ec 30 48 8d 54 24 50 e8 03 38 c9 ff 85 c0 0f  ..0H.T$P..8.....
kernel: [299104] ffff880308ed2020: 88 97 00 00 00 48 8b 5c 24 50 48 8d 54 24 58 48  .....H.\$PH.T$XH
kernel: [299104] ffff880308ed2030: 8b cb e8 e1 25 d2 ff 85 c0 78 74 48 8b 7c 24 58  ....%....xtH.|$X
kernel: [299114] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x66/0xc0 [xfs], xfs_inode block 0x13bfa0
kernel: [299115] XFS (dm-2): Unmount and run xfs_repair
kernel: [299115] XFS (dm-2): First 64 bytes of corrupted metadata buffer:
kernel: [299116] ffff880308ed2000: c7 00 00 00 48 89 5c 24 08 48 89 74 24 10 57 48  ....H.\$.H.t$.WH
kernel: [299116] ffff880308ed2010: 83 ec 30 48 8d 54 24 50 e8 03 38 c9 ff 85 c0 0f  ..0H.T$P..8.....
kernel: [299117] ffff880308ed2020: 88 97 00 00 00 48 8b 5c 24 50 48 8d 54 24 58 48  .....H.\$PH.T$XH
kernel: [299117] ffff880308ed2030: 8b cb e8 e1 25 d2 ff 85 c0 78 74 48 8b 7c 24 58  ....%....xtH.|$X
kernel: [299189] XFS (dm-2): metadata I/O error: block 0x13bfa0 ("xfs_trans_read_buf_map") error 117 numblks 16
kernel: [299195] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.
kernel: [299196] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3519 of file ../fs/xfs/xfs_inode.c.  Return address = 0xffffffffa02d4192
kernel: [299200] XFS (dm-2): Corruption of in-memory data detected.  Shutting down filesystem
kernel: [299200] XFS (dm-2): Please umount the filesystem and rectify the problem(s)
kernel: [372139] XFS (dm-2): xfs_log_force: error -5 returned

The server might have KVM installed which messed something up…
Serverfault

Check “journalctl -xb” to find out which LV is corrupted
Get the right LV for “dm-X”

dmsetup info /dev/dm-2

then have a look at your disks

df -h

and mounts

mount

the LV should not be mounted.

Try to repair the filesystem

xfs_repair /dev/mapper/VG02-LVdata

If this fails because of the journal log, try resetting the log

xfs_repair -L /dev/mapper/VG02-LVdata

If completed, rerun the first xfs_repair (without -L) again.
If successful, try to mount the device

mount -a

and check your filesystem

df -h
ll /data

Finally restart your system and pray…

Have a look at this website for more xfs_repair related info
fibrevillage.com - How to repair a xfs filesystem

You can connect to your MySQL with 127.0.0.1 but not with localhost

mysql -h127.0.0.1 -uroot -p
#Welcome to the MySQL monitor...

mysql -hlocalhost -uroot -p
#ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'

MySQL will try to connect to the unix socket if you tell it to connect to localhost. If you tell it to connect to 127.0.0.1 you are forcing it to connect to the network socket.

Verify if the socket is really your problem.

mysql --print-defaults

mysql would have been started with the following arguments:
--port=3306 --socket=/var/run/mysqld/mysqld.sock

mysqld --print-defaults

mysqld would have been started with the following arguments:
--user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306...

Compare the defaults for “socket”, both should be the same. If you get a different socket for your client, try to connect to your database by using the same

mysql --socket=/var/run/mysqld/mysqld.sock -hlocalhost -uroot -p

If that worked, check if you have specified a socket for both, the client and the daemon in your my.cnf

...
[client]
port            = 3306
socket          = /var/run/mysqld/mysqld.sock
...
[mysqld]
port            = 3306
socket          = /var/run/mysqld/mysqld.sock
...

If this is the case, check if your my.cnf is in one of the following folders. The default options are read from these files in the given order:

/etc/mysql/my.cnf 
/etc/my.cnf 
~/.my.cnf       

If you dont have your my.cnf in one of the folders, create a symlink at one of these locations referencing your config file

example:
ln -s /usr/local/mysql/etc/my.cnf /etc/my.cnf

The connection should work now.