Do not Panic! Remote Server (Hetzner) not rebooting any more – A Solution

GDFL 1.2
GDFL 1.2

I went through this experience recently. First of all, don’t panic! I panicked, and because of this, I made a mistake: I didn’t wait long enough for it to come online. Had I waited up to 60 minutes, it would probably have come online (see reason below). The story:

I had broken packages on my Ubuntu 10.04 server and decided to fix them by

While updating, I noticed that the package grub-pc  also was upgraded, apparently a new bootloader (or bootloader configuration) was installed. This made me feel uncomfortable, since I didn’t know if the server would reboot after this upgrade. So, because of the saying “The devil you know is better than the devil you don’t know” (and a desire to sleep peacefully at night!) I decided to reboot the server and see what would happen. To my big dismay, it did not come online. SSH connections failed with "Port 22: Connection refused" .

I panicked and asked the Hetzner Support (which is very responsive and supportive btw!) to install a LARA Remote Console so that I could see the text output of the booting screen. After some regular startup text, the screen became blank. I panicked even more and took immediate steps to move all data to a new server. It took me 6 hours to complete the most important parts, and another full week of restless work to finish it. We will see that it was not neccessary.

Most regular servers at the Hetzner datacenter are running Software RAID. It seems that after a reboot (especially if you send a Hardware Reset) the OS needs some time to re-sync or check the file system. I am not sure what caused the delay in my case. Re-syncing the entire RAID array can take up to 1-2 hours, depending on your hardware and disc space.

So, wait at least 1 hour for it to come online, especially after a hardware reset! When you activate Hetzner’s Rescue system (which is very good btw!) it will stay active for a minimum of 1 hour, so your server will be down for 1 hour at least, in any event. So you are not losing much by waiting a bit longer.

Now, in my case, I assumed that Grub2 was broken. So I activated the Hetzner Rescue System, booted into it, and reinstalled Grub2. I have found the following method here and it worked for me. First you have to mount the regular RAID filesystem under /mnt :

At this point, you are in your regular root directory. To reinstall Grub2 the Debian Way, I did:

To make really sure, I reconfigured the package:

It will ask you where to install the bootloader. I selected:

No errors were reported. I rebooted again and it did not come online immediately, for the reasons previously mentioned. I waited long enough (in my case, 15 minutes) and it did come online. So, rule number one is: Don’t panic!

A solution for MySQL Assertion failure FIL_NULL

Mysql screenshot

A defective RAM module recently caused data corruption in MySQL tables. MySQL would log the following to /var/log/syslog  in regular intervals, about every few minutes:

Reading MySQL documentation and various blogs didn’t help much. I ran CHECK TABLES  on all the tables and they all reported OK. Then I ran

and still all tables reported OK. Nevertheless the Assertion Failures continued. Then I stumbled across this excellent blog post, which suggested to dump evertying into a .sql file, wipe /var/lib/mysql (not without making a backup, mind you!), and re-import everything from scratch. This is what I did and it worked.

I recorded the passwords for each database and collected it into a SQL script like this:

When you dump all databases into a .sql file, it will not dump the permissions, so you will need to restore them later with this script. Next, the dumping part, then removal and reinstallation of mysql (Danger here: When you remove mysql-server, all packges which depend on it also will be removed!):

Here, I had to reset the MySQL admin password because it didn’t work any more, so I ran:

Then, I re-imported all databases from the dump file:

Then I run the SQL permission script that I mention above.

For me, this resulted in no more Assertion Failures. Yay!

 

 

Adventures with various Segfaults due to defective RAM

Crash

If you get various segfaults on your Linux server, like these:

or:

or

or

etc. etc., then, no, your system is not suddenly crazy. Nor are you. It is highly likely that you RAM is defective. You should reboot your server and run the  RAM test from your boot manager (Grub always has such a test) to see if it can detect faulty RAM.

If you are operating a server that you can’t reboot because you can’t tolerate downtime, there is an excellent tool called memtester , which is a memory test for a running system. It is part of the Debian distribution, installit with apt-get install memtester  Check top to see how much free RAM there is available. Say you have 10GB RAM free, then ask memterst to test 8GB of it (so that 2GB are remaining free for the running system to operate). In my case, memtester indeed detected faults.

I ran

It outputted stuff like this:

So, when I replaced the RAM, the Segfaults stopped. You can run memtester  regularly to make sure the RAM is okay. Healty RAM is a very crucial part of your successful hosting operation!

In my case however, the segfaults corrupted MySQL tables, which I had to clean up. All’s well that ends well!

 

How to install Guest Additions with Folder Sharing on VirtualBox – The Debian Way

Debian Wheezy as guest Operating System with Folder Sharing

I always forget where to download the proper version of the Virtualbox GuestAdditions, so here is a short tutorial about how to do it quickly and easily in the Debian way. I always need folder sharing between the guest and host OS, so you will see how this is done also. I will be using the current stable distribution Debian Wheezy as host system (without a running display manager, just the console), and  as the guest system also Debian Wheezy. It turns out that Debian has even packaged the Guest Additions for virtualbox.

First, install Virtualbox and the Guest Additions:

Then install Debian Wheezy as a guest system. Next, mount the Guest Additions ISO file by navigating through the menu system of Virtualbox:

  • Devices -> CD/DVD Drives -> Coose a virtual CD/DVD disk file …

Choose the ISO from this path:

Next, in your guest OS, mount the virtual CD, and run the installer:

Next shut down your guest OS (don’t to this in your host OS 😉 ):

Now set up the shared folder in the menu system of Virtualbox:

  • Settings -> Shared Folders -> +
  • Select Folder
  • Check “Auto-mount”

Re-start your guest OS, and you will have the shared folder auto-mounted in /media/sf_Public . Voila!

File permissions for successful SSH login via authorized_keys

Setting correct file permissions for ssh is a bit tricky

If you want to ssh into your server without being repeatedly prompted for the password you can copy your public ssh key into a file called authorized_keys  in the .ssh subdirectory of the home directory of the remove server account. However, this works only if the permissions for this file are set correctly.

First, if you have not done so already, generate the public key for your local user:

 

This will create a file ~/.ssh/id_rsa.pub

Append the only line in this file into the file ~/.ssh/authorized_keys  of the remote user account. Create the directory and file if it does not exist.

Now try to ssh into your remote account. If ssh is still asking for the remote user’s password, check the permissions of the following files and directories:

  • The permissions of the home directory of the remote user must be 755
  • The permissions of the remote .ssh directory must be 700
  • The permissions of the remote authorized_keys file must be 600

… of course all of those must be owned by the remote user, and not by root.

Now, you should be able to ssh into the remote account without being asked for the password!