Do not Panic! Remote Server (Hetzner) not rebooting any more – A Solution

GDFL 1.2

GDFL 1.2

I went through this experience recently. First of all, don’t panic! I panicked, and because of this, I made a mistake: I didn’t wait long enough for it to come online. Had I waited up to 60 minutes, it would probably have come online (see reason below). The story:

I had broken packages on my Ubuntu 10.04 server and decided to fix them by

While updating, I noticed that the package grub-pc  also was upgraded, apparently a new bootloader (or bootloader configuration) was installed. This made me feel uncomfortable, since I didn’t know if the server would reboot after this upgrade. So, because of the saying “The devil you know is better than the devil you don’t know” (and a desire to sleep peacefully at night!) I decided to reboot the server and see what would happen. To my big dismay, it did not come online. SSH connections failed with "Port 22: Connection refused" .

I panicked and asked the Hetzner Support (which is very responsive and supportive btw!) to install a LARA Remote Console so that I could see the text output of the booting screen. After some regular startup text, the screen became blank. I panicked even more and took immediate steps to move all data to a new server. It took me 6 hours to complete the most important parts, and another full week of restless work to finish it. We will see that it was not neccessary.

Most regular servers at the Hetzner datacenter are running Software RAID. It seems that after a reboot (especially if you send a Hardware Reset) the OS needs some time to re-sync or check the file system. I am not sure what caused the delay in my case. Re-syncing the entire RAID array can take up to 1-2 hours, depending on your hardware and disc space.

So, wait at least 1 hour for it to come online, especially after a hardware reset! When you activate Hetzner’s Rescue system (which is very good btw!) it will stay active for a minimum of 1 hour, so your server will be down for 1 hour at least, in any event. So you are not losing much by waiting a bit longer.

Now, in my case, I assumed that Grub2 was broken. So I activated the Hetzner Rescue System, booted into it, and reinstalled Grub2. I have found the following method here and it worked for me. First you have to mount the regular RAID filesystem under /mnt :

At this point, you are in your regular root directory. To reinstall Grub2 the Debian Way, I did:

To make really sure, I reconfigured the package:

It will ask you where to install the bootloader. I selected:

No errors were reported. I rebooted again and it did not come online immediately, for the reasons previously mentioned. I waited long enough (in my case, 15 minutes) and it did come online. So, rule number one is: Don’t panic!

5
0

, ,

20 Responses to Do not Panic! Remote Server (Hetzner) not rebooting any more – A Solution

  1. Jens C September 1, 2014 at 10:58 am #

    Thanks a lot, that saved my day!

    0

    0
  2. Sebastian E September 6, 2014 at 1:56 pm #

    Thank you very much, you saved me a weekend full of headaches!

    0

    0
  3. Adam P September 29, 2014 at 11:49 am #

    Thanks Michael!

    0

    0
  4. Jeroen October 13, 2014 at 7:56 pm #

    Thank you and thank you again!!!!

    0

    0
  5. Marco Antonio Blanco November 21, 2014 at 11:14 am #

    Great great great tutorial! Thanks a lot Michael.

    0

    0
  6. Enrico D December 12, 2014 at 12:50 am #

    Thank you very much for this excellent tutorial!!!

    0

    0
  7. Viet Quan February 3, 2015 at 4:04 am #

    Hello Michael.

    I think I have a similar problem with Hetzner. After reboot the server will be down, not sure for how long.

    Do this method works on Ubuntu 12.04 LTS 64bit? and on RAID 0?

    0

    0
    • Michael Franzl February 3, 2015 at 1:22 pm #

      It should also work on Ubuntu, because Ubuntu is based on Debian. Not sure about RAID 0 though…

      0

      0
  8. Rajesh April 24, 2015 at 7:43 am #

    Thank you. You saved my day.

    0

    0
  9. linuxman1 May 16, 2015 at 2:54 pm #

    Thanks,
    You provided me more details than the Hetzner’s wiki, Hetzener’s wiki some times is not detailed at all, so it doesn’t help much.
    BTW, I used this method to reinstall Grub over Centos 6 and it worked, from Debian rescue mode of Hetzner.

    0

    0
  10. Jonathan June 5, 2015 at 8:33 am #

    Thank you! You saved me a lot of time and probably money!!!

    0

    0
  11. Athan June 25, 2015 at 11:39 am #

    Thank you !!! saved me a lot of time and probably money !

    0

    0
  12. Bry July 25, 2015 at 3:39 pm #

    I have an issue, I only can access from rescue mode, after do apt-upgrade

    anyone can help me? because when I reboot my pc, I can’t access

    0

    0
  13. Bernd January 6, 2016 at 8:16 pm #

    My server needed 45 min to boot. I already got a LARA up and running. But i couldn’t find out what took so long. fsck is disabled and raid resync should not prevent booting. Thanks a lot for the tip!

    Cheers,

    Bernd

    0

    0
  14. Scottie February 15, 2016 at 11:52 am #

    WOOHOO!! Thanks for this!
    I had a server with Ubuntu + Software RAID 1. Drive sda died and was replaced, but the machine wouldn’t boot. Host temporarily set it to boot from sdb, which worked. After resync, I just did the steps starting at:
    dpkg-reconfigure grub-pc
    Picked both the new sda + sdb for GRUB to install on, and rebooted. DONE! 🙂

    0

    0
  15. Timo Zikeli July 9, 2016 at 11:42 am #

    Hello,
    I tried this and waited but it did not help. Is there any other public software which can block the boot?

    Thanks,
    Kind regards,
    Timo Zikeli

    0

    0
  16. Paul July 31, 2016 at 7:26 pm #

    Thank you, you saved my night!

    0

    0
  17. Daniel August 25, 2016 at 4:22 am #

    Thanks much appreciated. my harddrive died and i didnt eject it out of the sytem in time (hetzner support was damn quick to replace it).

    I booted the rescue system and partitioned the hard drive as explained in the hetzner wiki, but wasn’t sure about the whole chroot part at the end. I followed your chroot instructions and am currently resyncing the drives. Fingers crossed it works.

    If anyone reads this: Make sure when your drive dies to eject it from the system before calling support. it saves you a lot of headache as you won’t need to fiddle around with the rescue system etc….

    0

    0
  18. Larrybwoy October 20, 2016 at 11:41 am #

    This was awesome. You sir have my gratitude !

    0

    0
  19. J. Martin October 25, 2016 at 5:40 pm #

    Totally saved my bacon on this one. The Hetzner Wiki is not clear on this point, as it makes it sound like the rescue system goes away when you reboot. Also, there is the added issue of having to sync the raid drives if you made the mistake of using a hardware reboot. Always try the soft reboot first!

    0

    0

Leave a Reply

Powered by WordPress. Designed by Woo Themes