#1 28th Sep 2015 08:10:39

Lord_Chris
Lord Commander
Reputation: 145

From: England, United Kingdom
Registered: 14th Aug 2013
Posts: 2,169
Website

Unexpected downtime

Hi everyone!

I thought I'd post a topic on why the site has been down for the last week. I'd also like to thank everyone for their patience and apologise for the downtime. I'd also like to thank members such as EaglePrince and Tokamaps for getting in touch with me and trying to let me know the site was down.

On Saturday 19th September at approximately 10 AM UK time, the raid controller for the site failed, and so did the primary hard drive. The controller then attempted to kick into the secondary hard drive so no data was lost, and that also failed. These type of failings are not common at all, and in fact it's about as common as two car headlights burning out at the exact same time.

Our server monitoring team tried their very best, working very long, and very diligently to recover any data possible. That was the main thing that caused this whole process to last as long as it did, as they were trying to salvage data from what they had available. This included backups of the site that the server makes on a daily basis.

They then had to make the unfortunate decision that no more data could be salvaged as the were unable to read the data after the server crashed. In this instance, the very thing that protects against data loss, caused the data loss.

The server monitoring team restored what they could, but half of the website was missing including ~3000 files, our SSL certificate for the main website and IP address - as well as any subdomains such as the forum.

We therefore had to resort to a backup from the 17th of September, and everything posted between the 17th and 19th has been lost, including the update of the forum I did during this time. I will be doing some maintenance later on the site to update it again, but other than this, I can only be sorry for what happened.

Offline
  • 3

#2 28th Sep 2015 08:59:16

infinitysuns
Armed Peasant
Reputation: 0

Registered: 30th Jul 2015
Posts: 10

Re: Unexpected downtime

Sorry to hear about the failure. Thanks for all your hard work Lord Chris smile

Offline
  • 0

#3 28th Sep 2015 16:31:14

EaglePrince
Knez
Reputation: 146

From: Novi Sad, Serbia
Registered: 6th May 2014
Posts: 2,082
Website

Re: Unexpected downtime

I stand with infinitysuns. smile Glad to see the forum on its feet again.

Offline
  • 0

#4 29th Sep 2015 16:54:41

Nigel
Duke
Reputation: 51

Registered: 31st Aug 2014
Posts: 619

Re: Unexpected downtime

Same here. I am much relieved to see that the site is back.
Many thanks for all your efforts.

Edit 2:
Yes, the posts I make to the thread about the site downtime get redirected here..... reproducibly.

Last edited by Nigel (29th Sep 2015 16:55:51)

Offline
  • 0

#5 29th Sep 2015 17:28:56

Lord_Chris
Lord Commander
Reputation: 145

From: England, United Kingdom
Registered: 14th Aug 2013
Posts: 2,169
Website

Re: Unexpected downtime

Thanks Nigel, and you too infinitysuns for your support. smile

I've fixed the problem with replying to topics, thanks Nigel.

Offline
  • 0
Users in this topic: 0 guests, 0 registered users

Board footer

Powered by Aura