Past Incidents

Load is now decreasing on Servers and we are seeing improved performance. More changes are planned to increase performance further. The log replay for the merge is still in progress.

Due to the high load on the Servers there will be ongoing performance issues with the delivery of email. You may experience slowness both within Outlook and with new emails coming in.

You may be noticing some slowness and/or emails coming in maybe slower than normal. This is due to the system being under high load, we expect this to be resolved soon.

Hosted Exchange Email is now working. Please note there may be some missing emails from 2nd & 3rd of February which we will merge back in as soon as possible.

We are about to implement a different fix, we will restore an alternative system so customers can use their email and merge missing data once it is recovered. More updates soon.

We are working on another fix which should mean we will be up and running sooner. We will update you again shortly.

We are still working through the log replay. This currently sits at 27%. Based on this we can estimate that the service should be up by 4am 6th February. We will continue to update you on progress throughout the day.

What Has Happened - Explained

On Saturday evening 6:30pm one of our server nodes inside a cluster failed due to loosing 2 hard drives at a time. This node can loose a maximum of 1 drive which in most cases would be replaced straight away and rebuilt. Because it lost 2 drives the array failed completely causing the server node to fail itself. We take backups every night at 11pm. Due to the length of time between backups we decided to recover the logs from the broken node over the top of the existing backup to minimize any data loss, however this method takes time as it requires us to run all of the missed transactions between the time of the backup and the loss of the server, we decided this was the best approach forward. Due to the number of transactions this is taking longer than expected and as stated before we feel this is better than missing data/emails.

Note: This is a different problem that what we experienced late last year, however we are currently in the process of moving customers away from this platform to newer technology which has more forms of redundancy, reliability and features which eliminates these types of issues.

20% through log replay

Log extract complete, starting log replay.

New copy process is working faster, 10% in 15 minutes. Once this completes we will replay the EDB logs

Log files are just about recovered, machine is ready for them to be applied. Next update at or before 12am

We continue to restore backups and log files. At this stage our estimate is between 9 – 10pm this evening. More updates to follow closer to this time.

We are still in the process of recovering the backups. To avoid loss of data we are restoring log files along with the main database, this will take longer than initial estimates. We plan to have another update before 6pm. Email is queuing and will be delivered when the exchange server is back online.

We are still in the process of restoring the backup. ETA 3pm 04/02/2018 for server recovery

Recovery of the database failed due to another hard drive failing. We are now in the stages of restoring from a backup.

We have identified the issue to be a unclean shutdown database. We are trying to repair the database in the first instance, failing that we will run from backups. No ETA as yet, but we will post updates here.

Hosted Exchange 2010 is having some issues with its database, this looks like it maybe related to the failed node that happened last night. We will put updates here

We have found 2 failed disks on one of our nodes has failed, we are looking at restoring services soon.

We have had an alert a node has crashed. This may be effecting things like Hosted Exchange 2010 and a few others on this node

One of our nodes have failed, we are working to resolve the issue. Hosted exchange and some other services may not be available while it is down