On Saturday evening 6:30pm one of our server nodes inside a cluster failed due to loosing 2 hard drives at a time. This node can loose a maximum of 1 drive which in most cases would be replaced straight away and rebuilt. Because it lost 2 drives the array failed completely causing the server node to fail itself. We take backups every night at 11pm. Due to the length of time between backups we decided to recover the logs from the broken node over the top of the existing backup to minimize any data loss, however this method takes time as it requires us to run all of the missed transactions between the time of the backup and the loss of the server, we decided this was the best approach forward. Due to the number of transactions this is taking longer than expected and as stated before we feel this is better than missing data/emails.
Note: This is a different problem that what we experienced late last year, however we are currently in the process of moving customers away from this platform to newer technology which has more forms of redundancy, reliability and features which eliminates these types of issues.