Service interruption night of 25th to 26th August
When it rains, it pours....
On August 25th at 10:11 pm (EST) our monitoring systems recorded a problem accessing our servers. Our engineers were immediately notified and got to work on the problem. The problem was traced down to a critical piece of hardware - so-called 'load balancer', device which directs traffic to various servers handling the traffic to client sites. The device has malfunctioned and shut down. We worked with the data hosting center to have this problem resolved and unfortunately it took a while to restore the service.
Service was back to normal at 3:05 am on August 26th (total downtime 4:53). During this time none of the client sites were reachable.
This is the first time ever we had an incidents of the duration and scope like this and obviously this not acceptable for a web-based service like Wild Apricot.The only saving grace was that this happened during the night time in North America, which is the time of the lowest traffic to our client sites.
Please accept our sincere apologies and be assured that our team will be working to find the ways to prevent any similar occurrences in the future (even though our datahosting center engineers said it was an extremely rare freak situation they never had before).