August 31st, 10:45am EST: Experiencing service interruption
At about 10:45am EST Wild Apricot started to experience some issues and as a result, all website sites are down. We are investigating the problem and hope to restore service as soon as possible. Please, click "Follow" to get notified about updates
My name is Kevin and I am part of the support team at Wild Apricot
Last week we experienced extended and unplanned downtime. It was certainly a very disappointing and frustrating few days for all of our clients and their members.
We would like to start by expressing our sincere apologies for the disruption in service.
This downtime comes after a few other occurrences in the past months and we would like to share not only the root cause of this latest downtime but what we are doing to stabilize our environment moving forward.
So what happened? Code was introduced into the environment that missed a critical part of our quality assurance
How could this happen? A simple change was made to code that was thought to be too minor to have any impact and the proper testing was not performed.
How will we prevent this in the future? We have established a zero tolerance for any code being introduced into the environment without the strict scrutiny of our quality assurance teams.
Why did it take so long to fix? Bugs are not always easy to find, which is why we have such an intense scrutinization process. In this instance, our CTO reallocated existing engineers to help search and discover the bug. Our team worked in shifts much like a fire department until the issue was discovered and resolved.
What about the other downtime our customers have had over the past months? We realized quite some time ago that the stability of our hosting provider was not reliable enough to meet the standards that all of you expect. We began an intensive search for the right solution, which is not a easy task. We are happy to say that over the next few months, we will be migrating to a new platform that has a proven reputation of stability
To all of our customers that were affected , please accept our most sincere and humble apology.
We understand how critical it is that we deliver maximum up-time to support your campaigns and will increase our efforts to meet and exceed your availability
Now – back to making our service even better and more stable than ever.