Hi everyone,

My name is Kevin and I am part of the support team at Wild Apricot

Last week we experienced extended and unplanned downtime. It was certainly a very disappointing and frustrating few days for all of our clients and their members.

We would like to start by expressing our sincere apologies for the disruption in service.

This downtime comes after a few other occurrences in the past months and we would like to share not only the root cause of this latest downtime but what we are doing to stabilize our environment moving forward.

So what happened? Code was introduced into the environment that missed a critical part of our quality assurance

How could this happen? A simple change was made to code that was thought to be too minor to have any impact and the proper testing was not performed.

How will we prevent this in the future? We have established a zero tolerance for any code being introduced into the environment without the strict scrutiny of our quality assurance teams.

Why did it take so long to fix? Bugs are not always easy to find, which is why we have such an intense scrutinization process. In this instance, our CTO reallocated existing engineers to help search and discover the bug. Our team worked in shifts much like a fire department until the issue was discovered and resolved.

What about the other downtime our customers have had over the past months? We realized quite some time ago that the stability of our hosting provider was not reliable enough to meet the standards that all of you expect. We began an intensive search for the right solution, which is not a easy task. We are happy to say that over the next few months, we will be migrating to a new platform that has a proven reputation of stability

To all of our customers that were affected , please accept our most sincere and humble apology.

We understand how critical it is that we deliver maximum up-time to support your campaigns and will increase our efforts to meet and exceed your availability

Now – back to making our service even better and more stable than ever.

All the issues with websites availability were fixed on Sept 2nd.

September 6th, 9:26 AM EST: Our developers are currently working to get the root cause analysis completed, so it will be posted in this topic.

September 2nd, 2:00PM EST: Our developers have fixed the issue with intermittent outages.

Unfortunately there has been a problem with one of secondary services and further root cause analysis is being processed.

We apologize for the inconvenience this issue has caused.

September 2nd, 8:26AM EST: Our developers have found the root cause for intermittent websites availability and are preparing the hotfix publishing which should be released in approximately 3 hours.

This forum post will be updated once the hotfix is released.

We apologize for the inconvenience.

September 2nd, 3:12 AM EST: The situation with websites availability has been stabilised, but the issue is not yet resolved unfortunately. Our System Operations engineers are still working on the issue.

As soon as we have the situation fully resolved, we will share the details of the root cause and what we’re going to do about this. Again, we are very sorry for the inconvenience.

September 1st, 5:26 pm EST: We continue experiencing intermittent service issues.

We understand that this is very frustrating for you and we are sorry for all the inconvenience this has caused. Our System Operations engineers are still working to get the issue resolved.

We will continue to update this post. As soon as we have the situation fully resolved, we will share the details of the root cause and what we’re going to do about this. Again, we are very sorry for the inconvenience.

9:09AM EST Sept 1st: Wild Apricot websites are available, but the issue has not been fully resolved. Our System Operations team is working to get the issue resolved as soon as possible.
We apologize for any inconvenience caused.

1st September, 5:09AM EST. Websites are available, however there might be intermittent availability issues. Our System Operations engineers are still working to get the issue resolved.

We apologize for the inconvenience.

3:03 AM EST Sept 1st: Wild Apricot websites and functionality is available, but the issue has not been fully resolved. Our System Operations team is still working to get the root of the issue resolved while maintaining all websites availability.

We apologize for the inconvenience.

10:53 pm EST: We are still experiencing intermittent outages. Our systems operations team is working to get your site back up and running as soon as possible. We are sorry for all the inconveniences this has caused. We will continue to update this as soon as we have more information.

8:12pm EST: Wild Apricot sites are still down, our apologies. Our System Operations team are actively working on fixing this ASAP. We will continue to monitor and update you about this issue.
We’re sorry for the inconveniences this has caused.

5:25pm EST: Wild Apricot sites are down again. We are investigating the problem and hope to restore service as soon as possible – for now it looks like a DDOS attack. We apologize for any inconvenience this has caused.

1:45pm: Wild Apricot sites should all be back up now. We are still investigating the cause.