We experienced intermittent issues with the content delivery network we have. The symptoms of which are that your site doesn’t look correct, all the text loads, but you won’t see images or your stylesheets. 

The outage was for about 20 minutes. All systems are functioning now. 

As of 2:53 PM on January 27, 2013 All systems are functioning normally again. We had intermittent issues across our network.

Here’s what happened. 

One of our 4 memcached servers had run out of memory, and in the process locked up. This made it so that our database servers were seeing 8x the average calls. Since our monitors started telling us about higher than normal database activity, we started investigating the issue there.  

What did we learn?

It turns out, our monitoring on the memcached systems isn’t as good as we thought it was. Had we known that the one of the memcached server was out of commission, we would’ve been able to identify the problem, and fix it. Rather than investigating what was causing the spike in the database usage. 

We are currently experience an issue that causing sites to display database connection errors.

We are working on having this resolved as soon as possible and will update with more information as it is available.

Update:

All sites are back up now and the total down time was about 15 minutes.

If you have any questions, please send an email to help@zippykid.com and we can address your concerns via our help desk.

The maintenance took longer than expected, but everything is complete and tested. Thanks for your patience. 

The scheduled maintenance has started. 

Just a reminder, we have some maintenance scheduled at 3AM on December 22nd 2012. All services will be disrupted during this time period. 

The maintenance will take about 1 hour to complete. 

We’ve resolved the database issue. Everything is functioning once again. 

We’re experiencing a major issue with our database cluster. Sites are down, we’re working on the issue. 

Our customer control panel is currently down for maintenance, we’re working to fix it. All sites are functional, the only thing that’s not going to work right now is the ability to add a new site. 

We had an issue with our storage area network (SAN), this has been fixed, and everything is functioning again. 

If you’re still seeing an issue, please let us know. 

As of 10:54 PM on December 6, 2012.