We’ve noticed some major hardware issues on cluster 12, which is affecting around 100 websites. We’re working to fix this as fast as we can. We experienced this issue while doing prep work for the migration of this cluster to our TNG system. The transfer of this system to TNG is 95%, so we’ll be working to restore the sites onto the TNG platform by 7am CDT. Sorry for the issues, once this cluster is back, it’ll be much faster than before. 

We’re currently experiencing an issue with the database cluster that is impacting site availability. We’re working with Rackspace to isolate and address the issue.

We’ll provide an update to the outage as we learn more. If you have any questions, please email help@zippykid.com.

A load balancer on server 17 went out early this morning as a part of an outage in one of Rackspace’s cloud load balancer clusters.

UPDATE 12:10PM: The load balancer for server 17 is still down at the moment and we are working with rackspace to have service restored as quicky as possible

UPDATE 1:31PM: We are in the process of moving all sites on this server to our next generation architecture. Once this move is completed, sites will begin coming back up.

We’re experiencing issues on our Classic environment. We’re investigating the issue right now. 

We’re currently investigating an issue with the database cluster that’s causing connection issues. We’re working to restore stability to the system and we’ll update this post when we know more.

UPDATE 8:58PM CST: The database cluster is stable now. If you’re continuing to experience issues, let us know.

We’ve fixed the database glitch. All systems are back to normal. 

We’re aware of some anomalies with our database cluster. We’re looking into it and will update you shortly. 

Update as of September 17, 2012 12:22 AM : The issue has been resolved, all systems are back up and running.

Based on reports from various media organizations, there appears to be an issue with GoDaddy and at a minimum their DNS service (source, source, source)

What this means for you:

• If you’re hosting your site with us, and using our DNS (you’ve pointed the domain to our name servers)… absolutely nothing. Your site will continue to operate normally.
• If you’re hosting with us, but are managing the domain’s DNS at GoDaddy, you’ll likely see the site work sporadically.

Hopefully GoDaddy is able to resolve the issue and return service to normal for the users that are being impacted. As always, if you have any questions, please feel free to email help@zippykid.com and let us know.

We’re currently experiencing an issue that’s caused our database connection to go offline. This is impacting the connection for all sites to the databases. We’re actively working to correct this, however, we expect this issue to last for about 20 minutes.

We do apologize for the inconvience and if you have any questions, please don’t hesitate to email help@zippykid.com.

UPDATE 5:35PM CST: As we continue to evaluate the situation we’re expecting our earlier estimates to increase some. We’re hoping to have things up and running within the next 45 minutes. We’ll let you know if this changes, and our sincere apologies for this.

UPDATE 6:20PM CST: We’re still working to restore the connections to database servers. There are some delays in getting things recreated, but we’re hoping to start restoring connectivity in the next 45 minutes (I know… we said this last time…). We’ll update you when we have some more information.

UPDATE 7:00PM CST: We’re starting see database connections for the sites come back online. We’re expecting to see service return to normal shortly. If you’re still having issues with a site, please let us know and we’ll take a look. Our apologies for the issues, and if you have any questions, don’t hesitate to ask.

We are experiencing a load balancer issue that is affecting a small number of sites. We are working with Rackspace to have this issue corrected and will keep this updated as we have more information.

UPDATE 9:02 AM CST:  Rackspace has experienced a hardware failure which required migrating the cloud load balancers to new hardware. Affected sites are back up now, but may go up and down as the migration continues. We are continuing to monitor things and will update more as the migration completes.

UPDATE 9:44 AM CST: We are still seeing sites go up and down. The migration is still underway, we will update again with more information as we recieve it.

UPDATE 10:04 AM CST: Migration is complete and all sites on cluster 12 are back up. We have a small number of sites on cluster 17 that are still experiencing issues. We are tending to those issues and will update again shortly.

UPDATE 3:15 PM CST: There are still some lingering affects from this morning’s outage on cluster 17. Sites on this cluster may still be experiencing intermittent issues. We are aware of this and are working again with Rackspace to have this cleared up. All other clusters are up and stable.