We’re experiencing issues on our Classic environment. We’re investigating the issue right now. 

We’re currently investigating an issue with the database cluster that’s causing connection issues. We’re working to restore stability to the system and we’ll update this post when we know more.

UPDATE 8:58PM CST: The database cluster is stable now. If you’re continuing to experience issues, let us know.

We’ve fixed the database glitch. All systems are back to normal. 

We’re aware of some anomalies with our database cluster. We’re looking into it and will update you shortly. 

Update as of September 17, 2012 12:22 AM : The issue has been resolved, all systems are back up and running.

Based on reports from various media organizations, there appears to be an issue with GoDaddy and at a minimum their DNS service (source, source, source)

What this means for you:

• If you’re hosting your site with us, and using our DNS (you’ve pointed the domain to our name servers)… absolutely nothing. Your site will continue to operate normally.
• If you’re hosting with us, but are managing the domain’s DNS at GoDaddy, you’ll likely see the site work sporadically.

Hopefully GoDaddy is able to resolve the issue and return service to normal for the users that are being impacted. As always, if you have any questions, please feel free to email help@zippykid.com and let us know.

We’re currently experiencing an issue that’s caused our database connection to go offline. This is impacting the connection for all sites to the databases. We’re actively working to correct this, however, we expect this issue to last for about 20 minutes.

We do apologize for the inconvience and if you have any questions, please don’t hesitate to email help@zippykid.com.

UPDATE 5:35PM CST: As we continue to evaluate the situation we’re expecting our earlier estimates to increase some. We’re hoping to have things up and running within the next 45 minutes. We’ll let you know if this changes, and our sincere apologies for this.

UPDATE 6:20PM CST: We’re still working to restore the connections to database servers. There are some delays in getting things recreated, but we’re hoping to start restoring connectivity in the next 45 minutes (I know… we said this last time…). We’ll update you when we have some more information.

UPDATE 7:00PM CST: We’re starting see database connections for the sites come back online. We’re expecting to see service return to normal shortly. If you’re still having issues with a site, please let us know and we’ll take a look. Our apologies for the issues, and if you have any questions, don’t hesitate to ask.

We are experiencing a load balancer issue that is affecting a small number of sites. We are working with Rackspace to have this issue corrected and will keep this updated as we have more information.

UPDATE 9:02 AM CST:  Rackspace has experienced a hardware failure which required migrating the cloud load balancers to new hardware. Affected sites are back up now, but may go up and down as the migration continues. We are continuing to monitor things and will update more as the migration completes.

UPDATE 9:44 AM CST: We are still seeing sites go up and down. The migration is still underway, we will update again with more information as we recieve it.

UPDATE 10:04 AM CST: Migration is complete and all sites on cluster 12 are back up. We have a small number of sites on cluster 17 that are still experiencing issues. We are tending to those issues and will update again shortly.

UPDATE 3:15 PM CST: There are still some lingering affects from this morning’s outage on cluster 17. Sites on this cluster may still be experiencing intermittent issues. We are aware of this and are working again with Rackspace to have this cleared up. All other clusters are up and stable.

As of 5:00 AM July 23, 2012, our DNS provider Zerigo is still down. Therefore a lot of our customer sites are not resolving. This is making it look like the servers are down. We thought about emailing you about this issue as it was happening, but we realized since the name servers are down, email systems around the world wouldn’t know how to send you email either. So, right now this status blog, and twitter are the only way for us to let you know what’s going on. 

To keep up with the status of our partner, you can go to: http://www.zerigostatus.net

UPDATE 6:15AM CST: Zerigo is still working on mitigating the effects of the attack and bringing things online. We’re monitoring the situation, and hope they’ll get things under control.

UPDATE 7:40AM CST: Zerigo is still working on mitigating the issues they’re having. We’re continuing to monitor the situation.

UPDATE 11:35AM CST: There’s nothing new from Zerigo in terms of the status of the issues. We’re continuing to monitor things.

UPDATE 2:05PM CST: Here’s the latest from Zerigo on the issues:

We are still under a very large DDOS attack primarily affecting our European customers. Our #1 priority is getting our customers back online. We are working to bring our DNS services in the US to full capacity. Currently we are at 50% in the US. Our servers in Amsterdam and Singapore are serving our European customers at 33% capacity.

UPDATE 8:05PM CST: Another update from Zerigo:

Posted at 07/23/2012 23:57 UTC At present we’re waiting on the ISP’s operating our networks in Dallas and London to lift the null routes blocking traffic to c.ns.zerigo.net and d.ns.zerigo.net. SoftLayer reports that d.ns.zerigo.net will be available at 22:06 (GMT-5) (in about 3.5 hours) from now.

UPDATE 6:45AM CST: Finally some good news from Zerigo:

c.ns.zerigo.net and d.ns.zerigo.net are back online thanks to the ISPs in Dallas and London lifing the null routes.

DNS should be back online and responding normally for customers. If you’re still having issues, please don’t hesitate to email help@zippykid.com and we’ll take a look. We’ll be making a more detailed comment shortly (on the main blog/via email) about our plans moving foward.

We’re currently experiencing an issue with our DNS Provider that is causing sites to be unavailble for some users. We’re investigating the issue and will provide an update when we have more information.

UPDATE 11:00PM CST: Our DNS provider is currently experiencing a DDOS attack. This is causing issues with their servers that provide domain name resolution for sites using our DNS. We’ll provide an update when things are more stable on their end.

The old database server had a hiccup once again. We’re aware of the issue and are addressing it. Some of you may get an “Error Establishing database” connection message, but they will be going away very soon.