As of 5:00 AM July 23, 2012, our DNS provider Zerigo is still down. Therefore a lot of our customer sites are not resolving. This is making it look like the servers are down. We thought about emailing you about this issue as it was happening, but we realized since the name servers are down, email systems around the world wouldn’t know how to send you email either. So, right now this status blog, and twitter are the only way for us to let you know what’s going on. 

To keep up with the status of our partner, you can go to: http://www.zerigostatus.net

UPDATE 6:15AM CST: Zerigo is still working on mitigating the effects of the attack and bringing things online. We’re monitoring the situation, and hope they’ll get things under control.

UPDATE 7:40AM CST: Zerigo is still working on mitigating the issues they’re having. We’re continuing to monitor the situation.

UPDATE 11:35AM CST: There’s nothing new from Zerigo in terms of the status of the issues. We’re continuing to monitor things.

UPDATE 2:05PM CST: Here’s the latest from Zerigo on the issues:

We are still under a very large DDOS attack primarily affecting our European customers. Our #1 priority is getting our customers back online. We are working to bring our DNS services in the US to full capacity. Currently we are at 50% in the US. Our servers in Amsterdam and Singapore are serving our European customers at 33% capacity.

UPDATE 8:05PM CST: Another update from Zerigo:

Posted at 07/23/2012 23:57 UTC At present we’re waiting on the ISP’s operating our networks in Dallas and London to lift the null routes blocking traffic to c.ns.zerigo.net and d.ns.zerigo.net. SoftLayer reports that d.ns.zerigo.net will be available at 22:06 (GMT-5) (in about 3.5 hours) from now.

UPDATE 6:45AM CST: Finally some good news from Zerigo:

c.ns.zerigo.net and d.ns.zerigo.net are back online thanks to the ISPs in Dallas and London lifing the null routes.

DNS should be back online and responding normally for customers. If you’re still having issues, please don’t hesitate to email help@zippykid.com and we’ll take a look. We’ll be making a more detailed comment shortly (on the main blog/via email) about our plans moving foward.

We’re currently experiencing an issue with our DNS Provider that is causing sites to be unavailble for some users. We’re investigating the issue and will provide an update when we have more information.

UPDATE 11:00PM CST: Our DNS provider is currently experiencing a DDOS attack. This is causing issues with their servers that provide domain name resolution for sites using our DNS. We’ll provide an update when things are more stable on their end.

The old database server had a hiccup once again. We’re aware of the issue and are addressing it. Some of you may get an “Error Establishing database” connection message, but they will be going away very soon. 

We’re currently experiencing an issue with one of our Cloud Servers causing it to go offline. We’re working with Rackspace to get it back up as soon as possible. We’ll update this post when we know more or things are resolved. If you have any questions, please feel free to email help@zippykid.com.

UPDATE 4:35pm CST: The server has recovered from the issue and is now responding normally. There was an issue that caused the reboot signal to the box to fail, which took things down, but didn’t bring them back up. Things are running and should be stable, but we’re continuing to monitor.

We’re seeing similar database connection errors this morning across our systems. We’re working with Rackspace to determine the cause of the issue and resolve it ASAP. If you have any additional questions, please don’t hesitate to email help@zippykid.com and we’ll take a look.

UPDATE 6:30AM CDT: Rackspace is repairing a sync issue between the database servers. This is causing the current connection issues for sites. We don’t have an ETA for how long this should last, but we’ll update you when we know more.

UPDATE 6:45AM CDT: We’re expecting things to be more stable now and for sites to be responding. If you’re still expericing issues, please let us know.

As of Friday July 13th 2012 00:30 hours: It looks like things are back up and running. We’re looking to find out what happened, and how to avoid it. 

We’re experiencing database connectivity issues across the board. We’re working with Rackspace to solve this problem as we speak. Sorry for the down time. We’re aware of the issue and will fix this as soon as we can. 

Update as of Friday July 13th 2012: It looks like things are back up and running. We’re looking to find out what happened, and how to avoid it. 

There’s currently an issue with one of the host machines for our cloud servers. This is impacting a single server and things are offline for the time being. We’re working with Rackspace to determine the cause and provide an ETA to resolution. We’ll update you when we know more and sorry for the inconvience. If you have any questions, please don’t hesitate to email help@zippykid.com.

UPDATE 11:10 PM CDT: The issues with the host machine appear to have been resolved and the cloud server is coming back online. We’re still waiting on confirmation from RackSpace, but services look to be resuming. If you’re noticing any issues with your site, please don’t hesitate to email help@zippykid.com and we’ll be glad to take a look.

All of ZippyKid’s customers are now running the most up to date and secure version of WordPress core, version 3.4.1. The upgrade process started yesterday around 5 minutes after 3.4.1 was announced, and finished around 8am. Some of you may have noticed a short down time, or a maintenance message window. 

Due to the nature of some customizations to themes by some customers, we have not upgraded themes. 

We’re currently experiencing an issue with one of our clusters that’s causing high Disk IO. This is causing sites to perform slowly or not at all.

We’re working with Rackspace to determine the cause of this issue and get it resolved ASAP. We apologize for any issues this is causing and we’ll update you as soon as we know more.

UPDATE 2:30PM CST: The cause of the issue has been identified and we’re working with Rackspace to solve the problem. The solution will likely result in the migration of the cloud server from one physical host machine to another. This is currently resulting in sites to be down (a combination of the move and migration). We’re expecting things to return to normal once the migration has been completed. However, at this time we don’t have an ETA.

Sorry for the trouble, and if you have any questions, please don’t hesitate to email help@zippykid.com.

UPDATE 5:00PM CST: The Cloud Server is in the process of being migrated from the existing host machine to a new one. During most of this time things should remain up. During the last few minutes of the migration sites will go down while the move is completed and brought up in the new location. We don’t expect this outage or the ongoing issues to last much longer.

UPDATE 9:10PM CST: The final pieces of the migration to the new system are happening now. This is causing sites affected by today’s issue to be offline. We don’t expect this piece of the outage to last too long, but we’re closely monitoring the situation.

UPDATE 9:25PM CST: The Migration of the cloud server to a new host machine has completed and service should be returning to normal. We’re not expecting anymore outages or issues with this system. If you continue to experience problems, or have any questions, please don’t hesitate to email help@zippykid.com.

Thanks for your patience today while we worked through this issue!