We’re currently experiencing an issue with one of our clusters that’s causing high Disk IO. This is causing sites to perform slowly or not at all.

We’re working with Rackspace to determine the cause of this issue and get it resolved ASAP. We apologize for any issues this is causing and we’ll update you as soon as we know more.

UPDATE 2:30PM CST: The cause of the issue has been identified and we’re working with Rackspace to solve the problem. The solution will likely result in the migration of the cloud server from one physical host machine to another. This is currently resulting in sites to be down (a combination of the move and migration). We’re expecting things to return to normal once the migration has been completed. However, at this time we don’t have an ETA.

Sorry for the trouble, and if you have any questions, please don’t hesitate to email help@zippykid.com.

UPDATE 5:00PM CST: The Cloud Server is in the process of being migrated from the existing host machine to a new one. During most of this time things should remain up. During the last few minutes of the migration sites will go down while the move is completed and brought up in the new location. We don’t expect this outage or the ongoing issues to last much longer.

UPDATE 9:10PM CST: The final pieces of the migration to the new system are happening now. This is causing sites affected by today’s issue to be offline. We don’t expect this piece of the outage to last too long, but we’re closely monitoring the situation.

UPDATE 9:25PM CST: The Migration of the cloud server to a new host machine has completed and service should be returning to normal. We’re not expecting anymore outages or issues with this system. If you continue to experience problems, or have any questions, please don’t hesitate to email help@zippykid.com.

Thanks for your patience today while we worked through this issue!

We noticed that one of our MySQL Replicants (Slave) was out of sync by a few hours. So, some of you may have been serving up stale content. We’ve fixed the issue and things should be getting back in sync. 

We’re looking into why it took us so long to find this problem. Sorry for any confusion this may have caused you. 

On June 5th, 2012 at 10:00 PM CDT our engineers will perform maintenance to our Cloud Load Balancer infrastructure. In an attempt to roll out a new feature on Cloud Load Balancer, a server restart is required. While we do not anticipate any impact, there is a small possibility of as much as a 10-minute window where service may be interrupted. We are confident that this maintenance will be completed on, or before June 6th at 12:01 AM CDT.

We apologize for any inconvenience this may cause and thank you for your patience and understanding. If you have any questions please don’t hesitate to email help@zippykid.com.

We’re currently experiencing an issue with our Rack Connect system that connects our Cloud and Dedicated hardware. This is causing the connection interruption between our web servers and database servers. We’re working with Rackspace to resolve this as soon as possible.

Sorry about the issues, and we’ll be sure to update you when we know more.

UPDATE 1:00PM CST: The connection between our Cloud and Dedicated Environment has been restored now. We’re seeing our systems come back online now and don’t expect to see another issue related to this. If you’re still experiencing issues with your site, please don’t hesitate to email help@zippykid.com and let us know.

We’ve posted a more detailed explanation of the ongoing issues on our website:

http://zippykid.com/2012/05/01/explanation-of-recent-outages-over-the-past-we…

Feel free to give it a read. We’re still working on getting everything fixed as quickly as we can, and as always, if you have questions, email the helpdesk.

This is a continuation of the issues we’ve been having since late last week. Our systems have been under attack off and on during this time. It’s causing issues with sites being up, and database connections for the sites.

Some of the steps we’ve taken to help curb the attack so that we can find the source and get rid of it completely have caused other problems. Most of this revolves around updating/adding plugins and media items to your site.

We’re working with Rackspace to resolve these issues as quickly as we possibly can, but at this time we still don’t have an ETA.

Sorry about all the trouble and thanks for your patience during this time. If you have any questions, please feel free to email the helpdesk.

The issues we’re currently having (and have been experiencing the past several days) are related to an attack on our systems. We’re working with Rackspace to block the attack and return service to normal.

We apologize for the issues, and hope to return service to normal as soon as we can. If you have any questions, please don’t hesitate to email the helpdesk.

We still don’t have a clear cause for the unexpected network traffic we’ve seen the past few nights. We’re working with Rackspace and our own team to isolate and solve the problem.

I apologize for the issues, and be assured we’re working as hard as we can to get this figured out and return service to normal levels.

If you have any questions, please email the helpdesk. 

UPDATE 4:55PM CST: We’re experincing another burst of network traffic causing database connection issues and other problems. We’re working to resolve the issus and apologize for the problems.

UPDATE 6:40PM CST: We’ve blocked some access to the system that seemed to be causing the excess traffic. We’re still monitoring the situation for further issues.

We’re seeing an abnormally high amount of data being transmitted from one of our web servers. This is causing database connection issues and downtime for those on the system.

We’re working to figure out the cause of the network traffic and to restore services to normal.

UPDATE 8:40PM CST: The network appears to be more stable now. We’re still not sure of the cause, but we’re closely monitoring the situation to see if anything returns. As always, if you continue to have issues email help@zippykid.com and let us know.

UPDATE 9:50PM CST: The increased network traffic is back… we’re investigating again.

We’re currently experiencing an issue with a database server that caused sites across our systems to go offline briefly.

We’re still examining the cause of the issue, but for now we expect sites to be working properly. If you continue to have issues, please email the helpdesk and we’ll investigate.

UPDATE 8:22 CST: It appears as though the issue is not fully resolved, we’re continuing to investigate the problem.

UPDATE 8:50 CST: The issue should be resolved now. If you continue to see problems, please let us know and we’ll continue to investigate.

UPDATE 9:10 CST: We seem to be experiencing more issues with the database server. We’re continuing the investigation.

UPDATE 10:25 CST: We belive an adjustment we’ve made to the networking configuration of the system will solve the problems. We’re continuing to monitor the situation, but things appear to be much more stable.