Network Traffic Issues Continue

We still don’t have a clear cause for the unexpected network traffic we’ve seen the past few nights. We’re working with Rackspace and our own team to isolate and solve the problem.

I apologize for the issues, and be assured we’re working as hard as we can to get this figured out and return service to normal levels.

If you have any questions, please email the helpdesk. 

 

UPDATE 4:55PM CST: We’re experincing another burst of network traffic causing database connection issues and other problems. We’re working to resolve the issus and apologize for the problems.

 

UPDATE 6:40PM CST: We’ve blocked some access to the system that seemed to be causing the excess traffic. We’re still monitoring the situation for further issues.

Network Traffic Issues

We’re seeing an abnormally high amount of data being transmitted from one of our web servers. This is causing database connection issues and downtime for those on the system.

We’re working to figure out the cause of the network traffic and to restore services to normal.

 

UPDATE 8:40PM CST: The network appears to be more stable now. We’re still not sure of the cause, but we’re closely monitoring the situation to see if anything returns. As always, if you continue to have issues email help@zippykid.com and let us know.

 

UPDATE 9:50PM CST: The increased network traffic is back… we’re investigating again.

Database Server Issue

We’re currently experiencing an issue with a database server that caused sites across our systems to go offline briefly.

We’re still examining the cause of the issue, but for now we expect sites to be working properly. If you continue to have issues, please email the helpdesk and we’ll investigate.

 

UPDATE 8:22 CST: It appears as though the issue is not fully resolved, we’re continuing to investigate the problem.

 

UPDATE 8:50 CST: The issue should be resolved now. If you continue to see problems, please let us know and we’ll continue to investigate.

 

UPDATE 9:10 CST: We seem to be experiencing more issues with the database server. We’re continuing the investigation.

 

UPDATE 10:25 CST: We belive an adjustment we’ve made to the networking configuration of the system will solve the problems. We’re continuing to monitor the situation, but things appear to be much more stable.

Maintenance Window – Scheduled / Expected Downtime

Hi Everyone, 

We’re trying to mitigate the downtime as much as possible, but some of you may experience a downtime of 1 to 3 hours today around 6PM CST. 

What’s happening?

Rackspace is moving some of our servers onto bigger, better hardware. This is going to cause some bouncing around, as we move your data across data centers, change ips, and other fun and nerdy stuff. 

Sorry for the short notice, most downtime should not be more than 5 minutes, but there can be some sites that take a few hours to bring back.

 

UPDATE 9:20PM CDT: Some customers are still being affected by the migration process. We’ve touched base with Rackspace and everything is moving along as expected. We do not have an ETA for the complete restoration of services.

 

UPDATE 6:30AM CDT: The Migration at Rackspace (for a large majority of users who saw issues) finished around 2:50AM CDT. We apologize for the long day, but appreciate your pertinence while this happened. If you have any continued issues please contact help@zippykid.com