As of 2:30AM on June 20, 2013 we’ve been noticing severely limited connectivity between our dedicated and cloud infrastructure. This is causing a majority of our websites to be inaccessible. We’re working with Rackspace to figure out what the issue is and how to fix it.
We currently believe this is a routine maintenance at Rackspace gone awry.
Update June 20, 2013 4:10 AM CST: We’re working with Rackspace to get 95% of our sites back online. We’re hoping to have 100% restoration by 6:00AM CST.
We’re seeing connectivity issues between our cloud network and our dedicated hardware at Rackspace. This is currently causing performance issues for sites, as well as intermittent complete unavailability. We’re working with Rackspace to get this corrected ASAP.
We apologize for any inconvenience, and if you have any questions, please don’t hesitate to contact support.
UPDATE 10:00AM CST: We’re seeing services start to return to normal, but we’re still monitoring some issues across the systems. We’re hopeful this will be resolved shortly and will update you shortly.
UPDATE 11:30AM CST: Systems have returned to normal and all affected sites should now be stable.
We’re currently aware of an issue impacting the availability of our SFTP and phpMyAdmin servers. We’ve been working with our provider to get this issue resolved.
The issue impacts the ability to access site files via SFTP and the ability to access a site’s database through phpMyAdmin (available in my.zippykid.com).
There is currently no impact on the availability or performance of sites, only these services are being impacted.
We’ll provide an update as we have more information.
UPDATE 3:30PM CST: Services have come back online and we’re seeing the SFTP/phpMyAdmin services back online. If you’re still seeing issues connecting, please contact the helpdesk.
We’re currently experiencing some network connectivity issues between some of our hardware. This is causing sites to be unavailable or display database connectivity messages. We’re looking in to the issue now and working with our partners to identify the cause.
UPDATE 12:30pm CST: At this time we believe the network connectivity issues have subsided. We’re still investigating the cause of the issues, but things should be running normally at this time.
We’re removing a plugin that is known to put malware onto our customers sites off our systems. This will be in effect until the plugin has been re-instated at WordPress.org. Please see this blog post by Sucuri for more details on what this plugin does.
There’s an issue with some of our caching services causing intermittent site issues and downtime. We are working with our providers and operations team to get everything online. This blog will be updated with more information when it becomes available.
UPDATE 7:55PM CST: At this time we’ve reconfigured our caching setup to remove the impacted hardware from production. We’re still evaluating the root cause, but expect sites to be stable now.
In our ongoing efforts to give you the best WordPress Hosting on the planet, we need to make some system wide upgrades to our infrastructure. These upgrades will be happening on Sunday morning at 12:30AM CST, and will last 2 hours, till 2:30AM.
During this maintenance window, all websites will show a generic message letting you and your visitors know that the website is down for maintenance. We’re sorry for the downtime, but this is absolutely necessary, to keep up with our growth, and to continue to serve you in 2013.
For the geeks: We’re upgrading our NAS again, adding drives that are much faster, which should result in “Zippier” websites.
Thank you for your patience. The upgrade has gone smoothly, and all systems are up and running. We’ve gone through an internal check list.
There is a very small chance that we missed something, if something is not working for you, please open a ticket, and we’ll look into it immediately.
This is just a reminder, we will be conducting a major system upgrade in 5 hours. All systems will go offline.
The maintenance has been scheduled for 0000 CST on Friday February 22nd 2013.
We’re experiencing a major system outage due to human error at Rackspace. We’re working to fix this as soon as possible.
Pressable Status 