We had a brief issue with out Galaxy01 cluster that caused sites on these servers to send back a 502. We have addressed this issue and everything is now working as expected.

If you continue to experience issues, please let us know by submitting a ticket through your my.pressable.com panel.

Thank you!

Howdy,

We are currently experiencing an issue with a few servers that are not responding properly. This is causing sites on these clusters to not load properly or at all in some cases. We are working swiftly to bring these servers back online and resolving this issue.

If you have any questions, please submit a ticket to our help desk via http://my.pressable.com

UPDATE July 17th, 2014 at 3:01 PM Central:  Most customers are seeing sites back up and running by now. We have more servers coming back up as this is posted and service will be restored entirely to all sites shortly.

UPDATE July 17th, 2014 at 4:01 PM Central: All sites have been restored to full service by now. If for any reason you are still experiencing issues with a site being down, please let us know and we can address it accordingly.

UPDATE July 17th, 2014 at 5:30 PM Central: We are experiencing issues on our “thor” cluster still that is causing some sites to be very slow. We are working to restore sites on this cluster back to full speed and functionality.

UPDATE July 17th, 2014 at 5:30 PM Central: All servers in the “thor” cluster are now back in full functioning order. Sites will now run at their expected speed now. Thanks for your continued patience while we resolved issues here.

We’re working to resolve an issue where a bad web server configuration was pushed out to production web servers. This is causing all web servers to be down at the moment while we work to roll this back.

We apologize for any issues this causes you.

UPDATE 1:10PM CST: Our team is still working on the best method to roll the configuration back. We’ll provide on going updates as we have more information.

UPDATE 1:25PM CST: Our team is still working to resolve this issue.

UPDATE 1:32PM CST: Our team has rolled out a fix for this issue and sites should begin responding normally shortly. Our apologies again for any issues this caused you.

We’re currently investigating a connectivity issue inside our Rackspace Data Center which is causing dropped connections to sites. We’ll provide an update when we have more information.

UPDATE 4:55PM CST: One of our firewall edge devices was overloaded and shutdown causing the interruption in service. We’re still investigating the root cause of this, but things appear to be coming back online at this time.

UPDATE 5:05PM CST: We’re still investigating this with our provider, but we believe the edge device is having a hardware failure that is causing the problems. We’re working to prep a replacement as quickly as possible.

UPDATE 5:15PM CST: Rackspace is having a replacement edge device prepped and put in to network at this time. We’ll provide an update as soon as the new device is online and operational.

UPDATE 6:05PM CST: Rackspace confirmed that there was not a hardware failure, but rather a DoS attack was being directed at our edge devices. At this time the core network team has blocked these attacks and we’re continuing to gather more information. We expect things to be more stable at this time.

We’re still working to recover back to 100% from the database issues earlier this week. We’re aware of some current issues while this process continues. We appreciate your patience while our team works to resolve all outstanding issues.

Currently the team is still working to get services back to 100%. This means that we may still be experiencing performance issues and outages. We apologize for this, and we’re working as quickly as possible to get things stable.

As part of the outages, new site deployments are halted for the time being while we work to stabilize things. This impacts all new and existing customers… we’re sorry, but we need to get stabilized before we can bring on more sites.

Also, any customers who are still experiencing issues with their site content missing, or the site not looking like their own site, should email help@pressable.com. We’re working quickly to respond to these, but there may be a delay in responses.

Apologies again for the issues. We’ll continue to provide updates throughout the day in this post.

Also, a special thanks from the everyone on the Pressable team for all the kind words and support we’ve received through back channels and in comments to tickets. It really means a lot to have that support through these hard times.

UPDATE 4/24/14 at 3:45PM Central: At this time we’ve restored the ability to provision new sites on our systems. This means customers can add new sites to their accounts, or clone existing sites. Any sites that are stuck deploying or cloning will be removed and then you’ll be able to set those up again.

We are currently experiencing an issue with a database that is causing some sites in our network to be down and/or unresponsive. We are working quickly to address the database failure and will update this once service has been restored or as more information becomes available.

NOTE: This impacts the availability of our control panel, provisioning systems, and SFTP access for ALL customers.

NOTE 2: You may be seeing WordPress installation screens on your sites during the import process. This is because no database exists for your site at this time. However, once your site hits the restore process, whatever exists will be dropped from the database and your site will be restored.

UPDATE 4/22/14 at 5:00PM Central: We are continuing to work through this issue. Many sites are back up and running and more are coming up every moment. Once all sites are back up and running, we will update once more. If you have questions or concerns, please submit a ticket via my.pressable.com.

UPDATE 4/22/14 at 6:00PM Central: We are currently working through some additional issues with our database servers. We’re still expecting some downtime while we investigate the underlying cause of the current issues.

UPDATE 4/22/14 at 7:50PM Central: We are still investigating issues with our database servers that are causing sites to become unavailable, unresponsive and slow. Our team is working through the problems, but we still do not have an ETA for complete resolution.

UPDATE 4/22/14 at 8:35PM Central: Our team is taking measures to restrict access to the database servers while we continue to investigate ongoing issues. Most sites will be unavailable while our team continues to resolve this issue.

UPDATE 4/22/14 at 10:20PM Central: There is nothing new to report at this time.

UPDATE 4/22/14 at 11:45PM Central: Our team is still working to resolve issues. At this time there is no further information to report.

UPDATE 4/23/14 at 12:30AM Central: At this time our team is beginning a restore process on the database server. While this isn’t the ideal solution, we’ve exhausted other recovery options with the server. Due to the nature of the restore, sites will be rolled back to about 4/22/14 at 3:30pm Central. We expect to have things restored by 4/23/14 at 5:00am Central, but we’ll provide updates as the process continues.

UPDATE 4/23/14 at 5:00AM Central: At this time databases are still being restored by our systems. We do not have a new ETA, but we’re on the home stretch of getting the primary database server restored. A note about the process: You may be seeing WordPress installation screens on your sites during the import process. This is because no database exists for your site at this time. However, once your site hits the restore process, whatever exists will be dropped from the database and your site will be restored.

UPDATE 4/23/14 at 7:00AM Central: Databases are still being restored by our systems. We do not have an updated ETA at this time, but the restore process is still working as expected.

UPDATE 4/23/14 at 8:00AM Central: We’re continuing to work through the restore process.

UPDATE 4/23/14 at 9:30AM Central: We’re working through some lingering issues with individual sites at this time. We’re also prepping to have the slaves rebuilt from the restore point, we’ll provide more details shortly.

UPDATE 4/23/14 at 1:15PM Central: We’re still working to get everyone’s database imported from the backup. If you’re still having issues, please submit a ticket so that we’re able to resolve it for you. We’re also still experiencing some instability and performance issues while we work to recover to 100%.

UPDATE 4/23/14 at 8:30PM Central: Our team is still working to restore databases for sites who have been missed or had issues due to the automated processes used. If you’re experiencing issues, please submit a ticket so our team can get it in the queue to be processed.

We’re currently investigating an issue with our provider (Rackspace) that is causing sites to be slow or intermittently down. We’ll provide an update as soon as we have more details.

Update (4/22/14/ at 12:43 PM Central): Things have now stabilized and site availability and performance have returned to normal. If you continue to experience issues or have questions, please contact our helpdesk by submitting a ticket via https://my.pressable.com, thank you! 

We’re currently investigating an issue with connectivity between servers in our Dallas Datacenter. The issue may cause slow downs or sites to be temporarily unavailable. We’ll provide an update as soon as we have more information.