Author Archives: Jeff

Network Degredation Impacting Site Availability

We’re currently experiencing an issue with our network causing a complete degradation of services and loss of traffic. We’re working with our provider to determine the cause and source of the issues, but early signs point to a targeted attack against our systems. We’ll provide more details as they become available.

UPDATE 3:25PM CST: We’re still working with our provider to determine the source of increased traffic and to correct any issues.

UPDATE 3:55PM CST: At this time things appear to have stabilizied. However, we’re still working with our provider to determine the root cause of the issues and put any nesseceary measures in place to prevent any similar issues.

Site Availability issue on Galaxy01

We’re currently experiencing an issue with our Galaxy01 cluster of servers. This issue is causing intermittent 500/502 errors while processes fail to load. We’re currently evaluating the system and working to restore services to normal operational levels as quickly as possible. We’ll update this post with more information as we have it.

UPDATE 10:00AM CST: The team is still working to restore this cluster of servers to 100%. We’re getting closer as some new firewall rules come online and additional capacity. We’ll provide another update shortly.

UPDATE 10:45AM CST: At this time services are beginning to return to normal across the affected servers. We’re still waiting on some new rules to finish processing, but things are trending in a positive direction.

Slider Revolution Plugin Security Vulnerability

IMPORTANT: Sucuri has released details on an exploit for the plugin Slider Revolution Responsive WordPress Plugin. Details of the exploit can be seen on Sucuri’s blog.

IF YOU ARE USING THIS PLUGIN, PLEASE UPDATE TO THE LATEST VERSION IMMEDIATELY.

Our team is working on a way to block these attacks, but we can NOT update this plugin on sites as it’s not freely available on the WordPress.org repository.

RESOLVED: Service Interruption

We’re currently investigating an issue with site availability and system stability. We’ll provide an update as we have more details.

UPDATE 12:55PM CST: The team has identified an issue with one of the database servers and is currently evaluating the system.

UPDATE 1:20PM CST: The team is currently looking at an issue with one of the physical drives in the database server.

UPDATE 2:00PM CST: At this time we’ve restored basic services to sites. The team is still working to get the database servers back to 100%, but sites should be operational.

NOTICE: For the next several hours some of the database slaves will be serving stale data. This will occur while our team works to bring up new slave database servers. If you’re having issues with new content not showing up on your site, this is why. If you’re working to publish new content, and having problems, please contact help@pressable.com and our team can assist.

UPDATE 5:00PM CST: Our team is investigating load issues from the earlier database problems. This is causing sites to be down and unresponsive.

UPDATE 10:00PM CST: Site stability has returned, however, we’re still waiting for the database system to return to 100%.

IMPORTANT NOTICE: For the next several hours some of the database slaves will be serving stale data. This will occur while our team works to bring up new slave database servers. If you’re having issues with new content not showing up on your site, this is why. If you’re working to publish new content, and having problems, please contact help@pressable.com and our team can assist.

UPDATE 8/10/14 7:50AM: We’re still working to get the databases servers back to 100%. During this time database slaves will be serving stale data. This will occur while our team works to bring up new slave database servers. If you’re having issues with new content not showing up on your site, plugin changes not saving, or comments not posting it’s related to this issue. If you need to publish new content or make updates, please contact help@pressable.com and our team can assist.

UPDATE 8/10/14 12:33 PM: We are continuing to work on the database servers. During this time, we have put some changes into place on our Thor cluster that will help improve stability and performance. You may experience a 502 during this time and this will clear once all changes have been finalized. Please contact us through my.pressable.com if you have questions or concerns.

UPDATE 8/10/14 4:25PM: Just as a reminder, we’re not out of the woods with the database issues. We’re still working to get back to 100% and are dealing with stale data and sites not updating. If you’re having issues with new content not showing up on your site, plugin changes not saving, or comments not posting it’s related to this issue. If you need to publish new content or make updates, please contact help@pressable.com and our team can assist.

UPDATE 8/10/14 6:00PM: We’re still working to get back to 100% and are dealing with stale data and sites not updating. If you’re having issues with new content not showing up on your site, plugin changes not saving, or comments not posting it’s related to this issue. If you need to publish new content or make updates, please contact help@pressable.com and our team can assist.

UPDATE 8/10/14 7:20PM: At this time we’ve restored two new database slaves and put them in place for all customers. We’re continuing to monitor the database servers and make sure that things work properly, but early signs are positive. If you’re still having issues, please email help@pressable.com and we’ll take a look.

UPDATE 8/11/14 8:55AM: Overnight we noticed that one of the new servers we put in place wasn’t able to keep up with the demand of traffic. This caused momentary issues where it would serve stale data while other servers would serve the latest data. We’ve now replaced this machine and stability has returned to the systems. If you’re still having issues, please email help@pressable.com and we’ll take a look.

Limited functionality with database. – Emergency Maintenance

Symptoms

Some customers are experiencing limited functionality with the database systems, the symptoms appear to be that their posts and updates are not saving. Customers may also be experiencing issues approving comments or changing site/plugin settings. New site deployments and clones may also not be working.

What caused this?

This is due to a problem in our database cluster, currently when you save a new post, or update an existing one, we write to one server, and then the changes are propagated to 4 other servers. The link between the server you write to, and the servers that replicate the changes has been severed. This gives the illusion of things not being saved.

Time to resolve this issue

We are working to resolve this issue as soon as we can, in order to do so, we will need to make a change to the system where you will indeed not be able to make any changes for about 10 minutes.

We will provide an update to this post when the maintenance has begun. We do not expect any downtime, just the inability to save a post or new posts. Your website will continue to be served to your visitors without any issues.

UPDATE 10:10AM CDT: We’ve begun the maintenance on the database server and expect things to go smoothly. We’ll provide an update when the maintenance is completed.

UPDATE 10:35AM CDT: We’re currently experiencing an issue with the database maintenance that’s causing an interruption to service. We’re working to move traffic to one of the secondary servers and bring things back online.

UPDATE 10:55AM CDT: We’ve brought secondary servers online to help bring things back online. We’re still working through remaining issues with the emergency maintenance.

UPDATE 11:40AM CDT: Our team is still working on the ongoing database issue.

UPDATE 12:30PM CDT: The team is still hard at work resolving database issues.

UPDATE 1:30PM CDT: We’re expecting things to be more stable at this time and most sites to have returned to normal. If you’re still experiencing issues please let our support team know so we can investigate for you.

Database Maintenance: Friday, May 30, 1:00am CDT

On Friday, May 30th, 2014 at 1:00am we’ll be performing maintenance to our database servers. We do not expect this maintenance to be disruptive, however, some users may experience brief interruptions in service.

This maintenance is part of our ongoing efforts to continue improving the reliability and stability of our platform.

We’ll provide an update when the maintenance has been completed.

If you have any questions, please contact the support team through the https://my.pressable.com control panel.

Update May 30th, 0150

The maintenance was successful and is now complete.

RESOLVED: Web Server Configuration Issue

We’re working to resolve an issue where a bad web server configuration was pushed out to production web servers. This is causing all web servers to be down at the moment while we work to roll this back.

We apologize for any issues this causes you.

UPDATE 1:10PM CST: Our team is still working on the best method to roll the configuration back. We’ll provide on going updates as we have more information.

UPDATE 1:25PM CST: Our team is still working to resolve this issue.

UPDATE 1:32PM CST: Our team has rolled out a fix for this issue and sites should begin responding normally shortly. Our apologies again for any issues this caused you.

Connectivity Issue inside Rackspace Data Center

We’re currently investigating a connectivity issue inside our Rackspace Data Center which is causing dropped connections to sites. We’ll provide an update when we have more information.

UPDATE 4:55PM CST: One of our firewall edge devices was overloaded and shutdown causing the interruption in service. We’re still investigating the root cause of this, but things appear to be coming back online at this time.

UPDATE 5:05PM CST: We’re still investigating this with our provider, but we believe the edge device is having a hardware failure that is causing the problems. We’re working to prep a replacement as quickly as possible.

UPDATE 5:15PM CST: Rackspace is having a replacement edge device prepped and put in to network at this time. We’ll provide an update as soon as the new device is online and operational.

UPDATE 6:05PM CST: Rackspace confirmed that there was not a hardware failure, but rather a DoS attack was being directed at our edge devices. At this time the core network team has blocked these attacks and we’re continuing to gather more information. We expect things to be more stable at this time.

Still Working to Stabilize Database Servers

We’re still working to recover back to 100% from the database issues earlier this week. We’re aware of some current issues while this process continues. We appreciate your patience while our team works to resolve all outstanding issues.

Update on Database Outages this Week

Currently the team is still working to get services back to 100%. This means that we may still be experiencing performance issues and outages. We apologize for this, and we’re working as quickly as possible to get things stable.

As part of the outages, new site deployments are halted for the time being while we work to stabilize things. This impacts all new and existing customers… we’re sorry, but we need to get stabilized before we can bring on more sites.

Also, any customers who are still experiencing issues with their site content missing, or the site not looking like their own site, should email help@pressable.com. We’re working quickly to respond to these, but there may be a delay in responses.

Apologies again for the issues. We’ll continue to provide updates throughout the day in this post.

Also, a special thanks from the everyone on the Pressable team for all the kind words and support we’ve received through back channels and in comments to tickets. It really means a lot to have that support through these hard times.

UPDATE 4/24/14 at 3:45PM Central: At this time we’ve restored the ability to provision new sites on our systems. This means customers can add new sites to their accounts, or clone existing sites. Any sites that are stuck deploying or cloning will be removed and then you’ll be able to set those up again.