[RESOLVED] Issue With Slow and Improperly Loading Sites

Howdy,

We are currently addressing an issue that is causing some sites to load very slowly and not at all in some cases. Some sites are also seeing a “Coming Soon” Pressable page.

We are addressing this issue now and will update here once the issue is resolved.

If you have any questions or concerns, please submit a ticket via my.pressable.com

UPDATE 08/13/2014 @ 11:45 AM Central: Sites that are/were affected by this are now loading but are still a little a slow in some cases. We are continuing to address this issue where it is present and will update with further information as it becomes available.

UPDATE 08/13/2014 @ 1:15 PM Central: The team is continuing to address issues with slow and improperly loading sites.

UPDATE 08/13/2014 @ 2:20 PM Central: The team is continuing to address issues with slow and improperly loading sites.

UPDATE 08/14/2014 @ 1:45 PM Central: All servers are now stable and running properly. We have made many improvements that will help prevent similar issues going forward. We will be making a blog post soon that details these improvements and explaining the issues as they occurred shortly. If you have questions or concerns, please contact us via my.pressable.com if you have questions or concerns. Thank you for your patience and support, our team truly appreciates it!

RESOLVED: Instability on Web Clusters

Howdy,

We are experiencing suspicious activity that is causing instability on our some of our web server clusters.

We are working on expanding the stability/performance on this cluster and identifying the source of this suspicious activity. During this time, customers may experience intermittent 502s and slow loading sites.

We will update the status blog as soon as this is finalized.

UPDATE 2:50PM CST: The team is still working to bring new hardware and firewall rules online to help mitigate the attacks on our systems. We’ve brought new hardware online for some customers which is beginning to return some stability to the systems.

UPDATE 4:00PM CST: At this time we’ve brought on our new hardware and firewall rules for all of the customers on Thor. We’re still working to get these in place for our customers on Galaxy01.

UPDATE 5:55PM CST: At this time we’ve brought up new hardware and firewall rules for all customers. Stability should be returning to the web servers, but we’re still experiencing database issues.

RESOLVED: Service Interruption

We’re currently investigating an issue with site availability and system stability. We’ll provide an update as we have more details.

UPDATE 12:55PM CST: The team has identified an issue with one of the database servers and is currently evaluating the system.

UPDATE 1:20PM CST: The team is currently looking at an issue with one of the physical drives in the database server.

UPDATE 2:00PM CST: At this time we’ve restored basic services to sites. The team is still working to get the database servers back to 100%, but sites should be operational.

NOTICE: For the next several hours some of the database slaves will be serving stale data. This will occur while our team works to bring up new slave database servers. If you’re having issues with new content not showing up on your site, this is why. If you’re working to publish new content, and having problems, please contact help@pressable.com and our team can assist.

UPDATE 5:00PM CST: Our team is investigating load issues from the earlier database problems. This is causing sites to be down and unresponsive.

UPDATE 10:00PM CST: Site stability has returned, however, we’re still waiting for the database system to return to 100%.

IMPORTANT NOTICE: For the next several hours some of the database slaves will be serving stale data. This will occur while our team works to bring up new slave database servers. If you’re having issues with new content not showing up on your site, this is why. If you’re working to publish new content, and having problems, please contact help@pressable.com and our team can assist.

UPDATE 8/10/14 7:50AM: We’re still working to get the databases servers back to 100%. During this time database slaves will be serving stale data. This will occur while our team works to bring up new slave database servers. If you’re having issues with new content not showing up on your site, plugin changes not saving, or comments not posting it’s related to this issue. If you need to publish new content or make updates, please contact help@pressable.com and our team can assist.

UPDATE 8/10/14 12:33 PM: We are continuing to work on the database servers. During this time, we have put some changes into place on our Thor cluster that will help improve stability and performance. You may experience a 502 during this time and this will clear once all changes have been finalized. Please contact us through my.pressable.com if you have questions or concerns.

UPDATE 8/10/14 4:25PM: Just as a reminder, we’re not out of the woods with the database issues. We’re still working to get back to 100% and are dealing with stale data and sites not updating. If you’re having issues with new content not showing up on your site, plugin changes not saving, or comments not posting it’s related to this issue. If you need to publish new content or make updates, please contact help@pressable.com and our team can assist.

UPDATE 8/10/14 6:00PM: We’re still working to get back to 100% and are dealing with stale data and sites not updating. If you’re having issues with new content not showing up on your site, plugin changes not saving, or comments not posting it’s related to this issue. If you need to publish new content or make updates, please contact help@pressable.com and our team can assist.

UPDATE 8/10/14 7:20PM: At this time we’ve restored two new database slaves and put them in place for all customers. We’re continuing to monitor the database servers and make sure that things work properly, but early signs are positive. If you’re still having issues, please email help@pressable.com and we’ll take a look.

UPDATE 8/11/14 8:55AM: Overnight we noticed that one of the new servers we put in place wasn’t able to keep up with the demand of traffic. This caused momentary issues where it would serve stale data while other servers would serve the latest data. We’ve now replaced this machine and stability has returned to the systems. If you’re still having issues, please email help@pressable.com and we’ll take a look.

RESOLVED: Issue Causing Old Data To Display On Some Sites

We are experiencing an issue that is causing some sites on the network to show/display old data. This same issue is also causing some data to show as not saving properly.

We are working on addressing this issue and will update the status blog as soon as things have returned to normal.

If you have questions or concerns, please submit a ticket via my.pressable.com, thanks!

UPDATE 7:15PM CST: The issues causing some database servers to return older (stale) data has been corrected and all sites should be functioning normally now.

RESOLVED: Potential Mail Delivery Issue

We are experiencing issues with outbound email notifications from our systems. This means password reset, comment notification, registration, and other similiar types of emails are not being delivered. We are working with our email delivery provider SendGrid, to resolve this issue.

If you have any questions or concerns, please submit a ticket for through my.pressable.com

Thank you!

UPDATE 08/01/2014 @ 8:46 AM CST: We are still working on clearing this issue up. Email that is coming from websites is still not delivering properly and we are continuing to work toward resolution.

UPDATE 08/01/2014 @ 3:47 PM CST: We have notified SendGrid and asked them to make changes on their end that will get email flowing again. We will update here as soon as mail is back up and running.

UPDATE 08/02/2014 @ 10:00 PM CST: We have still been unable to get email service restored and are continuing on working toward a solution here. We apologize for the continued issues for those who are affected by this and will update again once further progress is made.

UPDATE 08/04/2014 @ 1:15 PM CST: At this time email services have been restored across our systems. If your site is still having issues sending email, please contact us so that we may investigate. Apologies for the delay in getting this resolved.

502 Errors on Galaxy01 Cluster

We had a brief issue with out Galaxy01 cluster that caused sites on these servers to send back a 502. We have addressed this issue and everything is now working as expected.

If you continue to experience issues, please let us know by submitting a ticket through your my.pressable.com panel.

Thank you!

Outage – Unresponsive Servers

Howdy,

We are currently experiencing an issue with a few servers that are not responding properly. This is causing sites on these clusters to not load properly or at all in some cases. We are working swiftly to bring these servers back online and resolving this issue.

If you have any questions, please submit a ticket to our help desk via http://my.pressable.com

UPDATE July 17th, 2014 at 3:01 PM Central:  Most customers are seeing sites back up and running by now. We have more servers coming back up as this is posted and service will be restored entirely to all sites shortly.

UPDATE July 17th, 2014 at 4:01 PM Central: All sites have been restored to full service by now. If for any reason you are still experiencing issues with a site being down, please let us know and we can address it accordingly.

UPDATE July 17th, 2014 at 5:30 PM Central: We are experiencing issues on our “thor” cluster still that is causing some sites to be very slow. We are working to restore sites on this cluster back to full speed and functionality.

UPDATE July 17th, 2014 at 5:30 PM Central: All servers in the “thor” cluster are now back in full functioning order. Sites will now run at their expected speed now. Thanks for your continued patience while we resolved issues here.

4th of July Holiday Support Hours

Howdy,

In observance of the holiday on July 4th, 2014, we will have a reduced support staff on hand and will be operating with minimal support response. We will resume operating under normal support hours on July 5th, 2014.

Minimal Support Response means that we won’t be available to respond to non-emergency requests. There will be support staff standing by, but we use our best judgement to determine emergency issues that need our attention immediately.

We hope everyone enjoys their holiday and look forward to serving you through the rest of the year.

Thank you!

Limited functionality with database. – Emergency Maintenance

Symptoms

Some customers are experiencing limited functionality with the database systems, the symptoms appear to be that their posts and updates are not saving. Customers may also be experiencing issues approving comments or changing site/plugin settings. New site deployments and clones may also not be working.

What caused this?

This is due to a problem in our database cluster, currently when you save a new post, or update an existing one, we write to one server, and then the changes are propagated to 4 other servers. The link between the server you write to, and the servers that replicate the changes has been severed. This gives the illusion of things not being saved.

Time to resolve this issue

We are working to resolve this issue as soon as we can, in order to do so, we will need to make a change to the system where you will indeed not be able to make any changes for about 10 minutes.

We will provide an update to this post when the maintenance has begun. We do not expect any downtime, just the inability to save a post or new posts. Your website will continue to be served to your visitors without any issues.

UPDATE 10:10AM CDT: We’ve begun the maintenance on the database server and expect things to go smoothly. We’ll provide an update when the maintenance is completed.

UPDATE 10:35AM CDT: We’re currently experiencing an issue with the database maintenance that’s causing an interruption to service. We’re working to move traffic to one of the secondary servers and bring things back online.

UPDATE 10:55AM CDT: We’ve brought secondary servers online to help bring things back online. We’re still working through remaining issues with the emergency maintenance.

UPDATE 11:40AM CDT: Our team is still working on the ongoing database issue.

UPDATE 12:30PM CDT: The team is still hard at work resolving database issues.

UPDATE 1:30PM CDT: We’re expecting things to be more stable at this time and most sites to have returned to normal. If you’re still experiencing issues please let our support team know so we can investigate for you.

Database Maintenance: Friday, May 30, 1:00am CDT

On Friday, May 30th, 2014 at 1:00am we’ll be performing maintenance to our database servers. We do not expect this maintenance to be disruptive, however, some users may experience brief interruptions in service.

This maintenance is part of our ongoing efforts to continue improving the reliability and stability of our platform.

We’ll provide an update when the maintenance has been completed.

If you have any questions, please contact the support team through the https://my.pressable.com control panel.

Update May 30th, 0150

The maintenance was successful and is now complete.