RESOLVED: Potential Mail Delivery Issue

We are experiencing issues with outbound email notifications from our systems. This means password reset, comment notification, registration, and other similiar types of emails are not being delivered. We are working with our email delivery provider SendGrid, to resolve this issue.

If you have any questions or concerns, please submit a ticket for through my.pressable.com

Thank you!

UPDATE 08/01/2014 @ 8:46 AM CST: We are still working on clearing this issue up. Email that is coming from websites is still not delivering properly and we are continuing to work toward resolution.

UPDATE 08/01/2014 @ 3:47 PM CST: We have notified SendGrid and asked them to make changes on their end that will get email flowing again. We will update here as soon as mail is back up and running.

UPDATE 08/02/2014 @ 10:00 PM CST: We have still been unable to get email service restored and are continuing on working toward a solution here. We apologize for the continued issues for those who are affected by this and will update again once further progress is made.

UPDATE 08/04/2014 @ 1:15 PM CST: At this time email services have been restored across our systems. If your site is still having issues sending email, please contact us so that we may investigate. Apologies for the delay in getting this resolved.

502 Errors on Galaxy01 Cluster

We had a brief issue with out Galaxy01 cluster that caused sites on these servers to send back a 502. We have addressed this issue and everything is now working as expected.

If you continue to experience issues, please let us know by submitting a ticket through your my.pressable.com panel.

Thank you!

Outage – Unresponsive Servers

Howdy,

We are currently experiencing an issue with a few servers that are not responding properly. This is causing sites on these clusters to not load properly or at all in some cases. We are working swiftly to bring these servers back online and resolving this issue.

If you have any questions, please submit a ticket to our help desk via http://my.pressable.com

UPDATE July 17th, 2014 at 3:01 PM Central:  Most customers are seeing sites back up and running by now. We have more servers coming back up as this is posted and service will be restored entirely to all sites shortly.

UPDATE July 17th, 2014 at 4:01 PM Central: All sites have been restored to full service by now. If for any reason you are still experiencing issues with a site being down, please let us know and we can address it accordingly.

UPDATE July 17th, 2014 at 5:30 PM Central: We are experiencing issues on our “thor” cluster still that is causing some sites to be very slow. We are working to restore sites on this cluster back to full speed and functionality.

UPDATE July 17th, 2014 at 5:30 PM Central: All servers in the “thor” cluster are now back in full functioning order. Sites will now run at their expected speed now. Thanks for your continued patience while we resolved issues here.

4th of July Holiday Support Hours

Howdy,

In observance of the holiday on July 4th, 2014, we will have a reduced support staff on hand and will be operating with minimal support response. We will resume operating under normal support hours on July 5th, 2014.

Minimal Support Response means that we won’t be available to respond to non-emergency requests. There will be support staff standing by, but we use our best judgement to determine emergency issues that need our attention immediately.

We hope everyone enjoys their holiday and look forward to serving you through the rest of the year.

Thank you!

Limited functionality with database. – Emergency Maintenance

Symptoms

Some customers are experiencing limited functionality with the database systems, the symptoms appear to be that their posts and updates are not saving. Customers may also be experiencing issues approving comments or changing site/plugin settings. New site deployments and clones may also not be working.

What caused this?

This is due to a problem in our database cluster, currently when you save a new post, or update an existing one, we write to one server, and then the changes are propagated to 4 other servers. The link between the server you write to, and the servers that replicate the changes has been severed. This gives the illusion of things not being saved.

Time to resolve this issue

We are working to resolve this issue as soon as we can, in order to do so, we will need to make a change to the system where you will indeed not be able to make any changes for about 10 minutes.

We will provide an update to this post when the maintenance has begun. We do not expect any downtime, just the inability to save a post or new posts. Your website will continue to be served to your visitors without any issues.

UPDATE 10:10AM CDT: We’ve begun the maintenance on the database server and expect things to go smoothly. We’ll provide an update when the maintenance is completed.

UPDATE 10:35AM CDT: We’re currently experiencing an issue with the database maintenance that’s causing an interruption to service. We’re working to move traffic to one of the secondary servers and bring things back online.

UPDATE 10:55AM CDT: We’ve brought secondary servers online to help bring things back online. We’re still working through remaining issues with the emergency maintenance.

UPDATE 11:40AM CDT: Our team is still working on the ongoing database issue.

UPDATE 12:30PM CDT: The team is still hard at work resolving database issues.

UPDATE 1:30PM CDT: We’re expecting things to be more stable at this time and most sites to have returned to normal. If you’re still experiencing issues please let our support team know so we can investigate for you.

Database Maintenance: Friday, May 30, 1:00am CDT

On Friday, May 30th, 2014 at 1:00am we’ll be performing maintenance to our database servers. We do not expect this maintenance to be disruptive, however, some users may experience brief interruptions in service.

This maintenance is part of our ongoing efforts to continue improving the reliability and stability of our platform.

We’ll provide an update when the maintenance has been completed.

If you have any questions, please contact the support team through the https://my.pressable.com control panel.

Update May 30th, 0150

The maintenance was successful and is now complete.

Memorial Day Support Hours

Howdy,

The year is nearly half way done and Memorial Day is upon us!

In observance of the holiday on May 26th, 2014, we will have a reduced support staff on hand and will be operating with minimal support response. We will resume operating under normal support hours on May 27th, 2014.

Minimal Support Response means that we won’t be available to respond to non-emergency requests. There will be support staff standing by, but we use our best judgement to determine emergency issues that need our attention immediately.

We hope everyone enjoys their holiday and look forward to serving you through the rest of the year.

Thank you!

RESOLVED: Web Server Configuration Issue

We’re working to resolve an issue where a bad web server configuration was pushed out to production web servers. This is causing all web servers to be down at the moment while we work to roll this back.

We apologize for any issues this causes you.

UPDATE 1:10PM CST: Our team is still working on the best method to roll the configuration back. We’ll provide on going updates as we have more information.

UPDATE 1:25PM CST: Our team is still working to resolve this issue.

UPDATE 1:32PM CST: Our team has rolled out a fix for this issue and sites should begin responding normally shortly. Our apologies again for any issues this caused you.

Connectivity Issue inside Rackspace Data Center

We’re currently investigating a connectivity issue inside our Rackspace Data Center which is causing dropped connections to sites. We’ll provide an update when we have more information.

UPDATE 4:55PM CST: One of our firewall edge devices was overloaded and shutdown causing the interruption in service. We’re still investigating the root cause of this, but things appear to be coming back online at this time.

UPDATE 5:05PM CST: We’re still investigating this with our provider, but we believe the edge device is having a hardware failure that is causing the problems. We’re working to prep a replacement as quickly as possible.

UPDATE 5:15PM CST: Rackspace is having a replacement edge device prepped and put in to network at this time. We’ll provide an update as soon as the new device is online and operational.

UPDATE 6:05PM CST: Rackspace confirmed that there was not a hardware failure, but rather a DoS attack was being directed at our edge devices. At this time the core network team has blocked these attacks and we’re continuing to gather more information. We expect things to be more stable at this time.

Load Spikes Causing Slow Performing Sites

We are investigating an issue that is causing some sites to respond slowly and/or have trouble resolving pages.

We will update here again as soon as we have more information but we expect this will be intermittent while we work on resolving this issue.

If you have any questions or concerns, please submit a ticket to our help desk via my.pressable.com

Update 01:00PM Central, May 13th, 2014: Things have stabilized but our team is working on adding more capacity. We anticipate this will be done soon, but for now, most sites are functioning normally.