Category Archives: emergency

Emergency Security Maintenance

Howdy,

We received notice this afternoon that our provider, Rackspace, has identified a vulnerability in Xen Hypervisor. This vulnerability has been patched by Rackspace but it requires a reboot of cloud servers in order to for it to take.

You can read more about the vulnerability and requirements here:

https://community.rackspace.com/general/f/53/t/5187
http://venom.crowdstrike.com

We have elected to reboot cloud servers in our network at our discretion rather than allowing Rackspace to reboot using a maintenance window approach.

We will begin the process of rebooting cloud servers tonight, May 13th, at 10:00 PM Central time. We expect this process to take several hours and customers will see intermittent outages of varying length.

We will update this page when the maintenance is completed.

If you have any questions or concerns, please contact our help desk.

We apologize in advance for any inconvenience this may cause and are working toward addressing this as smoothly as possible.

Thank you!

UPDATE – May 14th, 2015 @ 2:57 AM Central: We have completed the reboots that are necessary as a part of this vulnerability patch. All systems are back up, running, and stable. Thank you for your patience and please let us know if you have any questions. Thanks!

Rackspace Scheduled Critical Maintenance

This is a notice that Rackspace will be performing critical security related updates to many cloud server host machines in order to patch vulnerabilities in Xen Hypervisor.

You can read more about this maintenance here:

https://community.rackspace.com/general/f/53/t/4978

These patches/updates will require host machines to be rebooted, subsequently causing cloud servers hosted on them to require a reboot as well.

As it relates to our customers, here are the maintenance windows that we have been provided with and can expect we will begin seeing server reboots occur based on cluster:

  • Hyperion, Pegasus, Cartwheel Clusters
    • Tuesday, March 3rd 01:00 – Tuesday, March 3rd 05:00 EST COMPLETE
  • Galaxy01, Thor, Bode, Ursa, Hydra Clusters
    • Wednesday, March 4th 22:00 – Thursday, March 5th 06:00 CST
    • Thursday, March 5th 22:00 – Friday, March 6th 02:00 CST

To find out which cluster your sites are on, please reference our knowledge base article on identifying which cluster your site is on.

We definitely understand these kinds of outages are not ideal but we are hoping this early notice is helpful in the way of being able to notify your users, visitors, and customers.

If you have any questions, please feel free to contact the help desk via your my.pressable.com control panel.

Thank you!

Emergency Maintenance: December 24, 2014 at 3:00AM CST

On December 23, 2014 an issue was discovered with a database cluster serving sites using the “cartwheel” hostname. The issue discovered has been impacting performance of these sites as well as causing intermittent downtime issues. In order to correct the issue, we need to perform an emergency maintenance on these servers. This maintenance will require a period of downtime while we work to repair the servers and restore service levels back to normal levels.

MAINTENANCE INFORMATION: The maintenance window is expected to begin at 3:00AM CST on December 24, 2014 and expected to last approximately 2 hours. We DO NOT expect sites to be down during this entire period, but there may be periods of connectivity loss.

We will provide updates to the status of this maintenance in this post.

UPDATE 12/24/2014 @ 6:50AM CST: At this time we have completed the maintenance and are starting to see traffic return to sites. Our apologies about the short notice and extended periods of downtime. If you have any questions, please don’t hesitate to reach out.

RESOLVED: Potential Mail Delivery Issue

We are experiencing issues with outbound email notifications from our systems. This means password reset, comment notification, registration, and other similiar types of emails are not being delivered. We are working with our email delivery provider SendGrid, to resolve this issue.

If you have any questions or concerns, please submit a ticket for through my.pressable.com

Thank you!

UPDATE 08/01/2014 @ 8:46 AM CST: We are still working on clearing this issue up. Email that is coming from websites is still not delivering properly and we are continuing to work toward resolution.

UPDATE 08/01/2014 @ 3:47 PM CST: We have notified SendGrid and asked them to make changes on their end that will get email flowing again. We will update here as soon as mail is back up and running.

UPDATE 08/02/2014 @ 10:00 PM CST: We have still been unable to get email service restored and are continuing on working toward a solution here. We apologize for the continued issues for those who are affected by this and will update again once further progress is made.

UPDATE 08/04/2014 @ 1:15 PM CST: At this time email services have been restored across our systems. If your site is still having issues sending email, please contact us so that we may investigate. Apologies for the delay in getting this resolved.

Emergency Database System Maintenance

Leave a reply

We need to perform an emergency maintenance on our database systems to correct an issue with replication. This will bring the database servers offline briefly, but they should return to normal status shortly after that.

We’ll update this post when the maintenance is completed.

UPDATE 12:05PM CST: At this time the issue with the primary database server should be resolved. A small number of customers using HyperDB may still be seeing issues. We’re working to resolve those and will provide an update as soon as things are resolved.