Update to Rackspace Emergency Maintenance: Wednesday, October 28

This is an update to the emergency maintenance previously scheduled for October 19 to October 29. This notice applies only to sites which are hosted on the Ursa galaxy at Rackspace’s ORD data center; not all customers and collaborators will be affected by the update.

Affected site owners and collaborators have been notified via email. Additionally, you have the option to identify the galaxy (cluster) on which your site resides by following the instructions in this KB article. If you have any questions or concerns, please contact our help desk (support@pressable.com).

Rackspace will reboot a portion of its cloud infrastructure this Wednesday to patch a security issue. Rackspace first attempted to live-migrate the servers but those attempts failed.

Maintenance is scheduled to take place from 2:00am to 4:00am CDT on Wednesday, October 28. During this 2-hour window, we expect a brief interruption of service to your site(s).

We will be monitoring the situation closely. Additional updates regarding the emergency maintenance will be shared on our status blog.

Update to Rackspace Emergency Maintenance: Sunday, October 25 to Monday, October 26

This is an update to the emergency maintenance previously scheduled for October 19 to October 29. This notice applies only to sites which are hosted at Rackspace’s IAD data center; not all customers and collaborators will be affected by the update.

Affected site owners and collaborators have been notified via email. If you have any questions or concerns, please contact our help desk (support@pressable.com).

Rackspace will reboot a portion of its cloud infrastructure this Sunday to patch a security issue. Rackspace first attempted to live-migrate the servers but those attempts failed.

Maintenance is scheduled to take place from 9:00pm CST on Sunday, October 25 to 1:00am CSTon Monday, October 26. During this 4-hour window, we expect a brief interruption of service to sites which are hosted at Rackspace’s IAD data center.

We will be monitoring the situation closely. Additional updates regarding the emergency maintenance will be shared on our status blog.

Rackspace Emergency Maintenance: October 19 to October 29

Rackspace, our backbone provider, will be performing critical maintenance for a security issue on its cloud infrastructure beginning today. Rackspace will be patching live systems in an attempt to eliminate downtime. Maintenance is scheduled to continue through October 29.

We anticipate that all customers’ sites will remain fully functional during the maintenance window, however, Rackspace may need to reboot some of our cloud infrastructure if the initial live-patching fails. Should a reboot be required, we will notify customers via email and the status blog.

During the maintenance window we will provide additional updates via our status blog.

If you have any questions or concerns, please email support@pressable.com.

WordPress 4.3 Problem

A new problem has been discovered with WordPress 4.3 which is affecting some of the sites on our systems. The problem arises when cron jobs are periodically scheduled and results in unusually high CPU and database utilization, which can impact the performance and availability of sites. You can read more about the problem at WordPress Core Tracker. All sites are currently up, although a chance of this issue resurfacing at a later date is still present as cron jobs are processed.

We’re keeping an eye on the problem and working to quickly identify a solution. In the meantime, we’ll manually resolve any issues as they arise. If you feel that this issue has affected your site, please feel free to contact our help desk (support@pressable.com). Please stand by for an update once we’ve identified a remediation plan for sites that have been affected.

Independence Day Weekend Holiday Hours

From Wikipedia: Independence Day is commonly associated with fireworks, parades, barbecues, carnivals, fairs, picnics, concerts, baseball games, family reunions, and political speeches and ceremonies, in addition to various other public and private events celebrating the history, government, and traditions of the United States.

With that being said, Pressable is taking a much deserved holiday for the 4th of July weekend! What this means for you:

  • July 3rd: There will only be emergency support.

  • July 4th – July 5th: There will be minimal support staffing.

What this means in practice is that on July 3 we won’t be available to respond to non-emergency requests. Generally speaking that means we’ll only be available for system wide outages/issues impacting the customer base.

For the remainder of the holiday weekend, support staff will be limited in availability. This means response times may be slower, or some non-critical items may be held until the following Monday.

We apologize in advance for any inconvenience this may cause you. If you have questions please feel free to direct them to help@pressable.

RESOLVED: Outbound email issues.

UPDATE 7:26 P.M – We are starting to see email come through normally. If you are still having issues, please submit a ticket by emailing us at help@pressable.com.

We are having issues with outbound email from our servers, our engineers are working to resolve these issues. This means emails like comment notifications, password resets and and deployment emails will not be sent immediately, and recipients may get the email a few hours later than expected.

If you are having these issues, send an email to help@pressable.com, or check back here.

Christian Torres
Customer Success Agent
Pressable.com

Memorial Day Support Hours

In observance of Memorial Day on Monday May 25, 2015, we will have a reduced support staff on hand and will be operating with minimal support response on that day. We will resume operating under normal support hours on Tuesday May 26, 2015.

Minimal Support Response means that we won’t be available to respond to non-emergency requests. There will be support staff standing by, but we use our best judgement to determine emergency issues that need our attention immediately.

Emergency Security Maintenance

Howdy,

We received notice this afternoon that our provider, Rackspace, has identified a vulnerability in Xen Hypervisor. This vulnerability has been patched by Rackspace but it requires a reboot of cloud servers in order to for it to take.

You can read more about the vulnerability and requirements here:

https://community.rackspace.com/general/f/53/t/5187
http://venom.crowdstrike.com

We have elected to reboot cloud servers in our network at our discretion rather than allowing Rackspace to reboot using a maintenance window approach.

We will begin the process of rebooting cloud servers tonight, May 13th, at 10:00 PM Central time. We expect this process to take several hours and customers will see intermittent outages of varying length.

We will update this page when the maintenance is completed.

If you have any questions or concerns, please contact our help desk.

We apologize in advance for any inconvenience this may cause and are working toward addressing this as smoothly as possible.

Thank you!

UPDATE – May 14th, 2015 @ 2:57 AM Central: We have completed the reboots that are necessary as a part of this vulnerability patch. All systems are back up, running, and stable. Thank you for your patience and please let us know if you have any questions. Thanks!

XSS Vulnerability Affecting Multiple WordPress Plugins

The Sucuri Blog has notified users of multiple WordPress plugins that are vulnerable to Cross-site Scripting (XSS) attacks. Listed are some of the more popular plugins used in the WordPress community:

Security Advisory: XSS Vulnerability Affecting Multiple WordPress Plugins

The nature of this vulnerability makes it difficult to patch completely/comprehensively because so many plugins use the functions listed as being misused.

We highly recommend logging into your WordPress Dashboard and updating any plugins that have available updates.

If you have any questions or concerns, please contact our help desk by submitting a ticket via your https://my.pressable.com panel.

Concluded: EMERGENCY NETWORK MAINTENANCE: Saturday, April 11, 2015 – 11:30pm to 12:30am CST (ORD)

UPDATE 1:00 AM CST: AT THIS TIME, WE HAVE CONCLUDED THE EMERGENCY MAINTENANCE, SITES SHOULD BE FUNCTIONAL GOING FORWARD, AS WELL AS DEPLOYMENTS OF NEW SITES. CLONING IS STILL NON-FUNCTIONAL AT THIS POINT. IF YOU NEED A SITE CLONED, PLEASE SUBMIT A TICKET VIA YOUR MY.PRESSABLE.COM CONTROL PANEL, AND WE CAN CLONE IT MANUALLY FOR YOU. 

UPDATE 12:30 am CST: We are continuing to work with our partner to complete this emergency maintenance. While we believe that the portion of this that would cause sites to not function is over, it will not be out of the ordinary to see sites go down in the next 30-45 minutes for a short period of time. We will continue to deliver updates here as we look to finish the maintenance. 

Over the past week we’ve been working with our partners to address an issue preventing new infrastructure from coming online inside our Chicago (ORD) Network. While this issue has not been impactful to customer facing sites, it has created issues while we work to expand capacity and grow our systems. Unfortunately, we’re unable to move forward with correcting these issues without temporarily suspending network services.

MAINTENANCE TIMELINE

At approximately 11:30pm CST on April 11, 2015 we’ll begin working with our partners to bring services offline. This involves a reorder and reboot of our network edge devices to correct the issues identified. By 12:30am CST on April 12, 2015 maintenance on our network edge devices will be completed and services will be restored.

WHAT DOES THIS MEAN?

During the maintenance all sites located in our Chicago datacenter will be offline. While we only expect the downtime period to be 15 minutes, the maintenance is scheduled for 1 hour.

Sites will return several different types of errors ranging from 504 timeouts and 503 service unavailable messages. Once network maintenance is completed there may also be a period of 502 errors while we bring our infrastructure back online.

HOW CAN I TELL IF I’LL BE IMPACTED?

If your site is located in our Chicago datacenter you’ll be impacted by this maintenance. There are two easy ways to tell if your site is located here. First you can check if your site is in one of the following clusters:

  • Galaxy01
  • Thor
  • Bode
  • Hydra
  • Ursa

You can also very simply tell the datacenter by checking your SFTP hostname. If your SFTP hostname is:

  • sftp.rax.ord.openhostingservice.com

I HAVE MORE QUESTIONS!

If you have more questions, or are concerned about this maintenance, please do not hesitate to contact our helpdesk. You can do this by submitting a ticket from the https://my.pressable.com control panel, or emailing help@pressable.com.

Our sincerest apologies for the short notice on this maintenance. Our team and partners worked tirelessly to prevent a downtime like this, but ultimately it wasn’t possible.