Emergency Security Maintenance

Howdy,

We received notice this afternoon that our provider, Rackspace, has identified a vulnerability in Xen Hypervisor. This vulnerability has been patched by Rackspace but it requires a reboot of cloud servers in order to for it to take.

You can read more about the vulnerability and requirements here:

https://community.rackspace.com/general/f/53/t/5187
http://venom.crowdstrike.com

We have elected to reboot cloud servers in our network at our discretion rather than allowing Rackspace to reboot using a maintenance window approach.

We will begin the process of rebooting cloud servers tonight, May 13th, at 10:00 PM Central time. We expect this process to take several hours and customers will see intermittent outages of varying length.

We will update this page when the maintenance is completed.

If you have any questions or concerns, please contact our help desk.

We apologize in advance for any inconvenience this may cause and are working toward addressing this as smoothly as possible.

Thank you!

UPDATE – May 14th, 2015 @ 2:57 AM Central: We have completed the reboots that are necessary as a part of this vulnerability patch. All systems are back up, running, and stable. Thank you for your patience and please let us know if you have any questions. Thanks!

XSS Vulnerability Affecting Multiple WordPress Plugins

The Sucuri Blog has notified users of multiple WordPress plugins that are vulnerable to Cross-site Scripting (XSS) attacks. Listed are some of the more popular plugins used in the WordPress community:

Security Advisory: XSS Vulnerability Affecting Multiple WordPress Plugins

The nature of this vulnerability makes it difficult to patch completely/comprehensively because so many plugins use the functions listed as being misused.

We highly recommend logging into your WordPress Dashboard and updating any plugins that have available updates.

If you have any questions or concerns, please contact our help desk by submitting a ticket via your https://my.pressable.com panel.

Concluded: EMERGENCY NETWORK MAINTENANCE: Saturday, April 11, 2015 – 11:30pm to 12:30am CST (ORD)

UPDATE 1:00 AM CST: AT THIS TIME, WE HAVE CONCLUDED THE EMERGENCY MAINTENANCE, SITES SHOULD BE FUNCTIONAL GOING FORWARD, AS WELL AS DEPLOYMENTS OF NEW SITES. CLONING IS STILL NON-FUNCTIONAL AT THIS POINT. IF YOU NEED A SITE CLONED, PLEASE SUBMIT A TICKET VIA YOUR MY.PRESSABLE.COM CONTROL PANEL, AND WE CAN CLONE IT MANUALLY FOR YOU. 

UPDATE 12:30 am CST: We are continuing to work with our partner to complete this emergency maintenance. While we believe that the portion of this that would cause sites to not function is over, it will not be out of the ordinary to see sites go down in the next 30-45 minutes for a short period of time. We will continue to deliver updates here as we look to finish the maintenance. 

Over the past week we’ve been working with our partners to address an issue preventing new infrastructure from coming online inside our Chicago (ORD) Network. While this issue has not been impactful to customer facing sites, it has created issues while we work to expand capacity and grow our systems. Unfortunately, we’re unable to move forward with correcting these issues without temporarily suspending network services.

MAINTENANCE TIMELINE

At approximately 11:30pm CST on April 11, 2015 we’ll begin working with our partners to bring services offline. This involves a reorder and reboot of our network edge devices to correct the issues identified. By 12:30am CST on April 12, 2015 maintenance on our network edge devices will be completed and services will be restored.

WHAT DOES THIS MEAN?

During the maintenance all sites located in our Chicago datacenter will be offline. While we only expect the downtime period to be 15 minutes, the maintenance is scheduled for 1 hour.

Sites will return several different types of errors ranging from 504 timeouts and 503 service unavailable messages. Once network maintenance is completed there may also be a period of 502 errors while we bring our infrastructure back online.

HOW CAN I TELL IF I’LL BE IMPACTED?

If your site is located in our Chicago datacenter you’ll be impacted by this maintenance. There are two easy ways to tell if your site is located here. First you can check if your site is in one of the following clusters:

  • Galaxy01
  • Thor
  • Bode
  • Hydra
  • Ursa

You can also very simply tell the datacenter by checking your SFTP hostname. If your SFTP hostname is:

  • sftp.rax.ord.openhostingservice.com

I HAVE MORE QUESTIONS!

If you have more questions, or are concerned about this maintenance, please do not hesitate to contact our helpdesk. You can do this by submitting a ticket from the https://my.pressable.com control panel, or emailing help@pressable.com.

Our sincerest apologies for the short notice on this maintenance. Our team and partners worked tirelessly to prevent a downtime like this, but ultimately it wasn’t possible.

Site creation/Cloning Issues

We are currently having issues deploying new sites and cloning sites to our ORD Datacenter. Our team is currently looking into getting this issue resolved as quickly as possible. Please note that this only affects new site creations and clones from current sites. This does not affect functionality on current sites.

We will continue to post updates here as we look for a resolution. If you have any further questions, please submit a ticket via your my.pressable.com control panel.

UPDATE 4/4/2015: Deploying sites on our systems is functioning once again. We are still working on the cloning process at this point. 

UPDATE 4/9/2015: We are still working on getting clones working through our automative system. If you are currently waiting on a clone to finish or need one done, please submit a ticket via your my.pressable.com and we will be able to manually clone your site for you. 

Provisioning / Control Panel Issues

One of the machines behind our provisioning system has experienced a hardware failure, we are working on replacing the underlying hardware, and expect the machine to be back online in very soon.

In the mean time, functionality like adding, editing, removing a site, or collaborators of a site is not going to work. We apologize for any inconvenience.

Please note: This only affects our control panel, not your actual website.

Chicago Datacenter Issue – RESOLVED

Howdy,

We just finished dealing with an issue in our Chicago datacenter that was causing several other clusters to experience instability. Our “Ursa” cluster was taking on an extreme amount of traffic that looks to be, in large part, a bot attack.

This happened as a result of the “Ursa” cluster having a set of tools not running appropriately that detects and mitigates issues like this.

We’ve cleared this up and all sites are now back up and running appropriately.

If you have any questions or concerns, please contact our help desk via your https://my.pressable.com control panel.

Thank you!

RESOLVED: Chicago and Virginia Datacenter Outages.

Due to our upstream provider having connectivity issues, we are currently experiencing downtimes at our Chicago and Virginia Datacenters. We are currently working with our provider to correct this issue. We will continue to update this post as we get more information.

UPDATE 4:41 PM CST: After further investigation, it appears that this outage is only affecting customers in our Chicago Datacenter.  We are still trying to gather more information from our provider so we can provide a possible ETA. 

UPDATE 5:15 PM CST: We are still working with our provider in order to diagnose this issue. We should have a more detailed update very soon. 

UPDATE 6:30 PM CST: We are seeing some clusters begin to online. We are working on rolling out the rest of our clusters to full functionality now. We will update once that is done. 

UPDATE 7:47 PM CST: We have restored functionality across our systems. If you are continuing to see issues with your site/s, please submit a ticket via your my.pressable.com control panel. 

 

RESOLVED: WooCommerce SQL Injection Vulnerability

Earlier today the Wordfence Security team released the details of a WooCommerce SQL Injection Vulnerability. Our systems are already at work patching this popular plugin across sites on our systems. We’ll provide an update when the process has been completed.

UPDATE March 14th, 7:55AM CST: At this time all sites on our systems have been updated to the latest (patched) version of WooCommerce. If you have any questions, please don’t hesitate to reach out.

Rackspace Scheduled Critical Maintenance

This is a notice that Rackspace will be performing critical security related updates to many cloud server host machines in order to patch vulnerabilities in Xen Hypervisor.

You can read more about this maintenance here:

https://community.rackspace.com/general/f/53/t/4978

These patches/updates will require host machines to be rebooted, subsequently causing cloud servers hosted on them to require a reboot as well.

As it relates to our customers, here are the maintenance windows that we have been provided with and can expect we will begin seeing server reboots occur based on cluster:

  • Hyperion, Pegasus, Cartwheel Clusters
    • Tuesday, March 3rd 01:00 – Tuesday, March 3rd 05:00 EST COMPLETE
  • Galaxy01, Thor, Bode, Ursa, Hydra Clusters
    • Wednesday, March 4th 22:00 – Thursday, March 5th 06:00 CST
    • Thursday, March 5th 22:00 – Friday, March 6th 02:00 CST

To find out which cluster your sites are on, please reference our knowledge base article on identifying which cluster your site is on.

We definitely understand these kinds of outages are not ideal but we are hoping this early notice is helpful in the way of being able to notify your users, visitors, and customers.

If you have any questions, please feel free to contact the help desk via your my.pressable.com control panel.

Thank you!