Tag Archives: maintenance

Emergency Security Maintenance

Howdy,

We received notice this afternoon that our provider, Rackspace, has identified a vulnerability in Xen Hypervisor. This vulnerability has been patched by Rackspace but it requires a reboot of cloud servers in order to for it to take.

You can read more about the vulnerability and requirements here:

https://community.rackspace.com/general/f/53/t/5187
http://venom.crowdstrike.com

We have elected to reboot cloud servers in our network at our discretion rather than allowing Rackspace to reboot using a maintenance window approach.

We will begin the process of rebooting cloud servers tonight, May 13th, at 10:00 PM Central time. We expect this process to take several hours and customers will see intermittent outages of varying length.

We will update this page when the maintenance is completed.

If you have any questions or concerns, please contact our help desk.

We apologize in advance for any inconvenience this may cause and are working toward addressing this as smoothly as possible.

Thank you!

UPDATE – May 14th, 2015 @ 2:57 AM Central: We have completed the reboots that are necessary as a part of this vulnerability patch. All systems are back up, running, and stable. Thank you for your patience and please let us know if you have any questions. Thanks!

All Systems Operational

We’re happy to report that all systems are back online and operational. We will be coming forth with a detailed explanation of what happened in the coming days, but this is what we can share so far.

  1. This was a coordinated attack on our systems.
  2. This attack used a modified version of the “Slow-Loris” attack against our platform.
  3. Due to the sophistication of this particular attack, it went undetected by the network security team at our provider Rackspace.  It made it look like our infrastructure was being overloaded, when it was not.
  4. We identified this was an attack at 1:00AM on January 24th 2015, by 5:30AM, we had a solution in place that was blocking the majority of the attacks, this is when some customers on “Bode” started noticing their websites working again.

As of 1:30PM January 24th 2015, we have the majority of the attacks blocked, and have pushed the rules to block these attacks throughout our infrastructure.

We are working as fast we can to answer tickets specific to your site, and will keep you posted.

Currently, our systems are reporting at 100%, any issues you may be experiencing now are not related to this outage, and we encourage you to create a  support ticket so we can help you.

Once again, we’re very sorry for this to have happened, we’re working to find out why we were targeted and by whom, but more importantly, we’re working to ensure we are protected against this in the future.

Credits/Refunds/Recompense?..

We will be reaching out to all of our customers who were affected, sometime next week to make this right.  At this current moment, we have some ideas, but our focus is currently on stability and prevention.