One of the machines behind our provisioning system has experienced a hardware failure, we are working on replacing the underlying hardware, and expect the machine to be back online in very soon.
In the mean time, functionality like adding, editing, removing a site, or collaborators of a site is not going to work. We apologize for any inconvenience.
Please note: This only affects our control panel, not your actual website.
Howdy,
We just finished dealing with an issue in our Chicago datacenter that was causing several other clusters to experience instability. Our “Ursa” cluster was taking on an extreme amount of traffic that looks to be, in large part, a bot attack.
This happened as a result of the “Ursa” cluster having a set of tools not running appropriately that detects and mitigates issues like this.
We’ve cleared this up and all sites are now back up and running appropriately.
If you have any questions or concerns, please contact our help desk via your https://my.pressable.com control panel.
Thank you!
Due to our upstream provider having connectivity issues, we are currently experiencing downtimes at our Chicago and Virginia Datacenters. We are currently working with our provider to correct this issue. We will continue to update this post as we get more information.
UPDATE 4:41 PM CST: After further investigation, it appears that this outage is only affecting customers in our Chicago Datacenter. We are still trying to gather more information from our provider so we can provide a possible ETA.
UPDATE 5:15 PM CST: We are still working with our provider in order to diagnose this issue. We should have a more detailed update very soon.
UPDATE 6:30 PM CST: We are seeing some clusters begin to online. We are working on rolling out the rest of our clusters to full functionality now. We will update once that is done.
UPDATE 7:47 PM CST: We have restored functionality across our systems. If you are continuing to see issues with your site/s, please submit a ticket via your my.pressable.com control panel.
Earlier today the Wordfence Security team released the details of a WooCommerce SQL Injection Vulnerability. Our systems are already at work patching this popular plugin across sites on our systems. We’ll provide an update when the process has been completed.
UPDATE March 14th, 7:55AM CST: At this time all sites on our systems have been updated to the latest (patched) version of WooCommerce. If you have any questions, please don’t hesitate to reach out.
A new vulnerability has been discovered in the very popular WordPress SEO by Yoast plugin. Our systems have already begun updating this plugin across all impacted sites on our systems, and we expect this process to be completed shortly.
UPDATE 6:15PM CST: At this time all sites on our systems have been updated to the latest (patched) version of WordPress SEO by Yoast.
This is a notice that Rackspace will be performing critical security related updates to many cloud server host machines in order to patch vulnerabilities in Xen Hypervisor.
You can read more about this maintenance here:
https://community.rackspace.com/general/f/53/t/4978
These patches/updates will require host machines to be rebooted, subsequently causing cloud servers hosted on them to require a reboot as well.
As it relates to our customers, here are the maintenance windows that we have been provided with and can expect we will begin seeing server reboots occur based on cluster:
To find out which cluster your sites are on, please reference our knowledge base article on identifying which cluster your site is on.
We definitely understand these kinds of outages are not ideal but we are hoping this early notice is helpful in the way of being able to notify your users, visitors, and customers.
If you have any questions, please feel free to contact the help desk via your my.pressable.com control panel.
Thank you!
We are experiencing an issue with sites in our IAD Datacenter that is causing them to not load appropriately.
We believe this is related to issues that Rackspace is currently having with their cloud block storage services. We are working with them directly and awaiting further details regarding this issue and will update the status blog with more information as it becomes available.
If you have questions or concerns, please create a help desk ticket via your https://my.pressable.com panel or join us in our community lounge at http://chat.pressable.com for updates while we await further details.
UPDATE Feb 27, 2015 @ 5:00 AM Central: We were able to confirm that this is an issue occurring at Rackspace with their Cloud Block Storage service. You can find more details and information on their status page: https://status.rackspace.com/index/viewincidents?start=1425013200
We will update again as more information is available.
UPDATE Feb 27, 2015 @ 5:55 AM Central: Rackspace has resolved the issue on their end and we are now working on re-establishing stability on our end. We will update again as soon as this is taken care of.
UPDATE Feb 27, 2015 @ 6:10 AM Central: We have now restored functionality across our IAD datacenter and all sites are now functioning normally. If you continue to experience problems, please submit a help desk ticket via your https://my.pressable.com panel.
This is a notice that Rackspace will be performing software upgrades on network switches in our Chicago datacenter. These upgrades will require a reboot of these network switches, which will result in approximately five minutes of downtime for customers in this datacenter.
The maintenance is scheduled for Wednesday, February 18th during the window of 12:00am-6:00am CST.
Please note that this window can not be moved to another time due to the nature of our Provider. We apologize in advance for any inconvenience this may cause.
If you have any questions, please feel free to contact the helpdesk, via your my.pressable.com control panel.
We are currently experiencing an attack similar to the attack we had 2 weeks ago, we have isolated the target which is towards the cluster “galaxy01”.
We are working with our provider now to resolve this as quickly as we can.
UPDATE: 4:35 P.M.- We’re working on identifying which specific set of servers are being attacked and will have a fix in place as soon as we can identify this.
UPDATE: 5:07 P.M.- We have identified the IP that was causing this attack. We have banned that IP which has resulted in connections returning to normal, which in turn means that sites should be loading properly now. If you are not seeing your site/sites coming online, please submit a ticket via your my.pressable.com control panel.
If you have any other questions, please feel free to submit a support ticket via your my.pressable.com control panel.
We’re happy to report that all systems are back online and operational. We will be coming forth with a detailed explanation of what happened in the coming days, but this is what we can share so far.
As of 1:30PM January 24th 2015, we have the majority of the attacks blocked, and have pushed the rules to block these attacks throughout our infrastructure.
We are working as fast we can to answer tickets specific to your site, and will keep you posted.
Currently, our systems are reporting at 100%, any issues you may be experiencing now are not related to this outage, and we encourage you to create a support ticket so we can help you.
Once again, we’re very sorry for this to have happened, we’re working to find out why we were targeted and by whom, but more importantly, we’re working to ensure we are protected against this in the future.
Credits/Refunds/Recompense?..
We will be reaching out to all of our customers who were affected, sometime next week to make this right. At this current moment, we have some ideas, but our focus is currently on stability and prevention.
