Author Archives: Pressable

Issue With SFTP Connections

Our systems are currently experiencing an issue with SFTP connections. The problem impacts administrators and collaborators’ connections to their site’s files over SFTP. The issue does not affect visitors accessing sites. At this time, all sites should be loading as expected.

We’re actively working to identify the issue, and should have a resolution soon. If you feel that this issue is affecting your site, please contact our help desk (support@pressable.com) and we’ll promptly address your concern.

We will provide additional updates via our Status Blog.

 

UPDATE: This issue has been cleared and SFTP is now working properly across our network.

Reduced Holiday Support Hours

Our help desk will have reduced hours on the following days so that our employees can celebrate the holidays with family and friends. Please note these reduced support hours and consider addressing any potential issues in advance. Thank you, and happy holidays to you and yours!

  • Thursday, November 26: Emergency support only
  • Friday, November 27: Emergency support only
  • Thursday, December 24: Emergency support only
  • Friday, December 25: Emergency support only
  • Thursday, December 31: Emergency support only
  • Friday, January 1: Emergency support only

Our help desk will be working regular hours over the weekend following each holiday break. If you have any questions, please get in touch (support@pressable.com) and we’ll promptly address your concern.

Update to Rackspace Emergency Maintenance for MyPressable Control Panel

This is an update to the emergency maintenance previously scheduled by Rackspace, our backbone provider, for the MyPressable control panel.

Rackspace will reboot a portion of its cloud infrastructure this Tuesday and Wednesday to patch a security issue. Rackspace first attempted to live-migrate the servers which host the MyPressable control panel but those attempts failed.

Maintenance is scheduled to take place from 10:00pm CDT on Tuesday, October 27 to 4:00am CDT on Wednesday, October 28 and again from 10:00pm CDT on Wednesday, October 28 to 2:00am CDT on Thursday, October 29.

During these maintenance windows, we expect a brief interruption of service to the MyPressable control panel. The MyPressable control panel provides access to features/settings for individual sites on your account. If the MyPressable control panel is unavailable, visitors will still be able to access your site(s) and you will still have access to wp-admin.

We will be monitoring the situation closely. Additional updates regarding the emergency maintenance will be shared on our status blog.

We’d like to stress again that we do not expect the emergency maintenance to affect your site(s) but you will receive a separate notice if your site may be impacted by additional emergency maintenance. Thank you for hosting your site with Pressable. If you have any questions or concerns, please contact our help desk (support@pressable.com).

Update to Rackspace Emergency Maintenance: Wednesday, October 28

This is an update to the emergency maintenance previously scheduled for October 19 to October 29. This notice applies only to sites which are hosted on the Ursa galaxy at Rackspace’s ORD data center; not all customers and collaborators will be affected by the update.

Affected site owners and collaborators have been notified via email. Additionally, you have the option to identify the galaxy (cluster) on which your site resides by following the instructions in this KB article. If you have any questions or concerns, please contact our help desk (support@pressable.com).

Rackspace will reboot a portion of its cloud infrastructure this Wednesday to patch a security issue. Rackspace first attempted to live-migrate the servers but those attempts failed.

Maintenance is scheduled to take place from 2:00am to 4:00am CDT on Wednesday, October 28. During this 2-hour window, we expect a brief interruption of service to your site(s).

We will be monitoring the situation closely. Additional updates regarding the emergency maintenance will be shared on our status blog.

Update to Rackspace Emergency Maintenance: Sunday, October 25 to Monday, October 26

This is an update to the emergency maintenance previously scheduled for October 19 to October 29. This notice applies only to sites which are hosted at Rackspace’s IAD data center; not all customers and collaborators will be affected by the update.

Affected site owners and collaborators have been notified via email. If you have any questions or concerns, please contact our help desk (support@pressable.com).

Rackspace will reboot a portion of its cloud infrastructure this Sunday to patch a security issue. Rackspace first attempted to live-migrate the servers but those attempts failed.

Maintenance is scheduled to take place from 9:00pm CST on Sunday, October 25 to 1:00am CSTon Monday, October 26. During this 4-hour window, we expect a brief interruption of service to sites which are hosted at Rackspace’s IAD data center.

We will be monitoring the situation closely. Additional updates regarding the emergency maintenance will be shared on our status blog.

Rackspace Emergency Maintenance: October 19 to October 29

Rackspace, our backbone provider, will be performing critical maintenance for a security issue on its cloud infrastructure beginning today. Rackspace will be patching live systems in an attempt to eliminate downtime. Maintenance is scheduled to continue through October 29.

We anticipate that all customers’ sites will remain fully functional during the maintenance window, however, Rackspace may need to reboot some of our cloud infrastructure if the initial live-patching fails. Should a reboot be required, we will notify customers via email and the status blog.

During the maintenance window we will provide additional updates via our status blog.

If you have any questions or concerns, please email support@pressable.com.

WordPress 4.3 Problem

A new problem has been discovered with WordPress 4.3 which is affecting some of the sites on our systems. The problem arises when cron jobs are periodically scheduled and results in unusually high CPU and database utilization, which can impact the performance and availability of sites. You can read more about the problem at WordPress Core Tracker. All sites are currently up, although a chance of this issue resurfacing at a later date is still present as cron jobs are processed.

We’re keeping an eye on the problem and working to quickly identify a solution. In the meantime, we’ll manually resolve any issues as they arise. If you feel that this issue has affected your site, please feel free to contact our help desk (support@pressable.com). Please stand by for an update once we’ve identified a remediation plan for sites that have been affected.

Provisioning / Control Panel Issues

One of the machines behind our provisioning system has experienced a hardware failure, we are working on replacing the underlying hardware, and expect the machine to be back online in very soon.

In the mean time, functionality like adding, editing, removing a site, or collaborators of a site is not going to work. We apologize for any inconvenience.

Please note: This only affects our control panel, not your actual website.

All Systems Operational

We’re happy to report that all systems are back online and operational. We will be coming forth with a detailed explanation of what happened in the coming days, but this is what we can share so far.

  1. This was a coordinated attack on our systems.
  2. This attack used a modified version of the “Slow-Loris” attack against our platform.
  3. Due to the sophistication of this particular attack, it went undetected by the network security team at our provider Rackspace.  It made it look like our infrastructure was being overloaded, when it was not.
  4. We identified this was an attack at 1:00AM on January 24th 2015, by 5:30AM, we had a solution in place that was blocking the majority of the attacks, this is when some customers on “Bode” started noticing their websites working again.

As of 1:30PM January 24th 2015, we have the majority of the attacks blocked, and have pushed the rules to block these attacks throughout our infrastructure.

We are working as fast we can to answer tickets specific to your site, and will keep you posted.

Currently, our systems are reporting at 100%, any issues you may be experiencing now are not related to this outage, and we encourage you to create a  support ticket so we can help you.

Once again, we’re very sorry for this to have happened, we’re working to find out why we were targeted and by whom, but more importantly, we’re working to ensure we are protected against this in the future.

Credits/Refunds/Recompense?..

We will be reaching out to all of our customers who were affected, sometime next week to make this right.  At this current moment, we have some ideas, but our focus is currently on stability and prevention.

Resolved – Chicago DC – Unscheduled Maintenance October 25th 2014

Due to a miscommunication between us and our service provider Rackspace, the majority of websites hosted at our Chicago datacenter are offline.  We are on the phone with Rackspace working to restore the servers that are needed to bring these websites online as quickly as possible. 

As of Saturday October 25th 8:31AM. All services are running properly.