Botnet Attack Causing Issues Across Network

Howdy,

Our servers are currently under attack and this has caused some of our databases to run behind and result in sites not displaying, updating, or functioning properly.

We have already begun banning IPs before they hit out load balancers and are working toward reducing the impact and footprint of this.

We will update the status blog again once this attack has subsided.

If you have questions or concerns, please contact our helpdesk via your panel at  https://my.pressable.com.

Thank you!

UPDATE – 01/05/2015 @ 11:56AM Central: 

We are seeing the slave database servers as caught up and sites should be functioning properly now. If you are still experiencing issues, please contact us via https://my.pressable.com

UPDATE – 01/05/2015 @ 12:25AM Central: 

The slave databases have fallen behind again and we are attempting to get them caught back up. Although they had originally caught back up, they are now behind once more and you may be experiencing some issues (logging in to your site, content updates not display, etc.). We will update once we have seen sustained stability and synchronization on these database servers.

Please let me know if you have any questions, thanks!

Emergency Maintenance: December 24, 2014 at 3:00AM CST

On December 23, 2014 an issue was discovered with a database cluster serving sites using the “cartwheel” hostname. The issue discovered has been impacting performance of these sites as well as causing intermittent downtime issues. In order to correct the issue, we need to perform an emergency maintenance on these servers. This maintenance will require a period of downtime while we work to repair the servers and restore service levels back to normal levels.

MAINTENANCE INFORMATION: The maintenance window is expected to begin at 3:00AM CST on December 24, 2014 and expected to last approximately 2 hours. We DO NOT expect sites to be down during this entire period, but there may be periods of connectivity loss.

We will provide updates to the status of this maintenance in this post.

UPDATE 12/24/2014 @ 6:50AM CST: At this time we have completed the maintenance and are starting to see traffic return to sites. Our apologies about the short notice and extended periods of downtime. If you have any questions, please don’t hesitate to reach out.

Network Degredation Impacting Site Availability

We’re currently experiencing an issue with our network causing a complete degradation of services and loss of traffic. We’re working with our provider to determine the cause and source of the issues, but early signs point to a targeted attack against our systems. We’ll provide more details as they become available.

UPDATE 3:25PM CST: We’re still working with our provider to determine the source of increased traffic and to correct any issues.

UPDATE 3:55PM CST: At this time things appear to have stabilizied. However, we’re still working with our provider to determine the root cause of the issues and put any nesseceary measures in place to prevent any similar issues.

Site Availability issue on Galaxy01

We’re currently experiencing an issue with our Galaxy01 cluster of servers. This issue is causing intermittent 500/502 errors while processes fail to load. We’re currently evaluating the system and working to restore services to normal operational levels as quickly as possible. We’ll update this post with more information as we have it.

UPDATE 10:00AM CST: The team is still working to restore this cluster of servers to 100%. We’re getting closer as some new firewall rules come online and additional capacity. We’ll provide another update shortly.

UPDATE 10:45AM CST: At this time services are beginning to return to normal across the affected servers. We’re still waiting on some new rules to finish processing, but things are trending in a positive direction.

Server Outages

Howdy,

We are currently experiencing an issue with sites being down on a small portion of our network. This issue is a result of some of our servers going offline unexpectedly. We are working with Rackspace on getting these machines back up right now and things will resume working normally shortly.

If you have any questions or concerns, please contact our help desk via the https://my.pressable.com control panel.

UPDATE December 4th, 2014 @ 8:40AM Central: A large majority of servers are back online and functioning properly. We have a couple more servers waiting to come back online but most sites are back up and functioning properly again. We will update once more as soon as all servers are back online and functional.

UPDATE December 4th, 2014 @ 9:00 AM Central: All servers and sites are now back online and functional. If for some reason you are still seeing a site offline, please let us know and we can figure out what is causing it.

Thanksgiving Holiday Support Hours

Howdy,

2014 has nearly run its course and the holiday season is upon us. The Pressable staff wants to thank everyone for supporting us through this year and helping us find new ways to improve our services.

In observance of the holiday on November 27th, 2014, we will have a reduced support staff on hand and will be operating with minimal support response on both November 27th and 28th. We will resume operating under normal support hours on November 29th, 2014.

Minimal Support Response means that we won’t be available to respond to non-emergency requests. There will be support staff standing by, but we use our best judgement to determine emergency issues that need our attention immediately.

We hope everyone enjoys their holiday and look forward to serving you through the rest of the year.

Thank you!

Sporadic Functionality of Deploys and Clones

We are currently experiencing an issue that is causing clones and new site deploys to function sporadically.

Some customers are experiencing issues cloning sites and reporting unfinished clones. New site deploys are functioning for a large majority of our customers but there are still cases where new site deploys are not working.

We are attempting to resolve these issues as quickly as possible and will update this status blog as more information comes available.

If you have questions or concerns, please contact us by submitting a help desk request via your https://my.pressable.com control panel.

UPDATE Nov. 21, 2014 @ 10:41AM CST: Deploys across our systems are now working properly effective immediately. We are still working on fixing cloning across the board. We discovered that clones broke partially due to a fix we put in for POODLE recently. We will update again once this issue is resolved.

UPDATE Nov. 25, 2014 @ 9:20AM CST: Deploy functionality continues to work properly across our systems. The team is still working to resolve the incompatibilities with our cloning functionality and the increased security put in place post-POODLE.

UPDATE Dec. 1, 2014 @ 10:38AM CST: We are continuing to work on fixing the cloning issues from this past week. We have removed all clones that failed while this has been happening. We are asking for customers to refrain from attempting to clone until this issue has been patched fully. We will update again as soon as the cloning issue has been resolved.

Resolved – Chicago DC – Unscheduled Maintenance October 25th 2014

Due to a miscommunication between us and our service provider Rackspace, the majority of websites hosted at our Chicago datacenter are offline.  We are on the phone with Rackspace working to restore the servers that are needed to bring these websites online as quickly as possible. 

As of Saturday October 25th 8:31AM. All services are running properly.

POODLE SSL Vulnerability

We are aware of the recently announced vulnerability in SSL called POODLE. This vulnerability affects a very small percentage of traffic on our systems.

You can read more about this vulnerability at the following links:

https://isc.sans.edu/diary/OpenSSL%3A+SSLv3+POODLE+Vulnerability+Official+Release/18827

Click to access ssl-poodle.pdf

The “fix” for this issue is to disable SSLv3 on servers and/or clients that utilize it. Fortunately, it is in use in a very small percentage of traffic and is not likely to hinder traffic to sites we host in any significant capacity.

The recommended solution to this problem will make clients using Internet Explorer 6 running on Windows XP or older recieve an SSL error.

Currently, this fix will also break the Pingdom monitoring service SSL check. Pingdom is aware of this issue and is making the appropriate changes to their systems.

We are aware of the inconvenience this may cause, but we believe that this is the right thing to do.

You can see larger providers implement similar policies here.

https://chargify.com/blog/dropping-sslv3/
https://blog.cloudflare.com/sslv3-support-disabled-by-default-due-to-vulnerability/

If you have any questions or concerns, please contact our help desk by submitting a ticket from https://my.pressable.com.

RESOLVED: Intermittent my.pressable.com Control Panel Downtime

Howdy,

We created a previous post about our email provider outage potentially causing issues with email sending from our servers but wanted to specifically point out that this also affects the https://my.pressable.com control panel.

Users are seeing the control panel time out or be completely unresponsive on an intermittent basis.

These issues are also related to the same issues that are caupring problems with email in general because of the ties we in place for email on my.pressable.com.

As soon as Mailgun has these issues resolved, the control panel will function properly once more.

You can follow the status of the Mailgun issues here:

http://status.mailgun.com/incidents/9s93dc83gtlw

If you have quesitons or concerns, please send an email to help@pressable.com.

Thank you!

UPDATE 09/30/2014 @ 12:40 PM Central: Email issues with all sites have been resolved. http://status.pressable.com/2014/09/29/email-provider-outage-causing-issues/