Author Archives: Jeff

Rackspace Scheduled Maintenance: February 16-17

UPDATE: The scheduled maintenance has been completed. At this time, all sites are up. If you have any lingering issues, please contact our help desk (support@pressable.com).

Rackspace, our backbone provider, will be performing critical maintenance on the network that connects its cloud infrastructure from 9:00pm CT on Tuesday, February 16th to 5:00am CT on Wednesday, February 17. During this 8-hour window, we expect a brief interruption of service to sites which are hosted at Rackspace’s IAD data center.

We will update our status blog once the maintenance has been completed. If you have any questions or concerns, please contact our help desk (support@pressable.com).

Database Issue on the Hyperion Galaxy

UPDATE (4:55pm CDT): The database issue has been resolved. At this time, all sites are working as expected. If you have additional concerns about the issue, please contact our help desk (support@pressable.com) and we’ll promptly address your concern.

 

Our systems are currently experiencing an issue with database utilization which is affecting site on the Hyperion galaxy. Visitors are presently unable to access sites which are hosted on the Hyperion galaxy.

We’re actively working to resolve the issue, and should have things back to normal soon. If you need additional information about this issue, please contact our help desk (support@pressable.com) and we’ll promptly address your concern.

We will provide additional updates via our Status Blog.

Concluded: EMERGENCY NETWORK MAINTENANCE: Saturday, April 11, 2015 – 11:30pm to 12:30am CST (ORD)

UPDATE 1:00 AM CST: AT THIS TIME, WE HAVE CONCLUDED THE EMERGENCY MAINTENANCE, SITES SHOULD BE FUNCTIONAL GOING FORWARD, AS WELL AS DEPLOYMENTS OF NEW SITES. CLONING IS STILL NON-FUNCTIONAL AT THIS POINT. IF YOU NEED A SITE CLONED, PLEASE SUBMIT A TICKET VIA YOUR MY.PRESSABLE.COM CONTROL PANEL, AND WE CAN CLONE IT MANUALLY FOR YOU. 

UPDATE 12:30 am CST: We are continuing to work with our partner to complete this emergency maintenance. While we believe that the portion of this that would cause sites to not function is over, it will not be out of the ordinary to see sites go down in the next 30-45 minutes for a short period of time. We will continue to deliver updates here as we look to finish the maintenance. 

Over the past week we’ve been working with our partners to address an issue preventing new infrastructure from coming online inside our Chicago (ORD) Network. While this issue has not been impactful to customer facing sites, it has created issues while we work to expand capacity and grow our systems. Unfortunately, we’re unable to move forward with correcting these issues without temporarily suspending network services.

MAINTENANCE TIMELINE

At approximately 11:30pm CST on April 11, 2015 we’ll begin working with our partners to bring services offline. This involves a reorder and reboot of our network edge devices to correct the issues identified. By 12:30am CST on April 12, 2015 maintenance on our network edge devices will be completed and services will be restored.

WHAT DOES THIS MEAN?

During the maintenance all sites located in our Chicago datacenter will be offline. While we only expect the downtime period to be 15 minutes, the maintenance is scheduled for 1 hour.

Sites will return several different types of errors ranging from 504 timeouts and 503 service unavailable messages. Once network maintenance is completed there may also be a period of 502 errors while we bring our infrastructure back online.

HOW CAN I TELL IF I’LL BE IMPACTED?

If your site is located in our Chicago datacenter you’ll be impacted by this maintenance. There are two easy ways to tell if your site is located here. First you can check if your site is in one of the following clusters:

  • Galaxy01
  • Thor
  • Bode
  • Hydra
  • Ursa

You can also very simply tell the datacenter by checking your SFTP hostname. If your SFTP hostname is:

  • sftp.rax.ord.openhostingservice.com

I HAVE MORE QUESTIONS!

If you have more questions, or are concerned about this maintenance, please do not hesitate to contact our helpdesk. You can do this by submitting a ticket from the https://my.pressable.com control panel, or emailing help@pressable.com.

Our sincerest apologies for the short notice on this maintenance. Our team and partners worked tirelessly to prevent a downtime like this, but ultimately it wasn’t possible.

RESOLVED: WooCommerce SQL Injection Vulnerability

Earlier today the Wordfence Security team released the details of a WooCommerce SQL Injection Vulnerability. Our systems are already at work patching this popular plugin across sites on our systems. We’ll provide an update when the process has been completed.

UPDATE March 14th, 7:55AM CST: At this time all sites on our systems have been updated to the latest (patched) version of WooCommerce. If you have any questions, please don’t hesitate to reach out.

Connectivity Issues inside of Chicago DC

We’re currently investigating new connectivity issues inside of our Chicago Datacenter. These currently appear to be unrelated to issues that were present for the past 48 hours. We’ll provide an update as we have more information about the current problems.

UPDATE 1:30PM CST: Everything should be operating normally at this time. We’re still evaluating systems and the stability of things. If you’re still experiencing issues, please submit a ticket so our team can take a look for you.

RESOLVED: Chicago Data Center Outage

We wanted to provide this post as a notification that issues related to our Chicago Data Center Outage have been resolved. Our team will be continuing to work through the emails and tickets related to this issue and we’ll be providing a full postmortem tomorrow.

We sincerely appreciate your patience and understanding during this truly trying experience. The kind emails, tweets and messages we’ve received have been a true blessing.

RESOLVED: Network Connectivity Issues Impacting Availability

We’re currently experiencing an issue impacting our edge devices that’s causing sites and internal systems to be inaccessible. We’re investigating with our provider and will provide an update when we have more information.

UPDATE 10:20PM CST: Our provider is investigating the issue and believes it may be related to a DDOS attack against another customer of theirs which is saturating the internal networks at the datacenter. We’re continuing to work with our partners to restore services ASAP.

UPDATE 10:30PM CST: Our provider is informing us that the attack appears to have subsided for the time being. At this time our systems are beginning to return to normal and we’re continuing to monitor for any lingering issues.

UPDATE 10:50PM CST: At this time we’re seeing our systems operating at normal levels. We’ll continue to monitor for issues, but do not expect there to be any problems related to this incident.

Service Availability Issue on Galaxy01

We’re currently experiencing an issue with a cluster named “Galaxy01” that’s causing high load across the system which results in sites being down or displaying 502 error messages. We’re investigating the root cause of this issue and will provide an updates as we have them.

UPDATE 11:15AM CST: We’re continuing to investigate issues causing higher than normal load on the systems. We’re also working to bring new hardware online which we believe may help to alleviate some issues.

UPDATE 12:25PM CST: We’re currently in the process of bringing the new hardware online which is causing some downtime while it starts taking traffic. We apologize for the issue and expect the new hardware to be 100% functional shortly.

UPDATE 1:00PM CST: The new hardware is online and we’re starting to see usage return to normal. We’ll provide an update when things are more stable on the systems.

UPDATE 3:25PM CST: We’ve seen stability in the systems for the past hour and are continuing to monitor systems. If you notice any issues, please let us know.

Emergency Maintenance: December 24, 2014 at 3:00AM CST

On December 23, 2014 an issue was discovered with a database cluster serving sites using the “cartwheel” hostname. The issue discovered has been impacting performance of these sites as well as causing intermittent downtime issues. In order to correct the issue, we need to perform an emergency maintenance on these servers. This maintenance will require a period of downtime while we work to repair the servers and restore service levels back to normal levels.

MAINTENANCE INFORMATION: The maintenance window is expected to begin at 3:00AM CST on December 24, 2014 and expected to last approximately 2 hours. We DO NOT expect sites to be down during this entire period, but there may be periods of connectivity loss.

We will provide updates to the status of this maintenance in this post.

UPDATE 12/24/2014 @ 6:50AM CST: At this time we have completed the maintenance and are starting to see traffic return to sites. Our apologies about the short notice and extended periods of downtime. If you have any questions, please don’t hesitate to reach out.