Category Archives: maintenance

Scheduled Upstream Maintenance May 9th

From 5am to 11am UTC ( 12:00am to 6:00am CST ) Wednesday, May 9th non-service impacting network maintenance will occur by our upstream provider. This maintenance provides Pressable with a fully redundant network. And completes the maintenance our upstream vendor will be performing as result of our recent connection failure.

We do not forecast our vendor to require the entire window of time, and we do not anticipate any service interruption or impact to your sites. We will update this post witch additional details once the maintenance is complete.

5/10 5:23 UTC: Pressable has received word from the upstream provider the maintenance was complete at Wednesday, May 9th at 10:18 UTC without any service interruption. Pressable’s network now has fully redundant network connectivity.

Rackspace Scheduled Maintenance (UPDATED)

Update (3/23/16): Rackspace has scheduled additional maintenance for Thursday, March 23. Please check the maintenance schedule below to see if your site(s) will be impacted.

Rackspace will be performing scheduled maintenance on several servers over the coming week beginning on Monday, March 21. The maintenance schedule is as follows:

Note: You can identify which galaxy your site(s) are on by referencing the following knowledge base article:

How do I Identify what cluster my site is on?

Monday, March 21 at 9:00pm to Tuesday, March 22 at 3:00am Central

There may be brief disruption of service to customers on the following galaxies in our IAD datacenter:

  • Cartwheel
  • Hyperion
  • Pegasus
  • Shadowfax

Wednesday, March 23 from 12:00am to 5:00am Central

A small number of customers on the following galaxies in our IAD datacenter may experience a brief interruption in their SSH connectivity:

  • Hyperion
  • Pegasus
  • Shadowfax

Wednesday, March 23 at 10:00pm to Thursday, March 24 at 4:30am Central

There may be brief disruption of service and intermittent slowness to customers on the following galaxies in our ORD datacenter:

  • Galaxy01
  • Thor
  • Bode
  • Hydra
  • Ursa

Thursday, March 23 from 9:00pm to 11:00pm Central

There may be brief disruption of service to customers on the following galaxies in our IAD datacenter:

  • Cartwheel
  • Hyperion
  • Pegasus
  • Shadowfax

Friday, March 25 from 3:00am to 4:00am Central

There may be brief disruption of service to customers on the following galaxies in our IAD datacenter:

  • Hyperion
  • Pegasus
  • Shadowfax
  • Cartwheel

If you have any questions or concerns about this maintenance, please reach out to us by email (help@pressable.com) or submit a ticket for our Help Desk through the MyPressable Control Panel (https://my.pressable.com).

Thank you!

Emergency Security Maintenance

Howdy,

We received notice this afternoon that our provider, Rackspace, has identified a vulnerability in Xen Hypervisor. This vulnerability has been patched by Rackspace but it requires a reboot of cloud servers in order to for it to take.

You can read more about the vulnerability and requirements here:

https://community.rackspace.com/general/f/53/t/5187
http://venom.crowdstrike.com

We have elected to reboot cloud servers in our network at our discretion rather than allowing Rackspace to reboot using a maintenance window approach.

We will begin the process of rebooting cloud servers tonight, May 13th, at 10:00 PM Central time. We expect this process to take several hours and customers will see intermittent outages of varying length.

We will update this page when the maintenance is completed.

If you have any questions or concerns, please contact our help desk.

We apologize in advance for any inconvenience this may cause and are working toward addressing this as smoothly as possible.

Thank you!

UPDATE – May 14th, 2015 @ 2:57 AM Central: We have completed the reboots that are necessary as a part of this vulnerability patch. All systems are back up, running, and stable. Thank you for your patience and please let us know if you have any questions. Thanks!

Concluded: EMERGENCY NETWORK MAINTENANCE: Saturday, April 11, 2015 – 11:30pm to 12:30am CST (ORD)

UPDATE 1:00 AM CST: AT THIS TIME, WE HAVE CONCLUDED THE EMERGENCY MAINTENANCE, SITES SHOULD BE FUNCTIONAL GOING FORWARD, AS WELL AS DEPLOYMENTS OF NEW SITES. CLONING IS STILL NON-FUNCTIONAL AT THIS POINT. IF YOU NEED A SITE CLONED, PLEASE SUBMIT A TICKET VIA YOUR MY.PRESSABLE.COM CONTROL PANEL, AND WE CAN CLONE IT MANUALLY FOR YOU. 

UPDATE 12:30 am CST: We are continuing to work with our partner to complete this emergency maintenance. While we believe that the portion of this that would cause sites to not function is over, it will not be out of the ordinary to see sites go down in the next 30-45 minutes for a short period of time. We will continue to deliver updates here as we look to finish the maintenance. 

Over the past week we’ve been working with our partners to address an issue preventing new infrastructure from coming online inside our Chicago (ORD) Network. While this issue has not been impactful to customer facing sites, it has created issues while we work to expand capacity and grow our systems. Unfortunately, we’re unable to move forward with correcting these issues without temporarily suspending network services.

MAINTENANCE TIMELINE

At approximately 11:30pm CST on April 11, 2015 we’ll begin working with our partners to bring services offline. This involves a reorder and reboot of our network edge devices to correct the issues identified. By 12:30am CST on April 12, 2015 maintenance on our network edge devices will be completed and services will be restored.

WHAT DOES THIS MEAN?

During the maintenance all sites located in our Chicago datacenter will be offline. While we only expect the downtime period to be 15 minutes, the maintenance is scheduled for 1 hour.

Sites will return several different types of errors ranging from 504 timeouts and 503 service unavailable messages. Once network maintenance is completed there may also be a period of 502 errors while we bring our infrastructure back online.

HOW CAN I TELL IF I’LL BE IMPACTED?

If your site is located in our Chicago datacenter you’ll be impacted by this maintenance. There are two easy ways to tell if your site is located here. First you can check if your site is in one of the following clusters:

  • Galaxy01
  • Thor
  • Bode
  • Hydra
  • Ursa

You can also very simply tell the datacenter by checking your SFTP hostname. If your SFTP hostname is:

  • sftp.rax.ord.openhostingservice.com

I HAVE MORE QUESTIONS!

If you have more questions, or are concerned about this maintenance, please do not hesitate to contact our helpdesk. You can do this by submitting a ticket from the https://my.pressable.com control panel, or emailing help@pressable.com.

Our sincerest apologies for the short notice on this maintenance. Our team and partners worked tirelessly to prevent a downtime like this, but ultimately it wasn’t possible.

Rackspace Scheduled Critical Maintenance

This is a notice that Rackspace will be performing critical security related updates to many cloud server host machines in order to patch vulnerabilities in Xen Hypervisor.

You can read more about this maintenance here:

https://community.rackspace.com/general/f/53/t/4978

These patches/updates will require host machines to be rebooted, subsequently causing cloud servers hosted on them to require a reboot as well.

As it relates to our customers, here are the maintenance windows that we have been provided with and can expect we will begin seeing server reboots occur based on cluster:

  • Hyperion, Pegasus, Cartwheel Clusters
    • Tuesday, March 3rd 01:00 – Tuesday, March 3rd 05:00 EST COMPLETE
  • Galaxy01, Thor, Bode, Ursa, Hydra Clusters
    • Wednesday, March 4th 22:00 – Thursday, March 5th 06:00 CST
    • Thursday, March 5th 22:00 – Friday, March 6th 02:00 CST

To find out which cluster your sites are on, please reference our knowledge base article on identifying which cluster your site is on.

We definitely understand these kinds of outages are not ideal but we are hoping this early notice is helpful in the way of being able to notify your users, visitors, and customers.

If you have any questions, please feel free to contact the help desk via your my.pressable.com control panel.

Thank you!

Rackspace Scheduled Maintenance, February 18th 12:00am-6:00am CST

This is a notice that Rackspace will be performing software upgrades on network switches in our Chicago datacenter. These upgrades will require a reboot of these network switches, which will result in approximately five minutes of downtime for customers in this datacenter.

The maintenance is scheduled for Wednesday, February 18th during the window of 12:00am-6:00am CST.

Please note that this window can not be moved to another time due to the nature of our Provider. We apologize in advance for any inconvenience this may cause.

If you have any questions, please feel free to contact the helpdesk, via your my.pressable.com control panel.

Emergency Maintenance: December 24, 2014 at 3:00AM CST

On December 23, 2014 an issue was discovered with a database cluster serving sites using the “cartwheel” hostname. The issue discovered has been impacting performance of these sites as well as causing intermittent downtime issues. In order to correct the issue, we need to perform an emergency maintenance on these servers. This maintenance will require a period of downtime while we work to repair the servers and restore service levels back to normal levels.

MAINTENANCE INFORMATION: The maintenance window is expected to begin at 3:00AM CST on December 24, 2014 and expected to last approximately 2 hours. We DO NOT expect sites to be down during this entire period, but there may be periods of connectivity loss.

We will provide updates to the status of this maintenance in this post.

UPDATE 12/24/2014 @ 6:50AM CST: At this time we have completed the maintenance and are starting to see traffic return to sites. Our apologies about the short notice and extended periods of downtime. If you have any questions, please don’t hesitate to reach out.

Limited functionality with database. – Emergency Maintenance

Symptoms

Some customers are experiencing limited functionality with the database systems, the symptoms appear to be that their posts and updates are not saving. Customers may also be experiencing issues approving comments or changing site/plugin settings. New site deployments and clones may also not be working.

What caused this?

This is due to a problem in our database cluster, currently when you save a new post, or update an existing one, we write to one server, and then the changes are propagated to 4 other servers. The link between the server you write to, and the servers that replicate the changes has been severed. This gives the illusion of things not being saved.

Time to resolve this issue

We are working to resolve this issue as soon as we can, in order to do so, we will need to make a change to the system where you will indeed not be able to make any changes for about 10 minutes.

We will provide an update to this post when the maintenance has begun. We do not expect any downtime, just the inability to save a post or new posts. Your website will continue to be served to your visitors without any issues.

UPDATE 10:10AM CDT: We’ve begun the maintenance on the database server and expect things to go smoothly. We’ll provide an update when the maintenance is completed.

UPDATE 10:35AM CDT: We’re currently experiencing an issue with the database maintenance that’s causing an interruption to service. We’re working to move traffic to one of the secondary servers and bring things back online.

UPDATE 10:55AM CDT: We’ve brought secondary servers online to help bring things back online. We’re still working through remaining issues with the emergency maintenance.

UPDATE 11:40AM CDT: Our team is still working on the ongoing database issue.

UPDATE 12:30PM CDT: The team is still hard at work resolving database issues.

UPDATE 1:30PM CDT: We’re expecting things to be more stable at this time and most sites to have returned to normal. If you’re still experiencing issues please let our support team know so we can investigate for you.

Database Maintenance: Friday, May 30, 1:00am CDT

On Friday, May 30th, 2014 at 1:00am we’ll be performing maintenance to our database servers. We do not expect this maintenance to be disruptive, however, some users may experience brief interruptions in service.

This maintenance is part of our ongoing efforts to continue improving the reliability and stability of our platform.

We’ll provide an update when the maintenance has been completed.

If you have any questions, please contact the support team through the https://my.pressable.com control panel.

Update May 30th, 0150

The maintenance was successful and is now complete.