Scheduled Maintenance: July 17-18

Peer 1 will be performing maintenance on Pressable’s backbone infrastructure on Sunday, July 17th from 12:00am to 6:00am CT and again on Monday, July 18th from 12:00am to 6:00am CT. During each 6-hour maintenance window, we expect a brief interruption of service to our customers’ sites.

We will update our status blog once the maintenance has been completed. If you have any questions or concerns, please contact our help desk.

Service Interruption

 

RESOLVED @ 3:54 AM Central: Our hosting provider isolated the cause of the problems and made changes that should prevent further service disruptions.  We are not anticipating another loss of connectivity and will continue to monitor the situation.

UPDATE @ 3:41 AM Central:  The network connectivity issues returned momentarily, but things have recovered. The root cause appears to be some network instability between Dallas and San Antonio, Texas.  Will provide more details once they are available.

UPDATE @ 3:34 AM Central:  The network issues have been resolved and everything is online at this time.  We continue to investigate the root cause with our provider.

3:14AM Central: There is currently a problem with our hosting provider which is affecting all Pressable-hosted websites. We are actively troubleshooting the issue and will update this post with more details as they are available.

Database Character Set Issues

On Friday, June 10th, at 17:37 UTC (12:37PM Central) we accidentally deployed a platform-wide change that caused the DB_CHARSET value to be set incorrectly on most Pressable-hosted websites. DB_CHARSET is very important, as it tells the web server which character set to use when talking to the database server. It’s important that the values on both the web and database servers match and our change caused a mismatch.

We identified the problem and rolled back our changes at 19:52 UTC (2:52PM Central), but during the 2 hours and 15 minutes the problem existed, if you edited any content on your site, there is a chance it’s now corrupt. This corruption can appear in a few different ways, but one obvious way is that quotes might now appear as “gibberish”. Unfortunately, the problem is not limited to pages and posts, but can also affect widgets, settings, etc.

One of the benefits of our new 2.0 platform is a high level of automation and consistency across severs and websites. This should result in a more consistent, predictable, and improved user experience. Unfortunately, in this case, our newfound automation caused a major problem and for that we apologize. We are adding additional tests, checks, and monitoring to ensure a similar problem doesn’t happen in the future.

If your site has been behaving strangely since Friday, please get in touch and we’ll help you restore things to normal.

Rackspace Scheduled Maintenance for ORD Data Center: June 2

Rackspace will be performing maintenance on the network that connects the cloud infrastructure at its ORD data center on Thursday, June 2nd, 2016 from 2:00am to 3:00am CDT. During this 1-hour window, we expect a brief interruption of service to sites which are hosted at Rackspace’s ORD data center.

We will update our status blog once the maintenance has been completed. If you have any questions or concerns, please contact our help desk.

Emergency Maintenance: May 10th, 2016

We are about to begin an emergency maintenance period that will affect a large group of customers.

During this time, we expect affected sites to continue to stay up and available, but customers will be unable to upload or modify files.

This maintenance will begin immediately and we expect it to last around 30 minutes.

If you have questions or concerns, please submit a ticket to our help desk via your https://my.pressable.com control panel.

UPDATE 3:00PM Central: The maintenance period is lasting longer than initially anticipated. Affected sites remain in a “read only” state where files are not able to be uploaded or modified. We will update again as soon as more information is available.

UPDATE 3:30PM Central: Maintenance has now been completed and read/write functionality is available again to all sites. Please contact us if you have any questions or concerns and thank you for your patience during this maintenance.

SFTP Unavailability

UPDATE (4:36PM CDT): We’ve corrected this issue, and SFTP access is fully functional across all sites once again. If you are still having trouble, please submit a ticket to our helpdesk in http://my.pressable.com/

We are currently experiencing an issue that is causing SFTP to be unavailable for many customers. We are working diligently toward getting this resolved as soon as possible and will provide updates as they become available.

If you have questions or concerns, please contact us via your https://my.pressable.com control panel, or email us at help@pressable.com

Emergency Maintenance: IAD Datacenter

Update (1:26PM CDT) All sites on IAD are up, and the SFTP/Upload issues are now resolved. Please contact the helpdesk if you have any outstanding issues, questions, or concerns.

We are going to be immediately performing an emergency maintenance for clusters in our IAD datacenter.

This goal of the maintenance is to resolve issues related to customers not being able to upload to their sites via SFTP or the WordPress dashboard. These issues are occurring as a side effect last night’s scheduled Cloud Block Storage maintenance from Rackspace.

This maintenance will cause sites in IAD to be unavailable while we resolve these issues.

If you have questions or concerns, please contact our helpdesk via your https://my.pressable.com panel and we can address them accordingly.

Rackspace Scheduled Maintenance (UPDATED)

Update (3/23/16): Rackspace has scheduled additional maintenance for Thursday, March 23. Please check the maintenance schedule below to see if your site(s) will be impacted.

Rackspace will be performing scheduled maintenance on several servers over the coming week beginning on Monday, March 21. The maintenance schedule is as follows:

Note: You can identify which galaxy your site(s) are on by referencing the following knowledge base article:

http://kb.pressable.com/pressable-tutorials/identify-cluster-site/

Monday, March 21 at 9:00pm to Tuesday, March 22 at 3:00am Central

There may be brief disruption of service to customers on the following galaxies in our IAD datacenter:

  • Cartwheel
  • Hyperion
  • Pegasus
  • Shadowfax

Wednesday, March 23 from 12:00am to 5:00am Central

A small number of customers on the following galaxies in our IAD datacenter may experience a brief interruption in their SSH connectivity:

  • Hyperion
  • Pegasus
  • Shadowfax

Wednesday, March 23 at 10:00pm to Thursday, March 24 at 4:30am Central

There may be brief disruption of service and intermittent slowness to customers on the following galaxies in our ORD datacenter:

  • Galaxy01
  • Thor
  • Bode
  • Hydra
  • Ursa

Thursday, March 23 from 9:00pm to 11:00pm Central

There may be brief disruption of service to customers on the following galaxies in our IAD datacenter:

  • Cartwheel
  • Hyperion
  • Pegasus
  • Shadowfax

Friday, March 25 from 3:00am to 4:00am Central

There may be brief disruption of service to customers on the following galaxies in our IAD datacenter:

  • Hyperion
  • Pegasus
  • Shadowfax
  • Cartwheel

If you have any questions or concerns about this maintenance, please reach out to us by email (help@pressable.com) or submit a ticket for our Help Desk through the MyPressable Control Panel (https://my.pressable.com).

Thank you!

Issues with Sites on the Bode Galaxy

UPDATE (6:34pm CT): The load balancer and its host machine have recovered. At this time, all sites are up. If you have any lingering issues, please contact our help desk (support@pressable.com).

We are currently seeing issues with sites which are hosted on the Bode galaxy. The issues are related to a problem which Rackspace is currently experiencing on a load balancer utilized by the Bode galaxy.

We are working to route traffic to a new load balancer and are awaiting an estimate time of repair from Rackspace.

We appreciate your patience as we work to restore full functionality on the Bode galaxy. Please let us know if you have any additional concerns by reaching out to us through our help desk (help.pressable.com) or by email (support@pressable.com). 

Rackspace Scheduled Maintenance: February 16-17

UPDATE: The scheduled maintenance has been completed. At this time, all sites are up. If you have any lingering issues, please contact our help desk (support@pressable.com).

Rackspace, our backbone provider, will be performing critical maintenance on the network that connects its cloud infrastructure from 9:00pm CT on Tuesday, February 16th to 5:00am CT on Wednesday, February 17. During this 8-hour window, we expect a brief interruption of service to sites which are hosted at Rackspace’s IAD data center.

We will update our status blog once the maintenance has been completed. If you have any questions or concerns, please contact our help desk (support@pressable.com).