RESOLVED @ 8:33AM CST: Things have been stable for more than 8 hours. The root cause of the incident does appear to be a spike in traffic to our servers which caused a rather ungraceful failure of a couple load balancers. We’ve identified some issues that will help prevent this scenario in the future and will be implementing them over the next few days.
UPDATE @ 12:18AM CST: All sites continue to operate normally. We will continue to monitor things throughout the evening. Our initial findings show that this incident was most likely caused by an abnormal spike in traffic toward Pressable’s servers.
We are currently investigating an outage affecting a percentage of the sites on the Pressable platform. Currently, everything is back up and operating as expected. The partial outage lasted from around 11:00 Central to 11:30 Central time. Once we have more information, we will update this post.
RESOLVED @ 3:54 AM Central: Our hosting provider isolated the cause of the problems and made changes that should prevent further service disruptions. We are not anticipating another loss of connectivity and will continue to monitor the situation.
UPDATE @ 3:41 AM Central: The network connectivity issues returned momentarily, but things have recovered. The root cause appears to be some network instability between Dallas and San Antonio, Texas. Will provide more details once they are available.
UPDATE @ 3:34 AM Central: The network issues have been resolved and everything is online at this time. We continue to investigate the root cause with our provider.
3:14AM Central: There is currently a problem with our hosting provider which is affecting all Pressable-hosted websites. We are actively troubleshooting the issue and will update this post with more details as they are available.
On Friday, June 10th, at 17:37 UTC (12:37PM Central) we accidentally deployed a platform-wide change that caused the
DB_CHARSET value to be set incorrectly on most Pressable-hosted websites.
DB_CHARSET is very important, as it tells the web server which character set to use when talking to the database server. It’s important that the values on both the web and database servers match and our change caused a mismatch.
We identified the problem and rolled back our changes at 19:52 UTC (2:52PM Central), but during the 2 hours and 15 minutes the problem existed, if you edited any content on your site, there is a chance it’s now corrupt. This corruption can appear in a few different ways, but one obvious way is that quotes might now appear as “gibberish”. Unfortunately, the problem is not limited to pages and posts, but can also affect widgets, settings, etc.
One of the benefits of our new 2.0 platform is a high level of automation and consistency across severs and websites. This should result in a more consistent, predictable, and improved user experience. Unfortunately, in this case, our newfound automation caused a major problem and for that we apologize. We are adding additional tests, checks, and monitoring to ensure a similar problem doesn’t happen in the future.
If your site has been behaving strangely since Friday, please get in touch and we’ll help you restore things to normal.