Last Wednesday, April 18th, Pressable experienced an outage from an accidental fiber cut in our upstream provider’s connectivity. Over the past week we’ve worked with our upstream provider, as well as our engineering team, to ensure future outages don’t happen again.
We have evaluated and inspected the redundant fiber connectivity to our data center. During this process we confirmed the following important factors. First, the upstream provider had an undocumented and isolated area or fiber without redundancy. The outage occurred when the fiber in this area was accidentally cut.
Secondly, there is now 99.07% redundancy for the remaining connectivity. This provides confidence in our facility serving the uptime and availability you’ve come to expect from Pressable.
We are working as quickly as possible to get the final < 1% of our network fully redundant in the weeks ahead and will continue to update our valued customers as progress is made. In the meantime, if you have any questions feel free to reach out to me directly by emailing Jay@Pressable.com .
04/18 23:31 UTC: Our engineering team has finished their review and all systems are now fully operational. Again we want to apologize for the challenges this loss of connectivity has caused our customers and thank everyone for their patience.
04/18 23:08 UTC: The network has been restored and sites are coming back online. Our systems team is evaluating the status on our end. This post will be updated as more information is available.
04/18 22:15 UTC:
04/18 20:54 UTC: We have been in contact with our upstream provider who has informed us the outage is caused by a
04/18 19:49 UTC: Our upstream provider is still working to restore connectivity to the data center. We’re actively monitoring the situation and will restore full service once the issue is resolved. We apologize for the challenges this loss of connectivity has caused our customers.
04/18 18:49 UTC: We’ve had an update from our upstream provider that they’re aware of the outage on their end and currently working to get this resolved.
We expect the outage to be resolved as soon as they are back online. For their status updates follow along here:
04/18 18:20 UTC: We’re currently experiencing some connection issues on a large number of sites on our platform. We’re looking into the issue, and should have this resolved shortly. Thank you for your patience.
At approximately 7:24 PM Central on 4/11 (00:24 UTC on 4/12), we experienced a brief outage for some sites on our network. We isolated the issue and site availability was restored to normal at approximately 7:30 PM Central (00:30 UTC).
We will continue to monitor the situation but do not expect further issues. If you have any questions or concerns, please contact our help desk by submitting a ticket from your https://my.pressable.com control panel.
UPDATE: 19:25 UTC phpMyAdmin is up and working at this time.
19:14 UTC We are aware of and investigating an issue with access to phpMyAdmin in my.pressable.com
12:40 UTC We are aware of and investigating a partial outage impacting some customer sites.
13:05 UTC Outage resolved. Investigation into cause of outage ongoing.
RESOLVED @ 8:33AM CST: Things have been stable for more than 8 hours. The root cause of the incident does appear to be a spike in traffic to our servers which caused a rather ungraceful failure of a couple load balancers. We’ve identified some issues that will help prevent this scenario in the future and will be implementing them over the next few days.
UPDATE @ 12:18AM CST: All sites continue to operate normally. We will continue to monitor things throughout the evening. Our initial findings show that this incident was most likely caused by an abnormal spike in traffic toward Pressable’s servers.
We are currently investigating an outage affecting a percentage of the sites on the Pressable platform. Currently, everything is back up and operating as expected. The partial outage lasted from around 11:00 Central to 11:30 Central time. Once we have more information, we will update this post.
RESOLVED @ 3:22pm CST: The network issues have been resolved and everything is online at this time. We continue to investigate the root cause with our provider.
3:19pm CST: There is currently a problem with our hosting provider which is affecting all Pressable-hosted websites. We are actively troubleshooting the issue and will update this post with more details as they are available.