Category Archives: outage

404 Error Affecting Limited Amount of Sites

3:56 UTC – Domains have been added to all affected sites and they are now back online. Please contact help@pressable.com if you have any questions.

20:18 UTC – We continue to move files and databases into the restored sites. Once this process is complete, we are adding domains to the restored sites. We appreciate your continued patience and understanding.

17:37 UTC – Sites have been restored to the affected accounts. We are now in the process of adding the domains to the sites.

14:12 UTC – Progress continues to be made with the small amount of remaining sites to restore. Thank you for you understanding and we will continue to regularly update.

09:00 UTC – The engineering team has restored several sites. However, some of these sites may not have primary domains set or be in Production Mode. Once the primary domain is set and Production Mode is enabled, any issues you observe with domain redirects and missing JavaScript or CSS files should resolve themselves within approximately 10 minutes. Our apologies in advance for the inconvenience.

A very small number of sites may still be affected by this issue, however. We are working on restoring these sites and will post here once we have more information. Thank you for your continued patience and understanding. If you have questions or concerns, please contact the Help Desk: help@pressable.com.

08:12 UTC – The engineering team continues their work. We anticipate most affected sites to be functioning properly within the next hour.

07:12 UTC – Our engineering team continues to work on the restoration of the affected sites. We expect the majority of them to be online in the next 2 hours. We thank you for your continued patience.

06:12 UTC – Our engineering team is implementing a fix to restore all affected sites. We will have 80% of the impacted sites back online within 3 hours. We are exploring all methods of restore from our backups and will keep posting as information becomes available or every hour until this is resolved.

05:12 UTC – Our engineering team continues to work on restoring the affected sites. We appreciate your continued patience and understanding.

04:12 UTC – Our systems team is currently investigating an issue that caused some sites on our platform to become unavailable. This issue occurred as a result of routine maintenance. We have isolated the cause and our engineering teams are working to restore the affected sites as fast as possible.

April 18th Outage Update

Last Wednesday, April 18th, Pressable experienced an outage from an accidental fiber cut in our upstream provider’s connectivity. Over the past week we’ve worked with our upstream provider, as well as our engineering team, to ensure future outages don’t happen again.

We have evaluated and inspected the redundant fiber connectivity to our data center. During this process we confirmed the following important factors. First, the upstream provider had an undocumented and isolated area or fiber without redundancy. The outage occurred when the fiber in this area was accidentally cut.

Secondly, there is now 99.07% redundancy for the remaining connectivity. This provides confidence in our facility serving the uptime and availability you’ve come to expect from Pressable.

We are working as quickly as possible to get the final < 1% of our network fully redundant in the weeks ahead and will continue to update our valued customers as progress is made. In the meantime, if you have any questions feel free to reach out to me directly by emailing  Jay@Pressable.com .

Complete Platform Site Connection Issues

04/18 23:31 UTC: Our engineering team has finished their review and all systems are now fully operational. Again we want to apologize for the challenges this loss of connectivity has caused our customers and thank everyone for their patience.

04/18 23:08 UTC: The network has been restored and sites are coming back online. Our systems team is evaluating the status on our end. This post will be updated as more information is available.

04/18 22:15 UTC: The problem has been isolated to a fiber cut in San Antonio, Texas. Our provider has let us know they are actively working on repairing the damage. We expect to have services restored in the next 2 hours.

04/18 20:54 UTC: We have been in contact with our upstream provider who has informed us the outage is caused by a potential fiber cut. They are dispatching teams and working diligently to restore connectivity to our servers.

04/18 19:49 UTC: Our upstream provider is still working to restore connectivity to the data center. We’re actively monitoring the situation and will restore full service once the  issue is resolved. We apologize for the challenges this loss of connectivity has caused our customers.

04/18 18:49 UTC: We’ve had an update from our upstream provider that they’re aware of the outage on their end and currently working to get this resolved.

We expect the outage to be resolved as soon as they are back online. For their status updates follow along here:
https://status.cogecopeer1.com/pages/58b022e2ed91158a3a000df7

04/18 18:20 UTC: We’re currently experiencing some connection issues on a large number of sites on our platform. We’re looking into the issue, and should have this resolved shortly. Thank you for your patience.

 

4/11 Brief Outage (RESOLVED)

At approximately 7:24 PM Central on 4/11 (00:24 UTC on 4/12), we experienced a brief outage for some sites on our network. We isolated the issue and site availability was restored to normal at approximately 7:30 PM Central (00:30 UTC).

We will continue to monitor the situation but do not expect further issues. If you have any questions or concerns, please contact our help desk by submitting a ticket from your https://my.pressable.com control panel.

Thanks!

3/29 Partial Outage

12:40 UTC We are aware of and investigating a partial outage impacting some customer sites.

13:05 UTC Outage resolved. Investigation into cause of outage ongoing.

Investigating outage

RESOLVED @ 8:33AM CST: Things have been stable for more than 8 hours. The root cause of the incident does appear to be a spike in traffic to our servers which caused a rather ungraceful failure of a couple load balancers.  We’ve identified some issues that will help prevent this scenario in the future and will be implementing them over the next few days.

UPDATE @ 12:18AM CST:  All sites continue to operate normally.  We will continue to monitor things throughout the evening.  Our initial findings show that this incident was most likely caused by an abnormal spike in traffic toward Pressable’s servers.

We are currently investigating an outage affecting a percentage of the sites on the Pressable platform.  Currently, everything is back up and operating as expected.  The partial outage lasted from around 11:00 Central to 11:30 Central time.  Once we have more information, we will update this post.