Database Outage

We are currently experiencing an issue with a database that is causing some sites in our network to be down and/or unresponsive. We are working quickly to address the database failure and will update this once service has been restored or as more information becomes available.

NOTE: This impacts the availability of our control panel, provisioning systems, and SFTP access for ALL customers.

NOTE 2: You may be seeing WordPress installation screens on your sites during the import process. This is because no database exists for your site at this time. However, once your site hits the restore process, whatever exists will be dropped from the database and your site will be restored.

UPDATE 4/22/14 at 5:00PM Central: We are continuing to work through this issue. Many sites are back up and running and more are coming up every moment. Once all sites are back up and running, we will update once more. If you have questions or concerns, please submit a ticket via my.pressable.com.

UPDATE 4/22/14 at 6:00PM Central: We are currently working through some additional issues with our database servers. We’re still expecting some downtime while we investigate the underlying cause of the current issues.

UPDATE 4/22/14 at 7:50PM Central: We are still investigating issues with our database servers that are causing sites to become unavailable, unresponsive and slow. Our team is working through the problems, but we still do not have an ETA for complete resolution.

UPDATE 4/22/14 at 8:35PM Central: Our team is taking measures to restrict access to the database servers while we continue to investigate ongoing issues. Most sites will be unavailable while our team continues to resolve this issue.

UPDATE 4/22/14 at 10:20PM Central: There is nothing new to report at this time.

UPDATE 4/22/14 at 11:45PM Central: Our team is still working to resolve issues. At this time there is no further information to report.

UPDATE 4/23/14 at 12:30AM Central: At this time our team is beginning a restore process on the database server. While this isn’t the ideal solution, we’ve exhausted other recovery options with the server. Due to the nature of the restore, sites will be rolled back to about 4/22/14 at 3:30pm Central. We expect to have things restored by 4/23/14 at 5:00am Central, but we’ll provide updates as the process continues.

UPDATE 4/23/14 at 5:00AM Central: At this time databases are still being restored by our systems. We do not have a new ETA, but we’re on the home stretch of getting the primary database server restored. A note about the process: You may be seeing WordPress installation screens on your sites during the import process. This is because no database exists for your site at this time. However, once your site hits the restore process, whatever exists will be dropped from the database and your site will be restored.

UPDATE 4/23/14 at 7:00AM Central: Databases are still being restored by our systems. We do not have an updated ETA at this time, but the restore process is still working as expected.

UPDATE 4/23/14 at 8:00AM Central: We’re continuing to work through the restore process.

UPDATE 4/23/14 at 9:30AM Central: We’re working through some lingering issues with individual sites at this time. We’re also prepping to have the slaves rebuilt from the restore point, we’ll provide more details shortly.

UPDATE 4/23/14 at 1:15PM Central: We’re still working to get everyone’s database imported from the backup. If you’re still having issues, please submit a ticket so that we’re able to resolve it for you. We’re also still experiencing some instability and performance issues while we work to recover to 100%.

UPDATE 4/23/14 at 8:30PM Central: Our team is still working to restore databases for sites who have been missed or had issues due to the automated processes used. If you’re experiencing issues, please submit a ticket so our team can get it in the queue to be processed.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s