Limited functionality with database. – Emergency Maintenance

Symptoms

Some customers are experiencing limited functionality with the database systems, the symptoms appear to be that their posts and updates are not saving. Customers may also be experiencing issues approving comments or changing site/plugin settings. New site deployments and clones may also not be working.

What caused this?

This is due to a problem in our database cluster, currently when you save a new post, or update an existing one, we write to one server, and then the changes are propagated to 4 other servers. The link between the server you write to, and the servers that replicate the changes has been severed. This gives the illusion of things not being saved.

Time to resolve this issue

We are working to resolve this issue as soon as we can, in order to do so, we will need to make a change to the system where you will indeed not be able to make any changes for about 10 minutes.

We will provide an update to this post when the maintenance has begun. We do not expect any downtime, just the inability to save a post or new posts. Your website will continue to be served to your visitors without any issues.

UPDATE 10:10AM CDT: We’ve begun the maintenance on the database server and expect things to go smoothly. We’ll provide an update when the maintenance is completed.

UPDATE 10:35AM CDT: We’re currently experiencing an issue with the database maintenance that’s causing an interruption to service. We’re working to move traffic to one of the secondary servers and bring things back online.

UPDATE 10:55AM CDT: We’ve brought secondary servers online to help bring things back online. We’re still working through remaining issues with the emergency maintenance.

UPDATE 11:40AM CDT: Our team is still working on the ongoing database issue.

UPDATE 12:30PM CDT: The team is still hard at work resolving database issues.

UPDATE 1:30PM CDT: We’re expecting things to be more stable at this time and most sites to have returned to normal. If you’re still experiencing issues please let our support team know so we can investigate for you.