RESOLVED: Instability on Web Clusters

Howdy,

We are experiencing suspicious activity that is causing instability on our some of our web server clusters.

We are working on expanding the stability/performance on this cluster and identifying the source of this suspicious activity. During this time, customers may experience intermittent 502s and slow loading sites.

We will update the status blog as soon as this is finalized.

UPDATE 2:50PM CST: The team is still working to bring new hardware and firewall rules online to help mitigate the attacks on our systems. We’ve brought new hardware online for some customers which is beginning to return some stability to the systems.

UPDATE 4:00PM CST: At this time we’ve brought on our new hardware and firewall rules for all of the customers on Thor. We’re still working to get these in place for our customers on Galaxy01.

UPDATE 5:55PM CST: At this time we’ve brought up new hardware and firewall rules for all customers. Stability should be returning to the web servers, but we’re still experiencing database issues.