Cluster 1 Hosting Issues: Saturday update
By 21:00 on Friday evening, we had enacted all of the measures that we outlined in the previous update, i.e.
- doubling the size of Cluster 1 — increasing to 6x VMs;
- using our Web Application Firewalls (WAF), we "soft" split Cluster 1 into 3x sub-Clusters of 2x VMs each;
- all of the sites currently pointed at Cluster 1 were redistributed to one of the 3x sub-Clusters.
As of now, this seems to have been effective, with no more issues monitored on Saturday morning. We will continue to check in on Sunday morning too.
The real challenge will come with the massive increase in traffic that we see on a Monday, but we will be prepared and ready to counter any issues at that point.
We continue to scan for possible Root Causes, but have yet to establish anything definitive. We will continue to investigate, and communicate with our customers as appropriate.