As many people know, this week Facebook had a massive outage that affected all of their services. The cause of this is straightforward, simple, and incredibly silly to anyone who looks at it. In order to explain this fully, let us go into a quick breakdown of what happened:
So, how can a security engineer look at this situation and draw conclusions? Well, first thing to look at is how does a system cope with failure? If one system fails, is the failover on the same network or could it be impacted by the initial failure? This happens all the time in Security processes - if you have malware detected at your ID / network protection solution - do you have a secondary scanner on endpoints to pick it up? What does your response look like?
Thinking about all of these things is important when handling a security event or planning for a breach. As I've written about before, good Security Architecture is having plans and processes that can deal with cascading failure. If enough potential problems rest on a single point of failure, this can completely bring down a company (similar to Facebook's issues this week) - and this can be a more common problem without diversifying or using layered security.
Some considerations / goals of layered security:
What if you aren't sure how to proceed with the above? Well, don't be afraid to reach out to security vendors and ask what their processes are when one of their systems goes down. Most service providers have details they can provide that keep you appraised of what actions to take in case of an outage. Make sure this information is included in your overall plan - it helps to know who to call or what process to follow if it is a service you do not control fully internally.
And if those answers aren't clear, it is important to reach out to trusted security advisors or other security providers you work with in order to have a clear plan built for your business. Plan for the best, but also plan for the worst. It is important to have both plans in place and resolve any potential issues before they become glaring weaknesses in your security posture.