Monday October 2021 was an Internet outage day. Many Facebook users faced outage and felt they had “internjet” instead of Internet. Also ePanorama.net has few hours outage.
ePanorama.net went down few hours earlier. When I got my server up, in few hours Facebook was able to get their site up. Interesting correlation, but correlation does not always indicate causality. The reasons were different and size of incidents also. At this site it was just simple locale server error (out of disk space on critical part) that took some time to get access to and fix. At Facebook the situation was quite different – they had a configuration accident that locked out all of their servers out from Internet.
The crash of Facebook’s services for six hours showed how vulnerable we are. In the future, the internet will be as important as the electricity grid, and it will scare even M-Hyppö, F-Secure’s research director. According to the news agency Reuters, problems had also occurred on Twitter, Google and Amazon at the same time. F-Secure’s Mikko Hyppönen: Concentration of online services in Silicon Valley is a weak point of the Internet Facebook squat “absolutely exceptional”
Understanding How Facebook Disappeared from the Internet article https://blog.cloudflare.com/october-2021-facebook-outage/ gives a good explanation what happened to Facebook:
“The Internet is literally a network of networks, and it’s bound together by BGP. BGP allows one network (say Facebook) to advertise its presence to other networks that form the Internet. As we write Facebook is not advertising its presence, ISPs and other networks can’t find Facebook’s network and so it is unavailable. With those withdrawals, Facebook and its sites had effectively disconnected themselves from the Internet. As a direct consequence of this, DNS resolvers all over the world stopped resolving their domain names.”
You can find the official Facebook explanation posted to https://engineering.fb.com/2021/10/04/networking-traffic/outage/
“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.”
Facebook Outage: Yes, its DNS (sort of). A super quick analysis of what is going on
“More readable summary of the analysis below: The BGP routes pointing traffic to Facebook’s IP address space have been withdrawn. The Internet no longer knows where to find Facebook’s IPs. One symptom is that DNS requests are failing. But this is just the result of Facebook hosting its DNS servers inside its own network. Even with working DNS (for example if you still have cached results), the IPs are currently not reachable.”
More information links:
Facebook, Instagram, WhatsApp, and Oculus are down. Here’s what we know [Updated]
The root cause of the worldwide outage appears to be a flubbed BGP route update.
Zuckerberg Loses $5.9 Billion In A Day As Facebook Faces Rare Outage, Whisteblower Testimony
Facebook, WhatsApp and Instagram are slowly returning. Why did they disappear to begin with?
It’s always DNS, except when it’s BGP
These Three Letter Tech Acronyms Are Likely Behind Today’s Mega Outage At Facebook