Security News > 2021 > October > Facebook: Outage caused by faulty routing configuration changes
Facebook says that yesterday's worldwide outage was caused by faulty configuration changes made to its backbone routers that brought all its services to a halt.
"Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication," said Santosh Janardhan, VP for Engineering and Infrastructure at Facebook.
He also said that there is no evidence that Facebook users' data was compromised due to this downtime, with the company pinning the root cause behind this incident on a faulty configuration change.
While Facebook didn't provide any details and the massive outage appeared to be DNS-related at first, it was later learned that the problem was far worse and a lot harder to fix.
Multiple Facebook routing prefixes suddenly disappeared from the Internet's BGP routing tables, which immediately made it impossible to connect to any services hosted on those IP addresses, as Giorgio Bonfiglio, a Principal Technical Account Manager at Amazon AWS, explained.
Since Facebook's domain registrar and DNS servers are hosted on the company's own routing prefix, when the BGP prefixes were removed from routing tables, no one could connect to their IP addresses or the services running on top of them.