Security News > 2022 > July > Massive Microsoft 365 outage caused by faulty ECS deployment

In a preliminary post-incident report, Microsoft has revealed that this week's 5-hour-long Microsoft 365 worldwide outage was triggered by a faulty Enterprise Configuration Service deployment that led to cascading failures and availability impact across multiple regions.

ECS is an internal central configuration repository designed to enable Microsoft services to make wide-scope dynamic changes across multiple services and features, as well as targeted ones such as specific configurations per tenant or user.

What initially started like a minor Microsoft Teams outage ended up expanding downstream to multiple Microsoft 365 services with Teams integration that also leverage ECS, including Exchange Online, Windows 365, and Office Online.

As a result, users worldwide began reporting that they could not use Microsoft Teams and multiple Microsoft 365 services or features.

"A deployment in the ECS service contained a code defect that affected backward compatibility with services that leverage ECS. The net result was that for services that utilize ECS it would return incorrect configurations to all its partners," the company explained.

As a result of this incident, Microsoft says they're working on improving the resiliency of the Microsoft Teams service to fail back to a cached ECS configuration version in the event of a future ECS failure.

News URL

https://www.bleepingcomputer.com/news/microsoft/massive-microsoft-365-outage-caused-by-faulty-ecs-deployment/