Security News > 2023 > January > Massive Microsoft 365 outage caused by WAN router IP change
Microsoft says this week's five-hour-long Microsoft 365 worldwide outage was caused by a router IP address change that led to packet forwarding issues between all other routers in its Wide Area Network.
Redmond said at the time that the outage resulted from DNS and WAN networking configuration issues caused by a WAN update and that users across all regions serviced by the impacted infrastructure were having problems accessing the affected Microsoft 365 services.
"Between 07:05 UTC and 12:43 UTC on 25 January 2023, customers experienced issues with networking connectivity, manifesting as long network latency and/or timeouts when attempting to connect to resources hosted in Azure regions, as well as other Microsoft services including Microsoft 365 and Power Platform," Microsoft said in a preliminary post-incident report published today.
Microsoft now also revealed that the issue was triggered when changing the IP address of a WAN router using a command that had not been thoroughly vetted and that has different behaviors on different network devices.
"As part of a planned change to update the IP address on a WAN router, a command given to the router caused it to send messages to all other routers in the WAN, which resulted in all of them recomputing their adjacency and forwarding tables," Microsoft said.
As a result of the pause, some network paths continued experiencing increased packet loss from 9:35 UTC until the systems were manually restarted, returning the WAN to optimal operating conditions and completing the recovery process at 12:43 UTC. Following this incident, Microsoft says that it's now blocking highly impactful commands from being executed and that it will also require all command execution to follow guidelines for safe configuration changes.