Security News > 2023 > January > Massive Microsoft 365 outage caused by WAN router IP change
Microsoft says this week's five-hour-long Microsoft 365 worldwide outage was caused by a router IP address change that led to packet forwarding issues between all other routers in its Wide Area Network.
Redmond said at the time that the outage resulted from DNS and WAN networking configuration issues caused by a WAN update and that users across all regions serviced by the impacted infrastructure were having problems accessing the affected Microsoft 365 services.
"Between 07:05 UTC and 12:43 UTC on 25 January 2023, customers experienced issues with networking connectivity, manifesting as long network latency and/or timeouts when attempting to connect to resources hosted in Azure regions, as well as other Microsoft services including Microsoft 365 and Power Platform," Microsoft said in a preliminary post-incident report published today.
Microsoft now also revealed that the issue was triggered when changing the IP address of a WAN router using a command that had not been thoroughly vetted and that has different behaviors on different network devices.
"As part of a planned change to update the IP address on a WAN router, a command given to the router caused it to send messages to all other routers in the WAN, which resulted in all of them recomputing their adjacency and forwarding tables," Microsoft said.
As a result of the pause, some network paths continued experiencing increased packet loss from 9:35 UTC until the systems were manually restarted, returning the WAN to optimal operating conditions and completing the recovery process at 12:43 UTC. Following this incident, Microsoft says that it's now blocking highly impactful commands from being executed and that it will also require all command execution to follow guidelines for safe configuration changes.
News URL
Related news
- Microsoft 365 outage takes down Office web apps, admin center (source)
- Microsoft MFA outage blocking access to Microsoft 365 apps (source)
- Azure, Microsoft 365 MFA outage locks out users across regions (source)
- Microsoft investigates Microsoft 365 outage affecting users, admins (source)
- CISA orders federal agencies to secure Microsoft 365 tenants (source)
- Microsoft 365 users hit by random product deactivation errors (source)
- Microsoft fixes bug behind random Office 365 deactivation errors (source)
- Severe Security Flaws Patched in Microsoft Dynamics 365 and Power Apps Web API (source)
- Microsoft 365 apps crash on Windows Server after Office update (source)
- Hackers use FastHTTP in new high-speed Microsoft 365 password attacks (source)