Microsoft 365 Outage Caused by WAN Router IP Address Change, Says Microsoft

Published: Jan 30, 2023

Hero approved Microsoft 365

SHARE ARTICLE

Last week, Microsoft confirmed a multi-hour outage that prevented users from accessing Azure, Microsoft Teams, and other Microsoft 365 services. The company has since published a preliminary post-incident report to update customers about the root cause of the issue.

Microsoft explained that the disruption was caused by a router IP address change during a planned maintenance update on January 25. It had broken connections between client devices and Azure. The problem impacted Exchange Online, Teams, Outlook, OneDrive for Business, SharePoint Online, Microsoft Intune, PowerBi, Microsoft Graph, M365 Admin Portal, Microsoft Defender for Identity, and Microsoft Defender for Cloud Apps.

“As part of a planned change to update the IP address on a WAN router, a command given to the router caused it to send messages to all other routers in the WAN, which resulted in all of them recomputing their adjacency and forwarding tables. During this re-computation process, the routers were unable to correctly forward packets traversing them,” Microsoft wrote on the Azure status history page.

Microsoft 365 Outage Caused by WAN Router IP Address Change

Microsoft takes a proactive approach to prevent global Microsoft 365 outages

Microsoft’s engineers reviewed the recent changes and identified the problematic command that caused the domain name service (DNS) and WAN issues. The company later rolled back the network change to mitigate the issue across all Microsoft 365 tenants worldwide.

Microsoft’s services were fully back online around 12:43 UTC on 25 January after a downtime of more than four hours. The company confirmed that the global outage also affected Azure Government cloud services.

Microsoft emphasized that it has taken a couple of steps to prevent similar incidents from happening in the future. It has blocked all highly impactful commands from running on the network devices. Going forward, all command execution on devices must adhere to safe change guidelines. Microsoft intends to release the final post-incident report within the next few weeks.

SHARE ARTICLE