Microsoft Traces Day-Long Outlook Outage to a Bug Hiding Since July
Microsoft has revealed the root cause of the nearly day-long Microsoft 365 outage that knocked Outlook, Teams and other services offline on November 12–13: a software defect introduced in July, combined with a surge in traffic and reduced gateway capacity.
Microsoft Traces Day-Long Outlook Outage to a Bug Hiding Since July
Microsoft has explained what knocked Outlook, Teams and other Microsoft 365 services offline for almost a full day last week, and the answer reaches back months. A software defect introduced in July sat quietly in the company's networking gateway servers for more than three months before a surge in traffic and a shortfall in capacity finally triggered it, bringing much of Microsoft's cloud productivity suite to a halt.
Charles Lamanna, Microsoft's executive vice president of Copilot, Agents and Platform, laid out the cause in a post on X on Monday, November 16, drawing on the company's preliminary post-incident report. He apologised to customers, writing that Microsoft is "deeply sorry for the impact to all our customers" and recognises the disruption to their normal business and operations.
The outage began on Thursday, November 12, at 4:20 p.m. UTC and ran until 4:10 p.m. UTC on Friday, November 13, lasting nearly 24 hours. Outlook, Teams, Exchange Online, SharePoint, OneDrive, Microsoft 365 Copilot and Office apps were all affected at various points, with disruptions reported in 74 countries, according to the Associated Press.
Three factors combined to bring the gateway down
Lamanna said three factors together caused the outage of the networking gateway that sits in front of many Microsoft 365 services. None of them alone would have been enough. Together, they overwhelmed a system that had been running normally for months.
The first was the latent defect itself. In July, Microsoft rolled out a fix prepared specifically for its Microsoft 365 networking gateway servers. The change introduced a flaw that, under certain load conditions, caused lock contention inside an internal Windows process, leaving servers overwhelmed and unresponsive. It spread worldwide over several weeks and caused no noticeable problems for more than three months.
Microsoft's own testing never caught it. Lamanna said the company's load tests ran mostly on less powerful servers, and the problem was worse on newer hardware with more processor cores, where it was far more likely to appear. The company stressed that the defect is limited to its own gateway infrastructure and does not affect regular Windows PCs.
The second factor arrived about 15 hours before the outage. As part of regular failover operations, Microsoft Teams shifted from an alternative gateway to the primary gateway that also supports Exchange Online, the company's email platform, pushing more traffic through a single route.
The third was capacity. At the same time, nearly 14 percent of the gateway's server capacity was out of service because of an unrelated issue from the previous day. With higher load and reduced capacity, the gateway hit the dormant July bug, requests began to build up, and servers stopped responding. At the peak of the incident, more than 80 percent of the servers behind the networking gateway were unresponsive.
Why the fix took so long
The outage struck on a Thursday afternoon, during peak hours of use in the United States, and Microsoft's engineers worked through the night. But there was no quick undo. Because the bug had been in place for months, a simple rollback was not an option, and when servers were brought back into service they were immediately overwhelmed by the backlog of waiting traffic and became unresponsive again.
Microsoft eventually restored the earlier working code, restarted the servers and brought traffic back in stages. It also cleared delayed emails, expanded server capacity and added safeguards against similar cascading failures. The services were fully recovered on November 13, as reported earlier.
The company was clear about what did not cause the incident: there is no evidence of malicious activity, and neither a cyberattack nor AI agents played a part.
What comes next
The details released so far come from Microsoft's preliminary post-incident report, dated November 15. A final post-incident review with more detail and further preventive steps is expected on November 20. For the businesses, hospitals, governments and other organisations that spent most of a working day without email and meetings, the final report will show whether Microsoft's safeguards can stop a months-old defect from doing this again.
For more on how the services were brought back online, see our earlier report: Microsoft 365 Outage Over as Outlook and Teams Fully Recover.
Sources
- Engadget — Microsoft Reveals That An Old Bug Caused The Massive Outage That Took Its Services Offline
- Moneycontrol — Microsoft outage reason: Company reveals bug dormant since July caused global Outlook and Microsoft 365 disruption
- WCCFTech — Microsoft Reveals Root Cause Of Massive Microsoft 365 Outage, As Months-Old Windows Bug Crippled Over 80% Of Servers
- NDTV Profit — A Hidden Software Bug From July: Microsoft Finally Reveals What Triggered Global Outlook Outage
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0