Microsoft 365 outage: a July bug, a Teams failover and lost capacity
Three small problems combined into a 24-hour Exchange Online outage, Microsoft says. Its load tests used weaker hardware than production.
On this page · 1 min read
Key takeaways
- 1The outage ran from 11:20 ET on 12 November to 11:10 ET on 13 November. The cause was a bug shipped in July in the gateway servers’ networking update.
- 2It needed a trigger: Teams failing over onto Exchange’s infrastructure added traffic while about 14% of gateway capacity was already offline.
- 3Microsoft says its load tests used weaker hardware with fewer cores, and the bug gets worse on bigger, newer machines.
Original source · bleepingcomputer.com
Microsoft says months-old bug caused global Microsoft 365 outage
Published
What happened
Microsoft’s preliminary post-incident report in the Microsoft 365 Admin Center blames three problems for the outage that began at 11:20 AM ET on 12 November and lasted until 11:10 AM ET the next day, according to BleepingComputer.
The first was a previously unknown bug in an update for the networking gateway servers, shipped in July. It ran for more than three months without trouble, but could make servers stop responding under certain traffic conditions. Microsoft says its tests missed it because the lab traffic mix and server types were weaker than production: the contention problem is worse on machines with more cores and newer hardware.
The second was extra load. About 15 hours earlier, Microsoft Teams automatically failed over to the networking infrastructure used by Exchange Online. The third was capacity: nearly 14% of the gateway’s servers were already offline for an unrelated reason. With that combination, the bug triggered, and as servers stopped responding their traffic moved to the remaining ones, which then failed too. In under two hours the defect was hit across the fleet.
Teams, OneDrive and SharePoint could switch to other infrastructure and recovered earlier. Exchange Online could not, so mailboxes stayed unreachable. Restarting servers did not help at first because backed-up traffic overwhelmed them again. Microsoft has added capacity, improved monitoring and added safeguards against redirected traffic causing cascades, and has promised a final report.
Why it matters
Nothing here was exotic. A latent defect, a traffic shift and a small capacity loss lined up, and the system’s own protections spread the failure instead of containing it. Microsoft says more than 80% of servers were unresponsive at the lowest point, and that safeguards meant to stop problems crossing regions failed to hold.
In one sentence:a failover that works can still overload the next system, and recovery plans need the same testing as the failover itself.
What it means for you
- Run load tests on production-class hardware, including the highest core counts you deploy.
- Test what happens when traffic from a failed-over service lands on a system that is already down some capacity.
- Check that restarting servers under backed-up traffic does not just knock them over again, and have a way to shed load first.
- Keep a plan for mail and chat that does not depend on one vendor’s region or one product.
- Ask any provider what happens to your service when their own failover triggers.