eBay Deals


1 hour ago 14

At 9AM last Thursday, everything seemed normal. Microsoft employees were arriving on campus, ready for a regular day of emails and meetings. But barely an hour later, one of Microsoft’s worst outages in history started to unfold, leaving employees, businesses, and many governments without access to Outlook emails for nearly 24 hours.

It all started in Switzerland, according to a source familiar with the outage. Exchange Online servers that power Outlook started to fall offline. The problem quickly spread to Mexico, and then to other countries around the world, as Microsoft’s failover systems tried to shift email traffic elsewhere. Within a couple of hours, more than 80 percent of Exchange servers were unresponsive.

Sources tell me that Microsoft marked the outage as “sev0” in its internal incident tracking system, the highest level, which indicates a catastrophic level of service disruption. Hundreds of engineers began trying to figure out what had gone wrong, and how to restore service. It’s not unusual for servers to fail under certain circumstances, but Microsoft has a variety of systems that are supposed to route traffic toward healthy servers to keep the broader system online. In this case, the failover systems did their job, but they also spread the problem.

In the early hours of the outage, engineers were analyzing crash reports, trace logs, and kernel dumps to try and find clues to the giant outage. Some servers had entered into maintenance mode and became unresponsive due to unusually high CPU loads. Engineers tried to bring servers back online, but they were almost immediately overwhelmed by traffic waiting to access the servers.

Exchange servers boot up and run an OS over a network (Preboot Execution Environment), a process that allows Microsoft’s support staff and engineers to manage OS updates remotely. I understand that the sudden burst of traffic from servers being rebooted was overwhelming the systems monitoring them, causing a form of reboot loop where they kept trying and failing to come back online.

As engineers continued to hunt for the root cause, Microsoft was even struggling to roll back servers to a healthy state using its usual processes of reverting recent changes. As the outside world began to wonder if this outage was related to a cyberattack or sloppy AI coding, Microsoft’s engineers were already investigating potential attack vectors and liaising with DDoS specialists to understand whether this could have been an attack.

I’m told crash reports and logs quickly ruled out the possibility of an attack, and after recent rollbacks failed, Microsoft’s engineers began to take a closer look at the networking stack of the servers and discovered an unusual bug. It was unusual because the bug had been sitting on Microsoft 365 networking gateway servers since July without any impact to servers.

“A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions,” wrote Charles Lamanna, the leader of Microsoft’s Copilot, Agents, and Platform (CAP) team, in a post on X. The teams behind Office, Outlook, and Microsoft Teams all started reporting to Lamanna last month after the head of Office and Teams, Ryan Roslansky, left the company.

Microsoft’s preliminary post-incident report describes the bug as one affecting select Windows servers “due to lock contention inside an internal Windows process.” The bug doesn’t affect regular versions of Windows since it was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers.

Microsoft didn’t find the bug until this major outage, and it admits that was because of the traffic mix and hardware used in its testing labs. “The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on weaker hardware with fewer cores,” Lamanna wrote.

The bug became obvious because of two other incidents that increased load and reduced server capacity. Fifteen hours before this Outlook outage began, Microsoft Teams failed over from an alternative networking gateway to the main gateway that supports Exchange Online. “This increased the load on the networking gateway and changed its traffic patterns,” Lamanna wrote. Ordinarily this wouldn’t have been a problem, but at the same time around 14 percent of server capacity for this gateway was “out of service due to an unrelated issue from the previous day.”

The trifecta of a bug, Microsoft Teams failing over, and reduced server capacity all caused the network gateway to trigger the problem that had been lying dormant since July. It took Microsoft nearly 24 hours to get its Exchange servers stable again, and I’m told overlapping interventions from teams trying to fix issues complicated coordination at times.

Microsoft engineers had a running bridge call the entire time, and at one point it hit the maximum of 1,000 participants. That shows you how many teams were involved with trying to find the root cause of this huge outage, which started at the peak window of traffic in the US.

“Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again,” Lamanna wrote. “Our normally automated recovery processes were not designed to restore service from this broad unresponsive state. Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change.”

Microsoft is now working on a final post-incident review that will be published later this week. Even without that full review, it’s clear that the company will need to invest more into its testing processes to ensure that when something like this happens again, it can catch the problem a lot sooner. Outages happen daily, but this kind of cascading failure is a rare moment for Microsoft to learn from its mistakes and make sure the same sequence of events doesn’t bring down Outlook for millions of people again.

Notepad will return in December after next week’s Thanksgiving break.

I’m always keen to hear from readers, so please drop a comment here, or you can reach me at [email protected] if you want to discuss anything else. If you’ve heard about any of Microsoft’s secret projects, you can reach me via email at [email protected] or speak to me confidentially on the Signal messaging app, where I’m tomwarren.01. I’m also tomwarren on Telegram, if you’d prefer to chat there.

Thanks for subscribing to Notepad.

Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.

Read Entire Article