Why a Single Command Took Down Azure: The Myth of "Human Error"

Overview of Technical Program Management and Incident Analysis

• The role of a technical program manager involves modern incident analysis, which is a method of deeply analyzing certain incidents and creating detailed reports, such as 20-page Word documents, to deliver to senior leadership 10s.
• The modern incident analysis method uses specific tools and techniques to differentiate it from other post-incident frameworks like ITIL problem management, and it involves going in-depth on certain incidents to gather more information 1m42s.
• Internal terms such as "DR" and "OCE" are used to refer to an on-call engineer, who is the person responsible for receiving pages when issues arise, and "MOP" and "SOOP" are used interchangeably to refer to a standard or method of procedure 2m6s.
• The terms "incident" and "outage" have distinct meanings, with an incident being a notification that is triggered when a threshold is breached, and an outage being a declared event that has confirmed or imminent customer impact 3m30s.

Severity Classification and Public Perception of Outages

• A severity scale is used to categorize outages, with severity one being the largest impact, and a special category called "SEZ zero" which is reserved for extreme cases and involves additional work streams for internal and external communications 5m30s.
• The severity scale is not visible to external parties, but a good indicator of a SEZ zero outage is if it is being reported by major news outlets or social media, and these outages are often referred to by name, such as the "global WAN outage" 7m10s.
• The global WAN outage, which occurred on January 25th, 2023, was a significant event that resulted in the company being "hard down" for about an hour and 40 minutes globally, making it impossible for customers to reach the company's services 8m40s.

The Azure Outage and Its Impact

• The Azure outage had a significant impact on Microsoft customers, affecting not only Azure services but also other clouds within Microsoft, such as Office, M365, and Xbox, as these services also use the Azure weighing 10s.
• During the outage, a narrative developed that a change had gone wrong, which, although not entirely incorrect, was not the full story, and this narrative started to form internally as well after the dust had settled 1m42s.
• The incident management team, which deals with outages daily, forgets that not everyone in the company has the same level of experience with outages, and as a result, senior leaders and others who are not familiar with outages may develop and spread simplified narratives to explain the outage to customers, the board of directors, or external media 2m6s.

Simplified Narratives and Their Consequences

• These narratives can become the accepted story if left unchecked, and they often involve a simple explanation, such as "operator error," which is easy to tell and understand, but may not accurately reflect the complexity of the issue 3m42s.
• The tendency to simplify complex problems is a universal human need, and it is not unique to any particular organization or individual, but rather a natural part of the sensemaking process during an incident, where people try to synthesize data and come up with theories to understand what is happening 4m30s.
• The attractiveness of a simple narrative, such as "operator error," lies in its ease of explanation and understanding, but it can be misleading and may lead to incorrect conclusions, such as punishing the engineer who made the change or increasing the frequency of mandatory training 6m10s.
• The simplification of complex stories can be dangerous as it may lead to punishment of individuals without understanding the underlying factors that led to a decision or action, and may not address the actual issues, potentially causing harm to the organization's culture 10s.

The Incident Diagramming Process

• A diagram is used as an interviewing technique to break down the root cause of an incident, and it is not meant to be a specification of the incident, but rather a tool for whiteboarding and brainstorming to develop a narrative 2m6s.
• The incident in question occurred on January 25th, and it involved a single command that took down Azure, resulting in three distinct periods of impact, including a hard down period of about an hour and 40 minutes, and two long-tail recoveries in different regions 4m42s.

The Router Command and Network Expansion

• The command was run by an engineer on a router, which was part of a new role for the WAN and was provisioned with an external IP address as part of a network expansion to increase the size and speed of the global WAN 6m15s.
• The router was one of 12 router pairs that were part of a buildout process, during which it was determined that the architecture needed to be changed, and this change may have contributed to the incident 8m30s.
• To understand the outage, it is necessary to go back two months and examine the events and conditions that led up to the incident, including the provisioning of the router and the changes made to the architecture 9m10s.
• The routers in question were part of a software-defined wide area network, which required them to have an internal IP, and a change was needed to reIP the routers that had already been built out, a process that is not typically considered low-risk 10s.

SOP Development and Change Governance

• A Standard Operating Procedure (SOP) was developed for this operation because it was a new role and the existing SOP did not exist, and this SOP was subject to a change review process that included emulation, but it was created and modified outside of the usual governance process 1m30s.
• The routers in question were not production routers and were not serving customer traffic, but were connected to the IGP backplane, and there were inconsistent practices regarding the SOP change processes for non-production change work 2m40s.
• The SOP was used a couple of times on previous routers without peer review, cab review, or testing within the emulated environment, and during one of these operations, engineers encountered an issue with stale link state packets that required deleting the database within the router 4m10s.
• The router manufacturer provided guidance to run a specific command to resolve the issue, which was added to the SOP outside of the governance process, and this command was later run by an engineer on January 25th, following the SOP attached to the change ticket 6m20s.

Engineer Execution and Command Risks

• The engineer executing the change was unaware that the SOP had been changed and included the new command, and the execution of this command had significant consequences, highlighting the importance of treating SOPs and run books as guides that require human interpretation and decision-making 8m30s.
• Engineers are expected to scrutinize commands, especially high-risk ones, and question or escalate them if they seem unsafe, but in this case, an engineer ran a command without fully understanding its implications 10s.
• The Microsoft WAN uses three different router manufacturers, resulting in three different operating systems and versions, which was done for supply chain de-risking, and the engineer was familiar with the command on two of the operating systems but not the third, where it had a global scope 1m30s.
• The engineer's mental model was also affected by the expectation that unsafe commands would be blocked at the AAA level, but in this case, a full command audit had not been completed due to the onboarding process timing 2m6s.

Command Execution and Network Failure

• On January 25th, 2023, the engineer ran the command confidently, and 33 minutes later, ran it again on the other router, which set off a cascade of events that impacted the network 3m40s.
• The combination of the two commands caused the network to recompute its connectivity, resulting in a significant outage, with about 15 million routes needing to be recomputed, taking around an hour and 45 minutes to complete 5m30s.
• During the outage, three devices failed, and three were degraded due to defects, and the system for auto detection, auto rerouting, and auto recovery was paused because it was contributing to the problem 7m10s.
• The network was eventually healed, but not before causing significant disruptions, and the incident highlights the importance of careful command execution and auditing 8m40s.

Root Cause and Complexity of the Incident

• The root cause of a specific incident was an engineer executing a low-risk, locally scoped plan change to a non-production router pair, following an official standard operating procedure that had been previously updated by senior engineers, with a mental model informed by extensive experience with other router manufacturers, 10s.
• The engineer believed that any potential unsafe, globally scoped command would be blocked by the AAA system, and this incident highlights the complexity of identifying a single root cause or assigning blame, 42s.

Systemic and Tactical Repairs

• To address the issue, repairs can be identified by analyzing diagrams of the system and looking for nodes with many arrows coming from or going to them, which can indicate systemic or key issues, and the contributing factors closest to the impact are often representative of tactical repair items, 2m6s.
• Some of the repairs that have been done include making the network more robust against certain types of failures, such as IGP recomputing and BGP recomputing, by adjusting various buttons and levers, 4m30s.
• Further away from the impact, systemic or thematic issues can be identified, such as SOP change governance, and these are good targets for adding resilient behaviors to systems and organizations, 6m15s.

Mental Models and Training

• Mental models are also an important factor, and changing them requires training that is right-sized and meets the engineers where they are, rather than just making existing training mandatory, 8m40s.
• The incident is now studied as part of the onboarding training for the WAN team and the core networking team, to instill a respect for the complexity of the system and its potential for failure, 10m50s.

Post-Incident Communication and Culture

• Many of the points discussed have already been covered in post-incident reports and Azure incident retrospectives, which are live Q&A style panels with engineering leaders, 12m30s.
• The Azure team engages with customers through live events and Q&A sessions, and non-Azure customers can access recordings of these sessions on the azure.status.microsoft website 10s.
• When a major outage occurs, the team receives questions from customers about whether the responsible engineer was fired, which puts executives in a difficult position to respond without offending the customer 2m6s.
• The narrative of "human error" can develop internally and externally, even without people actively promoting it, and it's essential to actively curate a blameless environment to prevent this behavior 4m42s.

Challenges of Promoting a Blameless Culture

• The concept of blamelessness is not yet widely adopted, even within the tech industry, and it's crucial to educate people about its importance, especially in a large and complex organization like Azure, which consists of around 2,000 discrete services with different engineering teams 6m15s.
• Changing the narrative of blamelessness to the general public is a challenging task, and it requires working within the existing framework to implement gradual, grassroots changes, rather than trying to change the culture of an organization overnight 10m30s.

Approach to Postmortem Analysis

• The approach to postmortem analysis in Azure involves a combination of metrics, dashboards, and leadership updates, as well as efforts to promote a blameless culture, and it's essential to find ways to meld these approaches together effectively 12m40s.
• A typical analysis of an incident may take a month to complete and should not be rushed or time-boxed based on external commitments, allowing for a thorough examination of the incident and the creation of learning opportunities within the groups involved 10s.
• The goal of an incident analysis is to learn and build resilient behaviors within the groups involved, rather than simply meeting metrics or commitments, and this approach is similar to that used in safety-critical industries such as aviation, where the focus is on learning from incidents rather than assigning blame 2m6s.
• The use of practices such as the "five W's" is not recommended, as it can lead to a superficial analysis of the incident, and instead, a deeper analysis that considers the systemic and thematic issues involved is necessary 4m42s.

Resources and Deep Analysis of Incidents

• There are several resources available for conducting a deep analysis of incidents, including the Resilience and Software Foundation community, the Adaptive Capacity Labs blog, and books on safety science, such as the Stella report and the Howie guide 6m15s.
• The human toll of an incident, such as the trauma experienced by an engineer who made a mistake, is an important consideration, and it is common for engineers to be hesitant to take actions after being involved in an incident, highlighting the need for a supportive and blameless culture 10m42s.

Balancing System Complexity and Human Error

• Finding a balance between preventing incorrect actions and avoiding excessive complexity in systems is a challenge, as there may be many ways to incorrectly execute a command, and simply adding more code to prevent errors may not be an effective solution 12m10s.
• When conducting an incident analysis, it is important to consider the culture of the team involved and to gather background information before interviewing the engineer who performed the action, in order to understand the context and factors that contributed to the incident 14m20s.

Blameless Culture and System Design

• Engineers are rarely fearful of being fired due to their actions, and it is reinforced that the focus is not on punishment, but rather on understanding the situation, and this approach leads to the idea that the system is at fault when an engineer makes a mistake 10s.
• The system is considered to have failed the engineer if it allows them to run a command that causes significant damage, such as taking down an entire system, because it did not provide adequate protection or guidance 42s.
• Systems should be designed to help engineers make informed decisions and protect against potential mistakes by providing insight into the system's state and the potential effects of their actions 2m6s.