Crashes with Maneuvering Characteristics Augmentation System (MCAS)

A Boeing 737 crashed into the sea, killing 189 people, after faulty sensor data caused an automated manuevering system to repeatedly push the plane's nose downward.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The MCAS system lacked robustness by relying on a single sensor input without cross-checking, leading to catastrophic failure when that sensor malfunctioned.

Additional risk subdomains

• 7.4 Lack of transparency or interpretability: Boeing failed to disclose the existence of MCAS in flight manuals or training, leaving pilots unable to interpret or diagnose the system's automated actions.
• 5.2 Loss of human agency and autonomy: The automated system physically overrode the pilots' manual control column inputs, removing their ultimate decision authority to keep the plane level.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the MCAS software autonomously commanding the aircraft's nose down post-deployment, which was an unintentional failure of the system's safety goal.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is integrated into critical transport infrastructure. The reports describe an 'automated safety system' (MCAS) that 'repeatedly pushed the plane's nose down' which directly impacts the safety of public transport.

AI system and alleged parties

• AI system: Maneuvering Characteristics Augmentation System (Boeing)
• AI purpose: Automatic Fault Handling; Navigation Assistant
• Behaviour type: Autonomous
• Alleged developer: Boeing
• Alleged deployer: Boeing
• Alleged harmed parties: Airplane Passengers, Airplane Crew

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Severe. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Severe, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Severe, indirect Negligible
• Financial: direct Severe, indirect Severe
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Physical

Reported: The report explicitly describes the fatal crash of Lion Air Flight 610.

Directly caused: All 189 passengers and crew members on board were killed when the aircraft crashed into the Java Sea.

Property

Reported: The report explicitly describes the destruction of the aircraft.

Directly caused: The complete destruction of the brand-new Boeing 737 MAX 8 aircraft.

Financial

Reported: The reports describe the loss of the aircraft and a significant drop in Boeing's stock price.

Directly caused: The direct financial loss of the aircraft, valued at approximately 100 million dollars.

Indirectly caused: Boeing's stock price fell by 14 percent over three weeks following the crash, representing billions in lost market value.

Inferred additional harm: Massive financial losses from subsequent litigation, victim compensation, and the eventual global grounding of the entire 737 MAX fleet.

Psychological

Reported: The reports describe the chaotic and terrifying struggle in the cockpit before the crash.

Directly caused: The flight crew and passengers experienced extreme terror and acute distress during the 11-minute struggle against the automated system.

Indirectly caused: Widespread trauma, grief, and psychological suffering for the families and relatives of the 189 victims.

People affected

• Occurrences reported: 1
• People reportedly harmed: 189
• People reportedly exposed: 189

Potential causes

Management

• Prioritizing Market Competition: Retrofitting old aircraft to compete with Airbus created physical instability.
• Withholding Safety Information: Management failed to communicate the new automated system to airlines.
• Deficient Safety Culture: The airline failed to ground the plane after persistent sensor issues.

Technology

• Single Sensor Reliance for MCAS: MCAS triggered by a single faulty AOA sensor without cross-checking backups.
• Uncommanded Repeated Nose-Down: MCAS software repeatedly commanded stabilizer trim down, causing a deep dive.
• Control Column Override Disabled: Unlike older models, pulling back on the column did not stop MCAS runaway.

Data Inputs

• Erroneous AOA Sensor Readings: AOA sensor registered extremely high, incorrect values, triggering MCAS.
• AOA Sensor Disagreement: Left and right AOA sensors differed by 20 to 59 degrees during flight.

Human Factors

• Cockpit Confusion and Overwhelm: Multiple simultaneous alarms and alerts overwhelmed the flight crew.
• Lack of MCAS System Awareness: Pilots were not informed of the existence of the MCAS system in training.
• Erosion of Hand-Flying Skills: Over-reliance on autopilot systems reduced basic manual piloting proficiency.

Process and Methods

• Inadequate Maintenance Testing: Failed to perform a maintenance test flight to validate the sensor repair.
• Omission of MCAS from Manuals: Boeing omitted MCAS details from the flight crew operations manual.
• Incomplete Troubleshooting: Maintenance cleared the plane without checking the critical AOA sensor.

Regulatory Environment

• Delayed Safety Directives: Regulators only mandated manual updates and procedures after the crash.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide highly detailed, consistent, and official preliminary findings from aviation safety boards (NTSC, FAA) regarding the mechanical and software failures of the MCAS system and the AOA sensors.
• Ambiguities identified: The exact reason why the flight crew did not execute the stabilizer cutoff procedure remains partly ambiguous due to the missing cockpit voice recorder at the time of these reports.
• Alternative interpretations: Some initial statements from Boeing attempted to frame the incident primarily as pilot error or maintenance failure rather than a software design flaw.

The crash of Lion Air Flight 610 was a catastrophic critical infrastructure safety failure caused by the autonomous MCAS software reacting to erroneous sensor data. While resulting in tragic loss of life and massive economic consequences for the aviation industry, the incident was an unintentional technical accident with negligible direct implications for national security.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: United States, Indonesia
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing strategic concern regarding software automation safety and regulatory oversight rather than an active, fast-moving national security crisis.
• Autonomy: Full autonomy. The MCAS software acted autonomously to command the aircraft's nose down based on sensor inputs, and actively resisted the flight crew's manual override attempts.
• Novelty: Evolved capability. While software failures in aviation have occurred before, this incident represents a highly complex failure of an autonomous system overriding human operator control in a commercial transport setting.

Impact by dimension

• Physical security: Minor. The catastrophic failure of the MCAS flight control system resulted in 189 fatalities and the destruction of a commercial airliner. While representing a severe safety failure within the critical transportation infrastructure sector, it was an unintentional accident managed through standard aviation safety and regulatory channels rather than a hostile national security incident.
• Information security: Negligible. No information warfare, intelligence compromise, or systematic disinformation campaigns were associated with this mechanical and software safety failure.
• Sovereignty: Negligible. The incident did not involve threats to state authority, territorial control, or core government operations.
• Economic security: Minor. The crash and subsequent global grounding of the Boeing 737 MAX fleet caused significant economic damage to Boeing and the aviation sector, but did not threaten overall national economic stability or result in strategic technology theft.
• Societal stability: Negligible. The incident did not involve mass surveillance, algorithmic oppression, or civil unrest, and had no direct implications for societal stability or human rights.