Overfit Kaggle Models Discouraged Data Science Competitors

In the “The Nature Conservancy Fisheries Monitoring” competition on the data science competition website Kaggle, a number of competitors overfit their image classifier models to a poorly representative validation data set.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The AI model suffered from a lack of robustness and generalization capability, performing exceptionally well on training data (99%) but failing to maintain this performance on the unseen test set.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Pre-deployment

The VGG model unintentionally failed to generalize to the test set, which represents a pre-deployment development phase of the AI system.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The system is an image classification model used in a data science competition, posing negligible risk to users and society.

AI system and alleged parties

• AI system: VGG
• AI purpose: Image Classification; Target Identification
• Behaviour type: Tool
• Alleged developer: Individual Kaggle Competitors
• Alleged deployer: Individual Kaggle Competitors
• Alleged harmed parties: Individual Kaggle Competitors

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly exposed: 2300

Potential causes

Management

• Poor Competition Timeline: Releasing test set one week before deadline restricted proper evaluation.

Technology

• Overfitting of Pre-trained VGG: Pre-trained VGG reached 99% training accuracy but failed on test data.
• Model Shortcut Learning: Model learned boat backgrounds instead of actual fish features.

Data Inputs

• Dirty and Mislabeled Data: Real-world images were blurry, mislabeled, and dirty.
• Data Distribution Discrepancy: Training data features did not match the stage two test set distribution.

Human Factors

• Competitor Fatigue: Complex two-stage setup discouraged competitors from completing stage two.

Process and Methods

• Flawed Two-Stage Design: Releasing 12k test images late led to massive participant dropout.
• Inadequate Validation Split: Lack of local validation matching test set led to overestimating accuracy.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly outlines the technical discrepancy between training and test performance in a Kaggle competition, which is a classic machine learning robustness issue. There is no ambiguity regarding the lack of real-world harm, as this was a development-stage competition.
• Ambiguities identified: The exact reason for the VGG model's high training accuracy and low test accuracy is not fully explained in the text, though it is strongly implied to be overfitting or shortcut learning.

A 2017 Kaggle competition highlighted standard machine learning generalization failures (overfitting) when transitioning from training data to real-world test data. This incident represents a routine technical challenge in AI development and has zero national security implications.

• Overall national security impact: Negligible
• Scope: Unknown
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The event occurred in 2017 during a public competition and poses no active or immediate threat to national security.
• Autonomy: Human-controlled. The VGG network was used as a standard static image classification tool without autonomous action or decision-making capabilities.
• Novelty: Established threat. Model overfitting and failure to generalize to out-of-distribution datasets are well-established, fundamental challenges in machine learning.

Impact by dimension

• Physical security: Negligible. The incident involved a machine learning competition for fisheries monitoring and had no impact on physical systems, critical infrastructure, or human safety.
• Information security: Negligible. No classified systems, intelligence gathering, or information operations were targeted or affected by this academic data science competition.
• Sovereignty: Negligible. This was a public data science competition to classify fish species and did not involve or impact sovereign government operations or state authority.
• Economic security: Negligible. The event was a standard algorithm benchmarking issue on a public platform, presenting no threat to strategic industries or national economic security.
• Societal stability: Negligible. The failure of the image classification model to generalize had no societal impact, civil rights implications, or effects on public stability.