Apparently AI-Assisted Conviction Draft Compromised a Defendant's Fair-Trial Rights in Cali, Colombia

A criminal defendant's lawyers found a draft conviction proposing 18 years in prison in the court file before closing arguments in Cali, Colombia. Colombia's Supreme Court inferred that generative AI helped assess the complainant's testimony. It found that the draft prejudged guilt, granted relief for violations of due process and the right to an impartial trial, and ordered the case reassigned.

Risk classification

• Primary risk domain: 5 Human-Computer Interaction
• Primary risk subdomain: 5.1 Overreliance and unsafe use

The judge inappropriately relied on a generative AI system to perform critical judicial functions, specifically evidence evaluation and drafting a sentence, representing unsafe use in a critical legal context.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The AI system hallucinated hearing dates and fabricated future events, showing a lack of robustness and capability.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by a human judge's decision to use a generative AI tool to draft a judicial sentence and analyze evidence, which occurred post-deployment.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The AI system was used in the administration of justice and judicial proceedings, which is a critical area with significant implications for fundamental rights and public interest.

AI system and alleged parties

• AI system: None named
• AI purpose: Judicial Decision Support; Writing Assistant
• Behaviour type: Assistant
• Alleged developer: Large language model developers, Chatbot developers, AI-assisted legal drafting system developers
• Alleged deployer: Juzgado 7.º Penal del Circuito de Cali, Courts, Chatbot users
• Alleged harmed parties: Parties to legal proceedings, Judicial integrity, Epistemic integrity, Defendants, Criminal defendants

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Inadequate Supervision: Failure of the judge to supervise the work of the court official.
• Lack of Training on AI Ethics: No systematic training for judicial staff on responsible AI use.
• Poor Risk Management: Failure to assess risks of using generative AI in active criminal cases.

Technology

• AI Hallucination of Facts: AI invented dates of hearings that had not yet occurred.
• Automation of Human Reasoning: AI was used to perform core judicial evaluation of evidence.
• Biased Output Generation: AI structured analysis specifically to support a conviction.

Data Inputs

• Premature Input of Evidence: Unfinished trial data was fed into the AI for final sentencing.
• Transcription of Testimonies: Raw trial transcripts were used by AI without proper context.
• Unvalidated Draft Upload: Draft containing AI outputs was uploaded directly to the digital file.

Human Factors

• Overreliance on AI Output: The judge accepted AI-generated legal evaluations as valid.
• Accidental File Upload: A court official mistakenly uploaded the draft to the system.
• Lack of Digital Literacy: Insufficient understanding of AI limitations and risks by court staff.

Process and Methods

• Premature Sentence Drafting: Drafting the sentence before final arguments were presented.
• Lack of Review Protocols: No verification process before uploading documents to the digital file.
• Inadequate AI Safeguards: Absence of technical controls to prevent AI from drafting final decisions.

Regulatory Environment

• Non-compliance with Guidelines: Violation of Sentence T-323 of 2024 and Agreement PCSJA.
• Vague Enforcement Mechanisms: Lack of active monitoring for AI usage rules in local courts.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide highly detailed legal and factual accounts of the incident, including direct quotes from the Colombian Supreme Court's ruling, the specific text generated by the AI, and the procedural history of the case.
• Ambiguities identified: The specific commercial AI model or platform used by the judge is not identified in the reports.
• Alternative interpretations: None. The facts of the AI's use and the subsequent Supreme Court ruling are clear and uncontested.

A Colombian judge used generative AI to draft a criminal conviction and analyze evidence before the trial concluded, violating the defendant's due process. The Supreme Court of Colombia intervened, reassigning the case and issuing strict guidelines. The incident highlights emerging challenges with AI overreliance in judicial decision-making but poses minimal threat to national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: Colombia
• Alleged perpetrator: Individual

Threat characteristics

• Imminence: Long-term. The specific legal case has been resolved through reassignment, representing a long-term strategic concern regarding AI integration in judicial systems.
• Autonomy: Human-controlled. The AI functioned as an assistant to draft documents, requiring human oversight and final decision-making, though the judge failed to properly supervise it.
• Novelty: Evolved capability. While lawyers using AI is increasingly common, a judge using AI to prejudge a case and fabricate hearing dates represents an evolved capability and misuse of generative AI in legal settings.

Impact by dimension

• Physical security: Negligible. No physical systems, infrastructure, or human safety were threatened or impacted by this incident.
• Information security: Negligible. No intelligence capabilities, classified information, or systematic information warfare campaigns were involved.
• Sovereignty: Minor. A judge compromised a core judicial decision-making process by using generative AI to draft a conviction. However, the impact was limited and resolved through standard legal appeal mechanisms.
• Economic security: Negligible. The incident did not involve financial systems, strategic technology theft, or economic manipulation.
• Societal stability: Minor. The defendant's fundamental rights to due process and an impartial trial were violated, but this was an isolated legal case rather than a large-scale threat to societal stability.