Microsoft-Powered New York City Chatbot Advises Illegal Practices

New York City's chatbot, launched under Mayor Eric Adams's plan to assist businesses, has been reportedly providing dangerously inaccurate legal advice. The Microsoft-powered bot allegedly informed users that landlords can refuse Section 8 vouchers and that businesses can operate cash-free, among other falsehoods. The city acknowledges the chatbot is a pilot program and commits to improvements while the errors are addressed.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI chatbot generated and spread incorrect and illegal legal advice to users, leading to inaccurate beliefs about city regulations.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The chatbot exhibited a lack of capability and robustness by hallucinating false legal information and failing to perform reliably.
• 1.1 Unfair discrimination and misrepresentation: The chatbot's false advice encouraged illegal discrimination against housing voucher holders and unbanked individuals.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incorrect and illegal advice was generated by the AI chatbot after its deployment, representing an unintended failure of the system's capability.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is an AI chatbot, which falls under Limited Risk due to transparency obligations to ensure users know they are interacting with an AI.

AI system and alleged parties

• AI system: NYC AI chatbot (Microsoft)
• AI purpose: Chatbot; Question Answering
• Behaviour type: Assistant
• Alleged developer: New York City Office of Technology and Innovation, Microsoft
• Alleged deployer: New York City Government, Eric Adams administration
• Alleged harmed parties: New York City small business owners, New York City landlords and tenants, New York City employers and employees, Eric Adams administration

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Minor
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes potential differential treatment of individuals or groups caused indirectly by the chatbot's advice.

Indirectly caused: The chatbot's advice could lead to indirect differential treatment of low-income tenants and unbanked customers if acted upon by landlords and business owners.

Inferred additional harm: Potential increase in discrimination against housing voucher holders and cash-paying customers in NYC due to incorrect legal guidance.

Civil rights

Reported: The report explicitly describes potential violations of human or civil rights caused indirectly by the chatbot's advice.

Indirectly caused: The chatbot advised landlords that they do not need to accept housing assistance, which is illegal discrimination under NYC housing policy.

Inferred additional harm: Potential violations of housing rights for tenants relying on rental assistance if landlords acted on the chatbot's advice.

Epistemic

Reported: The report explicitly describes epistemic harm caused directly by the chatbot's generation of false and illegal business advice.

Directly caused: The chatbot directly provided false legal information to users, such as stating that employers can take a cut of workers' tips and that landlords do not have to accept Section 8 vouchers.

Inferred additional harm: Widespread dissemination of incorrect legal standards to thousands of NYC business owners and residents, eroding trust in official city information portals.

People affected

• Occurrences reported: 1
• People reportedly exposed: 2000

Potential causes

Management

• Premature Pilot Deployment: City launched a live pilot, relying on public testing to find errors.
• Prioritizing Launch over Safety: Focus on MyCity portal centerpiece over ensuring legal compliance.

Technology

• LLM Hallucinations: Chatbot produces inconsistent and inaccurate responses to users.
• Inconsistent Bot Outputs: The bot returns different answers to the same query over time.

Data Inputs

• Incomplete Training Data: Bot lacked synthesized knowledge of specific local laws and exceptions.
• Lack of Source Citations: Bot often failed to provide verification links for its answers.

Human Factors

• Overreliance on Bot Authority: Users trust the bot because it is hosted on an official city website.

Process and Methods

• Inadequate Pre-launch Testing: The chatbot was deployed publicly with major legal accuracy gaps.
• Deficient Safeguard Mechanisms: No real-time filters to block illegal or harmful business advice.

Regulatory Environment

• Absence of Municipal AI Standards: No strict regulations governing the deployment of public-facing city bots.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, consistent, and detailed examples of the chatbot's specific incorrect outputs, verified by multiple journalists and experts. The role of the AI system and its developer are explicitly stated.
• Ambiguities identified: It is unclear whether any users actually acted on the incorrect advice and suffered legal or financial consequences.

A municipal AI chatbot deployed by New York City provided inaccurate and illegal advice regarding local business, labor, and housing regulations. While demonstrating the risks of deploying unverified LLMs in government services, the incident has negligible national security implications and presents no threat to physical security, intelligence, or sovereign stability.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. This represents a localized municipal software issue that does not pose an immediate or near-term national security crisis.
• Autonomy: Human-supervised. The chatbot generates responses autonomously to users, but operates within a managed municipal IT framework with ongoing human oversight and administrative control.
• Novelty: Established threat. LLM hallucinations and inaccurate outputs from public-facing chatbots are well-documented and common occurrences.

Impact by dimension

• Physical security: Negligible. The incident involves a municipal information chatbot with no impact on physical systems, critical infrastructure, or human safety.
• Information security: Negligible. While the chatbot provided false information, this was a localized system error rather than a hostile foreign information operation or intelligence breach.
• Sovereignty: Minor. The chatbot gave incorrect advice on local regulations, representing a minor failure in local government service delivery, but did not disrupt core sovereign functions or electoral systems.
• Economic security: Negligible. No significant financial system disruption, strategic technology theft, or loss of competitive advantage occurred.
• Societal stability: Minor. The chatbot gave advice that could encourage illegal housing discrimination and labor violations, but there is no evidence of widespread societal instability, mass surveillance, or systemic human rights abuses.