Regulates AI companions by requiring protocols to address suicidal ideation and mandates user notifications about non-human interactions. Empowers the Attorney General to enforce compliance, with penalties benefiting a suicide prevention fund.
Good coverage — Comprehensive governance measures or mitigations with clear procedures relating to the risk subdomain are explicitly described.
All other risk subdomains had no coverage in this document.
Risk Coverage Summary
The document has good coverage of approximately 3-4 subdomains, with strong focus on exposure to toxic content (1.2), overreliance and unsafe use (5.1), and lack of robustness (7.3). The legislation specifically addresses risks from AI companions that could expose users to harmful interactions related to suicidal ideation and self-harm, and addresses the risk of users developing inappropriate relationships with AI systems.
Good coverage — Comprehensive governance measures or mitigations with clear procedures relating to the sector are explicitly described.
Minimal coverage — Brief mention but no specific mitigations, recommendations or governance measures relating to the sector are described.
All other sectors had no coverage in this document.
Sectors Summary
The document primarily governs the Information sector, specifically companies that develop and operate AI companion systems. These systems fall under data processing and information services. The regulation also has implications for Health Care and Social Assistance through its suicide prevention focus and crisis service referrals.
Hard Law
Legislative Status Summary
This is a binding state statute with mandatory requirements, explicit prohibitions, enforcement mechanisms through the Attorney General, and civil penalties of up to $15,000 per day for violations.
Proposer
Target
Enforcer
Monitor
Lifecycle stages covered by the documents, as defined in the OECD Framework for the Classification of AI Systems
AI Lifecycle Summary
The document primarily focuses on the Deploy and Operate and Monitor stages of the AI lifecycle. It mandates specific features that must be present in AI companion systems before deployment (detection protocols and user notifications) and requires ongoing operational monitoring for suicidal ideation and self-harm expressions during use.
Terms relating to technical scope that are mentioned or implicitly referenced in the document
Technical Scope Summary
The document explicitly defines and covers AI, generative AI, AI models, and AI companions. It specifically addresses emotional recognition algorithms as a component of AI companions. The document focuses on a specific application (AI companions) rather than general purpose or frontier AI, and does not mention compute thresholds or open-weight models.
Document data from AGORA, CSET Emerging Technology Observatory.