OpenAI faces a proposed class-action lawsuit alleging the company allowed outside contractors to read user conversations without adequate disclosure. The suit targets "Project Lily," an internal program where contractors review and score chatbot responses, claiming users were not clearly informed about human access to their chats. Plaintiffs seek damages and changes to OpenAI's data handling practices.
The lawsuit highlights significant privacy concerns for users of AI chatbots like ChatGPT, potentially impacting user trust and the future development and regulation of AI technologies. If successful, it could force companies to adopt more transparent data handling practices and require explicit user consent for human review of sensitive conversations.
OpenAI is facing a proposed class-action lawsuit filed by two ChatGPT users in the U.S. District Court for the Northern District of California. The lawsuit, filed this month, alleges that the artificial intelligence company allowed outside contractors to read real user conversations without providing adequate disclosure to users. OpenAI was served with the complaint on September 2.
The core of the legal challenge is directed at an internal OpenAI initiative known as "Project Lily." According to the complaint, contractors hired via third-party staffing firms, in roles such as "AI data reviewer" and "chatbot evaluator," are tasked with reading actual ChatGPT prompts and entire conversations. These contractors then summarize the user's intent and evaluate four different responses from the AI model on a scale of one to seven. This process is a fundamental aspect of how AI models like ChatGPT are trained and improved through reinforcement learning from human feedback (RLHF).
The lawsuit contends that users were not clearly informed that human reviewers, rather than solely automated systems, might be accessing their chats. While OpenAI employs an automated system to filter conversations before human review, the complaint asserts that this filter is not always effective, allowing personal details to reach contractors. Reports indicate that reviewers use a dashboard that includes a "user memories summary," which can reveal personal information such as general location, profession, or private life details, even though usernames are removed.
OpenAI has stated that these reviews are intended to address specific AI behaviors, such as the chatbot exhibiting overly human-like qualities or displaying "sycophancy"—agreeing with the user rather than providing accurate information. With ChatGPT boasting over 900 million weekly users, many of whom use the service for personal matters like health queries, financial questions, or as a diary, the privacy implications are significant.
The plaintiffs argue that OpenAI's privacy policy, while mentioning access by certain outside parties, does not specifically identify data annotation or human evaluation vendors. The complaint includes eight legal claims, including violations of California's Unfair Competition Law and Consumer Privacy Act, as well as common-law claims like intrusion upon seclusion. They are seeking damages, restitution, and punitive damages.
In terms of injunctive relief, the plaintiffs are requesting that OpenAI implement opt-in consent for any conversation to be reviewed by an outside party. They also want the "Improve the model for everyone" setting to be turned off by default, a clear warning to be displayed within the chat window, and potentially the deletion of work product derived from reviewed conversations, along with retraining of models that utilized such data. OpenAI is required to respond to the lawsuit by October 13.
Pick the topics you care about. Get only what matters, on your cadence.