Key facts
- A new website called In the Weights has been launched.
- In the Weights measures how well AI models recall individuals.
- The tool assigns a 'strength score' to LLMs.
- The assessment prevents LLMs from using web searches.
- In the Weights was created by former OpenAI employees Thomas Dimson and Joey Flynn.
A new online tool named In the Weights has been launched to evaluate the ability of artificial intelligence models, specifically large language models (LLMs), to recall and describe individual people. This website was created by Thomas Dimson and Joey Flynn, who are former employees of OpenAI. The core function of In the Weights involves querying various LLMs to determine how well they can identify and provide details about specific individuals. The system assigns a 'strength score' to each LLM based on its performance in these recall tasks. A key aspect of the evaluation is that the LLMs are prevented from using external web searches to find information about the individuals. This constraint ensures that the assessment focuses purely on the internal knowledge and memory capabilities of the AI models themselves. The development of In the Weights aims to provide a quantifiable measure of how well AI systems retain and access information about specific people, offering insights into their internal data representations and potential biases.