HUGGING FACE POLICY CHIEF QUESTIONS EMBEDDED AI MODEL TESTING

Irene Solaiman, Hugging Face’s chief policy officer, said placing outside evaluators inside AI labs would not, by itself, resolve problems with model benchmarking. Speaking at TechCrunch Disrupt, she described current evaluation methods as complicated and still maturing. Solaiman said the field had yet to agree on what a good evaluation should measure. Her remarks addressed proposals to give independent testers greater access to labs as confidence in public benchmarks and lab-reported scores comes under scrutiny.

Solaiman said some widely cited benchmarks lose their usefulness quickly as models improve, while others fail to reflect how people use AI products. She also identified contamination, in which test questions enter training data and can inflate scores. Labs can make comparisons harder, she said, by publicising favourable results without disclosing their methods or weaker runs. Access to OpenAI, Anthropic, Google DeepMind or Meta could improve an evaluator’s view of a model, according to Solaiman, but would not automatically make a test rigorous or reproducible.

Solaiman said third-party testing also raises questions about who funds evaluators, what access they receive and whether they can publish without a lab’s approval. Evaluators working inside a lab may depend on it for computing resources, model access and information, she said. Non-disclosure agreements could further constrain their work. She argued that independent testing needs safeguards against those pressures if it is to provide credible oversight. Solaiman called for more consistent evaluation methods before the industry relies on benchmarks to compare AI products.