Creating Better AI Benchmarks: How Many Raters is Enough?

Google Research explores the trade-off between the number of articles and human raters per article to improve the reproducibility of AI benchmarks and capture the nuances of human disagreement.