Hugging Face Papers ↗ · 2026-11-19
AltRA-Test: An Alternative Annotator Test for LLM Judges Without Shared Annotators
Media reports Hugging Face Papers Source language: Chinese
AltRA-Test is a statistical test that decides when LLMs can replace human annotators by comparing inter‑annotator agreement in resampled mixed human‑LLM panels. Experiments show it often needs about 100 items and makes better replacement decisions than baselines.
Full text
A statistical test, AltRA-Test, determines when large language models can replace human annotators by comparing inter-annotator agreement in resampled mixed human-LLM panels, and evaluations show it often requires about 100 items and makes better replacement decisions than baselines.