Hugging Face Papers ↗ · 2026-11-18
Squoosh Bench: Evaluating UX-1 on 318 Historical Website A/B Tests
媒体报道 Hugging Face Papers 原文语言:中文
研究团队通过318个历史网站A/B测试评估了模拟购物者预测的效果,其中基于人类浏览数据微调的模型Squoosh UX-1正确分类了267个结果,准确率达84.0%。相比之下,最强的通用配置Qwen3 Next 80B正确分类了196个结果,准确率为61.6%。
全文
Experimentation teams must decide which proposed website changes to test with live traffic. We evaluated whether forecasts from simulated shoppers could inform that choice. We screened 412 historical website A/B tests and evaluated four systems on the 318 that ran longer than 21 days. Each system predicted an original win, a variant win, or an inconclusive result. Page roles were masked and recorded outcomes were withheld during inference. Squoosh UX-1, a model fine-tuned on human browsing data, simulated 1,000 shoppers per arm and correctly classified 267 of 318 outcomes (84.0%, 95% Wilson interval 79.5-87.6%, treating experiments as independent). Qwen3 Next 80B, the strongest general-purpose configuration on outcome accuracy, correctly classified 196 of 318 (61.6%). UX-1 favored the variant in 126 experiments. In the live tests, 118 of those variants won (93.7%), four lost, and four had inconclusive results. The remaining 192 experiments included 19 additional variant wins.
View arXiv page View PDF Add to collection
正文以来源语言显示。