PolyRoute: A Twelve-Language and Code-Mixed Benchmark for Typed-Decision Routing, with Baselines and a Leaderboard

Research Hugging Face Papers Source language: Chinese

论文摘要(作者报告,未经本站复测):Typed-decision encoders answer a question about a text in one forward pass, but their behaviour outside English is rarely measured on a fixed yardstick.

Full text

Typed-decisionencodersansweraquestionaboutatextinoneforwardpass,buttheirbehaviouroutsideEnglishisrarelymeasuredonafixedyardstick.WereleasePolyRoute:322native-languagesupportmessagesin12languagesand9writingsystems,156code-mixedorromanisedmessages(Hinglish,Spanglish,Arabizi),967optionalnativetrainingrows,andaprotocolforathree-waydepartment-routingdecision.LabelscomefromQwen3.8-27Bwithablindsecondpass(91%ofcandidateskept);anindependentmodel,gemma4-31b,agreeson97.2%oftestitems,andthelargestcharacter5-gramsimilaritybetweenanytesttextandanytrainingtextis0.33.Aleaderboardof13systemsgives61.2%fortheEnglish-rootcheckpoint,77.3%fortheshippedmultilingualoneand96.6%forthebestfine-tune;TF-IDFwithlogisticregressionreaches91.9%.Onfourwithheldlanguagesthethreemultilingualsystemsscore77%,90%and93%.Oncode-mixedtexttheEnglishroot(85%)isnominallyaheadoftheshippedmultilingualcheckpoint(79%;p=0.134).Thesplitresolvesdifferencesonlyaboveabout4points.AlltextsandlabelsareLLM-generated.

View arXiv pageView PDF Add to collection

Sources

← Back to the daily brief