Humanized but Still True? Auditing Facts and Detection in AI-Text Humanizers

Research Hugging Face Papers Source language: Chinese

论文摘要(作者报告,未经本站复测):AI "humanizers" rewrite machine-generated text so that detectors classify it as human-written. Public leaderboards score evasion and embedding similarity, but not whether checkable details survive.

Full text

AI"humanizers"rewritemachine-generatedtextsothatdetectorsclassifyitashuman-written.Publicleaderboardsscoreevasionandembeddingsimilarity,butnotwhethercheckabledetailssurvive.Wecheckwhethernumbers,dates,personandorganizationnames,andquotedspanssurviveeachrewrite,usingarule-basedcheckervalidatedbytwoAIannotators(precision0.84-1.00,recall0.93-0.95;nohumanannotation).IntheOctober2026cycleofapublichumanizerleaderboard,all11commercialtoolschangedordroppedatleastoneoftheseunitsin35-65%ofthedocumentsthatcontainedone.Mostchangesweretoquotedspans;countingnumbersandnamesalone,10ofthe11toolsstillalteredatleastonein20%ormoreofthedocuments.TheSeptember2026cyclewassimilar(19-50%).Anopen12Bhumanizerkeptatleast97%ofnumbersandnamesinfact-denseU.S.federalagencytext,butreworded67-78%ofthequotedspans,againstitsownprompt.Unitretentionanddetectorevasionwerenotclearlyrelatedacrosstools(Spearmanrho=-0.46,95%CI-0.65to0.08),andtheopenmodel'sfourreleasedfiles(twoofthemquantization-awarelow-bitfiles)didnotshowthepre-registeredmonotonetrade-off.Onfederaltextswithmatchedhuman,AI-styledandhumanizedversions,ageneral-purposedecisionmodel,Jev,askedasinglezero-shotquestion,separatedhumanoriginalsfromhumanizedrewriteswithAUROC0.864(0.827-0.902),0.18-0.22higherthanBinoculars,Fast-DetectGPTandtheadversariallytrainedRADAR,whilecalling24%ofthehumanoriginalsAIata0.5threshold.OntheAI-styledrewritesby2026models,bothperplexity-baseddetectorsscoredbelowchance(AUROC0.37and0.39),alsowithlinebreaksremoved.Hypotheses,datarulesandanalyseswerefrozenbeforethemainruns;twoearlierdatacontactsaredisclosed.Wereleasethecheckerandtheper-itemresults.

View arXiv pageView PDF GitHub 1 Add to collection

Sources

← Back to the daily brief