Do Multimodal Models Decide Consistently on Equivalent Optical Spectra?

Research Hugging Face Papers Source language: Chinese

论文摘要(作者报告,未经本站复测):A spectrum can be given to a model as a table in nanometers or as a plot in photon energy, and the decision should not depend on which one it sees.

Full text

Aspectrumcanbegiventoamodelasatableinnanometersorasaplotinphotonenergy,andthedecisionshouldnotdependonwhichoneitsees.Whenthedataneededforanansweraremissing,themodelshouldsayso.Wetestrepresentationinvarianceandabstentiononmeasuredopticalspectraofsiliconmetasurfaces(MOCLIP)andgoldnanoparticles,withlabelscomputedfromthedata.Eachspectrumisshownasfourtables(nm,μm,scientificnotation,eV)andtwoplots(nm,eV),andpartialspectraremovedatainsideoroutsidethequeriedrange.Weevaluatethedecisionmodelsd3-nanoandJev1.13,threeGPT-6LunaconfigurationsandtheQwen3.5-2Bbasemodel.Changingonlytherepresentationmovesaccuracybyupto41percentagepointswithinonemodel,mostlyonphoton-energytablesandplots.Withthesameevidence,askinginphotonenergyinsteadofwavelengthhelpssomemodel-viewpairsby16-50pointsandhurtsothersbyupto25.Whentherelevantextremumisremoved,unadaptedmodelsrarelyanswer"undetermined".LoRAfine-tuningofd3-nanoimprovescomparisonsonthetrainedtableformats,butmuchofitsgainonunanswerablequestionscomesfromashortcut:itabstainswhenevertheinputistruncated.Trainingwithanswerabletruncationsavoidstheshortcutinallsixseeds,witha90-100pointgapbetweenabstentionafterinsideandoutsidecutsonheld-outstructures;withthesameamountofextraoriginal-taskdata,fiveofsixseedskeeptheshortcut.Ongoldspectrathetwotrainingsetsperformsimilarly.Multimodallanguagemodelsusedforscientificdecisionsshouldbetestedonequivalentrepresentationsandonmissingevidence,notonlyonaverageaccuracy.Wereleasethetasks,per-responseresultsandanalysiscode.

View arXiv pageView PDF GitHub 1 Add to collection

Sources

← Back to the daily brief