Hugging Face 模型 · tencent ↗ · 2026-08-14
UI-Mate-27B
模型发布 Hugging Face 模型 · tencent 原文语言:中文
官方机构模型仓库创建于 2026-08-14。仓库创建时间不等于正式发布日期。任务类型:image-text-to-text
全文
UI-Mate-27B
UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations
Show the workflow once. Let the agent adapt it to the task at hand.
Tencent HY Frontier
Project Page · GitHub · arXiv
Overview
UI-Mate-27B is an open-weight foundation GUI agent for long-horizon work across applications and operating systems. It observes live screenshots, reasons over the visible state, and produces structured keyboard and mouse actions for native desktop interaction.
It executes a task from a natural-language instruction and live screenshots alone, planning from what is currently on screen and re-planning as the interface changes.
Model Details
Parameters: 27B Base model: Qwen3.6-27B Input: task instruction, screenshots, and interaction history Output: reasoning, a concise action description, and structured computer-use tool calls Action space: mouse, keyboard, scrolling, waiting, user interaction, and task completion Training: supervised fine-tuning followed by online reinforcement learning in executable GUI environments License: Apache-2.0
UI-Mate is an agent checkpoint rather than a standalone visual-chat model. We recommend using the official prompt, response parser, and interaction harness from the UI-Mate repository.
Checkpoint Intended use UI-Mate-27B General computer use at 27B UI-Mate-9B General computer use at 9B UI-Mate-democua-27B Demonstration-guided computer use
Highlights
Strong general computer-use performance across Ubuntu and Windows benchmarks. Long-horizon execution across multiple applications. Live-screen grounding instead of coordinate replay. Structured actions compatible with pyautogui. OpenAI-compatible serving and client interface.
Evaluation
Benchmark UI-Mate-27B OSWorld-Verified · average score 77.0 WindowsAgentArena · average score 66.2 OSWorkerBench · strict success 41.00 OSWorkerBench · progress 76.86
See the project page for evaluation details and updates.
Quick Start
1. Serve the model with vLLM
vLLM automatically downloads the checkpoint from Hugging Face Hub on first launch.
The default agent retains five screenshots in context. The server must therefore admit at least six images because the newest screenshot arrives before the oldest one is collapsed.
Confirm that the endpoint exposes the expected model name:
2. Run the reference examples
Run a complete recorded trajectory:
Or provide your own screenshot:
3. Use the Python agent
Intended Use and Limitations
UI-Mate-27B is intended for research and development of screenshot-based GUI agents in controlled desktop environments.
Its behavior can be affected by application versions, screen layouts, display scaling, latency, and unexpected UI state. Benchmark performance does not guarantee reliable execution in arbitrary environments, and the model requires an external runtime to execute its predicted actions.
Safety
Computer-use agents can make mistakes, encounter prompt injection, or trigger consequential actions.
Prefer isolated or disposable environments. Avoid unattended, high-stakes, or destructive workflows. Require human confirmation before sensitive operations. Monitor the interaction trajectory and verify the resulting application state. Do not treat a model-reported success as proof that the intended outcome was achieved.
License
UI-Mate is released under the Apache License 2.0. Third-party components remain subject to their respective licenses. See the repository LICENSE for details.
正文以来源语言显示。