Youtu-Parsing-Omni

模型发布 Hugging Face 模型 · tencent 原文语言:中文

官方机构模型仓库创建于 2026-10-09。仓库创建时间不等于正式发布日期。任务类型:image-text-to-text

全文

One model that parses documents, images, charts, geometry, audio and video into structured JSON

📄 License • 📑 Technical Report • 🚀 Quick Start • 📊 Results • 📚 Citation

News

[2026-10] OmniDocBench evaluation code released (see OmniDocBench Evaluation). [2026-10] Technical report, model weights, vLLM plugin and inference examples released.

Introduction

Youtu-Parsing-Omni is a compact (5B) omni-modal parsing model. Given a single input — a document page, a natural image, a chart / flowchart, a geometry figure, an audio clip or an audio-visual video — it produces one structured JSON envelope that covers both perception (layout elements, text, tables, formulas, bounding boxes, timestamps, ASR, OCR, acoustic events, camera motion) and cognition (captions, narratives, reports). The output family is selected by the task prompt (--task in the examples, keys of prompts/youtu_parsing_omni.json).

Input --task modality / subtype Key contents Document page document image / document layout elements with bbox, text / LaTeX / OTSL tables / Markdown charts / Mermaid flowcharts, reading order Natural image natural_image image / natural_image entities and text with bbox, tags, captions, global description Chart graphics_chart image / document one chart element: Markdown table, notes, caption Flowchart graphics_flowchart image / document one flowchart element: Mermaid, caption Geometry figure graphics_geometric image / document one geometric element: points, lines, arcs, shapes, geometric relations and measurements Audio audio audio / – vocal / non-vocal segments with timestamps, speakers, ASR, timbre / scene captions, acoustic events Natural video natural_video video / natural_video temporal segments with visual elements, actions, interactions, camera motion, audio track Text-rich video textrich_video video / text_rich_video segments with OCR + ASR and a Markdown structured_report of the whole video

Highlights (see the technical report for details):

Unified schema – one JSON envelope for seven parsing families, driven by the task prompt. Omni encoder – image, audio and interleaved audio-visual video inputs (frames + audio track) in a single model. Strong results at a small size – state-of-the-art on OmniDocBench v1.6 (96.96 Overall), best open-weight model on OmniParsingBench (75.08 Avg., second only to Gemini-3-Pro), and competitive with specialized models on chemical-structure (ChemOCR) and music-score (PDMX-Synth) recognition (results). Easy to serve – a vLLM plugin, pinned serving settings, task prompts and inference examples are included.

Method

One encoder for every modality

Existing omni-modal parsers (a) encode pixels and audio with two separately pre-trained towers whose streams first meet inside the language model. Youtu-Parsing-Omni (b) replaces both towers with a single Youtu-Omni-Encoder, initialized from a pre-trained text language model to strengthen text reading. Thin modality-specific stems map pixels and log-mel frames to tokens; a shared bidirectional Transformer contextualizes an arbitrary packed mix of images, audio chunks and videos under one (t, h, w) positional encoding; modality-specific mergers and projectors then feed the tokens, together with the text prompt, to the Youtu-LLM decoder, which emits OmniSchema JSON. Only the stems and the merger / projector heads depend on the modality, so almost all perception parameters are shared.

Audio-visual fusion inside perception

The frames and audio chunks of a video are packed in temporal order. (a) A non-fusion layer gives every frame and every audio chunk its own attention window, while a few fusion layers start a new window at every audio-to-frame boundary, so a frame attends to the sound of the same instant. (b) Vision tokens take t from the frame index and (h, w) from the patch grid; audio tokens take t from their time offset with h = w = 0, so both modalities share one temporal coordinate system. Cross-modal alignment is thus computed inside the encoder rather than reconstructed by the decoder.

One schema for every task: OmniSchema-Aware On-Policy Distillation (OSAD)

All ten parsing tasks share OmniSchema, so after SFT and schema-routed RLVR a single self-evolving post-training procedure, OSAD, applies the same token-role-based objective across tasks. The student samples its own parses; rollouts that pass the admission gate (JSON validity, schema conformance, box syntax, repetition checks) are scored by a frozen copy of the model that additionally sees the reference parse. Because every output follows OmniSchema, each token is identified as content (Jensen–Shannon divergence) or structure (forward KL at a higher temperature), plus an all-token JS term. At every round boundary the student's LoRA adapters are merged and the student becomes the next teacher, so the parser and its supervision co-evolve without a hand-designed reward or a separately trained teacher.

Model Zoo

Model Parameters Download Youtu-Parsing-Omni 5B 🤗Hugging Face \ 🤖ModelScope

Results

OmniDocBench v1.6

Model Params Overall ↑ Text Edit ↓ Formula CDM ↑ Table TEDS ↑ Table TEDS-S ↑ Reading Order Edit ↓ Youtu-Parsing-Omni 5B 96.96 0.0271 96.80 96.79 98.20 0.1104 TeleOCR 1.2B 96.91 0.0267 96.59 96.82 98.18 0.1184 OvisOCR2 0.8B 96.47 0.0265 97.49 94.58 96.98 0.1120 PaddleOCR-VL-1.6 0.9B 96.34 0.0326 97.53 94.76 97.10 0.1278 MinerU2.5-Pro 1.2B 95.75 0.0360 97.45 93.42 95.92 0.1200 GLM-OCR 0.9B 95.22 0.0440 97.18 92.83 95.39 0.1330 HunyuanOCR-1.5 1B 94.74 0.0390 94.50 93.67 94.71 0.1290 Qianfan-OCR 4B 93.90 0.0400 95.08 90.53 93.31 0.1300 Youtu-Parsing 2.5B 93.74 0.0440 93.63 92.02 95.00 0.1160 Ovis2.6-30B-A3B 30B-A3B 93.70 0.0350 95.17 89.44 92.40 0.1350 Logics-Parsing-v2 4B 93.33 0.0410 95.65 88.42 91.98 0.1370 FireRed-OCR 2B 93.26 0.0370 95.44 88.04 91.06 0.1310 Gemini-3-Pro – 92.91 0.0640 95.99 89.15 92.96 0.1650 dots.ocr 3B 90.77 0.0480 89.95 87.18 90.58 0.1380 OpenDoc-0.1B 0.1B 90.67 0.0490 93.02 83.88 87.45 0.1400 DeepSeek-OCR 2 3B 90.25 0.0500 91.84 83.89 87.75 0.1440 Qwen3-VL-235B 235B-A22B 89.78 0.0630 92.55 83.07 86.75 0.1660 Dolphin-v2 3B 89.50 0.0690 91.01 84.40 87.44 0.1500 OCRVerse 4B 88.60 0.0630 89.61 82.44 86.27 0.1630 MonkeyOCR-pro-3B 3B 88.57 0.0740 88.74 84.35 88.62 0.1890 GPT-5.2 – 86.59 0.1140 88.21 82.95 87.93 0.1930

OmniParsingBench

Model Chart Geometry Natural Image Audio Natural Video Text-rich Video Avg. Gemini-3-Pro 92.79 85.43 69.73 76.74 74.82 76.80 77.44 Youtu-Parsing-Omni 95.02 74.33 62.84 78.08 74.18 75.62 75.08 Logics-Parsing-Omni 94.65 76.34 67.53 75.58 67.34 65.36 72.26 Qwen3-Omni-30B-A3B 87.07 58.60 59.35 75.23 65.19 59.59 66.44 Qwen3.5-397B-A17B 93.51 72.16 63.87 – – – – GPT-5.4 90.92 81.16 60.50 – – – –

ChemOCR & PDMX-Synth

ChemOCR

Model Params Category Avg. Sim. ↑ Tani@1.0 ↑ DeepSeek-OCR 2 3B General parsing MLLM 74.9 49.6 Youtu-Parsing-Omni 5B Unified parsing MLLM 74.6 54.2 ChemDFM-X 13B Chemistry MLLM 70.9 36.5 GPT-4o – Proprietary general MLLM 36.8 3.4 Qwen2.5-VL-7B 7B Open-source general MLLM 25.5 0.4 InternVL2.5-4B 4B Open-source general MLLM 4.4 0.0

PDMX-Synth

Model Category CER ↓ SER ↓ LER ↓ Youtu-Parsing-Omni Unified parsing MLLM 22.73 26.73 33.20 LEGATO Specialized OMR model 23.30 25.80 31.70 GPT-5.6 Proprietary general MLLM 45.02 54.74 87.52 Gemini-3-Pro Proprietary general MLLM 54.26 68.10 82.33 Logics-Parsing-Omni General parsing MLLM 72.44 81.93 81.34 GOT-OCR2.0 General parsing MLLM 93.98 98.54 97.76

Quick Start

1. Installation

vLLM serving and Transformers inference pin different transformers versions (5.2.0 and 5.10.2); install them into separate Python environments. peft must not be installed in the serving environment (the script removes it). ffmpeg must be available for audio / video inputs.

Manual installation

2. Serve with vLLM

This starts an OpenAI-compatible server on http://127.0.0.1:8000/v1 (served model name Youtu-Parsing-Omni) with the vllm-plugin-vita-omni model implementation and the runtime settings of the reported results.

Variable Default Meaning MODEL required Hugging Face model id (tencent/Youtu-Parsing-Omni) or a local checkpoint directory DATA_ROOT required the only directory the server may read local media from (--allowed-local-media-path) GPU 0 GPU the server runs on (CUDA_VISIBLE_DEVICES) PORT 8000 port of the server

Engine settings: --max-model-len 65536 --max-num-batched-tokens 65536 --max-num-seqs 1, --gpu-memory-utilization 0.70, --limit-mm-per-prompt '{"image": 8, "video": 4, "audio": 8}', --seed 1234, bfloat16; vLLM defaults for everything else. The server listens on 127.0.0.1 only and runs in the foreground (logs go to the terminal); Ctrl-C stops it. vLLM's internal port is 30000 (override with VLLM_PORT). The script also exports the media token budgets read by the plugin (YOUTU_VITA_*, see vllm-plugin-vita-omni). MODEL is passed to vllm serve as is: tencent/Youtu-Parsing-Omni is downloaded from the Hugging Face Hub (cached in HF_HOME); a local checkpoint directory works as well.

3. Parse a file

The media file must lie below DATA_ROOT.

Use --api-base to target another server. The task prompts are listed in prompts/youtu_parsing_omni.json. Requests must put the media before the text and enable the parsing mode of the chat template shipped with the model (chat_template_kwargs = {"enable_thinking": false, "enable_parsing": true}), which renders the media token directly followed by the prompt, without role markers. As in the reported results, decoding is greedy, stops at < image_pad > / < end_of_text > (stop_token_ids, looked up through the server's /tokenize endpoint) and keeps special tokens, which carry the coordinates.

Raw OpenAI API request

Audio files are sent as audio_url. For videos send one file:// video_url below --allowed-local-media-path and add "mm_processor_kwargs": {"nframes": 64, "use_audio_in_video": true, "use_vision_in_video": true} to extra_body; the plugin then decodes the audio track together with the frames (see Video requests).

Transformers

Transformers inference requires transformers==5.10.2 (bash scripts/setup_env.sh --transformers, in an environment separate from vLLM). A minimal example is in examples/infer_transformers.py:

--task takes the same values as examples/infer_vllm.py and selects the prompt from prompts/youtu_parsing_omni.json; the media path is not restricted to a DATA_ROOT here. Hub ids are downloaded to --download-dir (default ~/.cache/youtu-parsing-omni). The model is loaded with trust_remote_code=True, which executes the Python code shipped with the checkpoint; when --model is a Hugging Face Hub id, pin a reviewed commit with --revision . The reported results were produced with vLLM (scripts/vllm.sh).

Output format

The model answers with one fenced json { "modality": "image", "subtype": "natural_image", "elements": [ {"id": "e_000", "category": "entity", "bbox": " ", "tag": "snake bookmark", "content": {"text": "", "caption": "A silver snake-shaped bookmark ..."}} ], "global_description": {"caption": "...", "narrative": "...", "scene": "..."} } bash MODEL=tencent/Youtu-Parsing-Omni OMNIDOCBENCH=/path/to/OmniDocBench \ OUTPUT_DIR=outputs/omnidocbench_$(date +%Y%m%d_%H%M%S) bash scripts/eval_omnidocbench.sh

正文以来源语言显示。

参考来源

← 返回每日简报