Hy-Embodied-RxBrain-1.0

Model release Hugging Face 模型 · tencent Source language: Chinese

官方机构模型仓库创建于 2026-07-14。仓库创建时间不等于正式发布日期。任务类型:any-to-any

Full text

RxBrain Embodied Cognition Foundation Model with Joint Language–Visual Reasoning and Imagination Tencent Robotics X × Futian Laboratory × Tencent Hy Team

🔥 Updates

[2026-07] 🎉 We release Hy-Embodied-RxBrain-1.0 — the technical report, official inference code, and model weights.

📖 Introduction

RxBrain (Hy-Embodied-RxBrain-1.0) is a unified multimodal foundation model for embodied cognition — a single model that couples language reasoning with visual imagination to deliver three core capabilities:

🤖 Embodied Understanding & Reasoning — question answering and chain-of-thought over images and multi-frame video. 🔮 World State Prediction — imagine the near-future frames an action produces in the physical world. 🧩 Joint Subgoal Planning — decompose a task into steps, emitting for each step *both* the next action (language) *and* the goal image it should reach (vision).

These capabilities are unified through interleaved generation: within a single autoregressive sequence RxBrain alternates reasoning text and flow-matched imagined frames — a learned token decides when to imagine — so an embodied plan couples *what to do* with *what the world should look like*, step by step.

⭐️ Key Features

🧠 Unified Mixture-of-Transformers (MoT): A ~6.2B-parameter backbone with modality-specific pathways (text / vision / generation), so understanding and image synthesis share one autoregressive model instead of separate towers. 🎨 Flow-Matching Image Head: Imagined frames are produced by a flow-matching head decoding into a frozen FLUX VAE latent space, enabling text-to-image, multi-frame world-model rollout, and goal-image planning. 🔗 Interleaved Reasoning + Imagination: Text reasoning and generated frames are emitted in one sequence, coupling symbolic plans with visual goals.

📋 TODO

[ ] Open-source RxBrain-Bench, a comprehensive benchmark for embodied cognition. [ ] Open-source the fine-tuning code for Visual Question Answering (VQA), Multi Frames Generation, and Interleave Generation. [x] Release the technical report, inference code, and model weights.

🛠️ Dependencies and Installation

Prerequisites

🖥️ Operating System: Linux (recommended) 🐍 Python: 3.10+ ⚡ CUDA: 12.x, an NVIDIA GPU (required for flash-attn) 🔥 PyTorch: 2.10

Installation

Install the specific Transformers version required for this model (it provides the hunyuan_vl_mot backbone that unified_mot builds on):

Note: A stock transformers release does not yet include hunyuan_vl_mot; this pinned commit is required. We will merge the improvements into the Transformers main branch later.

Clone the inference code and install the remaining dependencies:

Model Download

Component Params Source Hy-Embodied-RxBrain-1.0 ~6.2 B 🤗 tencent/Hy-Embodied-RxBrain-1.0 FLUX VAE (ae.safetensors) 83.8 M Obtain from the FLUX distribution

Download the weights to a local directory — the loader reads the checkpoint files directly, so --ckpt must be a local path, not the Hub repo id:

The VQA (understanding) path needs only the main weights. Image generation (T2I / world-model rollout / interleaved planning) additionally requires the external FLUX VAE ae.safetensors.

🚀 Quick Start with Transformers

Load the Transformers processor together with the UnifiedMoT classes shipped in this repo, then run understanding (VQA). Run this from the repo root so the model package is importable, and point MODEL_PATH at your local download (see Model Download).

Note: RxBrain uses a custom interleaved text/image decoding loop rather than the standard model.generate API. The answer(...) helper (in vqa_inference.py) wraps that loop for the understanding case; image generation and planning have their own entry points below.

The same tasks are also available as ready-to-run scripts:

① Visual Question Answering (VQA) — image(s) + question → answer text

Pure autoregressive text understanding — no VAE / flow-matching needed.

② Text-to-Image (T2I)

③ Multi-Frame World-Model Rollout — imagine future frames from an observation

④ Interleaved Embodied Planning — text plan + goal images, step by step

Runs interleaved planning on a bundled scene. See demo_cases/README.md for details.

📊 Evaluation

RxBrain is evaluated on embodied understanding, spatial reasoning, and imagination/generation benchmarks. For detailed metrics and methodology, please refer to our technical report.

Sources

← Back to the daily brief