TPHC:通过张量并行分组更新改进 Hyper-Connections

Media reports Hugging Face Papers Source language: Chinese

Paper abstract (author-reported, not re-tested by this site): Hyper-Connections (HC) expand the residual stream of a Transformer into N parallel streams, and manifold-constrained HC (mHC) stabilizes them at scale.

Full text

TPHC combines tensor-parallel shards with hyper-connection residual streams to enable more independent stream updates, improving language model loss and zero-shot reasoning with modest training overhead.

Sources

← Back to Daily AI Report