Chorus: Shared N-gram Embeddings with Layer-wise Adaptation

Media reports Hugging Face Papers Source language: Chinese

Chorus shares a single set of n-gram embedding tables across all layers, enabling each layer to project and rescale the embedding using its hidden state. This approach achieves better quality with much smaller tables and lower optimizer memory.

Full text

Chorus shares one set of n-gram embedding tables across all layers, letting each layer project and rescale the embedding using its hidden state, achieving better quality with much smaller tables and lower optimizer memory.

Sources

← Back to Daily AI Report