Models & Architecture

Mixture of Experts

MoE

A model design that routes each input to a few specialized sub-networks instead of running the whole model every time.

In a mixture-of-experts model, the network is split into many expert sub-networks, and a router sends each input to just a small subset of them. This lets a model have a very large total parameter count while only activating a fraction of it per request, which saves compute. Several leading large language models use this design to grow capacity without a proportional rise in running cost.

Reference: Wikipedia: Mixture of Experts