Models & Architecture
Mixture of Experts
MoEA model design that routes each input to a few specialized sub-networks instead of running the whole model every time.
In a mixture-of-experts model, the network is split into many expert sub-networks, and a router sends each input to just a small subset of them. This lets a model have a very large total parameter count while only activating a fraction of it per request, which saves compute. Several leading large language models use this design to grow capacity without a proportional rise in running cost.
Reference: Wikipedia: Mixture of Experts
Related terms
Parameters
The adjustable numbers inside a model that get tuned during training and store what the model has learned.
Large Language Model
A model trained on massive amounts of text that predicts and generates language, powering tools like ChatGPT and Claude.
Inference
Actually running a trained model to get an answer, as opposed to training it in the first place.
Compute
The raw processing power used to train or run a model, a key driver of cost and capability.