What is mixture of experts (MoE)?

Mixture of experts (MoE) is a model architecture in which, instead of activating every parameter for each token, a routing network selects only a small subset of specialized experts to process that particular piece of input. The total model can have hundreds of billions of parameters, but for any single token only a fraction of them, often a tenth or less, is used. This cuts compute cost and inference latency compared to a dense model of the same nominal size, because the GPU runs fewer multiplications to generate each token. It is why models such as DeepSeek V3, Mixtral and Grok use this architecture: it lets capacity scale without the per-token cost a company pays in production scaling linearly with it. For a business buying access to a model through an API, the practical question becomes how many parameters are actually active per token, not how large the total parameter count is advertised as being.

See also: Foundation model · A model trained on broad, generic data and adaptable to many different tasks: the category LLMs and multimodal models belong to.

Why it beats a dense model on cost

In a dense model, every parameter participates in computing every token: doubling capacity roughly doubles both cost and inference time. In an MoE model you can increase the total number of experts, and therefore the model's overall capacity, without proportionally increasing the number of parameters activated per token. DeepSeek V3 reports 671 billion total parameters but activates around 37 billion per token: that is the technical reason behind API pricing that, according to public provider rate cards, ends up an order of magnitude below comparable dense models. The trade-off is infrastructure: you need more memory to keep all experts loaded and an efficient router, or the per-token cost advantage gets eaten up by deployment complexity.

A concrete enterprise case: a financial services firm processing millions of documents a month for data extraction and classification compares two options with similar performance on its internal benchmarks, a dense model and an MoE model. At comparable output quality, choosing the MoE model cuts monthly inference spend substantially, because the billed cost per token reflects the parameters actually activated, not the model's nominal size. The same logic applies to latency as experienced by end users in a high-volume conversational application.

What changes for decision-makers

For anyone evaluating models for a RAG architecture or an enterprise assistant, MoE means the total parameter count on a spec sheet is no longer a reliable proxy for cost or speed: what matters is the active parameters per token, measured against real cost and latency on your own workload, not on vendor benchmarks. It is also one reason it pays to keep more than one model option in production rather than locking into a single vendor: sparse architectures shift the quality-to-cost-per-token ratio quickly, and whoever already has an abstraction layer above their models can switch providers without rewriting the application.