Large language models examples: what the top systems prove

DeepSeek-V3 contains 671 billion total parameters but activates only 37 billion per token; Meta’s Llama 3.1 405B was trained on more than 15 trillion tokens; Google’s Gemini 1.5 Pro pushed the developer-facing context window to 2 million tokens. These figures describe different engineering strategies rather than a single race toward a larger model.

The current frontier is therefore defined by a three-way constraint: how much computation is required during training, how much of the network must be executed for each token, and how much state must be held in memory during inference. The popular large language models now represent distinct answers to those constraints. Some use sparse Mixture-of-Experts routing, some prioritize open weights and reproducibility, and others extend context far beyond the range that conventional transformer deployments were designed to handle.

The distinction matters because a model’s headline parameter count is an incomplete description of its economics. FLOPs per token, memory bandwidth, activation volume, key-value cache growth, quantization support, and the availability of production tooling determine whether an architecture can move from a research release into an operational system.

The shift toward Mixture-of-Experts: DeepSeek-V3 and parameter efficiency

DeepSeek-V3 is one of the clearest examples of the industry moving from dense scaling toward conditional computation. Its 671 billion parameters are not activated in full for every token. The model routes each token through a selected subset of experts, with 37 billion parameters active per token.

That difference changes the relevant comparison with a dense model. A dense 671B system would require computation across the full parameter set at every forward pass. DeepSeek-V3 instead maintains a very large total capacity while limiting the active computation. The model still carries the storage and deployment burden associated with a 671B-parameter network, but its per-token compute profile is closer to a much smaller active model.

This is not a free reduction. Sparse models introduce routing overhead, expert-dispatch communication, and more complicated parallelism. In a multi-GPU environment, selected experts may be distributed across devices, requiring tokens to be moved between accelerators before and after expert computation. The performance of the system is therefore determined not only by tensor-core throughput but also by interconnect bandwidth, batch composition, load balancing, and the degree to which expert utilization remains uniform.

A sparse architecture can reduce arithmetic work while increasing systems complexity. That trade-off is central to understanding current LLM examples in production. The model may require fewer FLOPs per token than a comparably capable dense model, but the serving stack must still manage a large parameter inventory and the communication pattern created by routing.

DeepSeek-V3 was pre-trained on 14.8 trillion tokens using 2.788 million H800 GPU hours. The significance of those figures is not that they establish a universal training price. Hardware-hour totals are difficult to compare across organizations because utilization, software efficiency, precision formats, checkpointing strategy, interconnect topology, and data-processing pipelines differ. Their value is directional: frontier-scale training is increasingly an exercise in infrastructure optimization rather than simply purchasing more accelerators.

Sparse activation changes the unit of comparison: total parameters describe capacity, while active parameters describe much of the computation paid for on each token.

The architecture also exposes a recurring weakness in simplistic model comparisons. Two systems can advertise similar quality while presenting very different resource requirements. One may be dense and easier to deploy on a homogeneous accelerator cluster; another may be sparse and more compute-efficient but more sensitive to network topology. Their token-level latency will depend on batch size, sequence length, routing balance, and memory pressure, not only on the nominal number of active parameters.

For developers, the useful questions are consequently operational:

• Is the serving environment capable of distributing expert layers without creating a communication bottleneck?
• Does the inference implementation support the model’s routing and parallelism strategy?
• Can the total weights fit within the available accelerator memory, or must quantization and weight offloading be used?
• Does the workload produce enough concurrent requests to amortize the cost of the larger model footprint?
• Is latency dominated by matrix multiplication, token dispatch, memory bandwidth, or the key-value cache?

These are architecture questions, not product-marketing questions. They determine whether a sparse model behaves like an efficient high-capacity system or like a large distributed application with unpredictable tail latency.

Scaling frontiers: Llama 3.1 and the 15-trillion-token milestone

Meta’s Llama 3.1 family illustrates a different scaling strategy. The largest model in the release, Llama 3.1 405B, was trained on more than 15 trillion tokens using over 16,000 H100 GPUs. Unlike a sparse model whose headline parameter count is separated from its active computation, the 405B label communicates the scale of a dense network more directly.

The training corpus and hardware allocation indicate the size of the optimization problem. A model trained on more than 15 trillion tokens is exposed to an unusually broad distribution of text and code, but token count alone does not guarantee data quality. Deduplication, filtering, mixture composition, curriculum design, and the allocation of tokens across domains affect the useful information extracted by the model. A larger corpus can increase coverage while also increasing the need for rigorous data engineering.

The use of more than 16,000 H100 GPUs adds a second layer of complexity. At this scale, training throughput is constrained by collective communication, synchronization, checkpointing, fault recovery, and the ability to keep all accelerators supplied with work. The theoretical FLOPs of the hardware are only an upper bound. Effective utilization depends on how much time is spent performing useful matrix operations rather than waiting for communication or recovering from stragglers.

Llama 3.1 is also significant because the 128,000-token context window was extended across the 8B, 70B, and 405B model sizes. That common context target makes the family easier to reason about from an application perspective: context capacity is not reserved exclusively for the largest checkpoint.

However, a 128K context window should not be confused with a 128K-token prompt that can be processed at negligible cost. The prompt must still be encoded, attention-related state must be created, and the key-value cache must be stored for subsequent generation. Memory usage rises with sequence length, batch size, number of layers, attention-head configuration, and numerical precision. Long context is therefore a capability with a cost curve, not a binary feature.

For a production system, the relevant distinction is between context capacity and context economics. A 128K-token request may be technically accepted but operationally unattractive if it causes high time-to-first-token, consumes substantial GPU memory, or reduces the number of concurrent requests that can be served. In many workloads, retrieval, document chunking, hierarchical summarization, or selective context packing remains cheaper than sending every available token to the model.

The main conclusions from Llama 3.1 are more structural than promotional:

1. Dense scaling remains viable at frontier size. A 405B-parameter model is impractical for many local deployments, but it remains a reference point for high-capacity open-weight systems and for distillation into smaller models.

2. Training data has become an infrastructure variable. Processing 15 trillion tokens requires not only storage but also filtering, tokenization, data movement, and repeated validation of the training mixture.

3. Context length must be evaluated together with memory behavior. The nominal window says little about latency or throughput without information about precision, batch size, and serving implementation.

4. Open weights alter the downstream optimization path. Organizations can adapt, quantize, distill, or host the model without depending entirely on a proprietary API endpoint.

The difference between the 8B, 70B, and 405B variants is especially important for deployment planning. Smaller checkpoints can fit into more constrained environments and may achieve lower latency at the cost of representational capacity. The largest model demands a distributed serving architecture and substantial memory bandwidth. Between those extremes, the 70B class often becomes an engineering compromise: large enough for demanding language tasks, but less operationally punishing than a 405B model.

Context windows: from 128K to 2 million tokens

Google’s Gemini 1.5 Pro established another frontier by offering a context window of up to 2 million tokens for developers. That is an order-of-magnitude shift from the 128,000-token windows associated with Llama 3.1, although the two numbers should not be interpreted as direct evidence that one model is universally more capable.

A context window defines the maximum amount of input and generated material that can be presented within a single interaction boundary. It does not establish how well the model retrieves information from every position, how much latency the request incurs, or how the system prices long inputs. Effective long-context performance requires separate evaluation.

The computational burden is substantial. In a conventional transformer, attention mechanisms create a relationship between tokens in a sequence, and the cost of processing long inputs grows with sequence length depending on the implementation and attention formulation. Modern systems use a combination of optimized kernels, architectural modifications, positional-encoding techniques, memory management, and sometimes sparse or approximate attention to make longer sequences practical. The engineering challenge is not merely increasing a maximum integer in an API.

During autoregressive generation, the key-value cache becomes a major factor. Previously computed attention states are stored so that each new token does not require the entire prefix to be recomputed. For long prompts, that cache can occupy a considerable portion of accelerator memory. Quantized cache formats, grouped-query attention, paged memory allocation, and efficient batching can reduce the burden, but each introduces implementation trade-offs.

The practical comparison looks like this:

The table does not establish a simple winner. A 2-million-token window is useful when the workload genuinely contains a large, interdependent body of information that cannot be efficiently retrieved in smaller segments. It is less useful when most of the input is irrelevant noise. Long context can increase the opportunity for distraction, duplication, and conflicting instructions while also increasing cost and latency.

The more durable development pattern is likely to be hybrid. Retrieval systems can narrow a corpus, while long-context models can process the selected material in a single reasoning pass. In that arrangement, the context window is not a replacement for indexing; it is a larger execution space for the final stage of retrieval-augmented generation.

This distinction is relevant in sectors where documents are long, structured, and operationally sensitive. A digital banking infrastructure guide may describe adjacent systems, but the same deployment principle applies: the value of capacity is determined by how the underlying data is selected, normalized, and routed into the system, not by the maximum size advertised in isolation.

Open weights versus proprietary black boxes

The most important division among examples of large language models is not always architectural. It is also institutional: who controls the weights, the serving interface, the evaluation environment, and the update schedule.

Meta’s Llama 3.1 models belong to the open-weight side of the market. The availability of weights gives organizations greater control over deployment, fine-tuning, quantization, and infrastructure placement. It also allows technical teams to inspect behavior under controlled conditions and to build specialized variants without sending every request to an external provider.

Open weights do not mean that a model is fully open in every research sense. Training data, filtering procedures, reinforcement-learning pipelines, infrastructure configurations, and some implementation details may remain unavailable. The term should therefore be treated precisely: access to weights is a material engineering advantage, but it does not imply complete reproducibility.

The deployment benefits are direct:

• Latency control: Requests can be served within infrastructure designed around the application’s geography, concurrency, and service-level objectives.
• Cost control: Quantization, batching, speculative decoding, and model routing can be tuned against actual workload characteristics.
• Data governance: Sensitive prompts and retrieved documents can remain inside a controlled environment, subject to the organization’s own security architecture.
• Version stability: A pinned checkpoint does not change merely because a hosted provider silently updates an API model.
• Model specialization: Fine-tuning and distillation can be performed for a domain, language, output schema, or tool-use pattern.

The costs are equally concrete. Hosting a large open model transfers responsibility for GPU procurement, cluster scheduling, observability, capacity planning, model updates, incident response, and security hardening. A 405B-parameter checkpoint may be accessible in principle but still be economically unsuitable for a modest inference workload. Memory capacity and memory bandwidth become hard constraints before model quality does.

Proprietary systems such as GPT-4-series models and other commercial frontier models offer a different trade-off. OpenAI officially released the original GPT-4 on March 14, 2023, but the company did not publish an official parameter count or complete architectural specification. That omission is not a minor documentation gap. Without those details, external engineers cannot calculate the model’s parameter-memory footprint, reproduce its training regime, or make a direct architecture-level comparison with an open-weight system.

The same caution applies to other closed models for which official parameter counts and architecture details have not been disclosed. A model can be evaluated behaviorally through benchmarks and production tasks, but its internal compute profile remains uncertain. Claims about efficiency should therefore be separated into two categories:

1. Observed product behavior: latency, throughput, context limits, tool support, output quality, and pricing under specified conditions.

2. Unverified internal explanation: parameter count, number of experts, training FLOPs, data composition, and hardware utilization.

Confusing the two produces false precision. An API may return tokens quickly because of provider-side batching, specialized hardware, caching, or serving optimizations that are invisible to the user. Conversely, a slower response may reflect queueing or network latency rather than model architecture.

For businesses deciding between commercial large language models and open deployments, the choice is less about ideology than control surfaces. An API reduces operational complexity but creates dependency on provider pricing, availability, policy, and model lifecycle. Self-hosting increases control but exposes the organization to the full cost of distributed inference.

Compute economics: training is only the first bill

The training compute figures attached to current models are useful because they show the scale of the frontier, but they do not describe the complete economics of an LLM. The cost of serving a model can dominate after launch, especially when the application generates long outputs, processes large prompts, or requires high availability.

Training is a one-time or periodic capital-intensive event. Inference is a repeated systems problem. Every request consumes compute and memory bandwidth, while the shape of that consumption changes with input length, output length, concurrency, batching, and precision.

A useful cost model separates at least five components:

• Prefill compute: The work required to process the input prompt. Long documents can make this stage dominant.
• Decode compute: The autoregressive generation of output tokens, usually more sequential and latency-sensitive.
• Weight movement: The transfer of model parameters from memory into compute units. For large models, memory bandwidth can become a stronger constraint than raw FLOPs.
• Key-value cache: The memory required to retain attention state across generated tokens and concurrent requests.
• Communication overhead: The movement of activations and routed tokens between GPUs in tensor-, pipeline-, or expert-parallel deployments.

Quantization affects several of these components. Reducing weights from higher precision to formats such as 8-bit or 4-bit can lower memory requirements and improve the chance that a model fits on available hardware. It can also reduce memory traffic, which may improve latency when the deployment is bandwidth-bound. But quantization introduces accuracy and stability trade-offs, and its effect is workload-dependent. A model that performs acceptably on short factual prompts may degrade more visibly on code generation, mathematical reasoning, multilingual output, or long-context retrieval after aggressive compression.

Parameter count remains relevant, but it should be treated as one variable in a larger equation. A dense 70B model and a sparse model with a higher total count can have different arithmetic intensity, different memory footprints, and different scaling behavior under concurrency. A model with fewer active parameters may still require extensive memory because inactive experts must be available for routing.

The same applies to context windows. If the prompt length increases, prefill cost rises and the key-value cache expands. A system designed for short customer-support exchanges may exhibit stable throughput at 4K or 8K tokens but behave very differently at 100K tokens. Tail latency can increase even when average latency remains acceptable, because long requests occupy memory and compute resources for longer periods.

This is why benchmark reporting needs more detail than a single quality score. A credible comparison should identify:

• model version and parameterization;
• context length and output length;
• precision or quantization format;
• hardware and interconnect;
• batch size and concurrency;
• time to first token;
• generation throughput;
• peak memory consumption;
• quality benchmark and evaluation protocol.

Without those fields, claims that one model is faster or more efficient are difficult to generalize. A result obtained at batch size one on a single accelerator answers a different question from a result obtained under production concurrency across a distributed cluster.

What these large language models examples prove

Taken together, DeepSeek-V3, Llama 3.1, Gemini 1.5 Pro, and GPT-4 represent four different dimensions of frontier progress.

DeepSeek-V3 demonstrates that total parameter count can be decoupled from per-token activation through Mixture-of-Experts routing. The architectural gain is conditional computation, balanced against dispatch and communication complexity.

Llama 3.1 demonstrates that open-weight models can be scaled to hundreds of billions of parameters and trained on token volumes measured in the tens of trillions. Its 128K context window also shows that long-context capability can be delivered across multiple model sizes rather than reserved for one flagship checkpoint.

Gemini 1.5 Pro demonstrates that context expansion has become a primary product axis. A 2-million-token window changes the class of documents and codebases that can be presented in one request, while making memory management, prefill latency, and input selection more important.

GPT-4 demonstrates the continuing relevance of proprietary systems whose internal architecture is not fully disclosed. Its March 2023 release marked a major commercial milestone, but the lack of official parameter and architecture details limits direct comparison with open models at the systems level.

None of these examples proves that scale alone is sufficient. Capability is shaped by data quality, optimization, alignment, inference software, tool integration, and evaluation design. Nor does a larger context window automatically produce better retrieval from every location in a document. Nor does a lower active-parameter count automatically translate into lower serving cost.

The more reliable conclusion is that LLM progress is now being expressed through specialized forms of scaling:

• sparse computation rather than uniform activation;
• larger and more carefully engineered training mixtures;
• longer context with more sophisticated memory systems;
• open weights that move optimization into the deployment layer;
• proprietary models whose quality may be high even when their architecture remains opaque.

For developers and infrastructure teams, the next generation of model selection will therefore look less like a leaderboard and more like capacity planning. The relevant question is not which model has the largest number attached to it. It is whether the model’s computation pattern, memory requirements, context behavior, licensing conditions, and serving stack match the workload.

That is the central lesson from the current large language models examples. The frontier is no longer a single vertical line of bigger models. It is a systems problem spanning training data, accelerator clusters, network fabrics, quantization formats, context management, and application architecture. The models that matter in production will be those whose theoretical advances survive contact with latency targets, memory limits, and the cost of moving every token through the server rack.