What are large language models and how to evaluate them

You open a chat interface, type a perfectly reasonable request, and get an answer that is fluent, fast and subtly wrong.

That gap is the real subject. Understanding large language models is not only about knowing that they predict the next token. It is about knowing where that prediction mechanism is useful, where it creates friction, and how to evaluate a model before it becomes embedded in an enterprise workflow.

A model with a spectacular benchmark score can still fail at your support queue, mishandle a spreadsheet, produce unreliable citations or become prohibitively expensive at scale. Conversely, a smaller model with lower headline performance may be the better product if it has lower latency, predictable output, stronger privacy controls and a smoother onboarding path.

What are large language models?

A large language model, or LLM, is a machine-learning system trained on very large collections of text and other data to model language. In practical terms, it breaks input into tokens, estimates what should come next, and repeats that process to produce an answer.

That description is technically accurate and operationally incomplete.

Modern LLMs are generally built on transformer architectures. Transformers use attention mechanisms to weigh relationships between tokens in a sequence. When the model processes a sentence, attention helps it connect terms that may be far apart: a subject with its verb, a requirement with its exception, or a programming function with the variable it modifies.

During pretraining, the model sees enormous numbers of examples and adjusts billions of internal parameters to improve its predictions. It does not store a clean database of facts in the way a conventional enterprise system stores records. Its knowledge is distributed across learned statistical patterns. That is why it can produce an elegant explanation of a complex topic, but also invent a source, merge two similar people or confidently misread a date.

The usual LLM workflow has several layers:

• Tokenization converts text into units the model can process. A token may be a whole word, part of a word, punctuation or a space-dependent fragment.
• Inference generates the response one token at a time, using the prompt and the model’s learned parameters.
• Context handling determines how much input the model can consider in a single request. A longer context window is useful, but it does not automatically mean the model will retrieve every relevant detail correctly.
• Post-training adjusts the raw model for instruction following, dialogue, safety behavior and preference alignment.
• Tool use and retrieval connect the model to search, databases, code execution or business systems. These additions often matter more to enterprise performance than a small difference in a public benchmark score.

This is the first distinction worth keeping clear: the base model is not the same thing as the product in front of you. A polished application may be an LLM wrapped in retrieval, system prompts, routing logic, moderation, memory and a highly opinionated UI. That UI wrapper can make a mediocre model feel effective—or hide serious limitations until the workflow becomes expensive.

An LLM does not deliver “intelligence” as a single score. It delivers a bundle of behaviors, each with its own failure modes, latency and cost.

How large language models work in real use

The standard explanation—predict the next token—is useful because it punctures the idea that an LLM is consulting a miniature human mind. But it should not lead to the opposite mistake: assuming that next-token prediction is trivial.

The model’s output reflects several interacting elements:

1. The prompt. Ambiguous instructions create ambiguous behavior. A request to “summarize the risks” means something different to a lawyer, an engineer and a procurement manager.

2. The context. The model may be given documents, conversation history, examples or structured data. The order and formatting of that material can influence the answer.

3. The model’s training and post-training. Different models have different strengths in coding, multilingual work, reasoning, style control and instruction following.

4. Decoding settings. Temperature and related parameters can make output more or less variable. Lower variability is often preferable for extraction; creative drafting may tolerate more.

5. External tools. Retrieval-augmented generation can ground an answer in current documents, while code execution can handle arithmetic more reliably than free-form generation.

6. The application layer. Rate limits, retries, caching, prompt templates and output validation all affect the user experience.

This is why a model can appear excellent in a demonstration and unstable in production. The demo is usually a narrow path through the system. Real users submit incomplete requests, attach badly formatted files, switch languages, interrupt the workflow and ask for something the product team did not anticipate.

A practical evaluation should therefore follow the task from input to final deliverable. Do not assess only the prose in the chat window. Assess whether the model produces a usable result without repeated corrections.

For example, in a contract-review workflow, the relevant questions are not simply whether the model can explain indemnification. They include:

• Does it preserve clause numbers?
• Does it distinguish an obligation from a recommendation?
• Does it identify missing information rather than filling the gap with a guess?
• Can it return findings in the format the legal team actually uses?
• Does it expose the source passage for each material claim?
• What happens when the document exceeds the preferred context size?
• How much manual editing remains after generation?

That last question is where product claims meet reality. A model that saves ten minutes on one document but creates a verification burden on every sentence may not be saving time at all.

Why benchmark scores are no longer enough

Public benchmarks remain valuable. They provide shared tasks, reproducible comparisons and a way to detect broad capability changes. The problem starts when one score is treated as a complete model profile.

MMLU, the Massive Multitask Language Understanding benchmark, covers 57 academic and professional subjects. It was influential because it tested a broad range of knowledge rather than a single narrow skill. But leading frontier models now cluster above 88% accuracy on MMLU, a sign that the benchmark is becoming less discriminating at the top end.

A crowded leaderboard does not mean the models are interchangeable. It means the test is no longer separating them effectively.

HumanEval takes a different route. It contains 164 Python programming problems, with generated code evaluated through docstrings and automated unit tests. That is more concrete than asking a model whether it is good at programming. Yet a model can perform well on small, self-contained coding problems and still struggle with an unfamiliar repository, hidden dependencies, security-sensitive changes or requirements that are not fully specified.

GSM8K, launched in 2021, became a widely used benchmark for grade-school mathematical reasoning. It helped expose differences in arithmetic and multi-step problem solving. It still cannot tell an organization whether a model will correctly reconcile a messy financial workbook or respect the conventions of a particular accounting system.

Then there is Chatbot Arena, released by LMSYS in May 2023. It uses randomized, blind pairwise comparisons: people interact with two anonymous model responses and select the one they prefer. The results are processed through rating systems such as Bradley–Terry and Elo. This captures something academic tests often miss: whether people actually prefer the model’s answers in open-ended use.

But preference is not the same as truth. A confident, well-written response may win a comparison against a cautious but correct one. Arena ratings are useful for conversational quality, not a substitute for domain-specific accuracy testing.

Evaluation is becoming more multidimensional for good reason. Stanford HELM measures seven dimensions, including accuracy, calibration, robustness, fairness, bias, toxicity and efficiency. That framework reflects the reality of deployment: a model must not only answer correctly, but also communicate uncertainty, resist variation in inputs and operate within practical performance limits.

There is another less visible problem: benchmark implementation. MMLU scores can differ depending on the evaluation harness, prompt format, few-shot examples, answer parsing and other execution details. Implementations from EleutherAI’s lm-evaluation-harness, Stanford HELM and original author repositories may not produce identical results. Comparing numbers without comparing methodology is leaderboard theatre.

What each evaluation method actually tells you

LLM-as-a-Judge systems are increasingly used because human review is slow and expensive. Advanced judging models can reach roughly 80% to 90% agreement with human evaluators in some settings, at a fraction of the cost. That makes them useful for regression testing and large-scale comparisons.

It does not make them neutral. Judges may prefer certain writing styles, reward verbosity or share the same blind spots as the model being evaluated. A judge should be calibrated against human reviewers and used with a clear rubric, not installed as an oracle.

The stress test I would run before choosing a model

When I evaluate an LLM product, I do not begin with its demo prompts. I begin with the ugliest routine task the intended users perform every week.

That task should contain real friction: long inputs, incomplete instructions, domain terminology, formatting requirements and a meaningful cost of error. If the model survives only polished prompts, the evaluation is measuring the marketing surface, not the product.

For an enterprise knowledge assistant, I would assemble a test set with:

• documents users genuinely need to search;
• questions whose answers are explicit in the source material;
• questions where the correct answer is that the information is missing;
• similar terms that must not be confused;
• outdated documents alongside current ones;
• requests for structured output, such as a table or JSON;
• multilingual or acronym-heavy examples if those occur in the workflow;
• adversarial prompts designed to provoke unsupported certainty.

Then I would run the same cases across the shortlisted models and record more than whether the answer “sounds good.”

1. Measure factuality and citation behavior

A useful model should separate what it knows from what it is inferring. In a retrieval workflow, check whether every material claim is supported by the supplied documents and whether the cited passage actually says what the answer claims.

The most revealing test is often the missing-answer case. Give the model a document that does not contain the requested information. Does it say that the evidence is unavailable, or does it manufacture a plausible answer?

This is where hallucination rate becomes a practical metric rather than a conference-paper phrase. You can define it for the workflow: unsupported factual claims per task, incorrect citations per document, or fabricated entities in a structured output. The exact measure depends on the use case, but it must be observable and repeatable.

2. Test instruction following under pressure

A model may follow a simple instruction perfectly and then lose constraints when the prompt becomes long. Ask it to preserve headings, exclude confidential fields, return a fixed number of findings and flag uncertainty. Then introduce irrelevant context and contradictory instructions.

Watch for silent failures:

• required fields disappear;
• the answer changes format halfway through;
• prohibited information is repeated;
• the model answers an earlier question instead of the latest one;
• an instruction is followed semantically but not operationally.

For developers, structured output validation is essential. If the downstream system expects a schema, a paragraph that vaguely resembles the schema is not a success. It is a broken integration waiting to happen.

3. Test latency, not just intelligence

Latency changes behavior. A model that is marginally stronger but takes long enough to interrupt a support workflow may deliver less value than a faster alternative.

Measure time to first token and time to complete response separately. They affect different experiences. A fast first token can reassure the user that the system is working, while a long generation still delays the final action.

Also test latency under realistic concurrency. Vendor demonstrations usually represent an ideal path. Production users arrive in clusters, retry requests, upload large files and trigger tool calls. Rate limits and queueing can turn a frictionless interface into a stalled one.

4. Test recovery, not only first-pass success

A practical assistant will make mistakes. The question is whether the correction loop is efficient.

Give the model feedback such as “you used the outdated policy; use the current version” or “the output must include the source section.” Does it repair the answer cleanly, or does it defend the original response, repeat the error and consume another round of tokens?

The strongest products make correction cheap. They expose context, preserve state and let users edit instructions without restarting the entire interaction. That is product design, not merely model capability.

5. Test cost at the level of a workflow

Pricing pages often encourage the wrong comparison. The visible unit may be tokens, seats, requests or credits, but the business unit is a completed task.

A model with low input pricing can become expensive if it needs long prompts, repeated retries and multiple verification calls. A model with a higher per-token price may be cheaper overall if it completes the task in one pass and requires less human correction.

Include these variables in the calculation:

• average input and output size;
• retrieval context included in each request;
• tool calls and reranking steps;
• retries caused by invalid output;
• human review time;
• batch discounts or minimum commitments;
• deployment and monitoring costs;
• data retention and security requirements.

For a self-hosted open-source LLM, replace API pricing with hardware, inference optimization, engineering and maintenance costs. Open weights remove one vendor dependency; they do not remove operations.

The right model is not the one that wins the benchmark. It is the one that completes the expensive task with the least total friction.

Open-source, hosted and specialized models

The choice between a hosted frontier model, an open-source LLM and a smaller specialized model is not a morality play. Each option shifts the burden to a different part of the stack.

Hosted models usually offer the quickest onboarding. The API is documented, capacity is managed by the provider and new capabilities arrive without an internal retraining project. The trade-offs include variable pricing, vendor dependency, data-governance questions and less control over model updates.

Open-source models offer more control over weights, deployment and customization. They can be attractive when data must remain in a controlled environment or when high request volume makes dedicated inference economical. But the UI wrapper is not the hard part. Teams must manage serving, quantization, hardware capacity, monitoring, security patches and model upgrades. A model that is easy to download may be difficult to operate reliably.

Smaller models deserve more attention than they receive in launch coverage. They can offer lower latency, lower infrastructure cost and easier deployment for classification, extraction, routing or narrow drafting. They may not be the best general conversational agents, but enterprise systems rarely need every component to be a general conversational agent.

A sensible architecture may route tasks:

• a small model handles intent classification;
• a retrieval system finds relevant documents;
• a stronger model drafts the answer;
• a deterministic validator checks required fields;
• a human reviews high-risk cases.

That design makes evaluation more complicated, but it also makes the product more honest. The LLM is one component in a system, not a magical replacement for the system.

A practical evaluation framework for enterprise teams

Before selecting a model, turn the use case into a testable contract. Not a grand statement such as “improve productivity,” but a description of what the system must do, what it must never do and how a reviewer will decide whether it succeeded.

The most useful evaluation set usually includes four categories:

1. Normal cases. Representative requests from everyday work, including typical document sizes and user language.

2. Boundary cases. Long inputs, missing fields, conflicting documents, unusual formatting and requests at the edge of the intended scope.

3. Adversarial cases. Prompts that attempt to override system instructions, extract restricted information or force an unsupported answer.

4. Failure-recovery cases. Situations in which the user corrects the model, changes the output format or supplies additional evidence.

For each case, define a rubric that can be scored consistently. Depending on the product, it might include:

• factual correctness;
• completeness;
• citation accuracy;
• instruction adherence;
• format validity;
• refusal or abstention behavior;
• latency;
• cost;
• amount of human editing required.

Do not collapse everything into one weighted score too early. A model that leads by two points in style but fails the citation requirement is not a close second. It is the wrong model for that workflow.

Keep the test set versioned. Models change. Providers alter system behavior, routing, context handling and safety filters. A result from last quarter is not a permanent property of the API. Run regression tests after model updates and after changes to prompts, retrieval, chunking or output schemas.

The same discipline applies to published research. When reading an AI research paper or model release, look beyond the headline table:

• What was the prompt format?
• Were examples included?
• Was tool use allowed?
• Were tasks contaminated or publicly available?
• How many runs were performed?
• Were failures and abstentions counted?
• Did the authors compare identical inference settings?
• Is the result statistically meaningful for the task you care about?

That is not needless scepticism. It is basic product hygiene in a field where a benchmark decimal can influence procurement decisions.

The difference between capability and reliability

Capability asks whether a model can perform a task under favorable conditions. Reliability asks whether it will perform that task consistently enough for someone else to depend on it.

Those are different properties.

A model can write excellent code but fail to preserve a required API contract. It can summarize a report accurately but omit the single paragraph containing the legal exception. It can answer difficult questions correctly while producing a high hallucination rate on simple questions that lack enough context. It can score well on a reasoning benchmark and still struggle with arithmetic embedded in a badly formatted spreadsheet.

Calibration matters here. A reliable model should not only be right; its confidence and wording should help the user understand when verification is necessary. In enterprise settings, a cautious answer that points to missing evidence can be more valuable than a polished answer that fills the gap.

This is also why user preference platforms and academic benchmarks should be read together. Human votes capture conversational quality. Static tasks provide controlled comparisons. Internal evaluations reveal workflow performance. None of them, alone, guarantees real-world success.

The strongest selection process triangulates across all three and then applies a deliberately uncomfortable stress test using the organization’s own data.

Verdict: evaluate the product, not the leaderboard

Large language models are prediction engines wrapped in increasingly sophisticated products. Their value depends on the entire chain: model architecture, training, context, retrieval, tools, interface, pricing and recovery from failure.

If you are trying to understand what are large language models, start with the mechanism. If you are choosing one for enterprise work, move quickly past the definition. The decision will be made by latency, citation quality, output validity, total cost and the number of corrections users must perform after the model finishes talking.

Public benchmarks such as MMLU, HumanEval, GSM8K and Chatbot Arena remain useful signposts. They are not a procurement verdict. Benchmark saturation, inconsistent harnesses and the difference between preference and factuality make that shortcut increasingly unreliable.

My practical rule is simple: take the ugliest important workflow, build a representative evaluation set, measure the complete user journey and compare the total friction. The model that looks least impressive in a launch demo may be the one that quietly does the job best. That is usually the model worth deploying.