Foundation models in generative AI: how to choose your stack

The first model that produces an impressive demo may become expensive at scale, awkward to deploy, difficult to customize, or unreliable in the exact workflow your users care about.

That is the practical answer to what are foundation models in generative AI: they are large, pre-trained neural networks designed to support many downstream applications across text, images, code, audio, and multimodal tasks. They are not finished products. They are the base layer beneath products, agents, search systems, copilots, and automation tools.

Choosing one is therefore not a beauty contest between model families. It is a stack decision. Performance matters, but so do latency, context handling, deployment control, integration effort, data exposure, and the cost of every request after the launch headlines have faded.

Defining the foundation: beyond the LLM hype

The term “foundation model” was coined by Stanford’s Institute for Human-Centered Artificial Intelligence in 2021. The idea was useful because it described a broader class of systems than language models alone: large models trained on broad datasets and then adapted to support multiple tasks.

An LLM is one type of foundation model, focused primarily on language. A vision model can also be a foundation model. So can a multimodal architecture that processes text and images, or other large neural systems adapted for generative and predictive applications. The relationship is straightforward:

• Every LLM can be considered a foundation model when it serves as a broad base for downstream applications.
• Not every foundation model is an LLM.
• Foundation models can span text, vision, code, and multimodal use cases.
• A model’s “general-purpose” label says little about how well it will perform in one specific production workflow.

This distinction is more than terminology. Teams often begin with an LLM because their product has a chat interface, then discover that the real problem is document retrieval, image understanding, structured extraction, or tool execution. The conversational UI is only the wrapper. The underlying workload determines the model requirements.

A support assistant that summarizes tickets has a different profile from a coding agent that edits repositories. A document-processing pipeline may care less about eloquent answers and more about stable JSON output, page-level grounding, and predictable latency. A marketing copy tool may tolerate some variation; an invoice extraction system cannot.

The model should follow the job, not the other way around.

A foundation model is not the product. It is the most expensive and consequential dependency inside the product.

The first practical mistake is treating a benchmark score as a product specification. Benchmarks are useful for narrowing the field, but they do not tell you whether a model will preserve your schema, respect your retrieval instructions, handle your domain vocabulary, or fail gracefully when the input is incomplete.

That is why the right definition of a foundation model for a business is operational: a reusable model layer that can be adapted to the company’s tasks, data, interfaces, and deployment constraints.

The enterprise shift toward open-weight architectures

The strongest argument for open-weight models is not ideology. It is control.

According to an IDC Spotlight Report cited in the available research, 61% of enterprises prefer open-source foundation models over proprietary options because of their customization and adaptability. That preference does not mean open models are automatically cheaper, more accurate, or easier to operate. It means enterprises increasingly value the ability to shape the system around their requirements rather than accept a fixed product boundary.

The terminology needs care here. “Open-source” and “open-weight” are often used interchangeably in product discussions, but they do not always describe the same thing. Open-weight models make trained parameters available for use or download, while licensing terms can still restrict redistribution, modification, or commercial deployment. The practical question is not whether a model is presented as open. It is what the license permits and what the release actually includes.

For a product team, open-weight deployment can create several advantages:

• The model can potentially run inside a company-controlled environment.
• Fine-tuning or task-specific adaptation may be possible.
• Inference behavior can be profiled without relying entirely on a vendor’s changing API.
• Data flows can be limited to infrastructure under the organization’s control.
• The serving layer can be optimized for a particular hardware configuration or traffic pattern.

But each advantage comes with an operational bill. A downloadable model still needs inference infrastructure, monitoring, version management, security controls, and people who understand how to keep the system available. A model that is inexpensive per token in a spreadsheet can become expensive when the team has to reserve accelerators, tune batching, manage memory, and absorb idle capacity.

Proprietary APIs reverse that trade-off. The provider handles much of the model infrastructure, and onboarding can be faster. The downside is dependency: pricing, rate limits, model availability, data policies, output behavior, and deprecation timelines are partly outside your control.

This is the point where “how to choose a foundation model” stops being a research question and becomes a product management question. The best model is the one that leaves the fewest unacceptable constraints in the rest of the system.

What to measure before choosing a model

Enterprise selection criteria in the cited research put performance first at 41%, followed by cost at 35% and computational efficiency at 29%. Those numbers reflect the obvious priorities, but each category hides several different measurements.

Performance is task-specific

A general benchmark can tell you that a model is capable. It cannot tell you whether it is capable enough for your workflow.

I would start by separating the tasks instead of assigning one overall quality score. For a typical generative AI product, the evaluation set may include:

1. Answer quality — Does the model produce useful, relevant responses, or merely fluent ones?

2. Grounding — Does it stay within the supplied documents or invent unsupported details?

3. Instruction following — Does it obey formatting, tone, length, and workflow constraints?

4. Structured output — Can it reliably return valid JSON, fields, labels, or function arguments?

5. Tool use — Does it call the right tool at the right moment and pass usable parameters?

6. Long-context behavior — Does performance hold when relevant information is buried inside a large input?

7. Failure behavior — Does the model admit uncertainty, request missing information, or confidently improvise?

8. Latency — How quickly does the user receive the first useful output, and how long does the complete response take?

The last two are routinely underweighted. A model with excellent prose but high latency can make an assistant feel broken. A model that is accurate in most cases but produces an unacceptable hallucination rate on high-impact tasks may require a different architecture, stricter retrieval, or human review.

The evaluation set should contain real examples from the intended product. Synthetic prompts are useful for coverage, but they often make models look cleaner than they are in production. Messy inputs expose the actual UX: missing fields, contradictory instructions, copied email chains, poor scans, ambiguous names, and users who do not phrase requests like benchmark authors.

Cost is more than the token price

A pricing page usually presents a clean number. Your production bill will not be clean.

The relevant cost may include:

• Input and output tokens.
• Embedding and retrieval services.
• Tool calls and external APIs.
• Fine-tuning or evaluation runs.
• GPU or accelerator capacity for self-hosted inference.
• Observability, storage, and logging.
• Human review of uncertain outputs.
• Engineering time spent handling failures and provider changes.

A cheap model that requires three attempts to produce a valid response is not necessarily cheap. A larger model that resolves a task in one pass may have a better total cost, especially when the alternative is a long chain of prompts, validators, retries, and manual intervention.

The right unit is often cost per successful task rather than cost per request. That calculation should include failure rates and retries. It also needs to reflect traffic shape. An API can be attractive for unpredictable demand because the team pays for usage. Self-hosting can become more efficient at sustained volume, but only when the infrastructure is utilized consistently and the organization can operate it well.

Computational efficiency affects the user experience

Computational efficiency is not just an infrastructure concern. It appears directly in the interface as latency, queueing, timeout behavior, and concurrency limits.

A model may deliver high quality but struggle with simultaneous requests. Another may be less capable on complex reasoning while remaining fast enough for autocomplete, classification, or high-volume extraction. These are different products, even if both are described as generative AI.

Measure at least:

• Time to first token or first useful result.
• Full response latency.
• Throughput under realistic concurrency.
• Context length at which latency becomes unacceptable.
• Memory requirements for the chosen quantization and serving setup.
• Behavior during traffic spikes.
• Rate-limit and timeout failure patterns.

A smooth onboarding flow can be ruined by a slow second screen. Users do not experience model quality in isolation; they experience the delay between action and useful feedback.

Foundation models versus proprietary APIs and open-weight models

The choice is often presented as a binary decision. In practice, teams can use a hybrid stack: a proprietary model for complex requests, a smaller open-weight model for classification or routing, and deterministic software for tasks that should not be delegated to a language model at all.

Still, the trade-off is easier to understand in a direct comparison.

Open-weight does not eliminate vendor dependency. Teams may still depend on a cloud GPU provider, an inference platform, a model distributor, or a specific software stack. It changes where the dependency sits.

The same applies to proprietary services. They are not necessarily a bad choice for sensitive or serious enterprise workloads. A managed API can offer mature access controls, regional deployment options, monitoring, and contractual commitments that would take a smaller team months to recreate. The right question is whether those controls satisfy the product’s actual requirements.

For selecting foundation models for business, I would ask four blunt questions:

• What data is sent to the model, and what data must never leave our environment?
• Do we need to modify the model, or is retrieval and prompt design sufficient?
• Can we tolerate a provider changing model behavior or pricing?
• Do we have the team to operate inference reliably at the required scale?

If the answer to the last question is no, self-hosting is not automatically a sign of technical maturity. It may simply move the risk from the API contract to the operations team.

Context windows: the feature that creates false confidence

Long context is useful, but it is not the same as reliable comprehension.

A larger context window lets a system accept more documents, conversation history, code, or retrieved material in one request. That can simplify orchestration. It can also increase cost, latency, and the amount of irrelevant material competing for the model’s attention.

For Gemini models, a rough sizing reference is that 100 tokens correspond to approximately 60 to 80 English words. That is only a planning metric, not a guarantee of output quality. Token counts vary with language, formatting, code, identifiers, and punctuation. A page full of tables or source code can consume context differently from ordinary prose.

Context planning should therefore begin with the information architecture, not the maximum advertised window. Ask:

• Which parts of the input are actually relevant to the current task?
• Can the system retrieve only the necessary passages?
• Does the model preserve instructions when the prompt becomes large?
• What happens when the user uploads several documents at once?
• Are conversation histories summarized, truncated, or passed in full?
• Does latency remain acceptable at the context size users will routinely generate?

A large context window can reduce the need for aggressive chunking, but it does not remove the need for retrieval quality. If the wrong evidence is selected, adding more evidence rarely fixes the answer. It can make the failure harder to diagnose because the model has more text from which to construct a plausible response.

This is also where UI design becomes part of model evaluation. If users can upload unlimited files, the interface is encouraging oversized prompts. If the product clearly shows which sources were used, users and reviewers can detect grounding failures faster. Good model behavior and good product design are connected.

The model is only one part of the stack

Teams often spend weeks comparing foundation models and then underinvest in the layers around them. That is backwards. The model provides capability; the surrounding system determines whether that capability becomes dependable.

A production stack may include:

• A routing layer that sends simple tasks to smaller, faster models.
• Retrieval that selects relevant internal documents.
• Prompt templates and version control.
• Output schemas and validation.
• Tool permissions and execution safeguards.
• Caching for repeated or stable requests.
• Tracing for prompts, outputs, latency, and failures.
• Human review for high-impact decisions.
• A fallback path when the primary model is unavailable.
• A test suite built from real user cases.

The UI wrapper matters as much as the API integration. Does the application show progress during a long generation? Can users correct a bad answer without starting over? Are citations visible where they matter? Is the system clear about whether it is drafting, extracting, recommending, or taking an action?

These details influence adoption more directly than a minor improvement on a general benchmark. A product with slightly lower model quality but excellent recovery flows can outperform a technically stronger system that leaves users stranded after an error.

The winning model is rarely the one with the best demo. It is the one that fails in the least expensive and least confusing way.

That principle should shape evaluation. Do not score only successful outputs. Track the cost of failure, the time required to detect it, and the path available to recover.

A practical selection process for a product launch

A disciplined process does not need to be bureaucratic. It needs to prevent the team from committing to a model before understanding the workload.

Start with a narrow task definition

“Build an AI assistant” is not an evaluation task. “Classify incoming support tickets into eight categories and produce a two-sentence draft response” is closer.

Write down the exact input, output, acceptable error, and user action that follows. If the model’s output triggers a payment, changes a record, sends a message, or makes a recommendation, define where human confirmation belongs.

Create a representative evaluation set

Use examples that reflect the actual distribution of requests. Include short and long inputs, clear and ambiguous instructions, incomplete records, domain-specific language, and adversarial or malformed content.

Keep a private holdout set for final comparison. Otherwise, prompt tuning can make the system look better on the examples used during development without improving its general behavior.

Compare model families by workflow

Run the same task through multiple candidates, but do not reduce the result to a single score too early. Record quality, latency, output validity, retries, and cost per successful task.

A useful table for internal decision-making might include:

The term “hallucination rate” needs a defined test. It should not mean a reviewer’s vague impression that an answer feels wrong. Decide what counts as an unsupported statement, how it will be labeled, and whether the risk is equally serious across all task types.

Test the deployment route early

If the shortlist includes an open-weight model, put it into the intended serving environment before making a final choice. Model-card claims and local inference performance are different things. Quantization, hardware, batch size, context length, and concurrency can materially change the result.

If the shortlist includes a proprietary API, test regional availability, quotas, error handling, response streaming, and versioning. A clean single-user demo does not reveal how the integration behaves when the provider throttles traffic or changes a model alias.

Model the cost at realistic traffic

Use more than one scenario:

• Low-volume launch with unpredictable demand.
• Normal daily traffic.
• A successful growth case.
• A spike caused by a campaign, outage, or seasonal event.
• A fallback scenario in which requests move to a more expensive model.

For an open-weight deployment, include idle capacity and operational labor. For an API, include retries, long contexts, tool calls, and observability. The final estimate will still be imperfect, but it will be less likely to collapse after launch.

What should not be delegated to the model

The excitement around foundation models can produce a familiar failure: using a probabilistic system where a deterministic rule would be clearer, cheaper, and easier to audit.

A model may help interpret an email, but a conventional parser may be better at validating a date. A language model may extract fields from a document, but a schema validator should decide whether those fields are structurally acceptable. A model can recommend an action, while application code controls whether that action is permitted.

The more consequential the output, the more the model should sit inside a constrained workflow rather than act as the workflow.

This does not require a theoretical discussion about AI safety. It is basic product engineering. Limit permissions. Validate outputs. Separate drafting from execution. Log decisions. Make failures visible. Give users a way to correct the system without losing their work.

The model’s role should be proportionate to its reliability on the specific task. A general-purpose foundation model can be remarkably capable and still be the wrong component for an exact calculation, a policy gate, or an irreversible transaction.

The final choice: capability versus control

There is no universal winner in the foundation models versus proprietary APIs debate. Managed proprietary models usually offer the fastest route to a working product. Open-weight architectures can offer greater control and adaptability, especially when the organization has the infrastructure and expertise to operate them. Hybrid designs often make the most practical sense: use different models for different workload tiers, and keep deterministic systems in charge of rules and execution.

The selection criteria are clear enough. Performance comes first, followed by cost and computational efficiency, but the real decision also includes latency, context behavior, deployment security, licensing, customization, and the quality of failure recovery.

The key is to evaluate the model as a user-facing dependency, not as an isolated research artifact. Ask what happens after the first click, under a long prompt, during a traffic spike, when the answer is wrong, and when the vendor changes the terms. That is where the stack earns or loses its place in a product.

My verdict is direct: start with the fastest credible path to representative user testing, but do not confuse fast onboarding with a final architecture. Prove the workload first. Then choose the level of model control your data, economics, and operational team can genuinely support. A foundation model should expand the product’s capabilities without becoming the product’s single point of failure.