How to choose the right LLM for your application

There is no universally best language model. There is only a model-and-system combination that fits a defined workload better than the alternatives you tested. A leaderboard can suggest candidates, but it cannot know your input distribution, acceptable error rate, privacy boundary, response-time target, or cost of a failed answer.

Write a task contract, remove candidates that violate hard constraints, and run the survivors on representative examples. The FIT-GAP framework covers Fit, Invariants, Tests, Governance, Availability, and Price.

1. Fit: define the job before the model

“We need AI for customer support” is not a task. Split the workflow into observable operations:

• classify the request;
• retrieve relevant policy passages;
• draft an answer grounded in those passages;
• detect missing account information;
• route a risky case to a person.

Each operation may need a different model or no model at all. A deterministic rule can classify a known account state more cheaply and predictably. Retrieval can supply policy text. A capable model can draft the explanation. Human approval can remain mandatory for refunds above a threshold.

Write a one-sentence contract: “Given the customer message and up to four approved policy passages, produce a polite answer that cites the passage identifiers, asks for missing data, and never promises an unapproved remedy.” Now success can be observed.

2. Invariants: establish non-negotiable gates

An invariant is a requirement a candidate must meet before quality trade-offs matter. Typical gates include:

• deployment in an approved region;
• a contract suitable for the data being processed;
• a maximum request or response latency;
• a minimum context requirement for realistic inputs;
• required structured-output or tool-use support;
• an acceptable license for weights and generated use;
• an operational path for abuse handling, access control, and deletion.

Treat a vendor statement as a lead to verify, not as the completed assessment. Record the source, date, exact model identifier, and account tier because product behavior and limits can differ by version or configuration. For open-weight candidates, review the actual model license and the licenses of required runtimes rather than relying on the label “open source.”

3. Tests: build a small evaluation that represents reality

The HELM project demonstrates why model evaluation benefits from multiple scenarios and metrics. Your internal evaluation can be much smaller, but it should preserve that principle.

Start with 30 to 100 examples if that is all you can curate carefully. Include:

• common inputs sampled from the real workflow;
• difficult but valid inputs;
• incomplete or contradictory information;
• instructions embedded in untrusted text;
• cases that require abstention or escalation;
• edge formats, languages, or lengths your users actually send.

Keep expected properties beside each example. Some tasks have an exact answer. Others need a rubric: groundedness, required facts, forbidden claims, tone, or citation correctness. Hide a portion of the set while tuning prompts so that repeated changes do not simply optimize for familiar examples.

Score the complete system, not a blank chat window. Use the intended system instruction, retrieval pipeline, tool permissions, output schema, and retry policy. Repeat nondeterministic cases. Save raw outputs and configuration so a later model update can be compared to the same baseline.

4. Governance: connect model behavior to consequences

The NIST AI Risk Management Framework organizes work around Govern, Map, Measure, and Manage. A practical model decision can mirror that lifecycle:

• Govern: name the owner, approver, prohibited uses, and review cadence.
• Map: document users, affected people, data, dependencies, and plausible failure consequences.
• Measure: run task evaluations, security tests, privacy checks, and operational load tests.
• Manage: add controls, choose escalation rules, monitor drift, and retain a rollback path.

The required control should rise with the cost of error. A private brainstorming assistant may tolerate a visibly uncertain draft. An automated action affecting access, money, health, employment, or legal rights needs a different risk decision and often qualified human authority. Model quality does not erase application responsibility.

5. Availability: measure the service you can actually operate

Quality on a saved test set is only one dimension. Measure end-to-end latency at realistic concurrency, rate-limit behavior, timeout frequency, regional availability, and recovery after a provider error. For a self-hosted model, include model loading, queue time, accelerator availability, memory headroom, deployment upgrades, and on-call capability.

Design a failure response before launch. Can the product retry safely? Can it fall back to a smaller model without changing the meaning of the task? Should it queue work, present a non-AI workflow, or stop? Silent fallback is dangerous when the replacement has not passed the same minimum gates.

6. Price: compare total cost, not token price alone

Build cost from the workload:

monthly requests × average input × average output × model rate

Then add retrieval, storage, reruns, failed requests, moderation, observability, engineering, and review time. For self-hosting, add accelerators, idle capacity, orchestration, power, deployment labor, security maintenance, and the cost of meeting peak demand. Use a low, expected, and high scenario instead of a single optimistic number.

Cost per successful task is more meaningful than cost per request. A cheaper model that triggers twice as many retries or corrections may cost more in practice. Conversely, a strong small model on a narrow classification task may outperform a flagship model economically without reducing acceptable quality.

A worked FIT-GAP example

Suppose a team extracts invoice fields from email attachments. Its invariant is strict JSON matching a schema; documents must remain in an approved processing region. The test set contains clean invoices, scans, multi-page files, missing totals, handwritten notes, and malicious text in a footer. Measures include field-level accuracy, schema validity, unsupported values, latency, and manual-review rate.

Candidate A is strongest on messy scans but has higher latency. Candidate B is cheaper and nearly equal on clean files but misses handwritten adjustments. Instead of declaring one winner, the team routes ordinary documents to B and uncertain scans to A, then sends low-confidence totals for human confirmation. The winning design is a routing policy, not a logo.

The decision record

Complete this one-page record before signing off:

Choose the simplest system that clears the gates with enough margin. Keep the evaluation and decision record. When a provider changes a model or a new candidate becomes fashionable, rerun the same workload instead of restarting the argument from marketing claims.