Generative AI & LLMs

LLM Testing: What to Check Before Launching an AI Application

Written by Tehreem FatimaReviewed by Umaid Asim

Published 17 September 20268 min read

LLM testing workflow from representative cases through application checks to a release decision, with production failures feeding future tests. — Figure 1. Review failures before making a release decision. Production feedback should improve the next test run.

LLM testing checks whether an application built around a large language model completes its intended tasks accurately, respects access boundaries and handles failure acceptably. It combines ordinary software tests with evaluation of generated answers, retrieved evidence and actions. The aim is to find consequential failures before users encounter them.

A convincing demonstration is a weak basis for a launch decision. An assistant might answer familiar questions correctly but invent a policy when information is missing, expose another customer’s records or claim that an action succeeded when nothing changed. Each problem needs a different check.

This article focuses on testing LLM applications, rather than using AI to generate tests for conventional software. It draws on current primary documentation, including IMDA’s January 2026 testing guidance, which distinguishes testing application outputs from investigating the components behind them.Source [1]

What should LLM testing cover

Start with the task and the consequence of getting it wrong. A drafting assistant and an agent allowed to modify customer records should not have identical acceptance criteria. Cover the relevant failure modes below, then test the complete user journey.

Useful answers and supported claims

Check whether the response answers the question, includes essential conditions and avoids invented details. Use approved reference material where a correct answer can be established. For an open-ended draft, use a rubric that describes what a useful result must contain instead of requiring one exact sentence.

For retrieval-augmented generation, or RAG, separate retrieval quality from answer quality. Did the application find the necessary source? Did the answer accurately use it? Ragas defines faithfulness as support for the response’s claims in the retrieved context. That is not an independent guarantee that the source itself is correct or current.Source [2]

Test missing evidence and conflicting document versions too. A policy assistant should identify uncertainty or request clarification when it cannot establish which rule applies. Our RAG architecture and retrieval quality guide explains the retrieval decisions behind these checks.

Structured outputs and connected systems

A valid JSON response can still contain the wrong customer ID or an unsupported amount. Validate required fields, types and allowed values, then check business meaning against the relevant records. Test the parser and application code independently of the model.

End-to-end testing should follow the request through the interface, retrieval, model response and downstream integration. AWS’s preproduction guidance recommends combining unit, integration and end-to-end tests with evaluations of generated output. A good model score does not replace those software checks.Source [3]

Data access and adversarial inputs

LLM security testing should include attempts to cross account boundaries, disclose protected information and override instructions through retrieved content. OWASP distinguishes direct prompt injection from indirect instructions embedded in external material. It recommends least-privilege access, controls implemented outside the model and human approval for high-risk actions.Source [4]

Use a sandbox and synthetic sensitive records for these tests. Verify that an unauthorised request is blocked by the application, not merely refused in the assistant’s wording. If the product accepts images or documents, include hostile instructions in those inputs as well. No collection of prompt injection tests proves that every future attack will fail.Source [4]

Agent actions and operational limits

When an agent can use tools, examine the result in the connected system. A message saying that a ticket was created is not evidence that the ticket exists. Anthropic’s 2026 evaluation guidance explicitly separates an agent’s transcript from the final state of its environment.Source [5]

Also test timeouts, unavailable dependencies and interrupted conversations. Measure the full request’s latency and cost, including retries and tool calls, rather than only the first model response. Put limits on repeated actions and check what the user sees when the application cannot complete the task. These are practical release criteria, not reasons to pursue one universal benchmark score.

Build a test set around real work

Useful LLM test cases describe a situation, the expected behaviour and the evidence needed to judge it. Begin with the workflows the product is intended to support, including cases where it should decline or ask a question.

For each case, record the input, relevant conversation history, user permissions, source version and expected outcome. Include ordinary requests, ambiguous wording, missing information, previous failures and deliberate misuse. If users work in different languages or regions, include those variations where they change the task.

IMDA’s starter kit treats risk selection and acceptance thresholds as context-dependent. Its coverage includes inaccuracy, bias, undesirable content, data leakage and adversarial prompts. Select the risks relevant to your application rather than assuming that a generic benchmark represents your users.Source [1]

Consider a hypothetical support assistant that reads a company’s return policy and can prepare, but not approve, a return request. The following examples illustrate how to turn that boundary into tests. They are not SensViz customer results.

Test situations for a hypothetical support assistant, with expected behaviour and the evidence to inspect
Test situation	Expected behaviour	Evidence to inspect
Customer asks about a return window	Uses the applicable policy and states its conditions	Answer, citation and policy version
Purchase date is missing	Requests the date before deciding eligibility	Conversation and proposed decision
Search finds no applicable policy	Explains the limitation and routes for help	Retrieved passages and response
Customer requests another account's order	Access is denied without exposing the order	Authorisation result and visible output
Retrieved text asks the assistant to ignore its rules	Embedded instruction does not cause an unauthorised action	Tool calls and application access checks
Request is retried after a timeout	Does not create duplicate return requests	Request identifiers and sandbox records

Illustrative test cases for a hypothetical support assistant. They show how a permission boundary becomes a test, and are not SensViz customer results.

Scroll the table sideways to see every column.

Keep sensitive production data out of test exports unless its use is authorised and appropriately protected. Record how test data was obtained. Retain a separate set of cases for final validation so prompt adjustments are not judged only on examples used to make those adjustments.

Match each check to the right evaluator

Use explicit checks where the answer is objective and human judgement where context matters. One overall score can hide a serious failure in a less common workflow.

Deterministic checks for clear rules

Use code to verify schema compliance, permissions, numerical calculations and whether required records exist. Keep these checks narrow and explainable. For example, a return request must belong to the authenticated customer regardless of how persuasive the assistant’s reply sounds.

Avoid strict text matching for answers that permit several correct phrasings. Reserve it for genuinely exact requirements, such as an allowed status value or a reference identifier.

Model graders for meaning

An LLM-as-a-judge evaluator can assess qualities such as relevance or instruction following against a written rubric. LangSmith supports model-based, code-based and human evaluation, as well as comparisons between application versions. These methods serve different purposes.Source [6]

Before relying on a model grader, compare its judgements with expert-labelled examples and inspect disagreements. Give it clear criteria, including examples of unacceptable answers. Recheck its behaviour when the judge model or rubric changes. Treat its score as fallible evidence, especially for ambiguous or consequential decisions.Source [5]

Human review for consequential failures

Have someone familiar with the domain inspect failures, disputed grades and a sample of passes. Ask whether a technically supported answer is actually useful. An answer can cite the right policy while omitting an exception that changes what the customer should do.

If reviewers disagree, clarify the expected behaviour before treating the disagreement as a model defect. Testing often reveals an unclear product requirement that better prompting alone cannot solve.

Three evaluation methods: code checks for explicit rules, model graders for rubric-based meaning and human review for context and consequences. — Figure 2. Match the evaluator to the question. Inspect disagreements rather than averaging away important failures.

Compare changes before approving a release

Run the current application and the candidate version against the same versioned cases. Keep the prompt, model configuration, retrieval setup and evaluator settings identifiable so the comparison can be investigated later. AWS recommends versioning the application stack and evaluation data together and using evaluation checks in delivery pipelines.Source [3]

Use repeated trials for variable behaviour rather than drawing a conclusion from one successful run. Review outcomes by task and failure severity. Anthropic describes multiple trials as a way to account for variability in agent evaluations; a successful attempt and dependable repeated success answer different questions.Source [5]

For the hypothetical support assistant, we would treat cross-account disclosure, unapproved actions and duplicate requests as release blockers. We would review an awkward but correct answer differently. This is an illustrative prioritisation, not a universal acceptance standard.

Set the release criteria before examining the candidate’s results. Record unresolved failures, their owners and the conditions under which a limited rollout is acceptable. Prepare a rollback or disable path for the affected feature. Passing the selected tests supports a decision within the tested scope; it does not certify the application as error-free.

Keep testing after launch

Production introduces requests that were absent from the original dataset. Monitor failures and user feedback alongside latency and cost, then turn reproducible problems into regression tests. LangSmith documents this connection between online evaluation of live interactions and offline experiments on curated datasets.Source [6]

Limit access to traces, redact unnecessary sensitive information and define retention rules. Review a sample of apparently successful interactions too, because a lack of complaints is not proof that the answer was correct.

Retest when prompts, models, data sources, retrieval settings or connected tools change. For systems that choose their own search steps, the distinction explained in agentic search vs RAG helps identify which intermediate decisions also need checking.

Where tools and SensViz fit

Choose testing software after deciding what evidence the team needs. LangSmith can organise datasets and experiments; Ragas provides metrics for evaluating grounding in retrieved material. Neither removes the need to define the expected behaviour. For a wider comparison, see our AI evaluation tools for production article.Source [2]Source [6]

SensViz’s generative AI and LLM development services cover application design, retrieval, integrations, output validation and evaluation. The service describes representative and difficult test cases, defined criteria and review controls matched to the use case. Agree the testing scope and ongoing responsibilities for the individual project.

If you are planning an AI feature, discuss the workflow with SensViz. Bring examples of acceptable answers, known failure cases and the actions the system must never take. Those details give the team a more useful starting point than a target score alone.