Generative AI & LLMs
AI Proof of Concept (PoC): How to Scope, Run, and Judge One
Written by Tehreem Fatima
Published 13 min read
An AI proof of concept, often shortened to AI PoC, is a short, controlled experiment that tests whether an AI idea works on your own data, at a quality and cost you can accept, before you commit to building it. It ends in a decision to build, change course or stop.
This guide is for teams that have an AI idea and need to know whether to put money behind it: a product owner weighing a new feature, an operations lead looking at automation, or a founder deciding what to build first. It follows one AI PoC from start to finish, with pass marks written before any results, an evaluation set made from your own cases, and a plain rule for deciding what happens next.
The sections run in the order of a real project: scope the question, run the experiment, judge the results and decide. Jump to the step you are on, or read straight through if you are planning your first AI PoC.
What Is an AI Proof of Concept?
An AI proof of concept (PoC) is a time-boxed experiment that tests whether an AI approach can solve one defined business problem using your real data, at acceptable quality, speed and cost. AWS describes a good one as a rigorous experiment that validates business value, data readiness, technical feasibility and risk mitigation, not a demo built to impress. Source [1]
AWS frames the purpose as a single question: “If we build this, will it matter?” Source [1] Microsoft puts it in similar terms, describing a PoC as a way to reduce implementation risk by checking technical feasibility and business value before full-scale development. Source [2]
The word is used loosely. Cloud guides define a PoC by its purpose, not by a fixed boundary with a prototype, a pilot or an MVP, so Table 1 gives the working definitions used in this guide. They are not an industry standard.
| Term | The question it answers | Who sees it |
|---|---|---|
| Proof of concept | Can this work on our data, and is it worth building? | The project team and the people who decide whether to fund it |
| Prototype | What would it look and feel like to use? | A few prospective users, for feedback on the design |
| Pilot | Does it hold up with real users doing real work? | A limited group of users, for a set period |
| MVP | Is this the smallest product people will actually use? | First customers or a first internal user group |
Term
Proof of concept
- The question it answers
- Can this work on our data, and is it worth building?
- Who sees it
- The project team and the people who decide whether to fund it
Term
Prototype
- The question it answers
- What would it look and feel like to use?
- Who sees it
- A few prospective users, for feedback on the design
Term
Pilot
- The question it answers
- Does it hold up with real users doing real work?
- Who sees it
- A limited group of users, for a set period
Term
MVP
- The question it answers
- Is this the smallest product people will actually use?
- Who sees it
- First customers or a first internal user group
When an AI PoC Is Worth Running
Run an AI proof of concept when the open question is whether AI can do the job well enough on your data, and the answer would change what you build or buy. If the outcome is already known, or a simple rule would do the job, skip the PoC and build or buy directly.
Microsoft warns that AI projects fail when organizations attempt implementations beyond their technical maturity or data availability. Source [2] A PoC is a low-cost place to find that out. Anthropic’s guidance goes further, recommending the simplest solution possible and more complexity only when needed, so check first whether the task needs AI at all. Source [3] Microsoft also suggests starting with internal, non-customer-facing projects to limit risk. Source [2]
If you are still deciding which use cases suit an agent, our guide to agentic AI use cases sorts examples by how much autonomy each one needs. Our generative AI integration article shows how to start with one workflow, confirm data access and decide who owns the model accounts.
How to Scope an AI Proof of Concept
Scope an AI proof of concept around one business question, one measurable goal and one dataset. Write the objective, the target and the measures that show it before choosing a model. AWS calls technical metrics that are disconnected from business impact a common failure point in generative AI projects. Source [1]
Pick One Problem and One Number
AWS gives a worked shape for this. Its example objective is improving customer support efficiency, its goal is reducing average handle time by 20%, and its measures include first-contact resolution, user satisfaction scores, hallucination rate, answer relevancy and latency. Source [1] The 20% is AWS’s illustration, not a benchmark, so set your own target from how the work performs today.
Table 2 shows three hypothetical AI PoC examples in that shape. The numbers to move are placeholders for your own.
| Question to test | Number to move (illustrative) | Simplest approach to try first |
|---|---|---|
| Can AI draft first replies to routine support tickets? | Share of drafts a reviewer accepts with light edits | Prompting, with the help-center text supplied in the prompt |
| Can AI pull fields from supplier invoices? | Share of fields extracted correctly, checked against a person's entries | Prompting with a fixed output format |
| Can AI answer staff questions from internal policy documents? | Share of answers an expert marks correct and supported by the source | Retrieval-augmented generation (RAG) over a small set of documents |
Question to test
Can AI draft first replies to routine support tickets?
- Number to move (illustrative)
- Share of drafts a reviewer accepts with light edits
- Simplest approach to try first
- Prompting, with the help-center text supplied in the prompt
Question to test
Can AI pull fields from supplier invoices?
- Number to move (illustrative)
- Share of fields extracted correctly, checked against a person's entries
- Simplest approach to try first
- Prompting with a fixed output format
Question to test
Can AI answer staff questions from internal policy documents?
- Number to move (illustrative)
- Share of answers an expert marks correct and supported by the source
- Simplest approach to try first
- Retrieval-augmented generation (RAG) over a small set of documents
Check the Data and Get Approvals
Data decides what is possible. Microsoft notes that data quality directly affects model performance and determines which use cases you can implement successfully. Source [2] RAND’s interviews with data scientists and engineers list a lack of necessary data as one of five root causes of AI project failure. Source [4]
Before any data goes into a system, AWS says the default should be synthetic or fully anonymized data. If real data containing sensitive or personal information is necessary, it says explicit approval from your information security and legal teams must be obtained before the data is ingested. Source [1] Settle where the data will be sent and what the provider keeps, which the risks section below covers.
Build an Evaluation Set Before You Build Anything
For generative AI PoCs, AWS says preparing an evaluation dataset replaces traditional training-data curation, and that reference answers should be validated by subject matter experts. Source [1] Anthropic suggests that 20 to 50 simple tasks drawn from real failures is a good start for evaluating an agent. Source [5] That is enough to catch obvious problems, but too few cases to prove a success rate with confidence.
Anthropic’s test for a good task is one where two domain experts would independently reach the same pass or fail verdict. Source [5] Include hard cases and past failures, not only the easy ones. OpenAI lists “it seems like it’s working” as an evaluation strategy to avoid. Source [6]
Choose the Simplest Approach That Could Work
AWS orders the options from simple to complex. Prompt engineering is “the simplest and fastest starting point for most PoCs,” RAG comes next when the model needs your knowledge, agentic designs come when the task needs reasoning, planning and tool use, and fine-tuning is a later consideration. For most knowledge-injection cases, AWS says RAG is a better starting point than fine-tuning. Source [1]
Anthropic’s guidance matches: add complexity only when it demonstrably improves outcomes. Source [3] So scope the PoC at the lowest rung that could meet your pass marks, and move up only if the evaluation shows it falls short. Our RAG architecture and retrieval quality guide covers the retrieval rung, and our guide to building agentic AI systems covers the agent rung.
Write Exit Criteria Before You Start
AWS says to set clear thresholds for quality, latency and cost, and to pivot or end the PoC if results do not meet them, so that poor performance does not lead to wasted effort or sunk costs. Source [1] Write the thresholds as pass marks: a minimum quality score on the evaluation set, a maximum response time, and a maximum cost per task, with the numbers taken from your business case.
Also name who makes the decision. Thresholds written after the results arrive tend to be fitted to the results, which defeats their purpose.
How to Run an AI Proof of Concept
Run an AI proof of concept in short, outcome-focused iterations, with feedback from business owners and subject experts from the first week. AWS advises validating only the core components before investing in full end-to-end integration. Source [1] Fix an end date at the start so the exit criteria have a deadline.
Set a Timebox You Can Defend
Choose the length from the question you are testing. It should be short enough that a failure is cheap, and long enough to rerun your evaluation set several times after each change. Agree a review date at which you decide whether to stop, and require a written reason to extend.
Microsoft suggests using PoC results to estimate timelines for each use case, adding 20 to 30 percent contingency to the initial estimates and planning for multiple development cycles. Source [2] That advice is about the full build that follows.
Keep a Feedback Loop Running
AWS recommends structured feedback from business leads and subject matter experts from the beginning, and says it can be as light as a shared spreadsheet. Source [1] Anthropic makes a related point about evaluations: the people closest to product requirements and users are best placed to define success. Source [5]
Keep a simple log with the case, what the system did, the reviewer’s verdict and the reason. The log doubles as new evaluation cases for the next round.
Track Cost and Speed From the First Run
AWS says to address data quality and retrieval issues before prompt tuning and to track unit economics early, because unit economics determine whether an application is financially viable at scale. Source [1] Record cost per task and response time on every run. They are the numbers a production budget will start from. Our AI agent development cost guide shows how to turn per-task figures into a monthly estimate, with a worked example.
How to Judge the Results
Judge an AI proof of concept by scoring its outputs on your evaluation set against the pass marks you wrote down before starting, and compare the score with how the work is done today. Use human reviewers to check the scoring, and read samples of real outputs instead of relying on the average alone.
Measure the Current Process First
To say AI helps, you need today’s number. Score how people, or the existing system, handle the same test cases using the same pass and fail rules. That is your baseline. The AWS example above only means something if you know the current average handle time.
Score With Graders You Have Checked
Anthropic says agent evaluations typically combine code-based, model-based and human graders, and that model grading often takes careful iteration to validate its accuracy. Source [5] OpenAI advises using human feedback to calibrate automated scoring, and names failing to calibrate automated metrics against human evaluations as an anti-pattern. Source [6]
Read the transcripts. Anthropic notes you will not know whether your graders work well unless you read the transcripts and grades from many trials. Source [5] It also says a zero percent pass rate across many trials is most often a sign of a broken task, not an incapable system. Source [5] If every attempt fails the same case, check the case and the grader before blaming the model. Our guide to LLM testing before launch goes deeper on test design.
What a Successful PoC Does Not Prove
A PoC result is evidence about the cases you tested. It is not a guarantee about everything that follows. AWS limits PoC testing to the core components before full end-to-end integration and says unit economics decide financial viability at scale. Source [1] A passing PoC may leave four things open:
- How the system behaves once it is fully integrated with your other systems.
- Cost and speed at production volume.
- Behavior on inputs unlike your evaluation set.
- Monitoring, governance and support after launch.
A successful demo or PoC is not proof of flawless production performance.
Decide: Go, Pivot, or Stop
The end of an AI proof of concept is a decision, not a demo: go, pivot or stop. Go if results meet your pass marks, pivot if they miss but you can name a change to data, scope or approach, and stop if the pass marks look out of reach. AWS frames the same choice as continue, pivot or terminate. Source [1]
Have the decision reviewed by someone with authority to release the system. NIST’s generative AI profile lists sharing the results of pre-deployment testing with relevant actors, such as those with release approval authority. Source [7] Table 3 lists what the PoC should leave behind, whichever way the decision goes.
| What to keep | Why it is worth keeping |
|---|---|
| Scorecard against the pass marks | Shows the decision rested on evidence and can be rerun |
| Evaluation set and grading rules | Becomes the regression tests if you build |
| Cost and response time per task | The starting point for a production budget |
| Data and approval notes | Records what data was used and who approved it |
| Rough build plan | Microsoft suggests adding 20 to 30 percent contingency to estimates drawn from PoC results Source [2] |
| Written decision | Names who decided, when, and why |
What to keep
Scorecard against the pass marks
- Why it is worth keeping
- Shows the decision rested on evidence and can be rerun
What to keep
Evaluation set and grading rules
- Why it is worth keeping
- Becomes the regression tests if you build
What to keep
Cost and response time per task
- Why it is worth keeping
- The starting point for a production budget
What to keep
Data and approval notes
- Why it is worth keeping
- Records what data was used and who approved it
What to keep
Rough build plan
- Why it is worth keeping
- Microsoft suggests adding 20 to 30 percent contingency to estimates drawn from PoC results Source [2]
What to keep
Written decision
- Why it is worth keeping
- Names who decided, when, and why
Why AI PoCs Stall
AI proofs of concept often stall for reasons that start before the model: a misunderstood problem, too little data, a focus on technology over users, weak infrastructure, or a task beyond what AI can do. RAND identified these five root causes in interviews with 65 experienced data scientists and engineers. Source [4]
Each cause has a counter in the steps above. Picking one problem and one number guards against a misunderstood problem. Checking the data first catches shortages early. Feedback from week one keeps the work tied to users. The list of what a PoC does not prove is a reminder to plan for infrastructure before the build, and exit criteria give you a way out when the task is too hard for AI.
In July 2024, Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs or unclear business value. Source [8] That was a forecast, not a measured result, so read it as a warning and not as a statistic.
For what comes after a successful PoC, our guide to enterprise AI agents covers rolling out in stages from one pilot to company-wide use, with a gate before each stage.
Data, Security, and Cost Risks to Settle First
Before a PoC sends any data to a model provider, check what the provider keeps and whether it trains on it, because terms differ by product and agreement. OpenAI and Anthropic both say API or commercial inputs are not used for training by default, and both describe retention of up to 30 days in standard cases. Source [9] Source [10] Source [11]
OpenAI says data sent to its API has not been used to train or improve its models since March 1, 2023, unless you explicitly opt in, and that abuse monitoring logs are kept for up to 30 days unless longer retention is required by law or to protect its services. Zero data retention needs prior approval. Source [9]
Anthropic says it deletes commercial API inputs and outputs from its backend within 30 days, unless you have agreed otherwise or an exception applies, such as a usage policy violation, where it keeps them for up to two years. Source [10] It also says it will not use commercial inputs or outputs for training by default, but may if a customer submits feedback or otherwise permits it. Source [11]
These terms change, so check your own agreement, the endpoint you will use and your account settings, and get security and legal sign-off before real data is used. Source [1]
How SensViz Can Help
SensViz is a custom AI and software development company with a regional office in Valencia, Lahore, and a UK office in London. Its custom AI development process runs from discovery through feasibility and data assessment, solution design and success criteria, then a prototype and evaluation step that tests the highest-risk assumptions with representative data before a larger production build. Source [12]
SensViz’s generative AI development page advises starting with one useful task before expanding into a broad AI program, and releasing to an agreed user group. Source [13] The same page says cost and timeline are confirmed during discovery before a delivery estimate or quote is given. Source [13]
If you are planning an AI proof of concept and want help scoping it, tell us about your project.
Frequently Asked Questions
Short answers to four questions that come up when planning an AI PoC.
Who Should Be on an AI PoC Team?
An AI PoC needs a business owner who can say what success means, subject experts who review outputs, and engineers who can build and test the system. AWS advises confirming the team can handle model evaluation, integration, performance tuning, safety measures and adversarial inputs, and gathering feedback from business leads and experts from the start. Source [1]
Should You Run a PoC Before Buying an Off-the-Shelf AI Tool?
Yes, in a lighter form. A vendor demo uses the vendor’s examples, so test the tool on your own evaluation set, with the pass marks and data approvals you would set for a custom build. If it misses them, you have learned that before signing a contract.
How Much Does an AI Proof of Concept Cost?
The cost is mostly people’s time, model usage fees and data preparation, and none of the guides cited here gives a typical price. The most reusable number a PoC produces is cost per task, measured on real runs. Our guide to what an AI agent costs to build and run shows how to turn it into a monthly estimate.
How Long Should an AI Proof of Concept Take?
There is no standard length. AWS asks for short, outcome-focused iterations but gives no number, so treat any range you see quoted as one firm’s estimate. Set the end date from your question and test set, then add 20 to 30 percent contingency when you estimate the full build from the results. Source [1] Source [2]
Sources
- AWS Prescriptive Guidance. Architecting a successful generative AI proof of concept. (opens in a new tab)
- Microsoft Learn. Plan for AI adoption, Cloud Adoption Framework. (opens in a new tab)
- Anthropic. Building effective AI agents. (opens in a new tab)
- RAND. The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed (RR-A2680-1, August 2024). (opens in a new tab)
- Anthropic. Demystifying evals for AI agents. (opens in a new tab)
- OpenAI. Evaluation best practices. (opens in a new tab)
- NIST. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1, July 2024). (opens in a new tab)
- Gartner. Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025 (press release, 29 July 2024). (opens in a new tab)
- OpenAI. Data controls in the OpenAI platform. (opens in a new tab)
- Anthropic Privacy Center. How long do you store my organization's data? (opens in a new tab)
- Anthropic Privacy Center. Is my data used for model training? (opens in a new tab)
- SensViz. Custom AI development service page. (opens in a new tab)
- SensViz. Generative AI development service page. (opens in a new tab)
