Begin with the decision, not the prototype
At the end of a two-week AI pilot, there should be a written go or no-go decision. If the planned conclusion is a demo meeting, the pilot has already been scoped incorrectly.
The decision names the workflow, user, acceptable failure boundary and business outcome. It also states what result stops the project. Without a stopping condition, every weak result becomes another prompt-tuning request.
The NIST AI Risk Management Framework likewise treats context, expected benefits, costs and risk as inputs to an initial go or no-go decision. The pilot is not there to prove a model can produce an impressive answer. It tests whether the proposed system works inside a business process.
Build the test before the feature
Start with cases from real work. Include ordinary inputs, awkward inputs and cases that should be refused or sent to a person. Remove personal information when production data is not approved for the experiment, but preserve the difficulty of the task.
Each case needs an expected outcome or a clear scoring rule. Some tasks have exact answers. Others need domain reviewers to compare outputs for correctness, completeness, tone or safe handling. OpenAI's evaluation API reflects the same structure: an evaluation combines a data source with explicit testing criteria and graders.
Run the current manual process against the same set. That baseline matters. An AI result can look clever and still be slower, less reliable or more expensive than the spreadsheet and experienced operator it is meant to replace.
Do not reserve evaluation for the end. The initial set should run whenever the prompt, model, tools or retrieval logic changes.
Build one vertical slice
The pilot should follow one case from input to useful output through the real boundaries of the future product. If the idea reads support requests, the slice should retrieve the relevant customer context, draft the response, record its sources and hand uncertain cases to an operator. A prompt pasted into a chat window is not that slice.
Use real interfaces where access is available, but keep consequential actions read-only or behind explicit approval. Log the input, retrieved context, model configuration, output, tool calls, latency, usage cost and reviewer decision. This record explains why a result failed and makes later comparisons possible.
The pilot must expose the integration path: required credentials, permission scopes, unavailable fields, rate limits and each system owner. A model can pass quality tests while the project fails because necessary data cannot be accessed safely.
The pilot that proves nothing
The common failure mode is a curated demo built from friendly examples. Someone adjusts prompts while looking at the answers, removes cases that behave badly and presents the strongest output. The room sees capability, but nobody learns reliability.
Another empty pilot hides humans behind the curtain. If an expert cleans every input, selects the right document and repairs the output before it appears, the system being evaluated includes that expert. Their time and judgment must be measured, not erased.
Changing several variables at once is another failure. A new model, prompt, retrieval setup and tools may improve the result, but the team cannot tell why. Controlled comparisons are more useful.
A failed pilot can still be valuable. It may show that retrieval is weak, the source data is unfit, the task needs human judgment or the economic case does not close. Reframing that evidence as almost ready only converts a cheap failure into an expensive project.
Make the call this week
Write the decision statement and stopping condition. Assemble the evaluation cases with the people who perform the work. Record the manual baseline, then run the same set through the smallest end-to-end slice.
Review errors by type, not only as an average score. Separate wrong answers, unsupported answers, unsafe actions, missing context and cases correctly handed to a person. Add measured latency, model and tool cost, integration blockers and the amount of human correction required.
The note should say go, no-go or change the problem. It should list the evidence, remaining risks and the next investment required. If nobody is willing to make that call, the pilot did not answer the question. It only delayed it.