If you don't define success before the project starts, the agency will define it for you — after the fact, in their favor. "The model achieved 94% accuracy on the validation set" sounds like success until you realize that 94% accuracy on your imbalanced dataset means the model predicts the majority class almost every time.
Define success at the outset, in terms you control, measured against a baseline you establish before the agency touches anything. Here's how to do it.
Why Post-Hoc Success Definitions Are a Problem
The most common failure mode in AI projects isn't technical — it's definitional. The project delivers what the agency promised to build, but what was built doesn't solve the business problem. And because no one defined the business problem in measurable terms at the start, there's no clean way to hold anyone accountable.
This happens for predictable reasons:
Agencies optimize for what's measurable to them (model metrics, code delivery, deployment milestones) rather than what matters to you (business outcomes, behavior changes, cost reduction).
Business outcomes take time to manifest, so agencies deliver technical outputs and move on before the long-term results are clear.
Vague project scopes invite vague success criteria. If the SOW says "improve customer support efficiency," both parties can claim success or failure post hoc and both be technically right.
The solution is to establish three things before any work begins: what the baseline is, what the target is, and who is responsible for measuring the gap.
The Baseline Problem
You cannot measure improvement without knowing where you started. Yet a surprising number of AI projects begin without a clear baseline measurement.
Establish your baseline before the agency starts work. This means actually measuring the current state of whatever you're trying to improve:
For a customer support automation project: Measure the current volume of tickets, average first-response time, average resolution time, percentage of tickets resolved without escalation, and customer satisfaction score. Measure these over a period of 4–6 weeks to account for variability.
For a predictive model project: Measure the current accuracy of whatever process you're replacing (human judgment, simple rules, existing software). If your demand forecasting is currently done manually, document the actual error rate over the past 12 months.
For a document processing project: Measure the current time required to process a document, the error rate of current processing, and the cost per document (labor + overhead).
These baseline measurements serve two purposes: they give you a real target to improve against, and they give you leverage if the agency's solution fails to deliver measurable improvement.
Choosing the Right KPIs
The KPIs that matter for an AI project are not the model metrics. Model accuracy, F1 score, AUC — these are internal quality measures that tell you whether the model is working as intended. They are not business outcomes.
The KPIs that matter are the ones that would appear in a business case or a budget review:
Cost reduction metrics:
- Cost per processed document
- Labor hours per task type
- Error-related rework cost
- Support cost per customer
Throughput metrics:
- Documents processed per day
- Tickets resolved per agent per day
- Time from order to fulfillment
- Lead response time
Quality metrics:
- Error rate in the process the AI is handling
- Customer satisfaction score on AI-handled interactions
- Return rate or correction rate on AI-generated outputs
- Escalation rate from AI to human
Revenue metrics:
- Conversion rate on AI-ranked leads
- Revenue per customer (for recommendation systems)
- Reduction in churn rate (for retention AI)
Pick 2–4 KPIs that are directly tied to the business problem you're solving. Avoid the temptation to track everything — focus creates accountability.
Setting Targets
Targets should be:
Specific: "Reduce first-response time to under 4 hours" not "improve response time."
Time-bound: "...within 60 days of launch" not just "after implementation."
Achievable but not sandbagged: If the agency tells you that a 10% improvement is achievable, don't let them negotiate you down to 5% just because it's easier to hit. Push back on sandbagged targets. Ask: "What do your comparable case studies show for similar implementations?"
Agreed in writing: The success criteria should be in the contract or in an attached project specification document that is referenced by the contract. Verbal agreements on success metrics are not agreements.
A reasonable target range for most AI projects:
- Process automation (ticket routing, document classification, data extraction): 60–80% automation rate on targeted document/request types, with accuracy above 90% on automated decisions
- Predictive modeling (churn, demand, fraud): 15–30% improvement in the relevant outcome metric (e.g., 20% reduction in churn among high-risk customers who receive AI-flagged interventions)
- Search and recommendation: 15–25% improvement in the target engagement metric (click-through rate, conversion, session depth)
- Generative AI (content, drafting, summarization): 40–60% reduction in time-per-task for the targeted workflow
These ranges come from honest post-project reviews across hundreds of AI implementations. They are achievable for well-scoped projects. If an agency is promising significantly higher numbers, ask them to show you the comparable case studies.
OKRs for AI Projects
OKRs (Objectives and Key Results) provide a useful framework for structuring AI project success criteria, especially for longer engagements where business outcomes take time to materialize.
An OKR-structured AI project success definition might look like:
Objective: Reduce the cost and time burden of invoice processing in the accounts payable workflow.
Key Results:
- KR1: Automate extraction of invoice fields (vendor, amount, date, line items) with >95% accuracy by week 10
- KR2: Reduce manual processing time per invoice from 8 minutes to under 2 minutes within 30 days of launch
- KR3: Process 85% of invoices without human review within 90 days of launch
- KR4: Reduce invoice-related data errors by 70% (measured by downstream correction requests) by month 6
Notice that KR1 is a model metric (accuracy on a specific task), while KR2-4 are business metrics measured over time. The model metric is an early checkpoint; the business metrics are the real success criteria.
OKRs also help with phasing accountability across the project timeline. KR1 is the agency's responsibility before launch. KR2-4 become shared responsibility between the agency (if there are post-launch bugs or issues) and your team (if the failure is adoption or process integration).
Shared Accountability: What the Agency Owns vs. What You Own
A key mistake in AI project measurement is placing all accountability on the agency for outcomes that depend heavily on your organization's adoption behavior.
The agency is accountable for:
- The technical quality of the AI system (accuracy, reliability, performance at specified load)
- Delivering the specified integrations and workflows
- Providing documentation and training
- Fixing defects during the warranty period
You are accountable for:
- Driving adoption of the new system across your team
- Providing adequate user training and change management
- Ensuring the AI system is actually integrated into real workflows (not used occasionally alongside the old process)
- Reporting and escalating problems promptly rather than working around them
Shared accountability:
- Business outcome metrics, because they depend on both technical quality and adoption
- Data quality, because both parties need to ensure the system is working with good data
This split matters because it prevents both parties from blaming the other when outcomes are disappointing. Build it into your measurement framework explicitly.
The Milestone Measurement Framework
For most AI projects, success measurement should happen at defined milestones, not just at the end:
Milestone 1 (data assessment complete, ~weeks 2-3): Confirm that the data is adequate for the modeling task. If not, document the gap and agree on remediation. This is the earliest checkpoint where you can catch a fundamental problem before significant investment is made.
Milestone 2 (working prototype, ~weeks 6-8): Test the model on a representative sample of real data and measure preliminary performance against your baseline. This gives you an early indicator of whether the target KPIs are achievable and enough time to course-correct.
Milestone 3 (user testing, ~weeks 10-12): Pilot the system with a subset of real users in real conditions. Measure task completion time, error rates, and user feedback. This tests not just technical performance but usability and adoption friction.
Milestone 4 (launch + 30 days): First measurement of business KPIs in production. At this point, you have real data on the metrics that matter. Compare to baseline.
Milestone 5 (launch + 90 days): The most important measurement. 90 days of production data tells you whether the system is actually working and whether adoption is where it needs to be.
Building measurement checkpoints into the contract (and tying payment milestones to them) creates accountability throughout the project rather than just at the end.
Post-Launch Monitoring
AI systems are not static. Models trained on historical data encounter distribution shift — the real world changes, and the model's training data becomes less representative over time. Monitoring for this is not optional.
Establish these monitoring practices from day one of production deployment:
Performance monitoring: Automate tracking of the model's business metrics on an ongoing basis. If accuracy drops below a threshold (e.g., more than 5% from baseline), trigger a review.
Data drift detection: Monitor statistical properties of the input data to detect when the production data is diverging from the training data distribution. Most ML monitoring tools (Evidently, Arize, WhyLabs, or custom dashboards) can do this automatically.
Model confidence monitoring: Most classification models output confidence scores alongside predictions. A rise in low-confidence predictions (many cases below the model's threshold) is often the first sign of a model struggling with new patterns.
Human review sampling: Randomly sample 2–5% of AI decisions for human review on an ongoing basis. This catches both model errors and edge cases that automated monitoring misses.
Include requirements for monitoring infrastructure in your agency RFP and contract. The agency should deliver not just a model but a system that tells you when the model is struggling.
When You Don't Hit the Target
Sometimes the AI system underperforms. The measurement framework you've built tells you clearly whether you hit the target, and it also tells you what happened.
If you've defined shared accountability correctly, you can trace underperformance to its source: Is the model technically deficient (agency's problem)? Is adoption lower than expected (shared problem)? Is the input data different from what the model was trained on (shared problem)? Did the business context change after launch (not clearly anyone's problem)?
Clean measurement data enables clean conversation about remediation. Without it, you're arguing about impressions. With it, you're discussing facts.
Use the aiagencymap.com directory to find agencies experienced in your use case — the more comparable projects they've done, the more realistic their success benchmarks will be. And request those benchmarks in writing during the proposal phase, before any work begins.
Ready to Find the Right AI Agency?
Browse 700+ verified AI agencies. Filter by tech stack, industry, location, and client ratings.
Browse AI Agencies