Unreliable data sinks more AI projects than flawed model design ever will. What makes it dangerous is how gradually that unreliability takes hold.
You’re a few weeks into an AI project and your annotation vendor has delivered the first batches exactly as promised. Then the cracks begin to show.
- Deliveries start arriving late.
- Annotation accuracy begins to decline.
- The same annotation task receives different labels from different annotators.
- Every issue turns into another discussion about who’s responsible for fixing it.
Taken individually, none of these problems looks serious. But they compound as annotation volumes grow and projects become more complex, until deadlines slip, costs increase and your AI team spends more time resolving data problems than building the product.
At that point, the tempting conclusion is that you chose the wrong vendor. More often, the vendor is perfectly capable and the expectations were never clearly defined. The Service Level Agreement (SLA) that should have specified quality, turnaround times, reporting, revisions and accountability instead left those terms open to interpretation, and two teams working from different assumptions will eventually produce different results no matter how competent both are.
This article walks through the questions to ask a data annotation vendor before signing an SLA. Asked well, these questions will reveal whether a vendor can deliver high-quality data on time and continue doing so as your project scales.
What is a Data Annotation SLA?
A Data Annotation Service Level Agreement (SLA) is a formal agreement that defines the measurable standards an annotation vendor commits to meeting. Instead of relying on assumptions, it specifies what successful delivery looks like before the first batch is annotated.
That specification covers more than delivery targets. A well-written SLA also does the following:
- It establishes accountability for both the client and the vendor.
- It defines how performance will be measured using agreed metrics and quality targets.
- It explains what happens if service levels aren’t met, including how issues are reported, escalated and resolved.
The accountability point deserves particular attention. Many annotation vendors now operate as orchestration layers, drawing on specialist freelancers and partners to produce bespoke training datasets. Your SLA should state that the vendor takes full responsibility for the quality of every dataset delivered, regardless of who performed the work.
Where these terms are missing, even experienced annotation teams can interpret the same requirements differently. The result is often inconsistent annotation, disputed responsibility and the very delays described earlier.
The questions that follow test whether a vendor can support each of these commitments with real process. Accuracy is the natural starting point, because every other clause in the SLA exists to protect it.
1. What Annotation Accuracy Can You Guarantee?
Annotation accuracy directly determines how well an AI model performs, which makes it the first commitment to examine. Ask what accuracy level the vendor can consistently achieve for projects like yours, and expect the answer to reflect your project’s complexity. Simpler tasks may target around 95%, while specialised work such as medical imaging or 3D point cloud annotation calls for higher standards.
Pay attention to who stands behind the number. A credible guarantee is backed by dedicated QA capacity, whether in-house QA teams, data scientists or solutions architects, rather than annotators checking their own work.
The SLA should then define what happens when accuracy falls below the agreed threshold, including rework obligations, additional reviews and resolution timelines. A guarantee without corrective actions attached carries little weight.
An accuracy target is also only as trustworthy as the method behind it, which is where the next question comes in.
2. How Will You Measure Annotation Quality?
A promised accuracy rate means very little unless the vendor can explain how it’s achieved. Ask them to walk you through their quality assurance process, from the moment an annotation is completed to the point it reaches your team.
Start with the basics:
- Is every annotation reviewed before delivery, or only a sample?
- Who performs the review, a dedicated QA team or another annotator?
- At what stages of the workflow are quality checks carried out?
For projects where accuracy is critical, confirm the vendor applies additional validation such as double annotation or expert consensus reviews. A mature QA process should be documented and repeatable, not dependent on individual reviewers.
Quality only tells half the delivery story, though. Data that arrives accurate but late creates problems of its own.
3. What Turnaround Time Can You Commit to?
Delayed annotations stall model training and push back product releases. The cost runs deeper than a slipped date, though, because models improve fastest when they receive feedback rapidly. Each delivered batch feeds training; evaluation feeds the next round of annotation.
The tighter that loop runs, the faster the model compounds its gains, so a vendor with slow turnaround doesn’t simply delay your schedule; it slows the rate at which your model can improve at all.
Ask when the vendor will deliver your data and how they arrived at that timeline. A realistic commitment accounts for dataset size, annotation complexity, available annotators and QA requirements, and larger projects should be structured in milestones so progress can be reviewed in sets.
Also establish how urgent requests are handled and how the vendor scales capacity as your project grows. Be cautious of anyone committing to aggressive timelines without explaining how quality holds as workloads increase.
Speed and accuracy commitments still leave one operational reality unaddressed. Even with both in place, annotators will sometimes disagree about the data itself.
4. How Will You Handle Annotation Disagreements?
No matter how detailed your annotation guidelines are, disagreements will occur because different annotators can interpret the same data point differently. The real question is how those disagreements get resolved. Ask the vendor to explain their review process:
- How quickly are disputed annotations resolved?
- Who makes the final decision when annotators disagree?
- Are resolved cases used to improve the annotation guidelines?
- Are disagreements reviewed by senior annotators or subject matter experts?
That last question matters most. The same feedback principle from the previous section applies here: a vendor who feeds resolved disputes back into the guidelines reduces repeat mistakes and improves consistency over time, while one who resolves disputes on a case-by-case basis will keep having the same arguments.
Disagreements are the everyday failures an SLA anticipates. The harder conversation is about what happens when the vendor misses the SLA.
5. What Happens If You Miss the SLA?
Even experienced vendors will occasionally miss a deadline or a quality target. What separates a reliable partner from a risky one is how they respond when it happens.
Ask the vendor to walk you through their process for an SLA breach. There should be a documented sequence for investigating the cause, correcting the immediate issue and putting measures in place to prevent recurrence. Depending on the agreement, remedies may also include service credits, rework or other contractual commitments.
The point of this clause isn’t punishment. A vendor willing to define consequences in writing is signalling confidence in their own processes, while one who resists is telling you something too.
Handling a breach well also depends on knowing exactly who to call when it happens.
6. Who Will Be My Point of Contact?
Clear communication keeps annotation projects on track, especially when requirements change or issues arise mid-delivery. Before signing, ask how communication will be managed and who owns it. A named project contact beats a shared inbox, because escalations move faster when responsibility sits with a person rather than a queue.
Agree on the cadence too. The vendor should provide regular updates covering progress, annotation quality, completed volumes and any risks to delivery, alongside periodic review meetings. There should also be a defined channel for urgent issues, since waiting for the next scheduled meeting isn’t always practical.
Communication covers how the project runs day-to-day. What it doesn’t cover is what happens to your data while all that work is underway.
7. How Will My Data Be Protected?
Once you share training data with a vendor, you’re trusting them with one of your organisation’s most valuable assets, and the SLA should define how it’s protected across its full lifecycle.
Start with storage. Establish where your data will live, confirm it’s encrypted both in transit and at rest, and find out whether third-party annotation platforms touch it at any stage of the workflow.
Access matters as much as storage. The vendor should restrict data access by job responsibility, monitor user activity and revoke permissions as soon as work is completed. For sensitive or regulated data, ask how they demonstrate compliance with requirements such as GDPR, HIPAA or ISO 27001.
Finally, agree on how long your data is retained after the project ends. An NDA can reinforce these controls, but it can’t substitute for them.
These protections take on extra weight as more people touch your data, and that’s exactly what happens when a project scales.
- Related article: Privacy & Compliance for Medical Data Annotation: HIPAA, GDPR & DICOM Best Practices
8. Can You Scale with My Project?
Annotation projects often begin with a small pilot before expanding to much larger datasets, and a vendor who performs well on a few thousand samples may struggle when volumes multiply. Ask how they’ll add capacity without letting quality or turnaround slip.
Adding annotators is the easy part. Genuine scalability shows up across a vendor’s whole operating model: people, meaning project managers who know how to run partners and freelancers rather than only internal teams; processes that integrate tightly with whoever performs the work; and technology built to source, vet and quality-assure contributors at volume. A vendor missing any of the three is scaling on hope, because as teams grow, keeping annotations consistent becomes the hardest part of the job.
Where those annotators come from matters too. Vendors drawing on deep, less-contested talent pools and investing in training annotators rather than competing for the same exhausted labour markets tend to maintain quality through growth.
Onboarding is the final test. New annotators shouldn’t touch production data until they’ve been trained on your guidelines and shown they can consistently meet the agreed-upon quality standards.
Scaling also changes the economics of the work, which raises the question of how productivity is measured.
9. How Do You Measure Productivity?
Speed alone doesn’t guarantee a successful project. A vendor chasing throughput at any cost will eventually pay for it in quality, so a good SLA defines productivity in a way that balances efficiency with accuracy.
Ask how the vendor measures output and how they manage their backlog. A growing backlog is an early warning that capacity isn’t keeping pace with demand, often before missed deadlines make the problem visible. The vendor should be able to explain how they’ll hold agreed productivity levels while still meeting the quality and turnaround commitments already defined in the SLA.
Every commitment covered so far shares one dependency, though. None of it can be verified unless the vendor shows you the numbers.
10. What Performance Reports Will I Receive?
An SLA shouldn’t leave you guessing how your project is performing. Regular reporting surfaces problems while they’re still cheap to fix, so ask what the vendor’s reports actually contain. Useful reporting goes beyond a single accuracy score and includes:
- Annotation quality broken down by batch.
- Review outcomes and quality trends over time.
- Recurring annotation errors and their root causes.
- Turnaround times measured against agreed service levels.
Then ask how reporting feeds improvement. The most valuable reports explain why issues occurred and what’s being done to stop them recurring, closing the same feedback loop that runs through everything from model training to dispute resolution.
Those are the ten questions. Knowing what to ask is only half the evaluation; the other half is recognising which vendor answers should give you pause.
Warning Signs During Vendor Evaluation
No two vendors operate the same way, but certain answers reliably signal that important processes are missing. When you hear one of these, dig deeper before moving forward.
- “Our annotation accuracy is always above 98%.” Ask how that figure is measured and whether quality reports can support it.
- “We’ll scale our team if your project grows.” Find out how quickly capacity can increase and whether the additional work goes to in-house teams or subcontractors.
- “We always deliver on time.” Ask how timelines are estimated and what happens when a deadline is missed.
- “Our data is completely secure.” Request specifics on encryption, access controls and any compliance relevant to your project.
- “We’ll fix any issues if they come up.” Ask how issues are escalated, who makes final decisions and how recurring problems are prevented.
- “We’ll keep you updated throughout the project.” Clarify what reports contain, how often they arrive and how urgent issues are raised.
The pattern across all six is the same. Confident generalities in place of documented process. A mature vendor can answer detailed questions without leaning on broad assurance, and everything they tell you should end up written down.
Data Annotation Vendor SLA Checklist
Before signing, run the SLA against this checklist. Each item corresponds to one of the commitments covered above, and each should be documented rather than promised.
- Annotation accuracy targets are clearly defined.
- The quality assurance process is documented.
- Turnaround times and delivery milestones are agreed.
- The revision and rework policy is documented.
- The disagreement escalation process is clearly defined.
- The data retention and deletion policy is agreed.
- Performance reports and SLA metrics are clearly defined.
- Remedies for missed SLA commitments are documented.
- The scalability plan for increased workloads is documented.
- A dedicated project contact and a communication process are agreed upon.
- Data security, access controls and compliance requirements are documented.
Conclusion
A Service Level Agreement won’t guarantee the success of your annotation project. What it provides is clarity: what will be delivered, how performance will be measured, and what happens when expectations aren’t met.
That clarity is also your best defence against choosing on price, the most common procurement mistake in this market. Affordability is often a false green flag. A cheap provider frequently means poor quality, extensive rework and an expensive data science team sitting idle while data problems get resolved. The opportunity cost of poor quality far outweighs the savings, which is why a costlier vendor with documented processes is usually the cheaper decision over the life of a project.
At Aya Data, those processes are the business. We control the quality assurance layer end-to-end, with in-house QA teams, data scientists and solutions architects standing behind every dataset and proprietary technology built to source, vet and quality-assure the specialist annotators who work on your project. We take full responsibility for the data we deliver, whether it’s medical imaging, 3D point cloud, LIDAR or geospatial work, and every commitment in this guide is one we’ll put in writing.
These failures rarely show up in model metrics. Catching them takes controls that follow the dataset from annotation through to final release.
