Medical image annotation identifies clinically relevant structures or findings in medical images and turns them into standards an AI model can learn from. The work often begins before an annotator ever opens an image.
Before an annotator marks a tumour on an MRI or delineates tissue on a pathology slide, a team has already decided which images belong to the dataset, what constitutes the target finding and how that finding should be represented. Those decisions shape annotation quality more than the annotation step itself.
A well-designed workflow works backwards from the clinical task. The imaging modality determines how the data should be inspected, and the target finding determines the level of expertise the protocol requires.
This article works through those decisions across the full annotation process, from preparing the training dataset through annotation and clinical validation to choosing suitable software and deciding when specialist annotation services make sense.
What Does Medical Image Annotation Actually Include?
Medical image annotation and medical image labeling are often used interchangeably, but they describe different outputs. labeling assigns a category to an image or finding, while annotation identifies the location or boundary of a clinically relevant structure.
A chest X-ray classification model may only need a label indicating whether pneumothorax is present. A detection model needs the abnormality localised, and a segmentation model goes further, requiring the affected region to be delineated at pixel level.
Annotation ranges from image-level classifications and bounding regions to anatomical landmarks and segmentation masks. For volumetric imaging, it may extend across multiple slices to represent a structure in three dimensions.
The appropriate method depends on what the model must learn, not which technique is easiest to produce. If the intended output is tumour volume, a bounding box provides localisation but cannot supply the boundary information needed to calculate that volume.
Teams also need to define reference standards before annotation begins. An expert’s interpretation may be sufficient for some findings, while other targets are better established using pathology, laboratory results, clinical outcomes or consensus between specialists.
This distinction welds annotation specification into dataset design. It should establish what clinical information the model needs to predict, how that information is represented, and the evidence used to judge whether each annotation is correct.
How Does the Imaging Modality Change the Annotation Approach?
Medical image annotation applies across X-ray, CT, MRI, ultrasound and pathology, and each modality changes what an annotator can see at once and how a finding needs to be represented.
X-rays produce a single 2D projection, which means overlapping anatomy can obscure or mimic a finding. An annotator working from one image has no way to resolve that ambiguity through additional views, so the protocol needs to state upfront how uncertain or partially obscured findings should be handled.
CT and MRI are volumetric, and an annotator typically works through a stack of slices rather than one static image. CT annotation depends on windowing, since the same scan viewed under a lung window, a bone window or a soft-tissue window reveals different structures from the same raw data, and the protocol needs to specify which window applies to the target finding. MRI adds a further layer because a single study can include multiple sequences, such as T1 and T2-weighted or FLAIR, and a finding may only be visible or fully interpretable when sequences are read alongside each other.
Ultrasound differs from the above in that it is often reviewed as a video or a set of frames captured during a live scan rather than a fixed volume. Image quality depends heavily on how the study was acquired, so an annotator may need to select representative frames before a finding can even be marked, and the protocol should define how that selection is made rather than leaving it to individual judgement.
Pathology annotation works at a different scale entirely. Whole-slide images are gigapixel files spanning multiple magnification levels, and a pathologist typically moves between a low-magnification view for tissue architecture and high magnification for cellular detail before reaching a diagnosis. An annotation tool built for a single 2D image cannot support that movement between scales, which is why pathology annotation often requires purpose-built whole-slide viewers rather than general-purpose image tools.
None of this changes what medical image annotation is trying to achieve. It changes what the annotator needs to inspect, what the protocol needs to specify and, later, what the annotation tool needs to support.
How Should a Medical Image Annotation Workflow Be Built?
A medical image annotation workflow needs defined entry and exit criteria at each stage. Moving images directly from collection into annotation leaves annotators to resolve problems that should have been addressed during preparation, while leaving quality control until the end lets the same annotation error spread across the dataset.
A practical workflow moves data through six stages: preparation, vetting, protocol design, annotation, clinical validation and controlled release.
1. Prepare the Medical Imaging Data
Start by bringing source images into a form that can be annotated without removing information the annotator needs to interpret them correctly. For DICOM studies, that means preserving the relationship between patient, study, series and individual instances rather than exporting every image as an unrelated JPEG or PNG.
Remove patient-identifying information before making images available for annotation. DICOM de-identification needs particular care, since patient information can exist within metadata and, in some cases, as text burned directly into pixel data. The DICOM standard provides confidentiality profiles for de-identification rather than treating anonymisation as simply removing a few header fields.
Preparation should produce an annotation-ready dataset without destroying clinically relevant context. Conversion may make a format easier for a particular tool to handle, but it should never strip image characteristics or study relationships the annotation task depends on.
2. Vet the Medical Training Dataset Before Annotation
This stage should produce a defined medical training dataset, rather than a repository of every image that happened to be available. Preparation determines whether files can enter the annotation environment, while vetting determines whether each case should actually be annotated.
The team should check studies against predefined inclusion and exclusion criteria before paying specialists to review them. An incomplete CT series, a corrupted scan or a study missing the required sequence can consume annotation time without producing usable training data.
Vetting should also catch duplicates and obvious dataset inconsistencies. Duplicate cases can distort dataset composition later and create leakage if related images end up split between training and evaluation sets.
3. Define the Annotation Protocol Before Assigning Cases
The annotation protocol translates the model objective into instructions an annotator can apply consistently. It should specify the target finding, the annotation unit and the acceptable boundary rules before production begins.
Clinical ambiguity needs explicit handling. If a finding cannot be determined confidently, the protocol should tell the annotator whether to mark it uncertain, exclude the case or escalate it for specialist review, rather than leaving an individual judgement call to become part of the benchmark data.
Annotator qualification should follow the protocol. A task requiring diagnostic interpretation may need a radiologist or pathologist, while repetitive work against a tightly defined reference can go to trained annotators followed by specialist review.
Run the protocol on a small sample before full production. Disagreement at this stage is useful, since it exposes instructions that two qualified annotators interpret differently while the cost of correcting them is still low.
4. Annotate Against the Agreed Protocol
Production annotation should execute the protocol rather than continually reinterpret it. Annotators need access to the same class definitions, examples and escalation rules throughout the project.
Edge cases should feed into a controlled clarification process. If one annotator encounters a finding the protocol doesn’t cover, resolving it centrally stops separate annotators from establishing different conventions.
Changes to the protocol should be versioned and communicated to everyone working on the dataset. Where a change affects previously completed cases, those annotations need to be flagged for reassessment rather than leaving two conflicting definitions of the same class sitting in one dataset.
5. Validate the Annotations Clinically
Quality assurance should determine whether annotations conform to the protocol and whether the underlying clinical judgement sits within acceptable thresholds. These are related checks, but they are not the same check.
Technical QA can catch issues such as missing annotations, invalid classes or malformed geometries. Clinical review addresses whether the correct finding was identified and represented appropriately.
Where interpretation is subjective, independent annotation can also reveal how reproducible the reference standard actually is. Inter-annotator agreement measures such as Dice similarity for segmentation or Cohen’s kappa for categorical labels can quantify that agreement, though the appropriate metric depends on the annotation task.
Cases that fail the agreed threshold should move to review or adjudication rather than being averaged into the dataset. The adjudicator determines the accepted annotation and identifies whether the disagreement came from the image, the protocol or the annotator.
6. Release a Controlled Version of the Dataset
A dataset should enter model development only after it meets the project’s acceptance criteria. Release needs to distinguish validated annotations from work that is incomplete, disputed or still awaiting review.
Preserve the annotation version alongside the protocol used to create it. If annotations are corrected later, the team should be able to trace which dataset release contained the earlier version and which models were trained against it.
Annotation provenance matters for the same reason. Recording who completed and reviewed an annotation gives a traceable route back to unusual cases and makes systematic errors easier to investigate.
The same controls apply when new cases are added. Treat each addition as another controlled release rather than continuously modifying the training data without a record of what changed.
Where Does Medical Image Annotation Quality Usually Break Down?
Medical image annotation quality can deteriorate even when the source images are suitable and the annotators are clinically qualified. The problem usually comes from variation in how the annotation target is defined or interpreted.
The Annotation Protocol Leaves Room for Interpretation
Instructions such as “delineate the lesion” can still leave uncertainty around poorly visible boundaries, adjacent tissue or ambiguous findings. Without explicit rules, annotators develop their own conventions, and one annotator may include an uncertain boundary while another excludes it, creating systematic differences across the dataset.
Testing the protocol on a sample before full production helps expose these ambiguities early. Edge cases identified later should also feed back into the protocol, rather than each annotator resolving them independently.
The Finding Itself Is Clinically Ambiguous
Some disagreement can’t be eliminated through better instructions. Lesions may have poorly defined margins, or an image may simply not contain enough information for specialists to reach the same interpretation.
The workflow should distinguish these cases from annotation errors. A missed structure may need correction, while a defensible difference between specialists may need consensus or adjudication instead.
This distinction matters because forcing annotators towards agreement can create an artificial reference standard. Research has shown that reference labels can shift depending on which radiologists sit on the expert panel.
The Annotator Does Not Match the Task
Medical expertise should match the level of clinical judgement required. Straightforward delineation against tightly defined criteria doesn’t need the same specialist input as identifying or defining a difficult pathological finding.
A tiered model can assign well-defined annotation work to trained annotators while routing difficult cases to the relevant specialists. Quality control should then examine recurring disagreement rather than simply calculating an agreement score, since repeated differences around the same finding can point to a protocol problem, an expertise mismatch or genuine clinical uncertainty.
What Should You Look for in a Medical Image Annotation Tool?
AI-assisted annotation can accelerate segmentation through pre-labeling, interpolation and model-assisted contouring. Beyond reducing annotation time, automation should focus on the parts of the process predictable enough to automate without weakening the reference data.
It works best where the annotation target is clearly defined and the same task repeats across a large dataset. A model can generate an initial segmentation, propagate an annotation across adjacent slices, or pre-label likely regions for an annotator to review, shifting human effort toward correcting difficult cases rather than producing every annotation from scratch.
MONAI Label’s developers report that this active-learning approach can reduce the time needed to fully segment a dataset by up to 75%, though the actual gain depends heavily on how well-defined the target structure is.
Still, the decision should depend on the task.
- Use automation for predictable, repetitive work. Established anatomical structures make stronger candidates than rare findings with poorly defined boundaries, since the target is already well understood.
- Establish the annotation protocol first. Automating a task before the target and boundary rules are settled just reproduces the same inconsistency faster.
- Measure whether automation actually reduces work. A pre-label that needs heavy correction can take as long to review as a manually created annotation, so the time saved may not be real.
- Keep human validation where clinical judgement is required. A model-generated annotation stays a proposed label until it passes the same acceptance criteria as a manually produced one.
When Should You Outsource Medical Image Annotation?
Outsourcing provides access to medical specialists, lower operational costs and faster turnaround, but it usually shifts the real question to whether the organisation has the clinical capacity and annotation infrastructure to run the workflow internally.
Outsourcing changes who executes the annotation process. It doesn’t transfer responsibility for defining what the resulting dataset needs to represent. Outsourcing medical image annotation should be treated as a build-versus-outsource decision, and it’s particularly relevant when:
- Specialist time is the bottleneck. A provider can structure the workflow so routine annotation work doesn’t consume time that should be reserved for difficult cases and clinical review.
- The dataset requires expertise the internal team lacks. This applies when a project spans several imaging modalities or needs specialists from a particular clinical discipline.
- Annotation volume changes significantly during the project. External capacity can expand or contract without building a permanent annotation team around temporary demand.
- The organisation lacks an established annotation operation. A provider can supply the production workflow, quality-control process and annotation infrastructure rather than requiring the organisation to build each component from scratch.
Choosing a medical image annotation service takes more than confirming a provider has worked with healthcare data before. Their annotation model must fit the clinical task, especially when it requires specialist judgement or multiple review levels. When evaluating who to outsource to, ask questions that reveal how the provider will actually run the project:
- Who will annotate the images, and how are they qualified for this specific task? Look beyond general medical credentials to relevant experience with the modality and clinical finding.
- How will you test the annotation protocol before production begins? A pilot should expose unclear instructions and confirm that different annotators can apply the protocol consistently.
- What happens when annotators encounter an ambiguous case? The provider should have a defined escalation or adjudication process, rather than leaving individual annotators to make undocumented decisions.
- How will annotation quality be measured and reviewed? Ask which acceptance criteria apply, who performs clinical validation and how recurring errors get traced back to their source.
- How will our medical images be accessed and stored? Confirm the controls around data access, storage, audit records and any regulatory or data-residency requirements relevant to the project.
- What annotation records will we receive with the completed dataset? The final delivery should make it possible to trace revisions, review status and the protocol version used to produce the annotations.
Conclusion
A medical training dataset keeps evolving after the first model ships. New imaging sources, previously unseen clinical cases or a shift in the model’s intended use can all expose gaps in the original dataset, and teams need a way to extend it without losing the standards that built the original dataset.
A repeatable annotation workflow is what makes that possible. Instead of rebuilding the process each time you need more data, the organisation already has a protocol for extending the dataset while keeping clinical and annotation consistency intact.
At Aya Data, we run this workflow end to end for organisations building and improving medical AI models, from data preparation and specialist annotation through quality review and controlled dataset delivery. If your team is weighing a build-versus-outsource decision on an upcoming imaging project, book a 15-minute discovery call to walk through your data, your annotation requirements and the specialist expertise your project needs.
Frequently Asked Questions
What is medical image annotation?
Medical image annotation identifies clinically relevant findings or structures in medical images and converts them into labels for AI model training. These can include classifications, lesions or segmentation masks.
What is the difference between medical image annotation and medical image labeling?
Medical image labeling assigns a category to an image or finding. Annotation goes further by identifying its location, boundary or structure within the image.
Who should annotate medical images for AI training?
It depends on the clinical judgement required. Well-defined tasks can use trained annotators with specialist review, while complex findings may need radiologists, pathologists or other clinicians.
How is medical image annotation quality measured?
Quality is evaluated through clinical review, consensus or adjudication alongside task-specific agreement metrics such as Dice similarity or Cohen’s kappa. The appropriate method depends on the annotation and the reference standard being used.
Can medical image annotation be automated?
Yes. Automation can generate initial labels or segmentations for repetitive, well-defined tasks, but clinically significant outputs still need human validation before they meet acceptance criteria.
What should I look for in a medical image annotation tool?
Prioritise support for your imaging data, your required annotation method and your review workflow. The tool should also provide appropriate controls for storing, accessing and modifying medical data.
How much annotated data do I need to train a medical imaging model?
There’s no fixed number, since it depends on the task. Some models reach useful performance with a few hundred well-annotated studies, while others need thousands, and consistency and diversity across the dataset usually matter more than raw volume.
What’s the difference between open-source and commercial medical annotation tools?
Open-source tools such as CVAT or 3D Slicer with MONAI Label give teams full control and no licensing cost, but they typically need in-house engineering capacity to set up and maintain. Commercial platforms trade that setup effort for built-in review workflows, consensus scoring and audit trails, which tends to matter more once a project moves past a pilot.
Written by
Head, Data Annotation
Akhil Singh is the Head of Business Unit, Annotation, at Aya Data and a seasoned AI and GTM leader. With over a decade of experience scaling B2B sales and operations across AI, data, and SaaS, he combines commercial discipline with deep domain expertise to drive growth across computer vision, GenAI, and Medical AI.
