Medical Image Segmentation: Why Better Models Cannot Fix Weak Clinical Standards

Three weeks into a project, the results land and the room goes quiet. A healthcare AI team pulls up the segmented output from a set of 3D CT scans, and several tumour boundaries come in below the clinical threshold the project requires. The disappointment is familiar. The model looked strong in testing, and now the numbers say otherwise. 

Each experiment costs more compute and changes nothing, because every model is learning from the same reference standard and repeating the same fault. The cause becomes clear only when the team looks at the source annotations: clinicians had labelled the difficult margins differently, with no rule for handling the uncertain cases.

We see this across medical annotation teams. Segmentation falls short, the model takes the blame, and the dataset that failed first goes unchecked. Inconsistent labelling produces ground truth that no amount of tuning can fix, so the dataset deserves a look before the architecture does. 

What Medical Image Segmentation Involves

Medical image segmentation divides an image into regions, each one representing a defined anatomical structure or an area of clinical interest. In a 2D image, the output assigns a label to every relevant pixel. In a 3D CT or MRI scan, it applies those labels across voxels to build a volumetric representation.

This level of localisation is what separates segmentation from image classification. A classification model can tell you that a scan contains a tumour, while a segmentation model has to identify where that tumour begins and ends. The boundary it produces then supports tumour measurement or guides treatment planning.

How much precision you need depends on how the output will be used. A small boundary difference carries little weight in an early research study, but the same difference in the same mask becomes clinically significant once it informs a diagnosis or a radiology plan. Segmentation quality is measured against the decisions that rest on it.   

How a Medical Segmentation Workflow Runs

A segmentation project moves through a defined sequence of stages, and each stage shapes what the model eventually learns from.

• Ingest and standardise the imaging. Scans arrive from the source in DICOM, carrying pixel data alongside metadata describing how each study was captured. Before anyone labels a thing, those studies are checked and standardised, because a slice thickness or a windowing setting that varies across the set will change how a boundary reads on screen.
• Annotate to an agreed standard. A clinician marks each structure of interest across the image, and in a 3D study that means carrying the label through every slice so the regions line up into a coherent volume. Clinical judgement does most of its work at this stage, and this is where two qualified readers can part ways on the same margin.
• Review the masks. A second reader checks each mask against the standard; disagreements route to a senior clinician, and the cases too uncertain to call get flagged rather than forced. What survives review becomes the reference standard the model trains against.
• Train and measure. The model learns the boundaries from the reviewed masks, and its output is measured back against that same standard. When results fall short, the workflow gives the team somewhere to look, because a fault can sit in the imaging, the annotation, the review or the model, and each one leaves a different mark.

The Trap of Constantly Chasing Models

[Alt text: four-step workflow for diagnosing poor medical image segmentation performance by reviewing model errors, comparing datasets, validating annotations, and auditing training data before changing AI models.]

When segmentation results fall below target, engineering teams tend to assume the model is responsible and reach for the training settings or a newer architecture. That instinct helps when the model cannot learn the required patterns, but it gets expensive when the real problem lies in the dataset.

Start by identifying where the model fails. If several models make the same mistakes on the same scans, review the images and their masks, because repeated errors around one boundary usually mean the annotation instructions were unclear.

Comparing results across data sources tells you more. Poor performance on scans from one hospital points to differences in imaging or data preparation. Errors that appear only in the difficult cases point to clinicians disagreeing on where the boundary should sit.

Review the dataset before you replace the model. Skip that step and each training cycle repeats the same problem with a different architecture.

Clinical Standards as Governed Decisions

Reviewing the dataset only helps if you know what a sound label looks like, and in medical imaging that is less obvious than it sounds. A clinician-created mask does not automatically qualify as reliable ground truth. Some structures have clear borders. Others sit close against neighbouring tissue and resist a clean edge, and image quality makes that call harder still when a lesion shows low contrast or an irregular margin.

Two qualified clinicians can therefore draw different boundaries around the same structure, and the same clinician can reach a different decision reviewing the scan a week later. Those differences rarely signal a wrong annotation. They reflect the genuine uncertainty in the image.

Teams need a defined process for handling that uncertainty. The process sets how many clinicians review each case and when a senior specialist decides between conflicting masks. It also sets the point at which a case is too uncertain to support a final label at all.

Standards are the output of an agreed clinical process, and their reliability depends on how consistently the team applies that process across the dataset.

How Governance Failures Enter a Segmentation Dataset

An agreed clinical process only holds if it survives contact with a live project, and this is where most datasets start to slip. Annotation quality changes as a project moves from initial setup into full production, and small gaps in the process can affect thousands of masks before anyone notices. The exposure runs beyond model accuracy. Medical imaging data sits under HIPAA and GDPR, diagnostic models are cleared as software as a medical device under FDA and EU MDR oversight, and the EU AI Act places high-risk medical AI under direct requirements for how its training data is governed and recorded. A process that cannot show how each label was produced and handled is a problem long before the model trains. Four governance failures account for most of that risk. 

• Unclear annotation instructions: When the guidance does not spell out how to label a difficult boundary, clinicians fall back on their own judgement and apply different rules. The result is variation across masks that were meant to follow one standard.
• Protocol drift: Annotation decisions shift as new cases surface. When those changes travel informally, some clinicians keep working to the original approach while others adopt the update, and the dataset ends up carrying both.
• Differences in imaging sources: Scans from different hospitals vary in quality and appearance. A process that does not account for those differences leaves clinicians applying the same rules unevenly across sources.
• Weak version control: Teams correct masks without recording what changed. Earlier versions then linger in the dataset or resurface in a later training run.

These failures rarely show up in model metrics. Catching them takes controls that follow the dataset from annotation through to final release.

A Governance Framework for Medical Image Segmentation

Governance is what turns those loose failure points into controls you can hold a project to. A framework defines how segmentation data is created, reviewed and approved, and it assigns responsibility for the decisions that shape the final masks. The seven steps below map that path from intended use through to a controlled release.

1) Define the Intended Clinical Use

Start by documenting what the segmentation output will support. A mask built for early research does not need the precision of one that guides treatment planning, and stating the difference upfront keeps the whole project calibrated to the right target.

Intended use sets the acceptance criteria and decides which errors trigger clinical review.

2) Create the Annotation Standard

The annotation standard defines each structure and explains where its boundary begins and ends. Difficult cases carry visual examples that show the approved approach, so annotators have something concrete to work against rather than a paragraph of prose.

A clinical lead owns the standard. Any change is documented before annotators apply it to new cases.

3) Calibrate Annotators Before Production

Every annotator labels the same sample before the full project begins, and that sample includes both routine scans and the cases most likely to cause disagreement. The team compares the masks, works through the differences, and opens full production only once the main sources of disagreement are resolved.

4) Measure Inter-Annotator Agreement

Agreement is measured on shared cases throughout the project, not just at the start. Dice or intersection over union shows how closely masks overlap, while boundary distance exposes the differences along the edge that overlap scores can hide.

5) Establish a Clinical Adjudication Workflow

The workflow names which disagreements go to a senior clinician for review. The reviewer either approves a final mask or records why the case cannot support a reliable label.

When the same disagreement keeps recurring, it points to a gap in the annotation standard, so the fix belongs in the standard itself rather than in a fresh ruling each time.

6) Automate Pre-Training Validation

Automated checks confirm that each mask matches its source image, and they flag missing masks and invalid labels before any of it reaches training. Failing cases return to the annotation team for review. Correcting them quietly, outside that process, is how bad masks slip back in.

7) Control Dataset Releases

Approved images and masks are grouped into a named dataset version. Each release records the annotation standard it was built against and any corrections made after review.

Every training run references a specific release. That single discipline lets the team trace a performance change back to its source, whether that source is the model or the dataset.

Conclusion: Better Segmentation Begins Before Training

Better medical image segmentation begins with the clinical standards, well before any model enters the picture. Those standards also have to reflect clear clinical intent and trace back to the decisions that produced them.

With those considerations in place, engineering teams assess model performance with real confidence. They stop burning cycles retraining against unstable labels and start working on the errors the model is genuinely responsible for.

At Aya Data, we hold to this methodology because it also makes clinical use safer: every mask passes through a defined review process before it reaches training. If you are building for healthcare AI, annotation governance is work we can take on with you from the outset. Book a 30-minute discovery call, and we will map it to your project.

Frequently Asked Questions