A labeled batch arrives, and someone has to decide whether it is good enough. In a public sector project, that decision is not only about the model. It may have to be explained to a supervisor, an auditor, or a successor who was never in the room.
"It looks fine" does not hold up in that conversation. Audit-ready annotation quality means being able to show, with evidence, why a batch was accepted or sent back. This post looks at how agencies can set that up, so that public sector annotation quality is measured the same way every time.
Quality is easiest to judge when the rules are written before the work begins. Acceptance criteria say what a batch must achieve to be accepted, and they belong in the project scope, not in a conversation afterward.
Good criteria are specific. They name each class or field, the level of accuracy expected, and what counts as an error, with examples. They also reflect the use of the data. If a model will route documents, a wrong category matters more than a slightly misplaced stamp.
A single overall percentage hides problems. A batch can score well overall while one important class is badly labeled. Criteria should therefore be set per class, and per region where the data spans several. They should also say how disagreements between the agency and the vendor are settled, so a dispute ends with a rule, not an argument.
Checking every label is rarely possible, so quality depends on sampling. A weak sample gives false comfort.
A sound approach mixes two kinds of checks. A random sample shows the general level of quality. A targeted sample looks at the places where mistakes are most likely: rare classes, difficult cases, new annotators, and regions or sources that behaved differently before.
Sampling should also happen after every batch, not only at the end. Problems caught early are cheap to fix, and problems found after thousands of items are not. It also helps to record which annotator, region, and source each sampled item came from, so patterns become visible. For an example of how regions and agencies can drift apart, see our guide on national infrastructure imagery annotation: keeping labels consistent across regions and agencies.
A misfiled confidential record and a slightly loose outline are both errors, but they are not the same size. Treating them alike leads to bad decisions in both directions. In a records project, an item filed under the wrong confidentiality grade is critical, while a stamp traced a few pixels wide is minor.
A simple tier system helps:
Acceptance rules can then reflect the tiers. A batch might need to contain no critical errors in the sample, with limits on major ones. This gives reviewers a shared language and keeps a pile of minor slips from sinking a good batch.
When two reviewers label the same items, how often they agree says something useful about the project. Low agreement on a class rarely means the people are careless. More often it means the guideline for that class is unclear.
That changes the response. Instead of blaming individuals, the team looks at the guideline, adds examples, and holds a calibration round. Tracking agreement by class shows where the rules need work, and a calibration round lets everyone label the same sample and compare answers.
A reference set helps as well. This is a collection of items labeled by experts and kept aside. Comparing ongoing work against it reveals drift over time, such as a team slowly changing how it applies a rule.
Finding errors is only half the job. What happens next should be as clear as the check itself.
Each defect is logged with its class and tier. The team looks for the cause, such as a vague guideline or a missed edge case, and fixes it at the source. Corrected items are checked again, and the batch is signed off by a named reviewer.
Silent edits are the enemy. If labels change without a record, no one can later say what was accepted or why.
A report turns all of this into something an agency can read and keep. At minimum, it should show:
Reports like this let someone who was not involved judge the data. A short written summary of what the numbers mean, and what will change next, helps readers who do not work with metrics every day. They also show whether quality is improving or slipping.
Domain specialists help define what counts as critical, since they know which mistakes affect real decisions. They judge edge cases fairly, explain why a class is ambiguous, and turn those lessons into clearer guidelines. They can also train reviewers to apply the tiers consistently. This reflects Globik AI's approach of using verified domain experts for work that depends on judgment.
Before accepting labeled data, settle these points:
Good labels are not enough on their own. Public sector teams also need to show how they know the labels are good. Written criteria, careful sampling, tiered errors, agreement checks, and clear rework give audit-ready annotation quality a firm base. With that in place, accepting a batch becomes a decision backed by evidence rather than a judgment call.
It means being able to show, with evidence, why a labeled batch was accepted or sent back for rework.
Written criteria give everyone the same standard, so quality is judged against agreed rules rather than opinions afterward.
Full checking is rarely practical, so a well-designed sample mixing random and targeted checks gives a reliable picture.
They separate critical, major, and minor errors so acceptance rules reflect how much each mistake matters.
It usually points to an unclear guideline for that class, which calls for better examples and a calibration round.
