Every minimally invasive operation produces hours of video. A small camera shows the surgeon’s field on a screen, and that same footage can be used to train AI. Models trained on it can recognize which step of the operation is under way, track the tools in use, and help teams review and learn from each case.
But surgical footage is among the hardest video to label. The view is narrow, the scene changes constantly, and the picture is often blocked. Good surgical video annotation needs labels that stay correct even when the camera cannot see clearly. This post explains what the work covers, why it is hard, and what healthcare AI training data needs to get right for the operating room.
Surgical video is usually labeled at three levels:
Some projects add actions, such as cutting, clipping, or suturing, and events, such as bleeding or a tool leaving the view. Models trained on these labels support case review, training tools, workflow analysis, and safety checks.
The operating field is crowded and unstable.
Annotators need rules for frames where the target cannot be seen. A label that says “no instrument” when a tool is simply hidden by smoke teaches the model a false lesson.
The most important decision is what to do when something is present but not visible. There are usually three honest choices.
Guessing is the worst choice. A model trained on guessed boundaries learns to be confident where it should not be. Teams should pick one rule for each case, write it down, and apply it evenly. An “uncertain” route that sends doubtful frames to a clinician helps a great deal.
Phase labeling and instrument labeling ask different questions.
Phase labeling asks what stage the operation is in. Labels cover stretches of time, so the key work is deciding where one phase ends and the next begins. Boundaries are often blurry, because surgeons return to earlier steps or overlap two steps. Clear start and end rules matter more than precise shapes, and reviewers check timing and order.
Instrument labeling asks which tool is in view and where. Labels attach to individual frames, usually as boxes or outlines. Tools look alike, move fast, and are often partly hidden, so reviewers check class names and shapes.
A single guide for both would blur these differences. Each needs its own examples and its own reviewers.
Phases are not the same across operations, hospitals, or surgeons. A step one team treats as separate may be part of a larger step for another. Without written definitions, two annotators will cut the same video at different points.
Each phase needs a definition that says what marks its start, what marks its end, and what counts as an interruption. Experts should label a sample independently, then compare, and refine the definitions until they agree. Where surgical technique varies, the guide should say so and give examples of each accepted variation.
Labeling anatomy is more than naming organs. Some structures are labeled because the surgeon must avoid them. Seeing where a vessel, duct, or nerve lies, and labeling it correctly, can matter more than any other label in the case.
Because tissue can look alike under blood or smoke, anatomy labels need clinician review. Annotators should not be asked to infer a structure from a few ambiguous pixels. A label that is wrong for a safety-critical structure is worse than a missing one, so the rules should state how doubt is handled.
A single operation can run for hours, which is far too long to label every frame. Teams must choose how often to sample, and choose more densely around key moments such as clipping, cutting, or bleeding. Frames from one case should stay on the same side of the train and test split, or the model’s score will look better than it really is.
Events such as bleeding, smoke onset, or a tool leaving the view need start and end times. These labels help models learn what happens before and after a problem, not only how it looks in a single frame.
Surgical video can include faces, tattoos, wristbands, monitors, and voices from the room, especially when the camera leaves the body. Teams should decide early what is masked or removed, who may view raw footage, and how access is logged. Out-of-body segments are often removed. Labels should refer to phases, tools, and structures, not to personal details about the patient or staff.
Surgical labels may also be tied to written records. Our guide to medical coding annotation shows how note text is linked to procedure codes, and the same careful evidence rules apply when footage and notes are matched.
Surgeons and trained surgical staff read this footage differently. They know which step comes next, how a tool is normally used, and which structures hide behind smoke or blood. They can tell a normal pause from a complication, and a safe plane from a risky one.
Specialists write the phase definitions, set the occlusion rules, and review hard cases. This reflects Globik AI’s approach of using verified domain experts for work that depends on judgment, where a generalist would be guessing.
Before starting a surgical video project, settle these points:
Surgery is not a clean scene, and the labels must be honest about what the camera could and could not see. Strong surgical video annotation uses phase definitions that experts agree on, clear rules for smoke, fluid, and blocked views, careful handling of safety-critical anatomy, and reviewers with surgical knowledge. With that foundation, AI can support review and training without learning from guesses.
It is the labeling of operation video so AI can recognize phases, instruments, and anatomy. Annotators mark what is happening and when, using rules written with clinical input.
Mark them as occluded, label only the visible part, or flag the frame as not judgeable. Guessing is avoided, because it teaches the model false boundaries.
They are the stages of an operation, such as access, dissection, the main procedure, and closure. Each phase needs a written start and end rule so annotators cut the video consistently.
Smoke, blood, and lens moisture blur or cover the scene, so tools and tissue become hard to see. Clear occlusion rules and clinician review keep labels honest.
Surgeons and trained staff know how steps flow, how tools are used, and which structures matter for safety. They write clear rules and review doubtful frames.
