The discrepancy in the surgical outcomes was striking. Within the span of a few months, Tamir Wolf had diagnosed his wife and previous boss with appendicitis, and the two had headed to two different New York City hospitals — mere miles apart — for their appendectomy, a common surgical procedure where the inflamed little organ is removed from the abdomen.
“The approach to treatment and treatment were very, very, different, leading to different outcomes,” Wolf says.
For his wife, a smooth run of emergency medicine as anyone could wish — 12 hours from ER door to theirs, free from the inflamed organ and any major complications.
For his then-boss, a horrible cascade, he says — septic shock, a protracted stay.
For Wolf, it is confirmation of a terrible truth in medicine: that — as he puts it — where you live determines if you live.
He had been the head of his unit’s medical branch as a lieutenant commander in Shayetet 13, an Israeli special forces unit comparable to the Navy SEALs, whose bat-winged insignia represents its swift and silent capabilities. He received a medal for his service in the 2006 war in Lebanon, for treating soldiers under fire.
Theator is platform to help surgeons improve on their performance using video, akin to a professional athlete or fighter pilot — the creation, collation, and application of what Theator calls “surgical intelligence.”
It is decisions that determine outcomes; the correct decisions were made in the case of his wife, and her operation and recovery went smoothly. Something along the line went wrong for his boss, one bad decision leading to another, a gathering snowball.
“These are two prominent hospitals within the same city,” Wolf says. “You’d think that for something as bread and butter … as an appendectomy, things would be different.”
These hidden disparities inspired Wolf to leverage technology to level the playing field.
Wolf and Dotan Asselmann, a computer vision and machine learning expert and entrepreneur who had served in Israel’s secret technology-focused Unit 81, founded Theator, a platform to help surgeons improve on their performance using video, akin to a professional athlete or fighter pilot — the creation, collation, and application of what Theator calls “surgical intelligence.”
“How do we ensure,” Wolf says, “that no matter where a patient goes in to seek treatment, they get the best possible care for them?”
The Inside Story
Many surgical procedures that could be recorded are, quite simply, not. They happen and exist only in the OR — then as whatever outcome the patient has.
“We realized that, even if there are millions — tens of millions — of procedures that are done under visual guidance, the vast majority of them are not captured,” Wolf says. “They’re not stored, they’re not analyzed.”
Left unused is a wealth of valuable surgical data, well known currency among football coaches: tape.
Sometimes, perhaps, surgical videos find their way to students as study aids, or — in particularly odd cases — conference presentations. But they are not consumed, analyzed, and acted on, the way that a point guard or pilot or amateur golfer would use it, spotting details and trends that could be missed, and figuring out how to improve performance.
What went right? What went wrong? Why? What do I do better than my peers? What do I do worse? Why am I faster, slower; have patients with better outcomes, worse outcomes — why? Am I hitting the surgical procedure steps properly? Am I doing things I don’t need to do? Do I spot the things I’m meant to as safety checks? How long are my patients usually on the table? Can I reduce that time?
A key issue: the video is raw.
The process of annotating the procedure, breaking it down into steps so that it can be more easily followed, is time-consuming and impractical for working surgeons.
“Not only that it’s not practical, it’s impossible,” says Ishai Levin, director of the Department of Gynecology at Sourasky Medical Center’s Ichilov Hospital in Tel Aviv.
Levin has calls day and night from his attendings to consult on and supervise surgeries, and a slate of operations to perform himself. Ichilov is a large hospital, and Levin’s department alone does around 6,000 procedures a year, he estimates.
Without annotation, the tape is difficult to analyze, and so often is not.
Surgical Intelligence
Theator aims to allow surgeons to analyze this tape and improve their performance by automating this process.
Using a computer vision program powered by deep learning AI, Theator automatically annotates the procedure and provides a number of analytical measures rapidly.
“Within 30 minutes I have a report,” says Matthew Tollefson, a Mayo Clinic urological surgeon who uses the program. Mayo is both a partner of Theator and an investor.
Tollefson receives a notification on his phone telling him the surgery has been analyzed, with steps demarcated, including unexpected events and important safety checks. He can benchmark this case with his own averages or worldwide averages.
The automated annotation is essentially pattern recognition — a place where computer vision algorithms shine, says Asselmann, co-founder and CTO.
After being initially trained on vast libraries of human-labeled surgical videos, the algorithm “learns” to identify a specific object in the footage, and is then able to say that, for a certain span of time, a specific action is happening.
“In our cases, it can be a dissection,” Asselmann says. “It can be something that has some critical meaning in context,” like seeing a specific organ or an unexpected bleed.
The process of annotating the procedure is time-consuming and impractical for working surgeons. Without annotation, the tape is difficult to analyze, and so often is not.
The AI then labels those actions; part of the collaboration with Mayo is breaking down procedures into steps for the algorithm to learn and identify, Tollefson says. There are still some mislabelings that the algorithm will smooth out with human assistance, as well.
For example, when dealing with dermoid cysts, the AI can confuse the oily liquid inside of them for pus when spilled, Levin says.
To help protect patient privacy, an algorithm ensures that anything occurring on camera outside of the patient’s body is blurred, hiding personal details of both the patient and the surgical team. The data then lives with Theator’s users, not the company itself.
Currently, Theator is limited to urological, gynecological, bariatric, and colorectal surgeries, although the intention is to continue adding more kinds of surgeries as the algorithm is trained and improves itself.
From Intel to Insights
“We talk about technology, but technology for the sake of technology is useless,” Wolf says.
Wolf’s vision is not just making Theator a surgical annotation tool — although it certainly could be, perhaps aimed at teaching hospitals as a training aid first and foremost. Instead, it’s the analysis of the data that Theator facilitates that could be most valuable.
“Everything really has to be immediate, and everything needs to be actionable” for surgeons to be able to incorporate video analysis into their hectic schedules, Wolf says.
Levin, the gynecology director in Tel Aviv, has used the platform to improve surgical practices on a department-wide level.
Seeing both ureters, the plumbing that moves urine from the kidneys, at the end of surgery is a key safety check that can help prevent ureteral injuries, Levin says.
Although rare, these injuries are most commonly caused by surgeries, and can be subtle to spot, but with severe complications including urine collecting in other points of the body, loss of kidneys, and even death.
“Everything really has to be immediate, and everything needs to be actionable.”
tamir wolf
After Levin began using Theator to see if the surgeons were completing their safety check of the ureter, he saw an improvement in the number of surgeons who ensured they took the step. The safety milestone went from being accomplished in 43% of surgeries to 67%.
“This is something that, in terms of safety, is tremendous,” Levin says — knowing that a surgery is going to now include a debrief provides an important sense of accountability.
Since beginning to use Theator, the department has also lowered the time of hysterectomies from an average of 100 minutes to 87 minutes, an important improvement, since less time spent on the operating table leads to better patient outcomes.
Levin compares it to the analysis fighter pilots do. “Every time they do a flight, every time they do training flights, every time they do combat flights, they review,” he says.
“It’s learning from your experiences and improving all the time,” Levin says. “This is something which is very important if you look at the medical profession.”
Tollefson, at Mayo, imagines eventually being able to identify more and better safety measures that can improve patient outcomes by being able to analyze the surgeries more easily.
“Right now, to be honest it’s fairly rudimentary,” Tollefson says. “Yes, you achieved it; no, you didn’t.”
Instead, surgeons may be able to receive feedback about the decisions they have made in terms of the likeliness of outcomes; you missed step x, so your patient has an increased chance of y.
Further down the line, Theator wants to be able to provide real-time decision support in surgery, using all of the surgical intelligence it has amassed to provide important feedback, input, and suggestions to surgeons while patients are on the table.
“Being able to say to surgeons, ‘in order to improve this patient’s erectile dysfunction, you should consider cutting here, rather than here’ — that’s like the big picture,” Tollefson says.
“The thing that’d be really great to get to at some point.”
We’d love to hear from you! If you have a comment about this article or if you have a tip for a future Freethink story, please email us at tips@freethink.com.
