Bringing AR to the OR

Surgery Goes to the Next Dimension: Bringing AR to the OR

November 10, 2022

Christina Cardoza

AR guided surgery

It seems like something out of science fiction. Doctors are using augmented reality—a technology more commonly associated with gaming—to guide them through delicate surgical procedures with a minimum of invasion and a maximum of information. This alliance, among technologies and among people, is part of an incredible transformation of healthcare in recent years that includes telemedicine and remote monitoring.

AR-assisted surgery is not the only potential benefit of the cooperation between medicine and augmented reality. Shae Fetters, Vice President of Sales at Novarad, a provider of advanced healthcare solutions; and Dr. Michael Karsy, Assistant Professor of Neurosurgery, Neuro-oncology, and Skull Base at the University of Utah, can speak with experience and expertise from both sides of the coin. They’ll discuss the advantages to patients and practitioners, the partnerships that make it all happen, and the future possibilities for this new vision of healthcare.

How is augmented reality transforming the healthcare space?

Shae Fetters: I think we’re really at the forefront of this evolution, and we don’t understand in full what the possibilities are at this point. What we’re not doing is looking to replace what doctors do. What we are doing is taking medical imaging—simple X-rays, CT scans, and MRIs—and merging them with AR to give doctors extra information that can make procedures go more smoothly. That could be a highly complex surgery, or something routine that might be safer with imaging on hand.

It can also be used in patient education. We find that a lot of patients are worried about what’s going to happen during a procedure, though doctors do the best they can to explain what’s in their heads and what’s in our complex bodies. Augmented reality helps patients see exactly what it is surgeons are talking about. That helps them feel more comfortable and more a part of their care. And if patients understand those things and feel comfortable with them, that helps get better results all around.

We also see it used in the training of new physicians, both in surgery and clinics. It substantially increases the effectiveness and the speed of learning. And that’s really just touching the surface of what the current possibilities are for AR in the healthcare space.

“What we are doing is taking #medical imaging—simple X-rays, CT scans, and MRIs—and merging them with #AR to give #doctors extra information that can make procedures go more smoothly.” – Shae Fetters, @NovaradCorp via @insightdottech

Can you talk about the use of augmented reality in the operating room?

Dr. Michael Karsy: Image guidance has been around for quite a long time in neurosurgery, and it’s now essentially standard of care—where we use a guidance system to allow patient DICOM images, MRIs, or CT scans to be used intraoperatively to help with the surgery. And as I was looking around to see where augmented reality was at in the healthcare space, Novarad was way ahead of everybody else. It had a technology that was FDA cleared for preoperative planning, as well as for intraoperative use for spine instrumentation.

I thought this technology could be the next step in imaging for patient care, and I thought it would be interesting to work on it with this group. I could already see the benefits of navigation for making spinal surgery minimally invasive, or in cranial applications to really home in on complex anatomy.

Shae also alluded to the fact that most surgeons develop in their mind’s eye a kind of X-ray vision of patient anatomy and what they expect to see in surgery. But that takes years to develop. I thought that with augmented reality you could essentially take these holograms and DICOM images, and apply them directly to a patient right in front of you. The software that Novarad has allowed me to install here at the university enables me to upload images to their cloud-based system, generate hologram images, and then download them directly to a headset just with Wi-Fi.

He also mentioned that we are really just at the beginning of how augmented reality is going to be used in healthcare. We don’t yet know the best applications for it; we don’t know exactly where it will add specific value for specific cases because this technology is so new. And it’s really exciting to work on it and see where it’s going to go.

How is Novarad making AR-guided surgery possible?

Shae Fetters: The company as a whole has been dealing with medical images and providing picture archival systems for hospitals for 30-plus years. We’ve now been able to change what the monitor is for doctors by condensing the images down to headsets.

Our solution, VisAR, has been the result of seven to eight years of trial and error, but we’ve got it to the point where you can look at a simple QR code and it downloads your patient’s study onto a headset in about 20 seconds. At this point, the whole process to download, to calibrate the lens, to let it learn the space it’s working in, and then to match the images to the patient’s anatomy accurately takes about two and a half minutes.

And working with Dr. Karsy we’re continuing to figure out better workflows so that it isn’t scary and complex for surgeons and different hospital staff; so it’s not a big deal when the doctor says before a case or partway through a case, “Grab my headset. We’re going to guide this with VisAR.” It’s not quite to the point where it’s absolutely amazing, but it’s still pretty mind-blowing to do this without needing multiple pieces of huge equipment in the OR.

What is the value for Novarad of working on this technology with partners like Intel?

Shae Fetters: Novarad is a fairly small company by industry standards, so it’s only been possible to produce this type of technology through our different partnerships. Microsoft has an amazing headset that they continue to develop: the HoloLens 1 and now the HoloLens 2. Because for us to really spend the resources and the time to develop the headset on top of all the software—well, I don’t think we could ever keep up with Microsoft. The inside parts of that headset are the chips by Intel. We don’t produce microprocessors; that’s not what we do. So it’s great to collaborate with companies that also have some great innovative minds and some amazing talent.

We also couldn’t have done this without brilliant minds like Dr. Karsy’s. And there are a lot of doctors across the country who are excited about this and want to share their different ideas. It’s very much an open and collaborative network that we’re working with. There isn’t competition between us, and there isn’t competition between the surgeons either; they want what’s best for patient care and for moving healthcare forward. It’s not a selfish venture.

Dr. Michael Karsy: From the clinical perspective, working with industry has been great; there’s no way one single academic center is going to develop 30 years of experience with PACS and imaging and DICOMs like Novarad has—and then develop augmented reality on top of it.

We really focus on what we do well, which is the surgery part and the clinical part and the management of that; and then we partner with a group that has expertise in radiology and augmented reality. That’s the way to innovate, and I think we’ve had really good success doing it that way.

What hopes do you have for AI-guided surgery going forward?

Dr. Michael Karsy: We’ve mentioned the Intel chips—and as they become stronger, have better processing speed, and are able to handle more operations they generate new technology that can do more, too. That’s kind of the same thing we’re seeing in healthcare: As soon as we have new technology—a better HoloLens, a better augmented reality system, software that works faster—it changes everything you can do. Now you have the ability to go back to procedures that have been done the same old way for decades, and you can do them differently: you can do them minimally invasively with smaller incisions, and you can get more accurate.

Imaging technology has been widely used in neurosurgery, and we can see how navigation has changed the way we do things. Then other kinds of surgeons come into our wards, see our navigation systems, and they’re wowed by it; such a thing doesn’t exist in many other surgical fields. I think AR will change all that. I think you’re going to start to see image guidance and minimally invasive procedures enter into other surgical subspecialties and spaces in healthcare where it didn’t exist before, because we just didn’t have a way or need to do it.

If there is a day where I don’t have to do another brain surgery that would be a great day, because it would mean that we’ve developed enough medical therapy and radiation and other things that surgery isn’t necessary. It will take a long time to do that. But that’s the ultimate goal of this—to reduce the harm to patients as much as possible. Technology drives the way forward for that, one hundred percent.

How do you see Novarad as part of that future?

Shae Fetters: If you think about X-ray and CT and MRI, and how we can put people in a little machine for a couple of minutes and then tell what’s going on inside them without needing to open them up—that’s a pretty revolutionary thing. We see AR as a similar move.

We really do see this as something that, in the next 10 years or so, is just going to be a standard of care where every surgeon, every doctor has their own headset and it’s a core part of how they treat patients and how they interact with them.

Any final thoughts?

Dr. Michael Karsy: AR technology is one of those things that the more you learn about it, the more you’re going to start to envision where else it could be applied. In healthcare there’s still so much that’s unknown, and to have a new technology that allows you to come up with new discoveries and implement them—that’s great. I think that’s the way things should be going forward, and I’m really excited to see this continue to develop in my career.

Shae Fetters: It can be difficult to find the right words to explain this technology to people. But I put a headset on someone’s head, and within 30 seconds to a minute I see their eyes open wide and they understand; they start to see the potential out there. So I encourage any physicians, any big healthcare systems that are looking to improve their patient care to reach out. It’s worth a conversation on how we can collaborate to provide better care for all patients in our communities.