Solutions

The Exciting Future (and Present) of AR and VR in Surgery

AR and VR have tremendous potential in surgery and it's perhaps developing quicker than any other field in medicine.

Martin Verbic

Mar 22, 2022 — 5 min read

The bias of this newsletter until now is that I've been heavily in contact with fields close to internal medicine, which are diagnostic-first fields. Procedures only come in second. That's why most articles I wrote are about AI and how it can improve the diagnostic reliability.

However, I recently came into more contact with surgery at medical school and started looking more into how technology could also be applied there. I mean, it makes sense that VR and AR are making its way into surgery, but I somehow dismissed it. It’s time to dive into it.

📢

Medical Notes is not affiliated with any of the companies mentioned below.

What are AR and VR?

I’m sure most people are familiar with both, but let’s nevertheless make a brief introduction.

Augmented reality (AR) is an interactive experience of a real-world environment. It’s like a browser add-on to what you see and experience in real life.

Virtual reality (VR) is a simulated experience that can be similar to or entirely different from the real world.

About a decade ago, Google was heavily developing their AR glasses called Google Glass. We would’ve though Google Glass would already be a part of our lives by now. However, they retired the consumer branch of the product a few years ago and seem to be focusing on companies. AR somehow didn’t stick, and we have very few (if any) widely available AR products to use in everyday life.

VR, on the other hand, did stick, especially in gaming. It even experienced the latest push with Facebook renaming itself into Meta and investing heavily in metaverse. But now it’s about time it makes its way into science and medicine.

Surgical education using VR and AR

VR is definitely interesting for immersive gaming and the “metaverse”. However, in my opinion, it attempts to give people a chance to build a perfect world without having to improve the real one. I find this somewhat disturbing, but still think that there’s a future for both AR and VR in other fields, such as medical education. But for this issue, we’ll focus on surgery.

Anatomy

The traditional way of studying anatomy is certainly still acceptable - with 2D atlas pictures and some practical work on cadavers. I honestly love the sketches in atlases such as Thieme. But on the other hand, I realise that VR and AR are a perfect match to replace this.

One of the best software programs for learning anatomy is Complete Anatomy. They’re basically a 3D anatomy atlas with capabilities of AR. You can visualise different structures in space using mobile devices. It’s clear that they want to develop a full AR/VR learning experience.

Virtual dissection with Complete Anatomy (source: 3D4Medical)

💡

Complete Anatomy was started by a company named 3D4Medical, which aimed to build the world’s best interactive anatomy atlas. They did, in my opinion, and were recently acquired by Elsevier.

Skillshare for surgery

Skillshare is an online platform where creators can share and sell their courses. What if that existed for surgery?

It already does. Proximie is a company developing a platform, which allow surgeons to “virtually scrub in” and watch surgery in real-time. This makes sharing and collaborating in the operating room much more inclusive. It allows for a seamless transfer of knowledge between surgeons.

It isn’t as immersive as a visit to an OR. For medical students (at least for me), it’s much more exciting to put on some scrubs and experience surgery live. However, there’s definitely value for Proximie’s platform. The latest confirmation of this is the COVID-19 pandemic, which significantly decreased the number of elective procedures, which restricted the access for (at least) medical students. Proximie was (and still is) a nice replacement.

💡

Proximie sometimes has events where they film different procedures. I watched a hip replacement surgery a few months ago. Follow them on social media to get notified for such events.

Simulation

The next ladder of surgical education is simulation. When researching for this issue, I particularly liked OssoVR, which are developing a VR platform for surgical training and assessment.

What’s particularly exciting to me is the assessment of surgical skills and proficiency. Even though VR simulation is nowhere near a live surgical procedure, a surgical team can still test different skills and train for specific surgical procedures. This makes surgery even safer for patients.

💡

Perhaps a team of surgeons from around the world that collaborate on an especially complicated case could train virtually and remotely before even meeting in person!

Augmented surgical practice

Education is only one part of AR and VR in surgery. But there are also numerous startups that develop virtual technologies for surgeons to use while operating. We can divide them into three levels: planning, operating and ensuring a single source of truth.

Planning

One of the main challenges in surgery is that medical images are usually in a 2D format. Even MRI and CT scans are made up of a collection of 2D images. I can imagine that a highly trained surgeon has no problems creating a mental 3D image out of multiple 2D planes. But occasionally, a 3D image is more than welcome.

ImmersiveTouch is a VR surgical planning platform that does just that - converts 2D images into 3D VR models. This makes it much easier for surgeons to visualise and plan how the procedure will be conducted.

Operating

VR and AR don’t stop at surgical planning. There are already cases of how predominantly AR is used during surgical procedures to ensure the best outcome for the patient. One example comes from a neurosurgeon from Johns Hopkins Medicine, who used AR for a spine surgery. They later released this video:

An AR startup called Proprio is developing a system that would do something similar to the procedure from Johns Hopkins. The main advantage of using AR during surgery is what was also showcased in the video - accuracy, which is achieved by imaging, navigation and guidance.

A single source of truth

Finally, I want to spend this last chapter discussing the single source of truth. This is an important concept in healthcare in general, whereby all doctors who treat the same patient rely on the same data. From my experience and from talking to doctors, this is not always the case and sometimes presents a big problem. And we can only imagine that not having all information during surgical procedures can be even more life-threatening.

SentiAR aims to provide the single source of truth for interventional procedures. Particularly exciting is a headset (a “wearable command centre”) that allows doctors to control 3D images using their eyes. Instead of having to constantly look at multiple CT and CTA scans, they have the single source of truth in front of their eyes.

So, there it is, a brief overview of what’s happening in the sphere of AR and VR in surgery. It definitely puts into perspective the development of surgery, which is arguably a little different from some other fields in medicine, such as internal medicine.