Virtual Surgical Intelligence: Microsoft Hololens in the OR

Modern imaging opens news doors to surgeries. Yet it also poses major problems for surgeons: They use two-dimensional images to navigate through a three-dimensional surgical environment, while they continuously have to switch their focus back and forth between the images and the patient. Now help is on the way in the form of interactive 3D projections and mixed reality (MR).

In this MEDICA-tradefair.com interview, Sirko Pelzl explains how Virtual Surgical Intelligence (VSI) can be applied in various medical fields and reveals how this supports doctors and patients.

Mr. Pelzl, what is VSI?

Sirko Pelzl: VSI is a mixed reality platform designed for hospital use. We started with an intraoperative application that uses mixed reality glasses to virtually display CT or MRI images on the surgical site. Examples include traumatology, dermatology or oral-maxillofacial surgery. We used this as a foundation to enhance the platform for various medical applications: surgical planning, patient education, training and education. The applications and functionality are fully specified by the physicians.

What are the advantages for physicians and patients?

Pelzl: For surgeons, VSI is a major step forward in surgical quality improvement. This marks the first time they can truly see 3D images during the surgery, allowing improved orientation and precision. Even though experienced surgeons are capable of doing this using two-dimensional images, VSI is a great help, especially for a new generation of physicians.

The technology can be used for various types of surgeries: Using the Microsoft "HoloLens" MR headset, surgeons can overlay 3D images of the liver on the surgical site and exactly locate the position of the ventricles or metastases for example. In dermatology, physicians can project a 3D SPECT-CT onto the patient to locate the sentinel lymph nodes. When used in traumatology, doctors can view broken bones and fractures from all angles. Based on our experience, all fields can benefit from this technology: doctors who try VSI immediately come up with great ideas on where and how they can use it.

What’s more, physicians can use VSI to show patients their own MRI scans and explain the surgical procedure in visual detail. We were able to illustrate in a recent study how greatly patients appreciated this education and communication.

Do surgeons need training or extensive practice to use VSI?

Pelzl: It does not take much time to familiarize yourself with the technology. We estimate between two and three hours for the new HoloLens 2 because it is better at recognizing user gestures than the original HoloLens and it supports more intuitive hand control. You actually do not need training for VSI. Our goal is to keep the system simple.

What technical requirements do hospitals need?

Pelzl: They only need a good wireless connection, although we now also have an offline option, eliminating the need for a network in the OR. Another reason we chose Microsoft HoloLens is because it can be easily integrated into hospital IT, has VPN connectivity and the valid security certificates.

How does the data processing work?

Pelzl: It happens almost automatically. A doctor only has to import image data in DICOM format using drag-and-drop, which automatically creates a 3D model. This works with any manufacturer like Philips, Siemens, GE or others. An AI algorithm subsequently edits the image, while the physician specifies the functionalities he or she requires. The physician then turns on the HoloLens and the image is displayed right in front of him or her.

What types of user feedback do you get?

Pelzl: At this point, we collaborate with more than 40 physicians from around the world. Whenever they put on the HoloLens for the first time and view the MRI or CT scans, it is apparent how much they like it. Needless to say, their feedback is great.

We also received positive feedback after we introduced VSI at last year's MEDICA TECH FORUM and had many interesting conversations, made new business connections and built relationships.

What are your future plans?

Pelzl: As I mentioned earlier, we plan to switch to the new HoloLens 2. We will also enhance usability and functionality in the future. In general, we intend to address the many emerging user needs that are gradually evolving.