Surgical planning with immersive mixed reality

The job of surgeons starts long before they step into the operating room. They must use two-dimensional MRI or CT scans to plan the surgical steps on a three-dimensional patient, relying on their experience, skill, and spatial sense. Using mixed reality (MR) to view human anatomical models allows for better visualization and navigation.

In this MEDICA-tradefair.com interview, Dr. Dominik Fischer talks about the use of MR in surgery, and describes the Mixed Reality View by Brainlab, which surgeons in California used to plan the separation surgery of conjoined twins in 2020.

Dr. Fischer, how does the Mixed Reality Viewer work?

Dr. Dominik Fischer: The system converts data from MRI or CT cross-sectional images into three-dimensional models. Freeing them from the screen, they can be viewed in the room using a mixed reality headset made by our partner Magic Leap. The system also includes a controller that facilitates the interaction with the 3D models, and a wearable computing platform that converts image data from PACS format into models.

What is the benefit of visualizing image data in this way?

Fischer: Image data is used not only for diagnostics but also for preoperative planning. Take a brain tumor, for example. There are different approaches to access the tumor, with the objective of not injuring brain regions that control language or motor function.

Surgeons use data from cross-sectional imaging to create a detailed surgical plan. They only get to see two-dimensional images of the body on the screen, even though it is much simpler, more illustrative, and helpful to view the anatomy in three dimensions. Thanks to the Mixed Reality Viewer, we create three-dimensional models that are more tangible as they can be touched and viewed from all angles, making this a highly immersive experience.

Where are the applications where doctors can use the Mixed Reality Viewer?

Fischer: Apart from surgical planning, physicians can also use the tool for patient education. The Mixed Reality Viewer supports a multi-user mode that allows multiple specialists to simultaneously view the same model. The physician can use the controller to virtually highlight sections, enlarge or rotate the model to explain the procedure.

What feedback do you get from users?

Fischer: All told, we receive excellent feedback. The Mixed Reality Viewer brings a new dimension to surgical planning. Surgery requires three-dimensional thinking and surgeons are experts in getting a good grasp and understanding of the operative field from two-dimensional images. The 3D models support and enhance this process.

Especially younger surgeons with less clinical experience benefit greatly from this technology. It also assists highly experienced surgeons who face very complex cases such as skull base tumors, intracranial aneurysms, traumatic fractures, or vascular surgery. In the fall of 2020, the Mixed Reality Viewer was used in California to prepare for the separation surgery of conjoined twins. This highly complicated and intricate surgery involved over 30 people and took more than 24 hours.

Medical education and training also benefit from this tool. Junior surgeons take years and hundreds of surgeries to hone their skills and become experts. Many surgeons have told us that the learning curve is much steeper using MR. The complexity of the three-dimensional structures of the human body becomes more tangible and transparent compared to learning from textbooks or watching other surgeons.

What future trends do you see in MR applications in surgery?

Fischer: I think mixed reality will play an increasing role in medical education. Students will work less with body donations and learn processes using virtual models instead.

The next step will be to bring mixed reality into the operating room. There are several conceivable applications in this setting. One promising application is to overlay preoperative image data on top of the patient's body to be able to virtually look inside the patient with MR. In the long run, surgical monitors and displays will become less important as an information source in the OR. We could replace most of them with MR displays and headsets.

Taking things even further, the different members of the operating team might have their own headset to access personalized content. While the surgeon views image data, the surgical instruments are tagged or labeled for the operating room assistants to avoid any mix-ups.

Aside from these options, the technology can also be used to foster collaboration among surgeons. During the preoperative stage, surgeons can review the models with their colleagues – regardless of their current location, thus enabling the integration of surgeons from another hospital or another country. During the intervention, surgeons could also review the next steps with experts who are outside the operating room. The models are ideally suited as a discussion platform, especially to promote interdisciplinary team settings.