Broader perspective: how Mixed and Virtual Reality transform surgery

For surgeons, nothing is more important than intimate knowledge and a spatial understanding of their operating field. Yet even three-dimensional imaging methods only provide limited assistance because the data is viewed on two-dimensional screens. When it comes to surgical planning or medical education, Mixed and Virtual Reality foster a better spatial understanding of the human body.

In Mixed or Augmented Reality (MR/AR), computer-generated imagery is projected into the user's field of view with the help of a headset, thus mixing digital information with real-world elements. Virtual Reality (VR) is a fully digital, computer-generated world users experience as an immersive 360°environment that puts them at its center. What originated in consumer electronics is now benefiting healthcare and medicine: MR and VR allow physicians to virtually travel inside a patient's body and get a three-dimensional view.

These applications are extremely helpful in the field of surgery as they give surgeons a better way to plan surgical interventions. "There are different approaches to access a brain tumor, with the objective of not injuring brain regions that control language or motor function," explains Dr. Dominik Fischer of Brainlab in a MEDICA-tradefair.com interview. Surgeons use data from two-dimensional images to plan complex surgeries, while making patient safety the utmost priority. Physicians use these 2D images and their spatial abilities to gain an understanding of the operating field. "It is much simpler, more illustrative, and helpful to view the anatomy in three dimensions," adds Fischer.

To assist surgeons in this endeavor, Brainlab teamed up with Magic Leap and created the Mixed Reality Viewer. This MR headset can be connected to PACS databases to convert data from MRI or CT cross-sectional images into three-dimensional models. Surgeons can use these models to plan surgeries, apply the multi-user mode to review complex cases with their colleagues or prepare and educate patients about the surgery.

Yet those are by far not all the benefits MR has to offer: "The next step will be to bring Mixed Reality into the operating room," says Fischer. One conceivable option is to display critical data in the field of vision of the operating team. In doing so, it would eliminate the need to divide their attention between a monitor and the actual task at hand. Preoperative image data could also be overlaid on top of the patient's body, providing improved visualization and navigation for the surgeon.

As part of the "sonAR" project, the Fraunhofer Institute for Computer Graphics Research IGD is currently developing this kind of intraoperative application. "We primarily focus on ultrasound-guided needle biopsies. The AR system allows physicians to take a biopsy by displaying and aiming at the target area," explains Matthias Noll of the Fraunhofer IGD in a MEDICA-tradefair.com interview. This is designed to make it easier for physicians to navigate the biopsy needle, since they no longer have to look back and forth between the patient and the ultrasound image.

VR technology can enable more effective learning and teaching in surgery by taking a page from aviation and using a practical simulator. Simulator dolls or models designed to deliver a lifelike representation of the body via different materials have been around for a long time. Meanwhile, digital simulators that either help train surgeons in an immersive virtual environment using VR headsets or facilitate screen-based simulations are less common.

An example of the latter are the surgical simulators by VirtaMed. The Swiss medical technology manufacturer specializes in arthroscopy simulators. These systems combine a realistic anatomical model of the respective joint and endoscopic equipment with specific tasks trainees are asked to perform on the screen. Assignments include the performance of simulated surgical procedures or more abstract exercises such as using a gripper to catch and place stars that hover over the screen.

In the spring, VirtaMed introduced a mobile surgical simulation lab, which brings medical training to hospitals in a converted cargo van. While this surgical simulation training could never fully replace a real surgery, the mobile laboratory gives physicians the opportunity to practice surgical skills and keep up with medical training requirements even during the coronavirus pandemic. It also reduces the need for patient contact in the operating room.

More complex surgical interventions require training that provides a more intensive level of realistic feel and response. The "Dynamic HIPS" project combines an immersive VR environment with instruments that give users realistic haptic feedback as they practice simulated hip replacement procedures. Surgeons in training not only see the operating room and the patient in front of them, but they also feel a response as they perform the individual surgical steps. "The simulator enables them to practice the surgical procedure in a virtual setting before they perform their first real-life surgery under supervision – all without the need for human body donations or animals," Mario Lorenz explains in a MEDICA-tradefair.com interview.

Whether the goal is to avoid close contact during the COVID-19 pandemic or to conserve educational resources, virtual reality simulation-based training offers one key benefit: Surgeons can use it to refine their skills in training and improve their operative performance, even if there are presently no acute cases in their hospital that require treatment. It is a more direct approach to introduce surgeons to surgical procedures and eliminates the need for them to repeatedly observe surgeries before they ultimately perform the procedure on their own. This fosters a greater appreciation and better understanding of the processes, while it also helps them to overcome potential fears and insecurities.