Realistic surgical simulations with AI, VR and haptics

A VR training environment with haptic feedback is intended to realistically depict surgical situations and allow risk-free training of surgical procedures. Prof. Björn Krüger describes the goals of the VIRTOSHA research project in an interview with MEDICA-tradefair.com.

Prof. Krüger, how does the simulation software work?

Prof. Björn Krüger: This training environment is to be developed over the next few years because surgeons have so far lacked haptic feedback during VR training. As many components interact here, we have formed a consortium with various partners. Together with Dr. Kristian Welle, Senior Consultant at the UKB, we want to use full immersion to convey what surgical procedures feel like.

Haptics are an important component when using robotic arms. We also want to integrate tissue and fluid simulation through our partners at the Cologne University of Applied Sciences. For an effective simulation, we want to convey how the tissue opens during an incision, for example.

The component of my working group is precise hand tracking. So far, the accuracy here has been in the region of around one centimeter. For operations, we are more in the sub-millimeter range. We have to use additional sensor technology for this, which is not yet available in the VR environment. This is brought together in a central data hub, which contains a digital twin of all components and then controls the individual components with the information from the tracking. The haptic arm not only performs movements, it also provides information about the forces acting on it.

How is AI being used here?

Krüger: AI methods are necessary for hand tracking, for example, so that a model of the hand can be inserted into the data of movement and camera images. AI methods are used in complex simulation methods. The calculations have to take into account soft and hard tissue as well as bones and fluids.

What are the advantages of the collaboration between the various project partners?

Krüger: In addition to the surgical expertise of Dr. Kristian Welle, the technical expertise of Prof. Arnulph Fuhrmann's group from the Institute for Media and Photo Technology at the TH Köln for computer graphics and simulations will also be brought into the interdisciplinary project. As a computer scientist with a professorship at the University Hospital for Digital Health, I form the bridge, so to speak.

As the project is to result in a product, two companies are involved: MindPort GmbH and Haption GmbH. MindPort will develop a workflow that can be modified and expanded without programming knowledge. For example, the simulation of a fibula fracture could be adapted by our doctors for a patella fracture without having to reprogram. This is made possible by MindPort's authoring system, which runs in a standard 3D environment.

Does the accuracy of the tissue simulation in the VR environment meet medical standards?

Krüger: Our project involves measuring parameters such as forces or the stability of bones, or the tensile strength and cut resistance of tissue. The robotic arm equipped with sensors is also used as a measuring device for this purpose. This gives us large amounts of data and allows us to assess how characteristics behave depending on age and gender. Through interpolation, we can generate data for new patient groups.

Which surgical areas could benefit in particular?

Krüger: We chose fibula fractures as the first example because they occur very frequently. Knee and hip operations can also be trained, as well as procedures involving artificial knee joints, which are more complex due to the implant.

Are there plans to evaluate the success of the training environment and integrate it into existing training programs?

Krüger: First of all, there will be an evaluation based on feedback from experienced surgeons. A complete evaluation of success is not planned or possible within the three-year project period. In the end, we want a result that we can pass on to our doctors in training.