Laboratory automation: from note book to gesture recognition -- MEDICA Trade Fair

Laboratory automation: from note book to gesture recognition

Interview with Marc Andre Daxer, Department of Laboratory Automation and Biomanufacturing Engineering, Fraunhofer Institute for Manufacturing Engineering and Automation IPA


For centuries, scientific research has succeeded by chronicling experiments with pinpoint accuracy. Yet despite all the progress in the actual laboratory, recording is often still done manually, in notebooks, logs or computer systems for instance. In the future, a gesture recognition system could perform this task for scientists.

Image: Smiling young man with short blonde hair and blue shirt - Marc Andre Daxer; Copyright: Fraunhofer IPA/Rainer Bez

Marc Andre Daxer; ©Fraunhofer IPA/Rainer Bez

In this interview with, Marc Andre Daxer talks about the development of a tracking system for laboratory use, defines the areas where it creates added value and reveals in which direction laboratory automation is heading.

Mr. Daxer, you developed a tracking system for the automated documentation of laboratory experiments. How does this system work exactly?

Marc Andre Daxer: Our tracking system recognizes gestures and hand motion. It was designed by our department – Laboratory Automation and Biomanufacturing Engineering – in collaboration with the Department of Machine Vision and Signal Processing. The demonstrator was subsequently implemented by my colleague Christian Jauch from our partner department.

We use a 3D camera in our prototype that is equipped with rudimentary gesture recognition technology already provided by the developer. We can utilize this feature to infer procedural steps from the recognized hand gestures in the right context. For instance, if I hold my hand as if I am holding a long, cylindrical object and make a pressure movement with my thumb, this should be interpreted as pipetting. When we subsequently combine this information with object recognition, which then identifies the pipette and the target vessel, it allows us to significantly increase the accuracy and reliability of the complete system. We continue to enhance these algorithms at the same time and by now have a gesture recognition accuracy of over 90 percent at our fingertips.

Where can this type of automated documentation be used?

Daxer: From an economic perspective, automating laboratory processes is only worth the trouble for high throughput and durable approaches, meaning procedures that are performed frequently – as is the case with standard or quality analyses. However, this is not the case in an experimental environment that’s focused on developing new processes because the processes need to continuously be changed and adapted.

Having said that, it makes sense to record gestures in manual laboratory procedures by using tracking. This makes it easier to capture all the parameters we need to repeat a process. In doing so, we are able to ultimately prevent too many gaps in a publication because we assume very implicit knowledge on the part of the reader. Tracking the process would ensure that it is repeatable.

This is especially interesting if certain details have not been recorded in the lab report. In one case, we had a client who not only performed a simple upward motion when she used the pipette but repeatedly pressed the pipette before she removed the liquid, thereby stirring the mixture. However, this unconscious stirring motion was not recorded in the report, so the process could not be repeated by other people. This was not apparent until gesture recognition detected it.

Bild: Ein Mann arbeitet an einer Laborbank, auf seinem Monitor läuft ein Programm zur Gestenerkennung; Copyright: Fraunhofer IPA/Heike Quosdorf

The tracking system from Fraunhofer IPA is able to document hand movements automatically through 3D image processing. It also analyses them in a smart way. This also saves researchers the trouble of a manual documentation; ©Fraunhofer IPA/Heike Quosdorf

How do you delineate the recognized gestures?

Daxer: We interpret consecutive individual images by using machine learning techniques and compare them with each other. What do the individual images look like before and after? When we analyze the image, are we able to detect a hand and reference it and teach software to do this? We are able to infer a motion when the individual images change, meaning the position of the referenced hand changes over many individual images. When we track the hand, it also allows us to identify gestures.

In practical application, we still encounter some problems when the hand leaves the camera’s vision range and then reenters without the hand being recognized as such. In this case, the camera might lose the hand as an object. This is why it is important for users to first present their hands. In doing so, users also receive immediate feedback, indicating that the system has identified and recognized the hands. Users can subsequently move the hands and the system tracks them accordingly. Needless to say, the area of operation covered by the system will be expanded by several cameras in the future.

What are the current trends in automated laboratory technology?

Daxer: I think assistance systems will gain popularity and support and guide users. Augmented reality will also play a role in this area. I am not necessarily thinking of smart glasses but am contemplating more pragmatic solutions such as glass panels that separate lab benches. For example, transparent screens could be integrated there or information projected with light.

These systems should also have the ability to capture the process and make respective data available. This makes sense from an entrepreneurial perspective. If I have processes that are frequently requested by clients, I have to improve throughput. Ideally, all data I need for fast automation has already been collected and is available. You could use it to equip laboratory automation systems, which perform the process while the manual laboratory procedures would be primarily dedicated to the development of processes. This would create a smooth transition between the use of assistance systems and the automation of laboratory processes.

What are the next steps in your work with the current tracking system?

Daxer: We already have several scenarios and settings in mind where we are able to create an added value with the tracking system. Materials quality control is one example of this: it is typically performed by two people using the four-eyes principle as it were. However, prolonged time spent on routine tasks can be very tiring and might result in human error. Meanwhile, unlike a human being, a tracking system is able to reliably check whether procedures were being adhered to.

In addition, our focus is not only on recording processes and individual steps but to also make them accessible again for other applications via other media such as AR or interactive tutorials. This could be utilized for employment training for example.

Overall, we want to continue to discover the possibilities of laboratory tracking systems to analyze and optimize processes or simply to accurately and reliably record them and make them available in a standardized format.
Image: Man with glasses and beard - Timo Roth; Copyright: B. Frommann

© B. Frommann

The interview was conducted by Timo Roth and translated from German by Elena O'Meara.