Augmented Reality for better laboratory results

Interview with Lynda Metref and Fabian Kappeler, Helbling Technik Wil AG

01.09.2020

Accuracy is paramount in laboratory settings and ensures that lab results are valid. Errors in a lab can render series of tests unusable and waste precious time and money. In the medical realm, this might even result in clinical trial errors. Augmented reality (AR) can help laboratory technicians to prevent errors and guide their work in the future.

Lynda Metref

In this MEDICA-tradefair.com interview, Lynda Metref and Fabian Kappeler talk about the concept of an AR-assisted pipetting system, explain how the company came up with the idea, and describe how this product could be used in the laboratory.

Ms. Metref, Mr. Kappeler, at the MEDICA 2019 trade fair, you introduced an AR-assisted (AR) pipetting system as part of the MEDICA LABMED FORUM. What prompted the idea for this product?

Lynda Metref: We wondered how pipetting could be done in the future? How can we use technology to support pipetting accuracy? And since pipetting is still frequently done manually, what are the challenges for lab technicians?

We introduced a system with a camera that tracks objects on the laboratory bench. With the help of augmented reality (AR), we can superimpose process details to assist the user. This enables the tracking of events during the lab work. Image recognition helps identify the reagent that was picked up with the pipette and helps assess whether the volumes are accurate.

Fabian Kappeler

Fabian Kappeler: Our Company offers a wide range of professional engineering services to our clients. This includes new ideas that are on the cusp of applied research and cutting-edge technology, which can then be turned into a product as part of a project development. In the laboratory arena, we have many years of experience developing automated laboratory equipment, robotics, and automatic analysis systems.

Metref: Right now, the integration of AR into pipetting procedures is still a concept we would like to explore with an industry partner. We pursue a user-centered approach and believe this is an important subject that could have future potential for new product development.

What was the catalyst for this development?

Kappeler: It was part of an in-house project in which we studied laboratory work and trends in the clinical laboratory industry during the course of an innovation week. We challenged ourselves as engineers to contribute to “Laboratory 4.0” or the “Laboratory of the Future”. We actualized the demonstrator as part of a hackathon and later realized that people were very interested in our “workshop idea” since it meets a market need. I am certain that it resonates with our audience.

Image: A young laboratory technician with AR glasses uses a pipette, he is surrounded by different bubbles with text; Copyright: Helbling Technik Wil AG

The smart, AR-supported pipetting system could support laboratory work in different ways. As a result, employees would ba able to concentrate on their actual work again.

How does the system support laboratory work tasks?

Metref: Pipetting is a skill that requires long-term concentration. Technicians must accurately carry out the process steps, protocol the steps at the same time, and master the practical skills of handling reagents and lab equipment. It is demanding work. They also have to deal with distractions, all of which is stressful. Automated process documentation could relieve the burden on users and significantly improve the quality of work, especially as it pertains to error reduction.

Image: A pipette, a microwell plate and a glass with markings are positioned in front of a screen that shows these items as well; Copyright: Helbling Technik Wil AG

Using a camera, image recognition and markings, laboratory items can be tracked. This can avoid mistakes during experiments.

Kappeler: Manual pipetting is used for small sample volumes and/or occasional procedures or a process that has to be modified. It takes concentration, and if you make a mistake, you may not notice it until much later with potentially enormous consequences. In research, this might mean that results ultimately do not match and the work you did for several weeks was all for nothing, or that precious, unique samples are now lost.

Our idea is to keep the benefits of manual pipetting, i.e., the efficiency and flexibility when pipetting small volumes, but still have the quality control and documentation of a typical automated pipetting system. These systems are usually programmed for high-throughput screening and large-volume samples that are always pipetted in the same way. However, automation does not efficiently support smaller batches.

Have you identified specific tasks where an AR system is especially beneficial?

Kappeler: We believe augmented reality is an exciting aspect in general. For example, it enables you to give users job instructions by superimposing the procedures for particular assays.

The entire process could be stored electronically as a protocol. The sequence of identified steps would then be automatically compared with the stored lab protocol. This allows the system to detect in real-time whether or not all steps have been performed accurately. This might even involve the system’s refusal to dispense from the pipette if it detects that the lab technician is about to make a mistake.

What do you think is next for "Laboratory 4.0"?

Kappeler: Digitization, networked instruments, and smart products are trends that permeate our world. The laboratory and its functions are still somewhat on the conservative end of this spectrum. Having said that, the aforementioned trends will eventually also make their way into the laboratory realm. Smart devices will increasingly support the various functions of laboratories, making life in the lab easier and results more accurate and reliable.