BMBF research project on skillful interaction with real tools in virtual worlds

The accuracy and dexterity of the hand is crucial in many areas where tasks need to be performed that require precision. So far, mixed-reality applications have been unsuitable for learning complex motor skills. The techniques used to determine the spatial position of the hands and held objects are too imprecise. The goal of the GreifbAR project is thus to develop methods that are more precise than previous systems.

Objectives and approach

The objective of the collaborative project is to develop an intelligent learning system for a mixed reality environment where users can practice high-precision motor skills with the help of a simulator. Such a learning system could be used to learn the complex handling of tools or how to tie surgical knots. For this purpose, new methods for hand-object interaction are being studied. They are based on the assessment of video recordings by artificial intelligence and enable fine object manipulation in mixed reality for the first time. As a result, interaction is perceived to be more realistic, improving the learning outcome.

Innovation and perspectives

The new training simulators are designed to improve the quality of training, reduce training costs and minimize risk in two application contexts. The intelligent learning system can be used for both familiarisation and ongoing employee training. It also lends itself for use in other fields of application.

Source: Federal Ministry of Education and Research

The Chair of Psychology and Human-Machine Interaction (Professor Susanne Mayr) is charged with psychological evaluation in the project. For this purpose, relevant parameters in the human-machine interface need to be identified and verified. The goal is to develop a system with the help of its prospective users that is as user-friendly as possible.

Principal Investigator(s) at the University	Prof. Dr. Susanne Mayr (Lehrstuhl für Psychologie mit Schwerpunkt Mensch - Maschine - Interaktion)
Project period	01.10.2021 - 30.09.2024
Source of funding	BMBF - Bundesministerium für Bildung und Forschung