As a way to represent and preserve Heritage Crafts in the context of Mingei, Virtual Humans are used to reenact and recreate the crafts in Virtual Environments. If we’re looking at the interaction between crafts and other (virtual) human beings, hand gestures play an important part. And when investigating hand gesture interaction, the challenge is to deal with tow moving objects and not a hand attached to a static object. Take partner dances for example. Partner dances like Salsa use a lot of hand-to-hand interaction, which can be beneficial and applied to hand-to-medium interaction, especially if the medium requires complex manipulation.
In this article we’d like to present a specific part of the experiment that relates to the Mingei project. In Figure 1 you can see a virtual partner that is defined as a 3D avatar whose upper body can be manipulated by the user via hand gestures. While seeing his or her own hands in the virtual world, the user can employ the trigger under his/her index to enable snapping his/her hands to the virtual partner ones, as in social dance. Then by pulling or pushing, the upper body will move accordingly by bending the thorax, but without motion in space or translation. Similarly to social dance, the lower body of the virtual partner, that embeds the current dance pattern, can change depending on the information transmitted by the leader of the dance (in our case, the user is the leader). To simulate the feeling of taking the lead in the dance, gestures in a left or right direction are detected at certain moments in time which correspond with the ‘leading motion’, as described in Figure 2.
The virtual partner will change his/her dance pattern according to the direction wanted by the user. The system has been developed using HTC Vive VR system and Unity 3D. In terms of gamification, the application embeds a visual environment, explications, and a learning exercises with start/stop signals. During the game, each user has to “hold” hands with the virtual partner and dance in synchronisation with the music. A total of 8 exercises with different tempo is defined based on previous studies. An overview of the immersive scenario is illustrated in Figure 3.
People with and without dance experience have both tried the system and its exercises, and both groups were very positive about the quality of the simulation and the competitive aspect, which offers them the opportunity to share the experience with friends. Overall, this system, which uses inverse kinematic, vibration and visualization of hands, shows to be of great interest for professionals and can be reused in another context, for example within cultural heritage such as handcraft materials or manipulations.
Written by MIRALab Sarl
References: Senecal, S., Nijdam, N.A., Aristidou, A. , Magnenat Thalmann, N. Salsa dance learning evaluation and motion analysis in gamified virtual reality environment. ia Tools Applications, 79, 24621–24643 (2020).