The essence of an expertly executed craft is too nebulous to define. This is the reason why Mingei uses a variety of methods to record and present traditional crafts to the public. However, the difference between an expert and a novice is not their ability to adhere to strict guidelines and instructions, but rather how well they can decide when to break the conventional rules and create something unique. Even more interestingly, those decisions are made subconsciously. The expert can decide what motion will be the most efficient and perform it precisely, seemingly without any preparation. Since the motion of the body is the most important tool of performing a craft, Mingei uses motion capture to document all the subtle details of an expertly execution.

Motion capture

Motion capture (MoCap) is the recording of the joint angles of the human body during a task. In our previous article, we explained our first steps in digitizing the crafters’ movements. Unlike standard video, the only thing that is recorded with MoCap is the human skeleton and its associated motions. Even though general information about the environment and the appearance of the person is lost, this method gives the motion of each joint in detail for all three dimensions. This is why it has been widely adopted for movies and video games as well as medicine for many years. However, until now, its use for preservation of crafts is relatively limited.

The workshop as recording environment

Very early in the MoCap sessions of Mingei, a few differences became apparent when recording for cultural heritage. The first and most significant difference was the environment. In most MoCap sessions in either entertainment or medicine, the recording environment is heavily controlled. The sessions are most of the times indoors, in a room specifically used for MoCap, and whatever tool is required, is usually a stage prop. In contrast, within the Mingei project, all the recordings had to be done in the actual workshop or field while the expert interacted with the equipment as they normally would. As a result, the MoCap was done to be as unobstructive as possible to allow the expert the freedom to perform.

At the glass-blowing pilot for example, the recording did not stop from the moment the expert picked up the molten glass until he was finished. On the other hand, silk-weaving required the expert to use different equipment during the whole process and therefore the MoCap was segmented based on that. Mastic cultivation tasks are performed throughout the year, with weeks passing between them. Therefore, the most distinct tasks were selected and recorded consecutively.

Eye for detail

Another difference regarding MoCap for cultural heritage crafts is that the expert’s gestures are almost exclusively unique. Even when they perform the same task, they will almost always change their motions to account for small variations in the material they are working on. What is more interesting, is that small changes in motion are important because they highlight the expertise (i.e. a novice won’t do them). This is in stark contrast with motions of an industrial worker that operates machinery, or an actor who will perform the same motions many times, or even a patient who will try to keep their movement consistent. The point here is that in cultural heritage crafts, small variations in motion encapsulate proficiency, while in other cases, they are mostly random.

In conclusion, the experience with Mingei showed that the MoCap recordings have some unique requirements. The environment and the end-product will have a lot more impact on the recording protocol than in other cases. In a different context, it is more important to identify a pattern from accumulated data of multiple individuals with a varying level of expertise. In crafts however, each motion of the expert is “valued” more even if it appears only a few times.

Written by Dimitrios Menychtas (Armines)
Top image by FORTH, other images by Armines

About movement sonification

Sonification is defined as the use of non-speech audio to convey information (Kramer 2016, 185–221), and the technique of rendering sound in response to data and interactions (Hermann et al. 2011, 1).

Bevilacqua et al. (2016, 385) describe the benefits of movement sonification in their paper as follows: “Movement sonification is considered the auditory feedback that is being created in relation to a movement and/or gesture performed. The idea of using auditory feedback in interactive systems has recently gained momentum in different research fields. In applications such as movement rehabilitation, sport training or product design, the use of auditory feedback can complement visual feedback. It reacts faster than the visual system and can continuously be delivered without constraining the movements.”

As such, sonification is currently used in a wide variety of fields, such as “driving a car or riding a bike blind, directly finding one’s way in an unfamiliar smoky environment or being able to improve the quality of one’s gestures in real-time” (Parseihian et al. 2016, 1).

In particular, movement sonification systems appear promising for educating complex techniques and skills in providing users with auditory feedback of their own movements. The purpose is guidance for improving movement performance during a learning process and the application can vary.

So, why researchers did put sound on purpose? Hermann et al. (2011, 3) describe that “the motivation to use sound to understand the world (…) comes from many different perspectives (…). Thus, the benefits of using the auditory system as a primary interface for data transmission are derived from its complexity, power, and flexibility.” In other words, the reason of distinguishing sound as relevant to communicate information is found in its inherent feature to have a resonance to one listener and diversity in the way it would be employed.

Considering that movement sonification can guide a user to reproduce an expert gesture, this method is proved suitable while focusing on enhancement of the learning experience and improving the movement performance.

The use case of glassblower’s craft

In the case of glassblowing, the key challenge is the development of a system that can at first track the human body, recognize the gestures that the person is executing and, in the end, compare them with the expert’s gestures and provide sound feedback. The sonification gives a motivation to the user to complete all his tasks/gestures by also reaching to a musical goal. Following the tempo of the sounds is also a helpful feedback on how well the gestures have been performed and in which way they need to be improved.

The researchers of Armines are currently applying movement sonification for vocational training as the most compatible method, taking into consideration the particularities of the craftsmanship. The motion capture of experts and their gestural skills related to glassblowing use case is getting implemented in collaboration of Armines and Cerfav, the National Innovation Center for Glass in France.

For this purpose, recording sessions of the gestural know-how of the glassblower are being organised within Cerfav with the use of high precision motion capture technologies. More precisely, this is done with a special suit equipped with sensors that the expert glassblower wears throughout the sessions of recordings. The data recorded are categorised, so it is specified which gestures are performed, where one gesture stops and another starts, and what are the tools used. This is what we call the “gesture vocabulary”.

The next step is to learn the gestures of a glassblower through gesture recognition and sonic feedback, where a user will try to reproduce the gestures of the professional glassblower. Whenever there is a deviation between the craftsman’s gesture and the apprentice’s gesture, a sound feedback is provided based on the pitch fluctuation of a predefined sound. In this way, the apprentice perceives the quality of the sound and knows if they performed a gesture in the same way as the expert, and if they have to repeat and correct the gesture.

In the video, the installation user is asked to perform one by one the gestures that the routine of the glassblower consists of. The sonification gives motivation to the user to complete all his tasks/gestures by also reaching a musical goal.

First, the user is invited to observe the expert gestures, to hear the correct sonification result of his movements. This is what we call here “the original sounds”. A mapping has been done between motion parameters and acoustic features: The tempo modality is affected by the movement of the right and left hand in the x-axis, while the panning of the sound is affected by the movement on the y-axis.

The gestures of the installation user are being recognized and also sonified. The quality of the sonification – in terms of how close the tempo or the pitch is to the original sounds – is the result of how well the gestures have been performed and recognized. The user is able to perform the gestures one by one, until he reaches the final gesture, thus the creation of the glass carafe. The sounds mapped to each one of the gestures are layered creating a complete music piece at the end of the gestural performance.

The learning process as an enjoyable user experience

Within this learning process developed in Mingei, one apprentice acquires the necessary skillset to perform and reproduce glassblower’s gestures and gradually improve their performance and technical prowess. Furthermore, the process itself is a positive enjoyable experience that brings pleasure and motivation as well as meaning and purpose for learning this craft.

Learning by receiving auditory feedback offers incentives for developing a natural and a non-intrusive experience, that informs the user in real time about their performance and gives them the stimulus to go ahead. Considering also that the end use for Mingei is setting up an installation within a museum, it is desirable that the experience of the visitor has a twist of entertainment and cultural engagement.

Encouraging the new uses of sound can prove to be an excellent, emergent method of embodied learning through learning by doing.

 Written by Ioanna Thanou (Armines)

References

Within the Mingei project, state of the art technology is used to preserve and represent traditional crafts. By digitizing processes, we can preserve the expertise of artisans, and simplify the transmission of craft skills and knowledge from generation to generation by using new innovative technologies, such as virtual reality, which is a technology already under use for educating complex techniques and skills. This expertise could be for example the handling of tools, where certain gesture, force or posture is required to execute the craft effectively.

ARMINES oversees two aspects of the Mingei project related to digitization. Firstly, we execute the motion capture of experts related to the pilots of mastic harvesting (top image), silk weaving  and glass blowing (image below). Secondly, we work on the development of a system that can track a human body in real time, and then recognise the gestures that the person is executing with the aim to compare them with the expert’s gestures and provide some feedback.

What is digitization?

We call digitization any kind of method that creates an electronic archive of the crafts. Standard coloured video is the simplest and most popular method. We also apply other methods  include 3D videos using depth cameras and motion capture to record the joint angles of the human body. Using different methods, it is possible to record and preserve crafts in a way that they can be reproduced. This allows historical arts to be preserved in a more organic way than just using photography and descriptive texts.

The first step of the digitization process is to breakdown the craft movement to components. For the crafts expert, the whole process is a continuous action. However, in order to digitize the motions properly, we had to sit down with the expert and separate each motion to subtasks.

Challenges of motion capture

Of course, the whole endeavor has its own caveats. Logically, the recorded motions have to be from a highly skilled expert, since their motions become the “default” motions to be preserved. In general, the challenges of digitization are related to the environment and the task that is being recorded. For example, mastic harvest is done outside, and the expert has to come into contact with dirt, dust, and tools that may affect the recording equipment. For weaving, the expert has to use bulky equipment (especially the loom) that can obscure the motions making the recording incomplete. The challenge is to record the complete motion, and simultaneously ensure that the digitization process does not interfere with the way the experts perform. As such, there is always a trial period until the best setup to record is found. Once this step is overcome, the digitization is fairly simple.

Another challenge is the presentation of the recorded information. In general, the more technically “rich” a dataset is, the more problematic is to present the data to the general public. For example, the motion capture data are strictly speaking a series of numbers. This requires special consideration on how to make the preserved crafts accessible to everyone. However, within Mingei, crafts that have been important for local economies across EU will be preserved in a functional way. Currently, historical preservation is concerned with items, with no way to preserve the methods that created them. Mingei will preserve the actual motion patterns and strategies making the items an example of a finished product.

Our next step is to continue the recording of experts and improve our methodologies for posture estimation and gesture recognition.

Written by ARMINES