The essence of an expertly executed craft is too nebulous to define. This is the reason why Mingei uses a variety of methods to record and present traditional crafts to the public. However, the difference between an expert and a novice is not their ability to adhere to strict guidelines and instructions, but rather how well they can decide when to break the conventional rules and create something unique. Even more interestingly, those decisions are made subconsciously. The expert can decide what motion will be the most efficient and perform it precisely, seemingly without any preparation. Since the motion of the body is the most important tool of performing a craft, Mingei uses motion capture to document all the subtle details of an expertly execution.

Motion capture

Motion capture (MoCap) is the recording of the joint angles of the human body during a task. In our previous article, we explained our first steps in digitizing the crafters’ movements. Unlike standard video, the only thing that is recorded with MoCap is the human skeleton and its associated motions. Even though general information about the environment and the appearance of the person is lost, this method gives the motion of each joint in detail for all three dimensions. This is why it has been widely adopted for movies and video games as well as medicine for many years. However, until now, its use for preservation of crafts is relatively limited.

The workshop as recording environment

Very early in the MoCap sessions of Mingei, a few differences became apparent when recording for cultural heritage. The first and most significant difference was the environment. In most MoCap sessions in either entertainment or medicine, the recording environment is heavily controlled. The sessions are most of the times indoors, in a room specifically used for MoCap, and whatever tool is required, is usually a stage prop. In contrast, within the Mingei project, all the recordings had to be done in the actual workshop or field while the expert interacted with the equipment as they normally would. As a result, the MoCap was done to be as unobstructive as possible to allow the expert the freedom to perform.

At the glass-blowing pilot for example, the recording did not stop from the moment the expert picked up the molten glass until he was finished. On the other hand, silk-weaving required the expert to use different equipment during the whole process and therefore the MoCap was segmented based on that. Mastic cultivation tasks are performed throughout the year, with weeks passing between them. Therefore, the most distinct tasks were selected and recorded consecutively.

Eye for detail

Another difference regarding MoCap for cultural heritage crafts is that the expert’s gestures are almost exclusively unique. Even when they perform the same task, they will almost always change their motions to account for small variations in the material they are working on. What is more interesting, is that small changes in motion are important because they highlight the expertise (i.e. a novice won’t do them). This is in stark contrast with motions of an industrial worker that operates machinery, or an actor who will perform the same motions many times, or even a patient who will try to keep their movement consistent. The point here is that in cultural heritage crafts, small variations in motion encapsulate proficiency, while in other cases, they are mostly random.

In conclusion, the experience with Mingei showed that the MoCap recordings have some unique requirements. The environment and the end-product will have a lot more impact on the recording protocol than in other cases. In a different context, it is more important to identify a pattern from accumulated data of multiple individuals with a varying level of expertise. In crafts however, each motion of the expert is “valued” more even if it appears only a few times.

Written by Dimitrios Menychtas (Armines)
Top image by FORTH, other images by Armines

How can the latest digital technologies help people to understand, protect and promote old and traditional crafts? To answer that question, Euronews dived into the Mingei project to learn about the innovative ideas that engage today’s youth with past traditions.

“Mingei is aiming to capture the motion and tool usage of heritage crafts practitioners, from living human treasures and archive documentaries, in order to preserve and illustrate skill and tool manipulation,” Euronews explains.[1] In the video, Xenophon Zabulis and Nikolaos Partarakis, computer scientists at FORTH in Greece, show how they create 3D scans of the tools that are used in traditional cheesemaking and weaving.

After digitization, the information will then be available through compelling presentations, using storytelling and educational applications, based on AR and MR and the Internet. In a second video, they showcase some interactive installations in a museum setting, that are engaging people with the crafts.

For example, they show an interactive comicbook that describes and shows people how glassmaking was done in France years ago. Another example is an old dial-up telephone that allows the museum visitors to hear the traditional songs that were sang during the mastic harvest at Chios.

“In order to maintain the traditional crafts, we must draw the interest of people in different ways. We have to modernize the stories,” Xenophon tells. Digital tools can help with that goal.

The videos are also available in French and German.

References

The history of craft goes back almost to the history of man: sharpening stone, carving wood, weaving fabric were amongst the first things human beings had to master to make their way up to our days. Most of the objects that we use in our life today are still the results of highly-skilled craftsmanship. Even those objects that are made with the help of sophisticated machines, would not be possible without the masterful skills of people who designed the machines and their usage in the process.

Preserving crafts, even those that are no longer in use today, is paramount to preserve our history and the many wonderful techniques that we have invented to tame matter and make the world a pleasant and comfortable place to live in. Preserving and representing crafts is the mission of Mingei. One of the major challenges that the Mingei project is facing, is to use computer-based technologies to represent an exquisitely, intrinsically human activity. How do we digitize something that is so intangible and dynamic?

Representing crafts using state-of-the-art technology

To achieve this ambitious goal, we use a three-step approach at Mingei: we collect, we connect and we open up. In each step, we put Information Technology at the service of the step’s goals, employing a wide range of techniques to maximize the quality of the result.

We collect

Humans are the primary sources of the knowledge about craft that Mingei collects. We talk with craftsmen and craftswomen to hear their stories. What needs to be done in the craft they master, and when, and where, and how? People knowledgeable about the crafts are our primary source of knowledge. We run co-creation sessions with them, in which we all interact to obtain from them all they know about the craft they master. And we record what the people we work with say, using digital audio-visual tools and techniques to obtain the most from our interaction with them. But we also track their movements, placing sensors on key parts of their bodies. This way we are able to document the most minute details of their actions during crafting, both verbally and physically.

The secondary sources of knowledge that we look at are books, articles, images, movies, web sites and so on, that report knowledge relevant to Mingei. Our humanist scholars, including anthropologists, historians and sociologists, all with an expertise in craft, study and research to learn what are the most relevant of those sources. They then explore them, to extract the relevant stories and notions which they then encode in digital form. All this knowledge, gathered from primary and secondary sources in digital form is imported into the Mingei knowledge base, which is the digital repository where the project stores the knowledge that it needs to preserve crafts.

During the knowledge collection phase, the pilots first provided pre-existing digital content, such as photographic documentation of museum exhibits, video documentaries and curated literature. Next, we created new digitisations by photographic documentation of each pilot sites, including the museums, machines, pre-existing photographs, catalogues, workshops, etc. All kinds of objects, such as weaving looms, garments, mastic trees, mastic villages, tools, traditional clothing, and glass instruments were digitized with 3D reconstruction technology. With use of motion capture technology, we recorded the meticulous and skilled movements of crafts practitioners. This collection of knowledge on the three pilot crafts of Mingei is supplemented with the knowledge from open repositories and online resources.

The video above is an example of 3D reconstruction. Here we see a handheld machine that is used for the cultivation of mastic on Chios. The image at the top of this article shows another 3D reconstruction, featuring a woman during the process of cleaning mastic on Chios.

We connect

The knowledge collected in the first step is formed by many elements of diverse nature, each addressing some particular aspect of some particular craft. Precious as it is, this knowledge does form yet a set of stories that can be used to document crafts: the elements it consists of need to be connected into coherent wholes that convey meaningful messages to the Mingei user audience. Performing this connection is the objective of the second step. This step uses semantic information as a medium and narratives as the tool to connect the knowledge elements. That is, we use stories as coherent wholes that convey the knowledge about crafts, and in particular stories shaped as semantic networks, to make them as readable and as easy to understand as possible using today’s information technology.  Examples are stories about the Jacquard’s loom, the Krefeld textile industry, the history of Bontemps’ life, the construction of Crystal Palace, the narrative of Isidore of Chios, the story of mastic chewing gum, and many more.

Every story created by Mingei is a rich network, consisting of two basic elements: the schema of the craft, and a set of executions of the schema. The schema is a description of the activities needed to make that craft, and of the order in which these activities must be done, that is, which activities need to be done before, or after, or in parallel to which activity. Like a blueprint or a manual of a craft.

An execution of the schema represents an actual performance of the craft, as a set of real actions carried out by some craft master in a certain place and at a certain time, in the order prescribed by the schema. Both schema and executions are represented as semantic networks, that is set of RDF (Resource Description Framework) triples. Some of these triples link activities and events to the knowledge elements that document them. These networks are then lent to the third step of the Mingei approach.

We open up

The RDF triples produced in the second step encode knowledge in a way that is known only to the people which created them. To communicate this knowledge to the different types of users Mingei addresses, a non-negligible effort is required. This effort is done in the third step of the Mingei approach. Here again, we resort to co-creation: we run sessions with museum experts to design apps that will allow users to discover, access, understand and enjoy the knowledge about crafts we have mustered. This is the way Mingei opens knowledge about crafts to the outside world. At the moment, we are co-designing immersive digital experiences, such as an app that enables the virtual creation of patterns and textiles, a digital city exploration of Krefeld, a mastic cultivation training app, and a digital glass experience.

At the same time, we make sure our semantic networks will be safely preserved for long-term access in the future by applying digital preservation techniques to them. In essence, we add further knowledge to our network and we archive them in special archives, so that they will be accessible and usable for a long time after the Mingei project is over.

This video shows an example of the digital experiences that Mingei is currently developing. The game is set up in the past, in the physical landscape of Chios created through satellite depth maps and exploits the aerial 3D scans of Chios villages created by Mingei.

Timeline

In order to maximize its performance, Mingei is running the three steps in parallel, with the beginning of the steps scaled in time, so that each step is able to receive its input from the previous one. At present, Mingei has completed the first two steps on all three pilots, and has begun the third one. The first finalized digital experience will be launched soon, so stay tuned to hear the news!

Written by Carlo Meghini (CNR)

Within the Mingei project, state of the art technology is used to preserve and represent traditional crafts. By digitizing processes, we can preserve the expertise of artisans, and simplify the transmission of craft skills and knowledge from generation to generation by using new innovative technologies, such as virtual reality, which is a technology already under use for educating complex techniques and skills. This expertise could be for example the handling of tools, where certain gesture, force or posture is required to execute the craft effectively.

ARMINES oversees two aspects of the Mingei project related to digitization. Firstly, we execute the motion capture of experts related to the pilots of mastic harvesting (top image), silk weaving  and glass blowing (image below). Secondly, we work on the development of a system that can track a human body in real time, and then recognise the gestures that the person is executing with the aim to compare them with the expert’s gestures and provide some feedback.

What is digitization?

We call digitization any kind of method that creates an electronic archive of the crafts. Standard coloured video is the simplest and most popular method. We also apply other methods  include 3D videos using depth cameras and motion capture to record the joint angles of the human body. Using different methods, it is possible to record and preserve crafts in a way that they can be reproduced. This allows historical arts to be preserved in a more organic way than just using photography and descriptive texts.

The first step of the digitization process is to breakdown the craft movement to components. For the crafts expert, the whole process is a continuous action. However, in order to digitize the motions properly, we had to sit down with the expert and separate each motion to subtasks.

Challenges of motion capture

Of course, the whole endeavor has its own caveats. Logically, the recorded motions have to be from a highly skilled expert, since their motions become the “default” motions to be preserved. In general, the challenges of digitization are related to the environment and the task that is being recorded. For example, mastic harvest is done outside, and the expert has to come into contact with dirt, dust, and tools that may affect the recording equipment. For weaving, the expert has to use bulky equipment (especially the loom) that can obscure the motions making the recording incomplete. The challenge is to record the complete motion, and simultaneously ensure that the digitization process does not interfere with the way the experts perform. As such, there is always a trial period until the best setup to record is found. Once this step is overcome, the digitization is fairly simple.

Another challenge is the presentation of the recorded information. In general, the more technically “rich” a dataset is, the more problematic is to present the data to the general public. For example, the motion capture data are strictly speaking a series of numbers. This requires special consideration on how to make the preserved crafts accessible to everyone. However, within Mingei, crafts that have been important for local economies across EU will be preserved in a functional way. Currently, historical preservation is concerned with items, with no way to preserve the methods that created them. Mingei will preserve the actual motion patterns and strategies making the items an example of a finished product.

Our next step is to continue the recording of experts and improve our methodologies for posture estimation and gesture recognition.

Written by ARMINES