In museums, objects are often exhibited separately. Tools are shown alongside other tools, and glass objects are exhibited together with other glass objects. How do you tell a coherent and captivating story, connecting the dots between different exhibited objects? In co-creation with CNAM (Conservatoire national des arts et métiers), Waag has been prototyping a digital experience for Mingei’s pilot on glass. Developer Lodewijk Loos takes you along the journey towards the first prototype.

Visiting CNAM

The goal of creating this digital experience at CNAM is to engage visitors and give them insight into the process of glass making. The prototype should work on site (in this case in the context of the museum), and should add to the already available real objects on display. However, the technology used for the prototype should be non-obtrusive to the local situation. Visitors who do not wish to use the technology, should not be bothered by it.

In order to get a grasp of the local context at CNAM, Meia Wippoo and Lodewijk Loos of Waag went to Paris in March 2020. There, we had a fruitful co-creation session with a team of museum professionals from CNAM and conceptualized a rough version of the prototype. In the glass section of the museum, we made some observations that were key to the first version of the prototype.

First of all, the glass objects are exposed in vitrines, they couldn’t be touched or picked up and could not be looked at from all angles. Of course, not being able to pick up objects in a museum is normal. However, as a lot of these objects are tools and utensils, being able to do so would contribute to the understanding of the object. We also know from experience in earlier projects, like meSch, that being able to pick up objects leads to more user engagement.

The next thing we noticed is that some objects were related to other objects that were displayed in different rooms of the museum. For example, the glass tools and a glass product were not in the same room. The reason for this is that there are different ways to classify object. The tools were in the tooling section and the carafe was in a section with artworks. However, these objects are part of the same story that we would like to tell: the process of glass making.

Another observation that we made was that some of the objects key to the story were not on display in the museum, for example a furnace and piece of wet paper were not there.

Augmented reality

As we decided upfront, the prototype should help to get insight in the process of glass making. With these observations, we could translate the story of glass in a more generic story. One could say that in the context of crafts, a general pattern is that objects are used with other objects (for example tools with materials), in different parts of the process. That is what we want our digital experience to give insight in.

We also concluded that the use of augmented reality (AR) technology could be of value for this prototype. With AR, it is possible to create the sense of picking up (virtual representatives of) objects, use them in another room, and show objects that are not physically there.

Mark the process

We returned back home and worked out several concepts. Next, we aggregated common interaction principles from our concepts. Our interaction principles showed similarities to (adventure) games. Adventure games are like a puzzle: you often have to pick up objects, sometimes not yet knowing what for, and use them at another location, sometimes in combination with another object.

One of our concepts focused at the carafe, named “Mark the Process”. This concept would lend itself for this type of adventure-like (mini) game. The central piece in this game would be the various parts and stages of completion of the carafe. This is how the process of making this type of carafe is currently displayed in CNAM. Wouldn’t it be nice if you had pick up the tools associated with this process in the one room, and place them at the right “step” in the other room? We also liked the idea of being able to collect museum objects and take them home for closer inspection.

The use of markers

With this concept in mind, we started implementing a proof of concept as a smartphone app. From previous AR projects, we had experience with the combination of Vuforia (AR framework) and Unity3D (gaming engine). The former is very well integrated in the latter, making it an ideal tool for (at least) prototyping. Vuforia support various ways of augmentation, both marker-based as marker less.

Markers are physical signs that are recognized by the app to instigate interaction. Using markers makes the app less dependent on local lighting conditions, which were not ideal or constant at CNAM. Recognising a marker, instead of an object itself, generally just works better. Additionally, using markers could make it easier for users of our app to see at which locations in the museum they could interact, because they serve as a visual clue. When you’re in a museum with thousands of objects, it is convenient that you can see immediately (without using a device) which ones are interactable. Finally, markers are easier in use. Augmenting an object by placing a marker in front of it is less challenging then having to scan the object and markers make it also easy to place objects in the void. In the longer run, the use of markers helps to accomplish a more generic application for different venues with different content, that allows its content to be authored by curators (as opposed to software developers).

Living room demonstration

Our original intention was to test the prototype at CNAM with random visitors of the museum. But during the development of the app, Covid-19 came around and it became clear that testing the app in a public venue with a real audience would not be possible anytime soon. Furthermore, the Covid-19 situation might even change the way we design things permanently. For example, it might have become undesirable to have devices in a museum that are handed out to visitors or to have installations with touch screens. An AR app that people can run on their own phone should be relative safe and convenient.

With this in mind, we slightly changed our prototyping strategy and made the decision to create a living room demonstration. Originally, the prototype was meant to include virtual copies of the museum objects. By the lack of museum objects in the developer’s house, we used general building tools and convincing 3D models from online repositories.

The prototype demonstrates a few of the principles. The user can pick up object and place them back again, objects can be collected in a treasure chest for later use, objects can be used with other objects by using them with a marker next to that other object, referenced media for the collected object is available as background information, information overlays (giving hints) can be shown and a collection of objects can be used to make simple puzzles. As a gamification element, the user receives badges after completing specific tasks or reaching certain goals.

Next steps

This simple approach allows for a lot flexibility to create puzzle-like games. For example, a timeline game could be created by changing the physical placement of the markers into another linear layout. One could also imagine having different kinds of visual markers for different kind of interactions. One type of marker could indicate that an object can be picked up, and another marker could indicate that an object can be used at that spot.

At this point it is also interesting to think about how these principles can be applied at the other pilot locations. Part of the Mingei project is a pilot in Chios (Greece) on the craft of harvesting and processing Mastic from the mastic tree. Would it be feasible to apply the prototype at the local situation over there by augmenting the Mastic tools and placing markers on and around a real tree? There is still enough work to be done and questions to be answered towards a generic AR application for on-site craft experiences!

Written by Lodewijk Loos (Waag)

The history of craft goes back almost to the history of man: sharpening stone, carving wood, weaving fabric were amongst the first things human beings had to master to make their way up to our days. Most of the objects that we use in our life today are still the results of highly-skilled craftsmanship. Even those objects that are made with the help of sophisticated machines, would not be possible without the masterful skills of people who designed the machines and their usage in the process.

Preserving crafts, even those that are no longer in use today, is paramount to preserve our history and the many wonderful techniques that we have invented to tame matter and make the world a pleasant and comfortable place to live in. Preserving and representing crafts is the mission of Mingei. One of the major challenges that the Mingei project is facing, is to use computer-based technologies to represent an exquisitely, intrinsically human activity. How do we digitize something that is so intangible and dynamic?

Representing crafts using state-of-the-art technology

To achieve this ambitious goal, we use a three-step approach at Mingei: we collect, we connect and we open up. In each step, we put Information Technology at the service of the step’s goals, employing a wide range of techniques to maximize the quality of the result.

We collect

Humans are the primary sources of the knowledge about craft that Mingei collects. We talk with craftsmen and craftswomen to hear their stories. What needs to be done in the craft they master, and when, and where, and how? People knowledgeable about the crafts are our primary source of knowledge. We run co-creation sessions with them, in which we all interact to obtain from them all they know about the craft they master. And we record what the people we work with say, using digital audio-visual tools and techniques to obtain the most from our interaction with them. But we also track their movements, placing sensors on key parts of their bodies. This way we are able to document the most minute details of their actions during crafting, both verbally and physically.

The secondary sources of knowledge that we look at are books, articles, images, movies, web sites and so on, that report knowledge relevant to Mingei. Our humanist scholars, including anthropologists, historians and sociologists, all with an expertise in craft, study and research to learn what are the most relevant of those sources. They then explore them, to extract the relevant stories and notions which they then encode in digital form. All this knowledge, gathered from primary and secondary sources in digital form is imported into the Mingei knowledge base, which is the digital repository where the project stores the knowledge that it needs to preserve crafts.

During the knowledge collection phase, the pilots first provided pre-existing digital content, such as photographic documentation of museum exhibits, video documentaries and curated literature. Next, we created new digitisations by photographic documentation of each pilot sites, including the museums, machines, pre-existing photographs, catalogues, workshops, etc. All kinds of objects, such as weaving looms, garments, mastic trees, mastic villages, tools, traditional clothing, and glass instruments were digitized with 3D reconstruction technology. With use of motion capture technology, we recorded the meticulous and skilled movements of crafts practitioners. This collection of knowledge on the three pilot crafts of Mingei is supplemented with the knowledge from open repositories and online resources.

The video above is an example of 3D reconstruction. Here we see a handheld machine that is used for the cultivation of mastic on Chios. The image at the top of this article shows another 3D reconstruction, featuring a woman during the process of cleaning mastic on Chios.

We connect

The knowledge collected in the first step is formed by many elements of diverse nature, each addressing some particular aspect of some particular craft. Precious as it is, this knowledge does form yet a set of stories that can be used to document crafts: the elements it consists of need to be connected into coherent wholes that convey meaningful messages to the Mingei user audience. Performing this connection is the objective of the second step. This step uses semantic information as a medium and narratives as the tool to connect the knowledge elements. That is, we use stories as coherent wholes that convey the knowledge about crafts, and in particular stories shaped as semantic networks, to make them as readable and as easy to understand as possible using today’s information technology.  Examples are stories about the Jacquard’s loom, the Krefeld textile industry, the history of Bontemps’ life, the construction of Crystal Palace, the narrative of Isidore of Chios, the story of mastic chewing gum, and many more.

Every story created by Mingei is a rich network, consisting of two basic elements: the schema of the craft, and a set of executions of the schema. The schema is a description of the activities needed to make that craft, and of the order in which these activities must be done, that is, which activities need to be done before, or after, or in parallel to which activity. Like a blueprint or a manual of a craft.

An execution of the schema represents an actual performance of the craft, as a set of real actions carried out by some craft master in a certain place and at a certain time, in the order prescribed by the schema. Both schema and executions are represented as semantic networks, that is set of RDF (Resource Description Framework) triples. Some of these triples link activities and events to the knowledge elements that document them. These networks are then lent to the third step of the Mingei approach.

We open up

The RDF triples produced in the second step encode knowledge in a way that is known only to the people which created them. To communicate this knowledge to the different types of users Mingei addresses, a non-negligible effort is required. This effort is done in the third step of the Mingei approach. Here again, we resort to co-creation: we run sessions with museum experts to design apps that will allow users to discover, access, understand and enjoy the knowledge about crafts we have mustered. This is the way Mingei opens knowledge about crafts to the outside world. At the moment, we are co-designing immersive digital experiences, such as an app that enables the virtual creation of patterns and textiles, a digital city exploration of Krefeld, a mastic cultivation training app, and a digital glass experience.

At the same time, we make sure our semantic networks will be safely preserved for long-term access in the future by applying digital preservation techniques to them. In essence, we add further knowledge to our network and we archive them in special archives, so that they will be accessible and usable for a long time after the Mingei project is over.

This video shows an example of the digital experiences that Mingei is currently developing. The game is set up in the past, in the physical landscape of Chios created through satellite depth maps and exploits the aerial 3D scans of Chios villages created by Mingei.

Timeline

In order to maximize its performance, Mingei is running the three steps in parallel, with the beginning of the steps scaled in time, so that each step is able to receive its input from the previous one. At present, Mingei has completed the first two steps on all three pilots, and has begun the third one. The first finalized digital experience will be launched soon, so stay tuned to hear the news!

Written by Carlo Meghini (CNR)

Digitisation of cultural heritage turns Europe’s cultural resources into an important pillar of the digital economy. The EU Work Plan for Culture also raises the issue of digitisation of cultural content and suggests that digital services can foster the expansion of trans-European tourism networks. Massive digitization of ancient documents and artefacts is becoming a reality; according to new research conducted by Axiell, museums’ audience engagement strategies are focusing on digitisation of the collection as well as on the use of social media and informative websites to deliver public value and educational opportunities.

Developing mobile apps

Museums are also leveraging on technologies such as smartphones and tablets, with 33% citing that visitors can use their own devices to access a complete guide of the museum and almost 40% increasing their investment to do the same.

Within the context of the Mingei project, imaginary is developing a collection of mobile apps to preserve those Intangible Heritage Crafts that are part of the history and economic life of the areas and communities in which they flourished. Visitors will be accompanied in distant worlds experiencing customs and traditions to make them meet ancient crafts: the art of silk weaving (Haus der Seidenkultur, in Krefeld), the mastic cultivation and processing (Piraeus Bank Group Cultural Foundation, Greece) and the reproduction of a carafe from 1842 (Conservatoire National des Arts et Métiers – CNAM, Paris).

A new reality

These mobile apps will enrich the museum experience and contribute to guarantee visitors’ engagement thanks to the usage of Augmented, Virtual Reality. Augmented reality is the process of using technology to superimpose images, text or sounds on top of what a person can already see. It uses a smartphone or tablet to alter the existing picture. Users stand in front of a scene and holds up their device. It will show them an altered version of reality. Since 2010, an increasing amount of AR applications in the Cultural Heritage domain is recorded [1] representing one of the most significant benefits from a supply perspective because an increasing number of tourists are nowadays looking out for unique and memorable on-trip experiences [2].

On the other hand, Virtual Reality offers total immersion in a different reality and replaces what the user sees with an alternate reality and requires specialist technology, such as headsets, controllers and sensors.

With these technologies, combined with a powerful storytelling based on the contents provided by museums, imaginary will create a remarkable blend of historical artefacts, characters, activities to offer dramatic, emotionally engaging stories that can be experienced while at a cultural site or remotely.

In conclusion, the digitization of both tangible and intangible cultural heritage means to safeguard our living heritage bridging our past and our future: an important community-building practice, it helps to ensure the longevity of invaluable customs and practices and it validates life experiences of older members of a community.

1. Jung, T., & Han, D. (2014). Augmented reality (AR) in urban heritage tourism—eReview of tourism research. Manchester: Manchester Metropolitan University.
2. Yovcheva, Z., Buhalis, D., & Gatzidis, C. (2013). Engineering augmented reality tourism experiences. In L. Cantoni & Z. Xiang (Eds.), Information and communication technologies in tourism 2015. Vienna: Springer International Publishing.

Written by Imaginary