Silk implementation timeline

Implementation timeline

 
 
 
 
 

Applying the Mingei protocol in the Silk pilot

STEP 1. Human resources and digital assets

In this step, we collect the documentation that will be eventually transformed into knowledge and history that we will digitally represent.

Human resources are invaluable in the description and explanation of craft practice and context. In this process, digital assets serve a range of purposes, from note-taking to reference practices by practitioners. Moreover, sometimes the contribution of a human resource may be available only through a recording.

Digital assets are carriers of documentation, information, and knowledge that required a comprehensive representation of craft. Moreover, digital assets that are appreciated by human observers (i.e., audio-visual) can encompass an insightful understanding of processes, environments and semiotics.

In this context, the following activities were conducted:

Knowledge collection and literature review

Craft understanding is initiated with the collection of existing knowledge of the following categories:

  • Basic knowledge including (a) curated text, (b) curated material, and (c) formal representation
  • Existing content includes (a) archives, (b) bibliographic research outcomes, (c) audiovisual material, and (d) digital representations
  • Contextual information (1) brief description of the craft (2) geographical location and range of the craft, (3) selected and representative communities concerning the craft, (4) craft workers / skilled workers / handicrafts education, (5) gender roles, (6) equipment, (7) craft products, (8) craft traditions incuding oral tradition / story telling / work songs / myths & legends, (9) social practices / social dimension, (10) economic dimension, (11) design dimension, (l2) artistic dimension, (13) rituals, (14) festive events, (15) religious dimension, (16) learning / education / transmission of knowledge, (17) geography of workshops, (18) emic (inside) Presentation
  • Narrative interviews with craft practitioners

Social and Historic Context

Data collection

In this step, the components of the crafting process are digitally recorded, post-processed, and curated in an online platform.  

These digital assets record objects and actions, acquired from documenting photographically and in 3D materials, tools, products, and workspaces and recording practitioner crafting actions, in recording sessions. The organization of these sessions is facilitated by the vocabulary and storyboard, in identifying the objects, sites, and practitioner actions to be digitized. Moreover, some of the photographic and video assets may be recorded in combination with ethnographic fieldwork.

Audiovisual recordings

Pertinent assets regard conventional audiovisual data acquired from the ethnography such as audio and video interviews photographic documentation, as well as documentation to be used for craft representation including photographs and video documentation of objects, spaces, and demonstrations.

Audiovisual recordings of the crafting process are important for overview and ethnographic study, as well as for documenting significant aspects of the process with a local spatial expression, such as tool gripping postures, detailed and minute manipulations, and so on.

Photographic documentation

Weaving Machines

Garments

Natural sources for pigments

Museum patterns catalogue

Ultra-high resolution surface scanning

For the needs of ultra-high resolution surface scanning of textiles a low-cost contactless overhead micrometer surface scanner was designed and implemented. The scanner regards the photographic digitization of planar and approximately planar surfaces and is proposed as a cost-efficient alternative to off-the-shelf solutions. The result is 19.8 Kppi,micrometer scans, in the service of several applications. Accurate surface mosaics are obtained based on a novel image acquisition and image registration approach that actively seeks registration cues by acquiring auxiliary images and fusing proprioceptive data in the correspondence and registration tasks. The device and operating software are explained, provided as an open prototype, and evaluated qualitatively and quantitatively.

The implementation of the scanner device can be seen in the following gallery:

Results of surface scanning are available in the following video demonstrations:

Video documentation

Recording of a human expert was performed during silk weaving. More specifically, the expert was recorded as he went through the different stages of the loom and silk preparation as well as the weaving process itself. For acquiring video documentation, in terms of equipment, we used three cameras for data acquisition. One digital camera (Nikon J1) with a 10-30mm lens that captured single images of resolution 3872 x 2592 as well as full HD (1920 x 1080) video at 60fps and two GoPro cameras with an ultra-wide lens capable of 2.7K (2704 x 1520) video footage at 70fps. To ensure that the footage is free of motion blur, we captured the scene at a high frame rate and additionally used tripods and a gimbal.

Motion capture

MoCap of Craft practitioners took place between April 2nd and 4th, 2019 at the HdS Museum, Krefeld, Germany. During that time, one expert was recorded performing the following tasks related to silk weaving: the creation of the punch cards, wrapping of the beam, preparation of the beam, and jacquard weaving with looms of different sizes (small, medium, and large).

Post processing of Digital Assets

Ethnographic documentation

Video analysis and identification of key operational sequences that signify craft actions and processes. Definition of process flow diagrams for the weaving processes.

Visual tracking results

For visual tracking, the OpenPose library and the MocapNET body tracking software was used. On the Silk datasets we observe that, when all limbs are visible, motions tend to translate well into the armature. Failure frames mostly consist of several limbs being occluded and not visible in the camera observations. Invisible joints lead to missing 2D joint estimations and missing 2D estimations, in turn, mean empty lines and columns in the NSDM matrices of MocapNET and not enough structural information for a clearly defined solution.

Visual tracking results – video documentation

Failure frames mostly consist of several limbs being occluded and not visible in the camera observations. Invisible joints lead to missing 2D joint estimations and missing 2D estimations, in turn, mean empty lines and columns in the NSDM matrices of MocapNET and not enough structural information for a clearly defined solution. This is particularly evident in datasets during the punching cards where despite the positioning of the cameras feet often are occluded going out of the field of view. This particular case could be remedied with specialized classes that handle joint estimation without feet. It should also be noted that both OpenPose, as well as MocapNET, handle input as standalone frames and that means that no temporal information is leveraged to mask possibly noisy input

Examples of the visual tracking outcomes are presented in the following gallery:

Visual tracking results – Documentaries

The post-processed audiovisual archives included the set of documentaries created by HdS showcasing the silk weaving process to the public. The documentaries include interviews of members of HdS describing the whole process as well as demonstrations of preparing and operating the looms.

MoCap segmentation

After the correction and noise removal from the MoCap data, the data was segmented by gestures. The recordings initially were done by tasks, being one recording a whole task, these recordings were afterward segmented by gestures. One task could have only one gesture that is repeated several times or might have more gestures that are repeated through the task.

Examples of segmented MoCap data visualisation are presented in the following video:

Digitisation Outcomes

Garments

Relevant Publications

Open Datasets

  • Carlo Meghini, Valentina Bartalesi, Daniele Metilli, Nikolaos Partarakis, & Xenophon Zabulis. (2020). Mingei Ontology (1.0). Zenodo. https://doi.org/10.5281/zenodo.3742829
  • Xenophon Zabulis, Panagiotis Koutlemanis, Nikolaos Stivaktakis, & Nikolaos Partarakis. (2021). A Low-Cost Contactless Overhead Micrometer Surface Scanner. https://doi.org/10.3390/app11146274
  • Panagiotis Koutlemanis, Xenophon Zabulis, Nikolaos Stivaktakis, & Nikolaos Partarakis. (2021). A low-cost contactless overhead micrometer surface scanner (supplementary material) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.4983052
  • Nikolaos Stivaktakis, Panagiotis Koutlemanis, Xenophon Zabulis, & Nikolaos Partarakis. (2021). A low-cost contactless overhead micrometer surface scanner (input data) (1.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.5012566

Applying the Mingei protocol in the Silk pilot

STEP 2. Knowledge Elements

In this step, basic craft knowledge or otherwise knowledge elements are formed and digitally represented. In plain words, this means that the digitisations are entered in the Mingei repository, properly classified and linked to the other entities in the repository. Collectively, the classifications and links formed in this step form the semantic metadata that have been previously introduced.

Forming a knowledge element requires a comprehensive understanding of the digitally represented asset and is regarded as a digital curation process. Digital assets are represented in the system by knowledge elements associated with the result of a curation process that yields metadata, annotations, and descriptions. In this process, the original meta-data (if any) that accompany the digital asset are useful.

Representation of knowledge on craft processes includes curated material which is also digitally represented. The basic elements of such representations are the tools, machines, workshops, etc. At the same time of importance is the social and historical context.

Social and Historical Context

Persons

Human resources are invaluable in the description and explanation of craft practice and context. In Mingei the profession, communal activity, and traditions of a place are of close relevant to the contextualisation of a craft. Of relevance is also the participation of persons in communities, guilds, social or artistic movements. In our case , we record and represent basic information on the life and times of a person, such as occupation, places of visits, time and place of life and death, and key events relevant to the topic of study, such as education, professional activity, collaborations with other persons and so on.

Places

Places are represented with 2 coordinates, latitude and longitude. Regions are polygonal representations upon the surface of the Earth. We call both “places”. e use a data base of location names (GeoNames).

More refined representation of locations may require addresses or even arbitrary user-defined locations on Earth that are not included in the database. The coordinates of these locations are provided through the digital curation process and supported through a GIS system.

Events

We consider events as the changes of state in cultural, social or physical systems brought by phenomena or influenced by other events. Thus, the definition of an event requires at least two observations of this state in time.

Events are basic elements of stories in general and, in particular, to the type of “stories” we wish to capture in our craft representation. Knowledge elements of the event type are employed to represent:

  • Socio-historical context relevant to the expression of a craft instance.
  • Actions in craft processes

Craft Context

Workshop tools

Workshop equipment

Digital assets

The most relevant digital data types are:

  • Text
  • Photographs
  • Video
  • Audio
  • 3D reconstruction
  • Human motion digitisation

Additional digitisation types of humans, matter, and actions (e.g. thermal, multispectral, X-ray), provide measurements of more material properties that photography can capture. Such properties are surface reflection and transmission properties, chemical composition, its degree of viscosity at a certain temperature, and other.

We provide some examples of primary digital assets from the Mingei dataset

  • Digital text of curated text, literature, and testimonies.
  • Photographs: historic photographs, photographic documentation of artefacts, printed matter, records.
  • Audio: interviews .
  • Video of craft practice.
  • 3D reconstruction of a workshop from a laser scan.
  • 3D reconstruction of an artefact from a handheld scanner.
  • Motion capture of craft practice.

Photographs

3D Digitizations

Videos

Motion Vocabularies

3D Reconstructions

Motion Vocabulary Items

Relevant Publications

Open Datasets

Applying the Mingei protocol in the Silk pilot

STEP 3. Craft Representation

In this step, the individual entities represented in the previous steps are linked to each other into an organic representation of the craft instance. This linkage implements the semantic representation of the craft.

The scope of this representation covers the following craft dimensions:

  • Tangible elements, such as materials, tools, and products
  • Craft actions and processes
  • Contextual knowledge that provides an understanding of
    • Artefact usage
    • The CH of a region and its people embedded in the artefact
    • The historic, geographical, economic, and social dimensions of the associated craft instance.

Social and Historic context

  • Representation of Fabulae

    Regards the structuring of ‘Events’ in MOP to ‘Fabulae’ presenting the sequence of ‘Events’ that are the backbone of a ‘Narrative’. Connection between ‘Events’ is established by two kinds of relations:

    1. Mereological relating events to other events that include them as parts, e.g., the invention of the flying shuttle is part of the life of the Industrial Revolution.
    2. Causal dependencies, relating ‘Events’ that in are predicated to have a cause-effect relationship, e.g., “the Industrial Revolution resulted in a reduction of the number of weavers”.

    Through MOP ‘Event’ entries are transformed into ‘Fabulae’.

Craft Representation

To encode craft understanding, activity diagrams are borrowed from Unified Modeling Language (UML) and used in the following sense. While UML represents computational actions that transform data are represented, in this work, physical actions that transform materials are. The transition types Transition, Fork, Merge, Join, and Branch are adopted and denoted as in UML.
Activity diagrams can be refined hierarchically, allowing the increase of representation detail at later stages. Moreover, their visual nature was found to support the collaboration with practitioners. Activity diagrams are essential in representing transitions between process steps, in which distinct tools are employed and actions take place. and, in particular, decision points and parallel tasks in the process.
The progression of sequential steps is modeled by a Transition link. Forks are used to represent the initiation of two parallel tasks. In Merge transition, two or more control paths unite and Join connects steps that should be completed before the transitions to the next step. Merge and Join transitions are structurally similar, but a join is a synchronization across a set of parallel flows, while in a merge only a single flow is active. Finally, Branch transitions connect a step with a decision step that accepts tokens on one incoming edge and selects one outgoing alternative. Branch nodes control the flow of a process by selecting one of several alternatives, based on the outcome of a condition evaluation.

Process schemas

Activity diagram are transcribed into transition graphs. The MOP UI facilitates the structured instantiation of process schemas and their steps. Data fields are used to enter appellations, informal descriptions, and step order. Transitions are instantiated via dynamic UI components that adapt to transition type.

Processes

Processes representations account for the events that took place during the execution of a process schema. Intuitively, a process is an individual flow of events, out of those possible in the activity diagram. The MOP UI enables the instantiation of processes representation, via the entry and chronological ordering of events, accompanied by the recordings that document them.

Relevant Publications

  • Digitisation of traditional craft processes, (in press, JOCCH-21-0060), X. Zabulis, C. Meghini, A. Dubois, P. Doulgeraki, N. Partarakis, I. Adami, E. Karuzaki, A. Carre, N. Patsiouras, D. Kaplanidi, D. Metilli, V. Bartalesi, C. Ringas, E. Tasiopoulou, Z. Stefanidi, ACM Journal on Computing and Cultural Heritage, DOI:10.1145/3494675.

Open Datasets

Applying the Mingei protocol in the Silk pilot

STEP 4. Narratives

A digital narrative or, simply, a narrative is an abstraction that represents a set of facts that have happened in the world. These facts are connected to each other in a way that makes them a story.

This abstraction includes a narration, which is the way that a certain author has told the story. Individual narratives may be focused at a subject and include a pertinent subset of the events that are represented in the fabula.

The narration tells the story, using a certain medium. In Mingei multiple media are employed, starting from verbal and visual and reaching up to immersive and interactive narrations. A medium, and correspondingly, a media object may be multidimensional; for example the audio and visual components of a mobile phone “video” recording, or the combination of video and 3D MoCap recordings.

A narrator is a person or a program that presents a media object, i.e., verbally, by visual rendering, sonification, VR, etc. Besides multimedia and immersive presentations (discussed in the next step), conventional, verbal and visual, media objects and channels are of fundamental importance. A textual media object can be exported as a narrative for a person or an avatar to read, and a formatted document with illustrations, or an “infographic”, can output channels. In this context, Mingei aspires the export of more imaginative and interactive uses of conventional verbal and visual outputs, including scripts for theatrical re-enactment of stories and role-play on contextual dimensions of craft instance, such as skill, trade, and labour. The choice of narration medium is relevant to the use of the content, the intended audiences, and the experiences to be rendered these are discussed in the next step. In this step, we focus on the narrative representation.

A narrative can include additional information besides the connection of the narration to the facts that are narrated. In particular, a digital narrative consists of three main elements:

  1. A representation of the fabula made of representations of the events that the narration is telling.
  2. One or more media objects.
  3. A digital representation of the reference function that connects the (representations of the) events of a story with a fragment of a media object that describes them and allows to derive the plot.

Each narration of a fabula employs the use of one or more media objects, which have been authored by one or more narrator(s). Through the Mingei Online Platform, a tool for authoring and visualising narratives is provided.

The Krefeld textile industry
The story of Abul- Abbas
The story of Jacquard Loom
The story of Hubert Gotzes Junior’s life
The story of Hubert Gotzes business in Chicago
The story of Hubert Gotzes’ company
The story of “Crefeld Höhere Webschule” (College for Weaving)
The story of Krefeld from its origins to town like silk and velvet
The story of Ecclesiastical Fabric Weaving in Krefeld
The story of the computer science
Patterns of ecclesiastical textiles
The story of the cloth in the shrine of Charlemagne and its motif
Hubert Gotzes Workshop: Production and Working Conditions
Silk Pilot Context
History of Haus der Seidenkultur

Relevant Publications

Applying the Mingei protocol in the Silk pilot

Step 5. Information tools

Virtual Humans

For the needs of the demonstrations 3D virtual humans, with their corresponding garments and accessories, have been carried out based on cultural and historical information and sources. VHs were created using references from the ethnographic research. For the definition of the 3D meshes and the design of the skin surfaces a customized toolchain was employed that combines automatic generation methods, with manual editing and refining. The approach utilizes VH creation software and 3D modeling software for clothes and accessories. The final model inherits and encapsulates the required structural components, such as bone and skin attachment data. This allows the VH to be dynamically animated and to exhibit real-time animation capabilities.

Another important aspect for attaining a high level of realism is the motion, behavior, and natural interaction of the VH with users, to avoid the “uncanny valley” effect. The VH must have the ability for verbal as well as nonverbal communication skills, be intelligent, have natural communication with the users, perceive information from the user, and physically react with them. The deformation of the skinned characters during their movements must be realistic, smooth, and not contain any discontinuities. The VH that supports all these capabilities will be an intuitive craft master acting effectively as an elderly companion and assistant for elder people.

Garment design was also considered during the implementation of avatars. To ensure appropriate rigging, since the generated models are automatically rigged an additional checking was performed to ensure that the rig is applied correctly, and the bones are well adjusted to the 3D body.

An example of a Virtual Human for the silk pilot can be seen in the following video:

Historical narratives – Explore Silk narratives from the Mingei Online Platform

Explore our historical narratives to learn more about Krefeld and its evolution to a major player in the European textile industry. The historic narratives have been produced by curated historic data provided by the museum of Haus der Seidenkultur and authored in the MIngei Online Platform.

Historical narratives – Explore the story of Hubert Gotzes parament workshop narratives from the Mingei Online Platform

In the ‘Town Like Silk and Velvet’ of Krefeld in Germany, you will find a former weaving workshop that stands proudly as a testimony of the rich silk industry of the Rhine region. The workshop has been transformed into a museum, the Haus der Seidenkultur, run by the Friends Association and operated by a number of dedicated volunteers. Learn more about the history of the Hubert Gotzes parament workshop from our collection of historical narratives. The narrative have been produced by curated historic data provided by the museum of Haus der Seidenkultur and authored in the Mingei Online Platform (MOP).

Motion Re-targeting

The processing of the motion data has been done on Autodesk MotionBuilder software which is dedicated to animation and the direct integration of motion capture technologies. The process requires different steps: (a) creation of an “actor” in MotionBuilder with skeleton definition corresponding to the BVH hierarchy, (b) mapping of the received animations on the actor, and (c) synchronization of the VH with the actor by adjusting the 2 models so that the measurements match and the animations are correctly reproduced. Examples of the outcomes are presented below:

Applying the Mingei protocol in the Silk pilot

Step 6. Experiences

Interactive Timeline application

Visit the silk Interactive Timeline application to view historical and social events related to the craft of Silk in Krefeld in a timeline format. Features

  • Web-based interactive craft presentation application
  • Supports connectivity to authored fabulae from MOP
  • Built on Unity 3D engine

Silk Games!

Interactive Ecclesiastical Vestments application

Explore ecclesiastical vestments.

A handbag inspired by traditional crafts

The idea of creating a contemporary accessory was born by studying the historical patterns and transforming them into text-based narratives. This was in line with one of the most important goals of HdS which is to raise awareness regarding the unique CH possessed by the museum and at the same time connect its legacy with European history and tradition. The design of the handbag was a challenging task. By closely studying the HdS patterns, the following issues were encountered that needed a careful design approach: (a) the patterns are almost monochromatic, (b) patterns are woven in bright yet intense color variations that require careful combinations, (c) due to the intense iconographic elements of the patterns the risk of creating a cluttered visual result is high.

Taking into account the above issues, a careful study on visual aesthetics was needed to decide upon the design approach to be followed. Initially, taking into account that the handbag should contain several patterns the size of the bag was decided to be approximately forty centimeters to thirty-five centimeters. For the bag, a minimal design approach was followed with a solid big base to ensure that it stands when placed on a surface and a long strap to be worn on the women’s shoulder but still be visible. The patterns were constrained into a strict geometric approach with some form of separator between the patterns. A black stripe for example could provide a good separator between colorful patterns. Negative space (visually empty space) was also employed. The final design was based on geometric abstraction as manifested in the art creation of the 20th century could and more specifically on the abstract style of Piet Mondrian.

Crafting the handbag was by itself an interesting process considering the identified requirements and concepts. One of the major challenges was the selection of the appropriate material for the handbag. From the outer side, a neutral material that can be easily hand-painted and provides a mental connection to art artifacts was preferred. To, this end it was decided that unpainted cotton canvas would be used. Cotton canvas is a strong fabric used in painting and can be painted in various media. At the same time is a material that can support the overlaying of delicate textile fabrics as the ones provided by HdS.

The interior of the handbag should provide stiff support to the exterior so as for the bag to “stand by its own” when placed on a surface and provide a smooth surface to the exterior to eliminate possible deformations of the patterns. It is important that the patterns remain visible and with limited deformation to support the recognition algorithms. Another, requirement for the interior is to be waterproof. This is a practical provision to support the actual usage of the artifact as a handbag. For the interior, a stiff, waterproof synthetic material was used. The same combination of canvas and stiff waterproof synthetic material was used for the handbag straps to give the same feeling as the rest of the handbag.

  • Placement of patterns
  • Placement of patterns
  • Side A
  • Side B

The selection of patterns was made to create maximum visual contrast. The rationale of the selection of pattern placement is twofold. First, we were interested in visual stimuli thus placement should produce stimulus input to the eye of the viewer and this is done through contrast variations. In the case of patterns with limited contrast to each other negative space is integrated to support the visual differentiation of patterns. Second, we are interested in the visual contrast of the greyscale representation of patterns. Such visual contrast is needed to assist the image processing algorithms in differentiating patterns that are not separated through negative space and thus could coexist in the visual field of an application.

Each motif has its history – a series of symbols that evolved in time and were artistically woven together to form traditional ecclesiastical fabrics. A few years ago some of these motifs were still being manually woven on wooden Jacquard hand-looms which were up to two hundred years old. Our AR application is providing three layers of pattern recognition. The first layer regards the bag which has its own story, its creation. The second layer regards multiple patterns which reveal the story of textile weaving in Krefeld. The third layer is the stories of the patterns have a story of their own (see Figure. 23).

The AR app is augmenting the virtual space with a canvas to present those stories. Furthermore, as these stories relate to the history of Europe and the social and historic dimension of textile weaving the bag is also a portal to a web of information provided through the MOP. To reach this information a button entitled “view more info” navigates the user to the respective webpage of the MOP and from where the journey to European History may begin.

Museum Tour Guide

A museum tour guide application that presents narrations regarding the socio-historic context of the museum exhibitions while guiding visitors in the museum. The application is built on top of ten hot spots in the museum each one connected to relevant narrations and audio visual presentations that unveiled the hidden treasures of the craft of textile manufacturing. Fron a technical perspective hot spots in the inside are linked to Silk Narratives authored in the MOP and other digitized museum assets. The application guides visitors through the ten hot spots in the museum and presents audio-visual content, based on the formulated Silk Narratives. Narrations by VHs enhance the presentation of narratives and contribute to the suspension of disbelief. In addition, scannable items and artifacts in the museum allow the user to access more information that is not included in the museum application per se. The museum tour guide is available in English and German.