A comparison of cross-sensory interactions between Spain and Portugal. The results of a Synaesthetic Design Workshop

Uma comparação das interações sensoriais entre Espanha e Portugal. Os resultados de um Workshop de design sinestésico

Gambera, D. Riccò, D. Duarte, E.

UNIDCOM/IADE - IADE, Universidade Europeia
Polimi - Politecnico di Milano
UNIDCOM/IADE - IADE, Universidade Europeia

Retirado de: http://convergencias.esart.ipcb.pt

ABSTRACT: Synaesthesia is the result of automatic processes of human perception that combine sensations of different sensory modalities. Throughout this paper, we are going to present the results of a Synaesthetic Design Workshop, made with design students of IADE, Universidade Europeia (Lisbon, Portugal) and University of Extremadura (Mérida, Spain) with the purpose of identifying different types of cross-sensory interactions between participants living in the two countries, that might be influenced by cultural and personal information. On this purpose the choise of Spain as country to have a comparison was driven by the fact that 13,95% of the Spanish population experiences some kind of synaesthesia (Melero, Peña-Melián, & Ríos-Lago, 2015) while in the rest of the world the percentage is around the 4,16% (Simner & Carmichael, 2015). Results suggest probable interactions between modalities with no obvious differences between genders from both the universities.

KEYWORDS: Synaesthesia, Synaesthetic Design, Cross-sensory Interactions, Perception, Design

RESUMO: A sinestesia é o resultado de processos automáticos da perceção humana que combinam sensações de diferentes modalidades sensoriais. Ao longo deste artigo, apresentaremos os resultados de um Workshop de Design Sinestésico, realizado com estudantes de design do IADE, Universidade Europeia (Lisboa, Portugal) e da Universidade de Extremadura (Mérida, Espanha), com o objetivo de identificar diferentes tipos de interações sensoriais que podem ser influenciadas para aspetos culturais e pessoais, entre participantes residentes nos dois países. Nesse sentido, a escolha da Espanha como país para fazer uma comparação foi motivada pelo fato que 13,95% da população espanhola experimenta algum tipo de sinestesia (Melero, Peña-Melián e Ríos-Lago, 2015) enquanto no resto do mundo, a percentagem é de cerca de 4,16% (Simner & Carmichael, 2015). Os resultados sugerem interações prováveis ​​entre modalidades, sem diferenças óbvias entre os sexos dos participantes das duas universidades.

PALAVRAS-CHAVE: Sinestesia, Design Sinestésico, Interações Sensoriais Cruzadas, Percepção, Design.

1. Introduction

In 1964, Stan Lee (1922-2018) and the cartoonist Bill Everett (1917-1973) published the first issue of Daredevil for Marvel Comics: it is the story of Matt Murdock a law student went blind after an accident with a truck full of radioactive materials. However, in spite of the loss of the sight, the contact with radioactive waste provoked a strong improvement of the other senses beyond human natural abilities. From that day Matt used these new abilities to fight the crime in New York City. The same author, in “Stan Lee’s Super-humans” (a documentary television series about people in real-life with extraordinary physical or mental abilities) presented the story of a real Daredevil; Ben Underwood (1992-2009) was an American teenager, blind since the age of three after cancer of the retina. Ben was famous for being an incredible echo-locator: at the age of five, he developed the ability to detect objects in the space by making frequent “clicks” with his tongue and perceiving their echoes. 

Echo-localization is just one example of how human senses work simultaneously to compensate the absence of information (in this case visual), to build the most detailed version of the world around them. As human beings, we use the information coming from the different modalities (senses) but only in a few cases our perception is related to one single modality: we are multi-medial most of the time (Park & Alderman, 2018). These arguments have been well explained in an article published by Giovanni Anceschi and Dina Riccò in 2000:

“Even when we deal with mono-medial stimuli (such as a sound), what is triggered is not only the sensory system that is directly stimulated (i.e. the auditory system) but also other modalities, though not directly stimulated, in a process of completion of the information” (pp. 2).

As in the case of echo-localization, phenomena of cross-sensory interaction are particularly evident in people affected by a partial/total loss of a sensory modality (in this case visual modality). Nonetheless, as we can experience in our everyday life, it happens very often to use a sense to compensate another when the specific modality is “blocked” even without permanent damage. As an example when it is too dark, we put our hands in front of us to touch possible obstacles, or when there’s too much noise around, we try to read the movements of the speaker’s lips, etc. “There is a vision of touch or a vision of hearing. The hand watches, the eye touches. The hand watches, but not like the eyes, and the eye touches but not like the hand” (Rognoli & Levi, 2005). When it is possible to observe a transfer of competence between the different modalities we are talking about «sensory substitution». These phenomena represent only one typology of the greater set of cross-sensory interactions that human beings can experience. In particular, those specific kinds of experiences of sensory substitutions are specifically defined with the term “acquired synaesthesia”. The word «synaesthesia» (from the Greek syn, "together", and aisthēsis, "sensation", literarily “perceiving together”) is often used to describe some rhetorical expression in poetry, such as a “warm colour” or a “cold sound”. It is possible to find great use of those figures of speech in the French poetry of the XIX century. To be precise those expressions are defined “linguistic synaesthesias” and are clear representations of the natural and automatic mechanisms of human perception that combine elementary sensations of different modalities. In synaesthesia, the sensory information is also integrated with our past (personal experience) and with the imaginative abilities of our mind. Richard Cytowic used to define synesthesia as the “union of the senses” (Cytowic, 2002). Nowadays we know that “synaesthesia” corresponds to the general name for a great set of over 80 related cognitive processes (Córdoba, Riccò & Day, 2012). Besides those people that suffered the loss of a sensory modality (acquired synaesthesia), phenomena of cross-sensory interactions are also common in a small group of the population able to experience similar phenomena since the childhood with a great frequency. Those people are called “genuine syanesthetes” and represent around 1/23rd(4,16%) of the general population, according to a study that across the relatively wide range of synaesthesias tested (Simner & Carmichael, 2015). For this second group, we use the term “genuine synaesthesia” to define a condition characterized by specific additional experiences in response to normal sensory input, without any damage to sense organs. The most common example of genuine synaesthesia is l’audition colorée, namely a condition in which the hearing of a sound produces the visualization of a colour. Famous artists, such as poets Charles Baudelaire (1821-1867) and Arthur Rimbaud (1854-1891), painters such as Paul Klee (1879-1940), Vassily Kandinsky (1866-1944) or musicians as Alexander Scriabin (1872-1915) and Arnold Schönberg (1874-1951) are amongst known famous genuine synesthetes. A third group includes all the other synesthetic effect that can be provoked - “induced synaesthesia” - with the use of some kind of substances (drug-induced synesthesia) or with the exposition to sensory stimuli opportunely designed. However, phenomena of cross-sensory interactions are nothing exceptional: they are at the basis of the functional perception model of any human being. As observed by Maurice Merleau-Ponty in “Phénoménologie de la perception” (1945, p.66).:

“Nor are these even exceptional phenomena. Synesthetic perception is the rule, and we are unaware of it only because scientific knowledge shifts the centre of gravity of experience, so that we have unlearned how to see, hear, and generally speaking, feel, in order to deduce, from our bodily organization and the world as the physicist conceives it, what we are to see, hear and feel.” 

Based on this context, we investigated how are those aspects related to the work of designers. Usually, designers invest great efforts in solving problems and create something aesthetically appealing. Nonetheless, when the users are in front of a new object, they can perceive it with all their senses. As pointed by Munari in 1981, even if at first look the object is found to be okay, if it is unpleasant to the touch, if it is too heavy or too light, if it is cold or if it doesn’t have formal relations with human anatomy, it will be discarded for a similar object, with same characteristic, but pleasant to all the senses. Nowadays, experiences in fields as sound design, tactile design, and material experience are gaining great importance and poly-sensorialism is one of the most important drivers in the change of epistemology that is today opening new sensorial approaches in design as well as in material science (Del Curto, Fiorani, & Passaro, 2010). Nevertheless, the potentialities of incorporating phenomena of cross-sensory interactions, as synaesthesias, in Design are not completely explored and a grammar of those phenomena still not exists: the coordination of all the different modalities doesn’t occur or, when exists, takes place only in later phases of the design development. Considering that phenomena of cross-sensory interaction can be found in every human being, we envisioned a Synaesthetic design approach to be at the basis of a successful strategy for Human-centred Design. As we previously mentioned, phenomena of cross-sensory interaction could be induced. Since the expressive qualities of any design objects are liable to be designed, designers are responsible for designing not only the stimuli (i.e. colour, light, texture, softness, sound) but also the cross-sensory interactions that could be triggered by those stimuli. 

Achieving a better configuration of the designed artefacts is the objective of Synaesthetic Design. These methodological issues were object of study in a field of research called Synaesthetic Design by Anceschi and Riccò in 2000. The Synaesthetic Design research is nowadays a multidisciplinary field of Design research, embracing fields as Communication (Anceschi & Riccò,2000), Automotive (Haverkamp, 2013), Healthcare (Duarte, Riccò & Gambera, 2019), Multimedia and Material Design (Rognoli & Levi, 2005). 

“The aim of the synesthetic design is to coordinate all sensation stimulated by an artefact in a manner that results in a pleasant, harmonious overall appearance while coinciding with the particular function(s) desired. This goal can be achieved basing the design choices on the systematic connections between different modalities”.Haverkamp, M. (2013, p.14).

 

2. Workshop

A workshop untitled  “Synaesthesia a way to design across the senses to promote well-being” was run in two universities (Faculty of Industrial Design Engineering and Product Development, University of Extremadura, Mérida, Spain, the 11th of March 2019, and IADE - Faculty of Design, Technology and Communication of Universidade Europeia, Lisbon, Portugal, the 26th of March 2019) with the main objective to identify phenomena of cross-sensory interaction within different modalities. The fact that 13,95% of the Spanish population experiences some kind of synaesthesia (Melero, Peña-Melián, & Ríos-Lago, 2015) backed the choice of a Spanish university for this study. The eventual differences between the two groups, if found, could suggest some differences regarding the occurrence of cross-sensory interactions between the two countries. The gathered data is expected to be useful for the creation of a design model based on synesthetic connections.

 

3. Participants

A Total of 54 Design students of the bachelor’s degree participated: 44,4% were IADE Universidade Europeia students and 55,6% were students from Universidad de Extremadura and EASDM (Escuela de Arte Superior de Diseño de Mérida), 70,4% were females and were 29,6% males. Their age ranged from 17 to 38 (IADE Mean age = 20,00; SD = 2,39 – UEX Mean age = 22,23; SD = 3,38). After a preliminary analysis of the data collected, 10 participants (5 from each university) were excluded from the sample due to the pieces of evidences of a clear misunderstanding of the task. As an example were excluded from those forms, those one that reported in the description of the stimulus a confusion between some sensory systems (e.g. vestibular and visceral).  As a result, 29 students from Universidade Europeia and 35 from Universidad da Extremadura composed the sample.

 

4. Method

The experience started with a theoretical introduction regarding the topics of sensory modality; sensation and perception. After the introduction, a complete taxonomy of senses, consisting of 32 sensations, grouped in 8 sensory systems (Riccò, 1999) was provided, as it follows: 

Table 1 — Taxonomy beyond the five senses.


Source: (Riccò, 1999).

 

The same information was displayed on 8 totems placed on the workspace, reporting the different sensations per sensory system together with a list of qualities (i.e. bright, dark, sour, soft, etc). Afterward, the topic of synaesthesia was introduced and synesthetic perceptions were explained with practical examples. Then, eight different stimuli (one for any sensory system) were placed over a set of tables arranged in a circle. The stimuli provided were:

— An ASMR record of a woman breathing (auditory stimulus); 

— A Menthol candy (gustatory stimulus); 

— Ethylic alcohol spilled on cotton balls (olfactory stimulus); 

— A physical exercise: leg push-ups or side rises (proprioceptive stimulus); 

— Cotton balls impregnated with water (tactile stimulus); 

— 10 twirls on themselves (vestibular stimulus);

— A painful clothespin on the nose (visceral stimulus); 

— Three bulbs emitting a cold light:  6.500 k, 806 lm (visual stimulus).

With the selection of the stimuli, we tried to replicate some stimuli that are frequent in healthcare facilities, coherently with our doctoral research: i.e. incessant lights, the smell of chemical agents, unpleasant textures (Nanda, 2017). Participants needed to try any stimulus for one minute to memorize the perception. Afterward, they were invited to refer the stimulus, describing it with sensory qualities of other modalities and providing an optional explanation. At the end of the workshop, participants’ answers were used to create a map of the interactions. For this purpose, participants were divided into 8 groups (one group for any sensory system), and the connections were made visible with the use of wool threads. This mapping was intended to provide a visualization of how human perception combines all the information proceeding from the different modalities (fig.5). The workshop lasted 3 hours.

 

Fig. 1 — Participants experiencing the visual stimulus.


Source: Authors.

 

Fig. 2 — Example of visual stimulus displayed.


Source: Authors.

 

Fig. 3 — Example of visual stimulus displayed.


Source: Authors.

 

Fig. 4 — Participants experiencing the visual stimulus.


Source: Authors.

 

Fig. 5 — Mapping cross-sensory interactions with the use of wool threads of different colours.  — Participants experiencing the visual stimulus.

Source: Authors.
 


5. Results

Table 2  — Percentage of interactions per stimulus.
 

Source: Authors.
 

The data gathered (see table 2) suggest the existence of a cross-sensory interaction between most of the different modalities, for each one of the stimuli provided. The more frequent interactions are:

— Proprioceptive, tactile, visceral and visual information in response to the auditory stimulus (fig.6). The record of a woman breathing caused different reactions: a relaxing sensation (often associated to the loss of strength), as well as the sensation of rejection (visceral discomfort). Furthermore, several people visualized different kinds of images as, for example, the sea or the sand. A Tactile sensation was often associated with the feeling of the wind on the skin.

— Olfactory and tactile information in response to the gustatory stimulus (fig.7). These are common associations, used every day to discriminate the tastes and the consistency of food.

— Olfactory, tactile and visceral information in response to the olfactory stimulus (fig.8). Several participants reported augmented salivation after being exposed to the smell of ethylic alcohol. In the case of tactile information, it is mostly referred to cotton balls where the ethylic alcohol was spilled. In several cases, the smell of alcohol was also referred to a sensation of pain (visceral).

— Vestibular and visceral information in response to the proprioceptive stimulus (fig.9). Some participants experienced a loss of balance (vestibular) and muscular or articular pain (visceral).

— Visual information in response to the tactile stimulus (fig.10). Most of the cases refer to the colour of the cotton balls.

— Proprioceptive and visual information in response to the vestibular stimulus (fig. 11). These are common interactions used in daily motor activity. The interactions reported are loss of sight after twirls and a sensation of movement (proprioceptive). In some cases, symptoms of nausea were reported (visceral information).

— Tactile, proprioceptive and olfactory information in response to the visceral stimulus (fig.12). In this case, the tactile information regarded the sensation of pressure (tactile) and strength (proprioceptive) of the clothespin applied on the nose.  The olfactory information provided was the absence of smell, due to the nostrils occlusion.

— Auditory, proprioceptive, tactile, vestibular and visceral information in response to the visual stimulus (fig.13). It is interesting to observe how the visual stimuli (three bulbs emitting light of 6.500 k and 806 lm) were referred to a great variety of sensory systems. Proprioceptive, vestibular and visceral information referred to a sensation of strength (proprioceptive), a sensation of pain (visceral) and a loss of balance (vestibular). In this case, the light was considered too aggressive.
 

Table 3 — Percentage of interactions according to gender and groups (on average).


Source: Authors.
 

When comparing the recurrence for any interaction by gender, a higher number of interactions are found for females in response to the following stimulus: auditory (+0,2%), olfactory (+1,3%), tactile, vestibular (+5,5%), visceral (+1,8%) and visual stimuli (+4,2%). On the other hand, a higher recurrence of masculine interactions if found within the proprioceptive stimulus (+2,7%). When comparing genders by university of origin, the results are as follows. In the case of IADE, Universidade Europeia, a higher number of interactions are found for females in response to the following stimulus: auditory (+0,4%), vestibular (+9,6%), visceral (+0,6%) and visual stimulus (6%); while masculine interaction presents more recurrence in gustatory (+6,4%), olfactory (+1,7%), proprioceptive (3,1%) and tactile (+2,9%). In the case of UEX, a higher number of interactions are found for females in response to the following stimulus: auditory (+0,1%), gustatory (+1,8%), olfactory (+3,3%), tactile (+7,2%), vestibular (+2,8%), visceral (+2,7%), and visual (+3,3%) stimuli. The prevalence of masculine interaction regarded the proprioceptive stimulus (+2,1%).  

 

Fig. 6 – Auditory stimulus: number of interaction per gender.

Source: (IADE | UEX).

 

 

Fig. 7 – Gustatory stimulus: number of interaction per gender.

Source: (IADE | UEX).

 

Fig. 8 – Olfactory stimulus: number of interaction per gender.

Source: (IADE | UEX).

 

Fig. 9 – Proprioceptive stimulus: number of interaction per gender.

Source: (IADE | UEX).

 

Fig. 10 – Tactile stimulus: number of interaction per gender.

Source: (IADE | UEX).

 

Fig. 11 – Vestibular stimulus: number of interaction per gender.

Source: (IADE | UEX).

 

 

Fig. 12 – Visceral stimulus: number of interaction per gender.

Source: (IADE | UEX).

 

Fig. 13 – Visual stimulus: number of interaction per gender.

Source: (IADE | UEX).
 

 

 

6. Conclusions

The results of the workshop show a higher number of interactions between visual, tactile, auditory, olfactory modalities (usually privileged by designers) and the ones that, on the contrary, are often neglected by designers (gustatory, vestibular, visceral). The data between universities do not suggest relevant differences between the two groups. The comparison of results gathered by gender (both when considering the entire sample or when analysing data by the university) seems to show a small prevalence of cross-sensory interactions in female responses for vestibular, visceral and visual stimuli and a small prevalence of masculine responses regarding the proprioceptive interactions. 

The choice of a workshop format to collect data limited our capacity to get a homogenous sample in terms of gender. Due to the reduced number of male participants, we decided to not conduct inferential statistics. Future replications of the same experience will give us the possibility to enlarge the sample and test data for significant differences between genders. The mapping of this information is seen as a way to collect the required number of cross-sensory combinations to be strategically included in a Synaesthetic model for design development.

 

Acknowledgments

This work was supported by the grant SFRH/BD/129136/2017 to Davide Antonio Gambera, given by the FCT, Fundação para a Ciência e a Tecnologia (Portuguese Science Foundation). 

We would like to acknowledge the University of Extremadura, Centro Universitário de Mérida, the EASDM (Escuela de Arte Superior de Diseño de Mérida) and the organization of DIME 2019, Mérida Design Week and IADE, Universidade Europeia for the kindness and availability of guest our research activities. Personal thanks go to prof. Cayetano Cruz of UEX, Centro Universitário de Mérida, Cristina Pinheiro, Mariana Castro and Sónia André from IADE, Universidade Europeia.

A preliminary version of this paper was published in DDC’19 Conference (Gambera, Duarte & Riccò, 2019) in Duarte, E. (Ed.) (2019). Design Doctoral Conference’19: TRANSformation. Proceedings of the DDC 6th Conference. Lisbon: IADE, Universidade Europeia / EDIÇÕES IADE. ISBN: 978-989-8473-27-1 

 

References

Anceschi, G., & Riccò, D. (2000). Research of Communication Design: a synesthetic approach. In S. Pizzocaro, A. Arruda, & D. De Moraes (Eds.), Design plus Research, Proceedings of the Politecnico di Milano conference, Politecnico di Milano, may 18-20 (pp. 1–7). Milano. https://doi.org/10.13140/2.1.2121.4089 CITATIONS 0 2

Córdoba Serrano, M. J., & Riccò, D. (2012). Sinestesia: los fundamentos teóricos, artísticos y científicos. Fundación Internacional Artecittà.

Cytowic, R. E. (2002). Synesthesia: a union of the senses. MIT Press. 

Del Curto, B., Fiorani, E., & Passaro, C. (2010). La pelle del design. Progettare la sensorialità. Milano: Lupetti. 

Duarte, E., Gambera, D. A., & Riccò, D. (2019). Beyond the Five Senses: A Synaesthetic-Design Approach to Humanize Healthcare Environments (pp. 16–22). Springer, Cham. https://doi.org/10.1007/978-3-030-24067-7_2

Haverkamp, M. (2013). Synesthetic design: handbook for a multisensory approach. (W. de Gruyter, Ed.). Köln: Birkhäuser. 

Kandel, E., Schwartz, J., & Jessell, T. (2013). Principles Of Neural Science (5th editio). McGraw-Hill. https://doi.org/10.1007/s13398-014-0173-7.2

Melero, H., Peña-Melián, Á., & Ríos-Lago, M. (2015). ¿Colores, sabores, números?: La sinestesia en una muestra española. Revista de Neurologia.

Merleau-Ponty, M. (1945). Phenomenology of perceptionPhenomenology of Perception. Gallimard, Paris 1945  https://doi.org/10.4324/9780203720714

Munari, B. (1996). Da cosa nasce cosa: appunti per una metodologia progettuale. Roma, Bari: Laterza. 

Nanda, U. (2017). A Sensthetic Approach to Designing for Health., 42(2 OP-Journal of Interior Design; Jun2017, Vol. 42 Issue 2, p7-12, 6p), 7. https://doi.org/10.1111/joid.12098

Park, C., & Alderman, J. (2018). Designing Across Senses. A Multimodal Approach to Product Design. O’reilly.

Rognoli, V., & Levi, M. (2005). Materiali per il design: espressività e sensorialità. Milano: FrancoAngeli. 

Simner, J., & Carmichael, D. A. (2015). Is synaesthesia a dominantly female trait? Cognitive Neuroscience. https://doi.org/10.1080/17588928.2015.1019441

Reference According to APA Style, 5th edition:
Gambera, D. Riccò, D. Duarte, E. ; (2019) A comparison of cross-sensory interactions between Spain and Portugal. The results of a Synaesthetic Design Workshop. Convergências - Revista de Investigação e Ensino das Artes , VOL XII (24) Retrieved from journal URL: http://convergencias.ipcb.pt