The role of intersemiotic translation in multimodal communication in educational textbooks
Loading...
Files
Date
2021-11
Authors
Els, Danilda
Journal Title
Journal ISSN
Volume Title
Publisher
University of the Free State
Abstract
Our social world is a multitude of semiotic resources which we use in combination with language to make and convey meaning. The ever-increasing integration of language with other semiotic resources is turning our social world into a multimodal-communicative environment. Multimodality is using and combining various semiotic resources, for example, images and linguistic expression, for communication. It looks at all meaning-making resources available and how different resources are used in multiple contexts. Moreover, it includes the study of how these resources are organised in these contexts to form coherent meaning. These organised sets of resources are called modes. Through the selection and arrangement of modes, people create meaning, which they can then communicate. The modes in multimodality are the various semiotic resources used in combination with language, which creates a message. Since these resources can be translated into a written text and vice versa, an argument can be made that a multimodal text resembles an intersemiotic translation. Intersemiotic translation is the interpretation of language and/or linguistics signs through other non-verbal semiotic resources. It can be further hypothesised as resemiotisation, which focuses on how meaning conceptualisation occurs when there is a shift between different semiotic resources. Resemiotisation is thus concerned with "tracking" how meaning translates from one semiotic sign to another through intersemiotic translation when meaning-making takes place through a multimodal text. One field in which multimodal communication plays an ever-increasing role is education. Education as an academic field has transformed tremendously over the years. Changes are not just pedagogical but also, even more importantly, seen in the composition of learning material. Learning and teaching from predominantly text-based textbooks are gradually falling out of favour, and a greater emphasis is placed on adapting educational material to become more multimodal. One of the theories emphasising the importance of multimodal texts is the theory of multiliteracies. The pedagogical approach behind this theory is that a wide range of communicative, cultural and linguistic tools should be incorporated into teaching methodology. This framework thus creates the space for studying the relationship between the different modes in a multimodal text, as well as the composition. This research aimed to conduct a descriptive content analysis of an academic literacy textbook that communicates predominantly via multimodal texts. The primary research question this study wanted to answer was what relationship, if any, exists between the written text and the other semiotic resources used for communication? Thus, the communication process that was looked at here was specifically multimodal communication. This was done through descriptive analysis. The text that was analysed was from a multimodal literacy textbook used in the field of Natural Sciences at the University of the Free State, South Africa. Additionally, the research also looked at how semiotic resources compare across different subject fields. For the comparison, another multimodal academic literacy textbook was used; this one was from the field of Humanities, also used in classes at the University of the Free State. The aim of the comparison was to see whether similar semiotic resources were used in the meaning-making process or if perhaps certain semiotic resources were used in particular subject fields only. The study investigated the role intersemiotic translation played in creating multimodal communication by studying one chapter of a textbook to answer the research question. The analysis and subsequent comparison focused on finding answers for the following questions: What semiotic resources are used when creating multimodal texts, and lastly, whether there is a relation between text type and resource? In other words, does a particular text type favour a particular semiotic resource. The research design was qualitative. Explorative and descriptive content analysis was done to illustrate the relationship of semiotic resources used in the analysed text. This showed how these resources were compared across different fields and how they created the message together with written text. The data that was analysed in this study was one chapter from a first-year literacy course textbook used at the University of the Free State. The analysis and subsequent comparison established whether specific semiotic resources were consistently used across subject fields, whether one is preferred over another, and how intersemiotic translation transformed communication. This research proved helpful in understanding the interaction between semiotic resources and language and how multimodal texts communicate meaning.
Description
Keywords
Dissertation (M.A. (Language Practice))--University of the Free State, 2021, Multimodality, Intersemiotic translation, Resemiotisation, Multiliteracies