Creating multimodal compositions has become an everyday practice due to the increased use of digital technology that enables the combining of resources and makes it easier than ever to make meaning using a wide range of modalities and media. As Kress (2003) points out, technology development puts meaning-making in a new light. Using various modes to create meaning and mediate it using digital devices affect the nature of what is communicated (cf. Lemke, 2006). Today’s young people are keen users of these possibilities (e.g., Statens Medieråd, 2017), mostly in informal settings in which they read and create texts that are often a combination of writing, pictures, moving images, and sound, mediated through digital devices such as computers and mobile phones. However, formal settings, such as schools, often cling to traditional ways of making meaning through written, paper-based texts (e.g., Kress, 2010). Together with other societal changes, such as increased flexibility in the labour market, globalisation, individualisation, and increasingly multi-cultural societies (New London Group, 2000, Kalantzis and Cope, 2012), contemporary meaning-making affects teaching, learning, and education. This article aims to discuss how a multimodal social-semiotic approach to meaning-making can contribute to language education. The discussion builds on the analysis of the poetic designs created by two students in upper secondary school. The analytical focus is informed by multimodal social-semiotic theory, and the research question concerns how meaning in poetic designs can be interpreted when using a multimodal socio-semiotic approach.
Despite the changing conditions for meaning-making, verbal communication in the form of traditional reading and writing are still necessary skills. They are crucial in society for people to take an active part in democratic processes and citizenship. To acknowledge how young people experience contemporary meaning-making, while also supporting their reading and writing development, is a challenge for all teachers, particularly language teachers. In a conference proceeding (Godhe and Magnusson, 2017), we have previously discussed the challenges that face language education due to changes in the communicational landscape. In this article, we develop the discussion, focusing on how a multimodal socio-semiotic approach can contribute to understanding the meaning-making taking place in language education, in this case in the subject of Swedish. Classroom practices are governed by conceptions of the subject and the subject culture but also by conceptualisation of the subject in the curricula (Godhe, 2014). In 2017 revisions were made in the Swedish curricula aiming to support students’ development of digital competence (Skolverket, 2017a, 2017b). These revisions address the changes in the communicational landscape and, therefore, form the basis for questioning how and to what extent multimodal teaching and learning is supported in curricula and syllabi.
In a study of how multimodality is expressed in L1-subjects in Nordic curricula, that is, Danish, Finnish, Norwegian, and Swedish, Elf et al. (2018) state that multimodal teaching is connected to digital technology. The study takes into account both research and classroom practices, and although the concept of multimodality forms part of the L1-subjects in all four countries, there are differences in how it is conceptualised. An important conclusion they make is that whereas multimodality previously was connected mainly to students’ receptive skills, there now appears to be a shift towards students’ multimodal productions. Additions in curricula concerning the production of multimodal compositions are considered by Elf et al. (2018) to be a historically new situation in the Nordic context. This shift in focus in curricula, stresses the importance of formulating qualitative aspects of multimodal productions in a school setting in order for teachers to be able to assess them. Recent revisions in Swedish curricula aims to support students’ digital competence (Skolverket, 2017a, 2017b). Adjustments have been made in the overall goals for the educational system and syllabi for different subjects, for example, Swedish. In the syllabi, the aim of the subject and the core content have been adjusted, but not the knowledge requirements, which means that qualitative aspects of multimodal compositions are still absent from the curricula. Openings towards a broader concept of text and multimodal meaning-making can be found in the core content for Swedish in compulsory school, but not for upper secondary school (Godhe, Magnusson and Sofkova Hashemi, submitted). As pointed out by Elf et al. (2018), digitalisation and multimodality are closely related, but multimodality is not explicitly mentioned in the Swedish curricula; neither do the recent changes open up for incorporating multimodal meaning-making to any great extent.
Multimodal theory formation, developed in the 1990s, is undergoing dynamic changes where its concepts and research possibilities are scrutinized, as well as its limitations (Jewitt, 2014a, 2014b). Our interest is ontologically based in multimodal, socio-semiotic theory and its description of how contemporary meaning-making can be understood (cf. Bezemer and Kress, 2016; Kress, 2010). By drawing on Kress and van Leuween (cf. 2001) and Jewitt (cf. 2005), among others, we pay attention to how individuals use the modes as signs of interest and how possibilities and restraints connected to the resources at hand are socially and culturally situated. In this view, digitalisation is one important resource that makes the shift between modes, that is, transduction (Kress, 2010), easier. Kress and van Leeuwen (2001) discuss how semiotic principles are used in and across modes. To represent the meaning conveyed in one mode in a different mode is also discussed as synaesthesia (Cope and Kalantzis, 2010), a possibility that is also affected by digitalisation. Also, Shipka (2011) explores the meaning-making possibilities across multiple modes and refers to this as semiotic remediation practice.
From a multimodal perspective, meaning-making is possible in different modes and media in a non-hierarchic, ecological way (Barton, 2007; Kress, 2010). All modes (in Kalantzis and Cope, 2012: audio, visual, gestural, tactile, spatial, spoken, and written; in Bezemer and Kress, 2016 also layout) potentially can convey full meaning, that is, ideational, interpersonal, and textual meaning. Drawing on Systemic Functional Grammar (Halliday, 1985), the meaning potential in a multimodal perspective is applicable in all modes. Ideational, interpersonal, and textual meanings refer to different functions in communication and depend on the use of semiotic resources. The ideational function expresses our experiences of the outer world. The interpersonal function deals with interaction and our relationships with others, and the textual function refers to how these ideational and interpersonal meanings are organised. These three metafunctions are used as analytical tools in the analysis of the students’ poetic designs. How we make use of them is further explained in the section in which the analysis is conducted. This non-hierarchical approach to understanding meaning-making can be used for discussions of how, and to what extent, formal education uses and recognises different modes as learning in, for example, assessment and teachers’ planning of teaching.
With a multimodal approach mediation applies to the meaning potential of all modes. In other words, meaning is made, distributed, interpreted, and remade through various communication resources (cf. Jewitt, 2008; Kress and van Leeuwen, 2001). To understand the conditions under which modes and semiotic resources are used, it is essential to consider affordance. As explained by Bezemer and Kress (2016), modes have different affordances, that is, possibilities and restraints. Material affordance are connected to what can be realized in a mode whereas semiotic affordance concerns how the sign-making is shaped historically and socially. Neither the material nor the semiotic affordances affiliated to modes are fixed, but contextually situated. In the poetic designs analysed in this article, the affordances are affected by, for example, the teaching, the students’ knowledge and experience of poetic meaning-making, of the contexts of the classroom setting as well as cultural and technical aspects.
To make meaning is to participate actively in an ongoing process that can be understood as design. Kalantzis and Cope (2012) use design as a key concept in which people are regarded as active designers of meaning and social futures (Jewitt, 2008). As Bezemer and Kress (2016) highlight, design is prospective, which means that new meanings are always created in the design process. Design refers to meaning-making and learning at the same time, although learning stems from the pedagogical tradition, while meaning-making stems from semiotics. Kress (2010: 182) considers meaning-making as learning:
Learning is the result of the transformative engagement with an aspect of the world which is the focus of attention by an individual, on the basis of principles brought to her or him to that engagement; leading to a transformation of the individual’s semiotic/conceptual resources.
Conceptualising education and learning as a process of designing meaning-making differs from views on learning as a process of transferring knowledge (cf. Säljö, 2010). Moreover, it makes it possible to distance multimodal meaning-making from the linguistic tradition and the concept of text. Traditionally, text is defined as paper-based, printed materials, meaningful and coherent, with a beginning and an end (Björkvall, 2009), which not always adheres to multimodal and digital communication. To use text in a broader sense sets out verbal language as the starting point for meaning-making which is problematic in a multimodal approach supporting a non-hierarchic relationship between modes (cf. Bezemer and Kress, 2016). From now on in this article, we aim to avoid this by talking about the students’ multimodal meaning-making as poetic designs.
Although the individual actively designs meaning, patterns and conventions are inherited and affect the design process. In this view, teachers design learning processes and environments, and students design their learning (cf. New London Group, 1996, 2000; Selander and Kress, 2010; Selander, 2017). When looking at meaning-making as multimodal design (Cope and Kalantzis, 2000; Jewitt, 2005; Kress, 2003, 2010) and as a design process (Kress and van Leuween, 2001), the meaning-making is understood as design in different stages: available design (for example, assignments and resources for learning), designing (students’ work with the assignments), and redesign/new available design (the students’ poetic designs). An available design is connected to both the intended and the perceived meanings (Selander and Kress, 2010), stressing the contextual aspects. It is important to note that the concept of design, can be used regardless of mode or medium.
According to Selander (2017), subject traditions affect what teachers and students perceive as subject content and what is considered learning, but learning itself cannot be observed. What can be observed are signs of learning, as a change from one point to another. As we understand Selander, he does not fully agree with Kress’s view on learning and meaning-making as synonym concepts. We agree with Selander that, in teaching, it is important to consider how to define learning and which aspects of learning are recognised in educational settings, but for our purposes in this article, we consider signs of learning as results of engagement and communication, both with oneself and others (Bezemer and Kress, 2016). Poetic design implicates acts of interpretation and translation within a mode, as well as transduction between modes, when searching for ways to express meaning. In the poetic designs analyzed in this article, the signs of learning are connected to the students’ choices of modes and semiotic resources and how they choose to present their poetic designs.
To talk about meaning-making as design has been questioned by, for example, Bazalgette and Buckingham (2013). Their critique is based on the connotations of design as something that implies communication processes that are always rational and controlled. We understand their point, but we consider design to be an apt concept for now, as it offers an alternative to the linguistic tradition and points to the importance of agency in contemporary meaning-making. Moreover, when the importance of the meaning-making context is taken into consideration, the design itself becomes situated, that is, the design is affected by various discursive aspects (cf. Hodge and Kress, 1988).
Discussing students’ work in school as design brings the whole design process within the focus of interest. The fact that multimodal designs are becoming increasingly important and common in today’s communication landscape also challenges the conception and evaluation of literacy in education and what is recognised as learning. Similar to text, literacy is a concept from the linguistic tradition and as such is problematic to use. In developing a ‘pedagogy of multiliteracies’ (e.g., Kalantzis and Cope, 2012; New London Group, 1996, 2000), and among others (e.g., Adami, 2015; Jewitt, 2005, 2006; Lemke, 2006), this has been discussed, but in this article, we use literacy without further elaboration.
Among other factors, such as teaching and assignments, assessment practices are crucial in determining what is recognised as learning (Godhe, 2014). How the teacher works with assessment affects how learning is perceived and established (e.g., Aagaard and Silseth, 2017; Drotner and Erstad, 2014; Selander and Kress, 2010). In a multimodal perspective, assignments can encourage students to use multimodal meaning-making in their design process. The extent to which they do so may depend on how the assignment is expressed and prepared through teaching and how the assessment criteria are formulated and communicated. Therefore, it is important to notice the lack of adjustments concerning multimodality and digitalisation in knowledge requirements and assessment criteria in the Swedish curricula (Godhe, Magnusson and Sofkova Hashemi, submitted). Literacy practices in school are affected by institutional contexts, so the activities in which teachers and students engage must relate to this setting. Whether meaning-making, expressed in a variety of modes, is recognised as learning depends on, and is affected by, rules in the educational setting. Such rules include curricula, but also traditions in certain communities, such as the teachers’ subject communities and established ways of working within the subject (Selander and Kress, 2010). The fact that meaning-making is always situated stresses the context of the teaching; however, this is only briefly addressed in this article since the article focuses on how a multimodal socio-semiotic perspective can be used to analyse students’ poetic designs and what the implications for language education are.
Several researchers have explored how to integrate multimodal learning and meaning-making in education across the curriculum (e.g., Baldwin, 2016; Danielsson and Selander, 2014, 2016; Shipka, 2011), following the New London Group (1996) and the theoretical work of Kress and van Leuween (e.g. 2001) and Jewitt (e.g. 2005), closely relating to research on multimodality in language education. Studies following those by Kenner (2004), Kress (2003) and Kress et al. (2004) has shown how both learners of their first language (L1) and their second language (L2), are supported in their literacy development by a multimodal framework (Adoniou, 2013; Early and Marshall, 2008; Potts and Moran, 2013; Sofkova Hashemi, 2017). Smith’s (2014) review on L2 education and multimodal approaches to writing showed that a multimodal approach to teaching in L2 was beneficial to student learning in a number of areas, including academic writing, when combined with explicit instruction. Research on L1 supports the benefits of multimodality for developing writing and reading both in early L1 education (Mackenzie and Veresov, 2013; Sofkova Hashemi, 2017) and throughout the school years (Oldham, 2005; Pantaleo, 2012; Svärdemo Åberg and Åkerfeldt, 2017). There are also studies which report on how a multimodal approach is difficult to enact in the classroom due to issues related to school traditions, teachers’ competence, the challenges of power relations in the classroom and even the students’ resistance (Aagard and Silseth, 2017; Cederlund and Sofkova Hashemi, 2018; Gilje, 2010; Godhe, 2014; Olin-Scheller, 2006).
Studies show that what is recognized as learning in language education is still very much connected to verbal writing, both in teaching and assessment (Lankshear and Knobel, 2003; Oldham, 2005; Tønnessen, 2010). Results from studies by, among others, Aagard and Silseth (2017), Godhe (2014) and Silseth and Gilje (2019), reveal how assessment practises fail to align with teaching. While teaching practices may be multimodal, assessment practices are generally not, and assessment criteria used to evaluate verbal texts, do not adequately address the complexity of multimodal compositions (e.g., Cope, Kalantzis and Magee, 2011; Godhe, 2014; Oldham, 2005). Neither do they facilitate a shared understanding amongst teachers of qualitative aspects in multimodal texts, nor how to evaluate them. Similar results are found by Åkerfeldt and Svärdemo Åberg (2017) who show that the students’ choice of mode is also dominated by the verbal.
Despite a growing body of research within the field, the need for further research on a multimodal approach to language education is called for by several scholars (cf. Bearne 2009; Flewitt and Wolfe, 2010; Jewitt 2007; Pahl, 2009; Pahl and Rowsell 2006; Rowsell and Decoste, 2012; Unsworth and Thomas, 2014; Tønnessen 2010). Already in 2007, Jewitt pointed out the need for research to study the potentials of modes other than the verbal. Metalanguages for literature education in school, based on systemic functional grammar have been suggested and studied by Macken-Horarik (2016) and Unsworth (2008). Chandler (2017) and Danielsson and Selander (2014, 2016) have introduced models for multimodal meaning-making applicable in all subjects, including language education. The use of a metalanguage for multimodal meaning-making has to be studied further in order to develop, what Grapin (2018) distinguishes as a ‘strong version’, where multiple modes are seen as essential contributors to learning, in contrast to ‘a weak version’, where nonverbal modes are used to support the verbal. We aim to contribute to this by focusing on poetry and poetic designs as part of language education.
Poetry, which is a part of the subject of Swedish throughout the school years, can be seen as being multimodal and especially suitable for developing teaching with a multimodal approach to literacy (cf. Dymoke and Hughes, 2009). Multimodal possibilities in poetry have been exemplified as poetic meaning expressed in wikis, blogs, by hypertexts and stage performance (Xerri, 2012; Alghadeer, 2014). Research in L2 teaching (cf. NATE, 2009; Newfield and D’Abdon, 2015), shows that by exploring the possibilities of poetic meaning through different modes and media, students’ engagement is stimulated, as well as their literacy development (cf. Xerri, 2012). Working with poetry in a Swedish school context is often accompanied by interpretations of music, the auditive mode, and pictures, the visual mode (cf. Höglund, 2017). Furthermore, poetry through the verbal oral mode, as spoken word and stage poetry, is a growing art form, both live and in digital environments, and may be familiar to students. The following examples outline how students design multimodal meaning working with poetry in the subject of Swedish. We will briefly introduce the context to set the scene and then present two students’ solutions to the poetry assignment, focusing on how the students use semiotic resources in their poetic designs.
The examples are taken from the work done in a class in the second year of upper secondary school in Sweden and a teaching unit on poetry that author Magnusson was teaching. Permission to use the students’ designs was requested and granted at the end of the school year as we found the outcomes of the students’ work to be interesting examples of multimodal meaning-making. However, the teaching unit was not initially thought of as a data source. Of course, this limits the possibilities for analysis since we have only a brief teacher log, the assignments, and the poetic designs as they were presented at the end. More extensive data could have been collected about the students’ thoughts and designs if there had been plans for observing and analysing the teaching unit from the start. Nevertheless, we argue that the poetic designs, as they were presented, are interesting examples of how semiotic resources are used in meaning-making activities and how the affordances are used in developing new meanings (cf. Lemke, 2002) in the redesigns (cf. New London Group, 1996, 2000). As examples, they serve our purpose for discussing the importance of multimodality in language teaching.
The teaching unit was developed by the teacher and students together and included the writing of poetry through history; who wrote the poems, what they wrote, how and for whom, and concluded with the students creating their poetic designs. The students worked in groups to get to know poetry through the literary eras, based on the textbook used in class. They used different textbooks, online encyclopaedias on their personal computers, and an educational TV program on literary styles focusing on poetry. They also watched spoken-word poetry performances on YouTube, and there was a class discussion on how the spoken-word poets used the sound and light in their performances, and how the poems the students had chosen as typical epoch examples, could have been performed on stage in their historical settings. Approximately six hours were designated for working with poetry through history before sharing their findings with the other groups. This formed a part of the teaching unit and worked as a prompt for engagement (Bezemer and Kress, 2016). It can also be regarded as part of the available design, which the students had access to when they were given the assignment to create a poem of their own. According to the teacher, the aim was to teach from a multimodal perspective to encourage the use of different modes for communication. However, there was no explicit teaching of multimodal meaning-making in the sense of exploring the meaning-making possibilities of semiotic resources.
After the presentation of the group assignments, the students worked with the assignment: ‘Create poetry! What do you want to express? Can you express it in different ways?’ The assignment included information on the locally formulated assessment criteria for the actual unit, one being: ‘The student can graphically present different types of texts and motivate choices of illustrations, disposition, and layout’. No preference of mode or definition of how to use semiotic resources was provided (note: create instead of write). Instead, the direct questions used in the assignment can be seen as opening up for the students’ own choice of how to express meaning, thereby creating a space for the students’ agency when designing their poetry. We suggest that how the assignment was expressed and presented to the students rendered possible recognition of multimodal meaning-making as learning. However, as Bezemer and Kress (2016) note, agency is constrained by the resources of modes and, more important in this setting, the competence of the sign-makers. The students’ poetic designs varied in the media used for presentation as well as the extent to which they were multimodal. Nine of 17 presented poems were explicitly multimodal, that is, several modes were used by the students and also noted as ‘explained and motivated’, according to the teacher log, but the exact way that this was done is not elaborated on. Two of the students’ poetic designs have been chosen as examples of how meaning in poetic designs can be interpreted when using a multimodal socio-semiotic approach.
The analysis uses the completed poetic designs which means that it is aimed at how the students, have designed meaning multimodally, with the semiotic resources available. We do not have access to how the students were thinking and planning through the design process. Therefore, we focus on the completed poetic designs, including how they were presented, based on notes in the teacher log. We analyse the poetic designs in order to answer the question of how meaning in poetic designs can be interpreted when using a multimodal socio-semiotic approach. However, we do not claim that this is the only possible interpretation or that the conclusions drawn are applicable generally, but our interpretation is validated by referring to other similar research and concepts used in multimodal theory formation.
As has been laid out in the theoretical frame, we use a multimodal socio-semiotic approach. The analysis will use metafunctions as the main analytical tool and examine how the modes and semiotic resources are used to convey meaning. Our analysis starts with the textual metafunction, followed by the interpersonal and ideational metafunctions to show how the designs are constructed.
When we analyse the textual meaning, we examine the complete layout of the design, how the relationship between modes is structured, and how the modes work together to make meaning. When working with the textual meaning, Kress and van Leeuwen (2006) deal with information value, which considers the placing of elements, and salience, which considers what it is that makes some resources more salient than others, for example, for the visual mode, size or colour. In the students’ poetic designs, the audio mode is also used, and salience here can be dependent on frequency, silence, and volume. Framing is another concept used when examining the textual meaning. Framing concerns how the different parts of the design are connected or disconnected by space, distance, and other culturally accepted signs.
The analysis of the interpersonal meaning examines how the poetic designs build relations to the receiver through choices made among modal semiotic resources, such as, angle, distance, perspective, volume, font size, and interaction. In the poetic designs, these aspects form a more or less explicit offer to the receiver (Kress and van Leeuwen, 2006). Altogether, the choices made express socially determined values (Danielsson and Selander, 2014, 2016), which is an important aspect of the interpersonal meaning. Ideational meaning focuses on how different modes are used to convey interpretations of the world and can be both narratives, with events and actions, and conceptual, which is more of a fixed state and about characteristics (Kress and van Leeuwen, 2006).
Poetic design A (Figure 1) was displayed on a computer screen in front of the class, starting with the sound of wind moving through poplar trees for 30 seconds. A recorded, neutral voice slowly read the poem twice, accompanied by soft music by the music duo Say Lou Lou, without lyrics. At the end of the presentation, the sound of the blowing wind returned at a higher and increasing volume for 30 seconds and then stopped.
The student uses an assembly of modes (cf. Bezemer and Kress, 2016) for the design: audio, verbal speech, writing, visual and layout modes are put together in a PowerPoint presentation with one still picture. The audio mode has the functional load, and the semiotic resources of sounds from nature and music are intertwined throughout the design. They are salient throughout the presentation, even though they are in the background while the poem is read.
In the picture, the bedding is foregrounded and emphasises the visual mode. However, the visual mode is somewhat downplayed by the use of colour, here as nuances of grey and white. The visual mode could, on the one hand, with the photo of a used bed with clean cotton sheets, be said to add to a coherent design. On the other hand, the choice of the dark grey striped sheets could also be seen as a contrast to the smooth and calm audio resources and interpreted as a disruption.
The verbal mode as writing, displayed in a plain grey font, is not prominent, and the same message can be heard by the voice reading the text. Moreover, the verbal mode is made less prominent since it is organised in the layout mode to the right end of the picture, whereas the left-hand side displays the unmade bed, which is connected to the sound of the blowing wind. The verbal writing and speech use alliteration, with nouns that connote summer and references to a laid-back musical act. The blowing wind, which returns at a higher volume and rises in volume for 30 seconds after the recorded, neutral voice has read the poem, ends abruptly and frames the presentation by the circular use of the audio mode.
The information value is shaped by the salient audio mode and the resources of nature sound, human voice, and music, which, together with the choice of motive and colours, create the design. On the interpersonal level, this design is an offer of a view into an enclosed moment but also an invitation to interpretation through the sound of loud wind that suddenly stops at the end. In a Swedish cultural context, a summer day, to many people, refers to holiday, sunbathing, and rest. The design explicitly invites one to a relaxed, calm feeling when referring to and picturing a summer day, but it could also carry implicit disruption and disturbance, especially since the rising volume is sudden. Ideationally, we see this poetic design as a mainly conceptual representation where the chosen characteristics of ‘summer’ are communicated through an assembly of modes and where the communicated overall meaning is different from the meaning communicated through each separate mode. However, the design could also be interpreted as a narrative – what is implied through the rising volume of the wind in the poplar trees and by the tanned dark back, perhaps of an implied lover?
In Poetic design B (Figure 2), the student uses verbal written, visual, and layout modes displayed on a smartphone screen. The poem was presented by the student walking through the classroom showing the smartphone to the other students, two at a time. As he walked around the classroom, more and more giggling and laughter could be heard. He closed the presentation by reading the poem aloud, leaving out the dates and timings.
The presentation is similar to the one in poetic design A in that it is an assembly of visual, verbal written, and layout modes, to some extent also supplemented by the audio mode during the presentation, but there are also differences. For example, the presentation includes the gestural mode (student walking around the classroom) and spatial mode in the fixed resources of the smartphone, where the placing of the messages and the set date and timings are part of the framing, making it obvious that the ‘the platform is an active producer of text’ (Bezemer and Kress, 2016: 112). This means that the student, the designer, has limited agency in ordering and arranging the signs when using the display for presentation. The space in time between the first and second message, from January to December, is left for interpretation and also left out when the student reads the poem after showing it to his peers. Accordingly, the space in time becomes a semiotic resource in the verbal written mode (cf. Bezemer and Kress, 2016: 22). The verbal written mode carries the functional load in this design being the most salient, but it is the combination with the affordances of the smartphone that forms the information value. Since smartphone users are aware of the possibility to erase messages, questions concerning whether this has been done or not could, therefore, be raised. Also, the placing of text in the layout is set by the smartphone. The colours and font type and size are also fixed, or at least they have not been altered from the most common settings. Possible other resources, such as emoticons, are not used, which also affects the design since they can be expected in personal text messages. The affordances of the platform thus shape the interpersonal meaning. What is offered depends on to what extent these affordances are known and an integrated part of the interpretational context. Just reading the fixed messages, noticing the use of space in time as a semiotic resource, forms a different offer compared to if you are aware of the possibilities of the platform (cf. Bezemer and Kress, 2016). At the ideational level, the poetic design can be interpreted as a narrative representation although the events and actions are implicit and left to the interpreter to understand based on the actors’ short messages displayed on the screen.
We have shown how the meaning of the poetic designs could be interpreted through an analysis from a multimodal socio-semiotic perspective. By examining the use of semiotic resources, the analyzis show how the use of ‘an assembly of modes’ (Bezemer and Kress, 2016) and semiotic resources affect the meaning. The use of metafunctions as an analytical tool reveals how modes and semiotic resources shape the perceived meaning through the textual, interpersonal, and ideational metafunctions. As there is no record of the students’ intended meaning in the teacher log, this has not been possible to include in the study. Despite this, we argue that the study contributes to an understanding of how a multimodal socio-semiotic approach can be used when analysing a learning unit on poetry and when interpreting students’ poetic designs. Moreover, the result can be used as an example when discussing the implications for teaching.
Although a multimodal approach to communication, as a part of the classroom environment, was new to the students, they used various modes, semiotic resources, and media in their designs in a way that can be understood as synesthetic meaning-making (cf. Cope and Kalantzis, 2010) where meaning is re-represented across modes in a semiotic remediation practice (Shipka, 2011). We consider the students’ meaning-making in the poetic designs to be signs of learning (Selander, 2017; Selander and Kress, 2010) visible to us through their choices of modes and their use and combination of semiotic resourses and media. It is notable that in the poetic designs, that is, their redesigns, the students appear to have used available designs not only from the formal setting in school, that is, the resources provided by the teacher, but to a great extent also from informal setting outside school. It is not unusual for teachers to encourage the use of different modes and digital media for student work, but it is more unusual to recognise it as learning (Borgfeldt, 2017; Godhe, 2014; Oldham, 2005; Selander and Kress, 2010; Silseth and Gilje, 2019). In the example above, the explicit invitation to create poetry from a multimodal perspective, both in assignment instruction and in assessment criteria, are likely to have encouraged the students’ exploration of meaning-making possibilities through different modes and semiotic resources. Accordingly, the teaching unit on poetry is regarded as an example of how multimodal meaning-making can be recognised as learning when it is referred to in teaching, assignment, and assessment.
To understand and include different modes and semiotic resources in teaching, assignment, and assessment requires competences that go beyond those needed for verbal meaning-making. While competences that are central to oral and written verbal meaning-making are well-known to language teachers, knowledge of how to make meaning in other modes are not. Teachers need to develop knowledge about all modes and how semiotic resources can be used so that they can distinguish how meaning is made in multimodal compositions through the use and combination of different modes. Moreover, the teacher’s knowledge and competence play an important role in supporting agency in the design process (Bezemer and Kress, 2016).
To develop teaching for contemporary meaning-making, understanding how semiotic resources can be used and how they are connected to various modes is required. By talking about meaning-making at general and at specific levels, the importance of this understanding can be clarified (cf. Tønnessen, 2011). On a general level, meaning-making concerns understanding available designs in a similar way, regardless of mode or media. This understanding deals with questions of sender, message purpose, and transmission, and it concerns questions about the perception of ideational, interpersonal, and textual meanings (cf. Kress, 2010; Bezemer and Kress, 2016). An understanding at the general level is necessary to grasp the basics of the available design. When drawing on multimodal socio-semiotic theory and considering all modes as equal and having full-meaning potential, it is also essential to understand how the semiotic resources in the various modes work at the specific level of meaning-making and the functional specialisation of modes (cf. Bezemer and Kress, 2016). For a language teacher, the specific meaning-making of verbal oral and writing modes are competences that are well established within the subject and known and practised by both teachers and students. In terms of the poems, the focus may be on voice, figurative language, intertextual connections, meter and rhyme to examine the specific level of verbal oral and written meaning-making. In the analysis presented above, what is considered as contributing to the meaning-making in the poetic designs is broadened by the use of the multimodal social-semiotic approach. When considering, for example, the audio, visual, and layout modes, there is a need to understand how semiotic resources work in these modes, that is, on the specific level, in order to be able to interpret how these modes contribute to the full meaning potential. Using the specific level in examining an available design promotes an understanding of how the semiotic resources create meaning.
To fully understand the multimodal meaning in the poetic designs, the teacher and students must understand the specific level, not only for the verbal modes but also for the audio, visual, spatial, and layout modes. As shown in the analysis, this would, in poetic design A, involve the need for discussions about font size and placement of the written text, the foregrounding of the bedding to the left, the choice of sound and recorded voice, how the different modes work together, and which mode has the functional load and why (cf. Bezemer and Kress, 2016). In relation to poetic design B, there is a need to understand how the medium that is used creates meaning. To establish a metalanguage for meaning-making in education based on the metafunctions and to teach about metafunctions and meaning-making on general and specific levels is a possible strategy to cultivate multimodal meaning-making. We consider this approach to be one way forward in striving to integrate and embrace multimodal meaning-making fully in language education. This approach is necessary to adopt if language subjects are to be considered important subjects to young people and contemporary communication. It is no longer possible for a teacher in Swedish to focus exclusively on traditional available designs, such as written texts, and ignore a large proportion of the available designs that students come into contact with on a daily basis. To continue to develop ways that recognise and support multimodal meaning-making and a wide conceptualisation of literacy is, in our opinion, crucial for education in general and language education in particular. Since language subjects are about communication and how to make meaning, they need to evolve in consort with contemporary communication.
A multimodal socio-semiotic perspective challenges the verbal written mode as central to meaning-making in, for example, traditional lyrical analysis, as it considers all modes as carrying meaning (cf. Newfield and D’Abdon, 2015; Bezemer and Kress, 2016). The multimodal framework makes it possible to teach and learn about contemporary meaning-making by using the metafunctions and by naming and addressing both the general and the specific levels of each mode, including the verbal modes.
Meaning-making in language education, as well as in school in general, need to be based on a non-hierarchical, inclusive view of modes and media to create a readiness and flexibility in unison with the demands of a rapidly changing society. Taking this stance means that we must deal with challenging changes in how language subjects are conceptualised.
A commonly agreed-upon conceptualisation of language subjects is that writing is considered to belong to the language subjects, while other modes, such as visual and audio, adhere to other, more practical or esthetical subjects (Elmfeldt and Erixon, 2004). By embracing multimodal meaning-making and incorporating it as valuable learning in language subjects, students and teachers could access and assess students’ complete and widened literacy competences, instead of focusing on some practices, while largely ignoring others. Recognising how students consume and produce meaning and raising students’ and teachers’ awareness of the purpose of different kinds of available designs is essential to develop the meaning-making abilities that students need in order to become active citizens in the society of today and tomorrow.
In Swedish schools, access to digital devices is high, but the use of these devices is restricted in many subjects (Skolverket, 2016). Teachers consider themselves in need of professional development focusing on the pedagogical use of digital tools. It is important that professional development also include teaching and assessing from a multimodal, socio-semiotic perspective so that meaning-making in different modes and media are embraced and evaluated in teaching. Moreover, further adjustments in curricula are needed where qualitative aspects of students’ multimodal compositions are explicitly addressed in order for teachers to be able to assess the students’ designs.
To sum up, based on an example from a teaching unit on poetry in the subject of Swedish, we argue for a need to widen language subjects to include multimodal meaning-making fully and on both general and specific levels. If language teaching does not seriously consider and evaluate contemporary meaning-making, activities carried out in language subjects run the risk of having little or no relevance to the meaning-making that students engage in outside of school or to contemporary communication in general.
The authors have no competing interests to declare.
Aagaard, T., & Silseth, K. (2017). Teachers’ assessment of digital stories: challenges and dilemmas. In Nordic Studies in Education, 37(03–04), 231–242. DOI: https://doi.org/10.18261/issn.1891-5949-2017-03-04-08
Adoniou, M. (2013). Drawing to Support Writing Development in English Language Learners. In Language and Education, 27(3), 261–277. DOI: https://doi.org/10.1080/09500782.2012.704047
Bazalgette, C., & Buckingham, D. (2013). Literacy, media and multimodality: a critical response. In Literacy, 47(2), 95–102. DOI: https://doi.org/10.1111/j.1741–4369.2012.00666.x
Bearne, E. (2009). Multimodality, literacy and texts: Developing a discourse. In Journal of Early Childhood Literacy, 9(2), 156–187. DOI: https://doi.org/10.1177/1468798409105585
Bezemer, J., & Kress, G. (2016). Multimodality, learning and communication: a social semiotic frame. London: Routledge. DOI: https://doi.org/10.4324/9781315687537
Borgfeldt, E. (2017). “Det kan vara svårt att förklara på rader”: perspektiv på analys och bedömning av multimodal textproduktion i årskurs 3. [“It’s hard to explain on a line”: perspectives and analysis of multimodal text production in year 3] Diss. Göteborg: Göteborgs universitet.
Cederlund, C., & Sofkova Hashemi, S. (2018). Multimodala bedömningspraktiker och lärares lärande [Multimodal assessment practices and teachers’ learning]. In EduCare, 2018(1), 43–68. DOI: https://doi.org/10.24834/educare.2018.1.3
Chandler, P. D. (2017). To what extent are teachers well prepared to teach multimodal authoring? In Cogent Education, 4(1). DOI: https://doi.org/10.1080/2331186X.2016.1266820
Cope, B., & Kalantzis, M. (2000). Multiliteracies: the beginning of an idea. In B. Cope & M. Kalantzis (Eds.), Multiliteracies: literacy learning and the design of social futures (pp. 3–8). London: Routledge.
Cope, B., Kalantzis, M., & Magee, L. (2011). Towards a semantic web: connecting knowledge in academic research (1. publ.) Oxford: Chandos. DOI: https://doi.org/10.1533/9781780631745
Danielsson, K., & Selander, S. (2016). Reading Multimodal Texts for learning – a model for Cultivating Multimodal Literacy. In Designs for Learning, 8(1), 25–36. DOI: https://doi.org/10.16993/dfl.72
Drotner, K., & Erstad, O. (2014). Inclusive Media Literacies: Interlacing Media Studies and Education Studies. In International Journal of Learning and Media, 4(2), 19–34. DOI: https://doi.org/10.1162/IJLM_a_00092
Dymoke, S., & Hughes, J. (2009). Using a poetry Wiki: How can the medium support pre-service teachers of English in their professional learning about teaching about poetry and teaching poetry writing in a digital age? In English teaching: Practice and Critique, 8, 91–106.
Early, M., & Marshall, S. (2008). Adolescent ESL Students’ Interpretation and Appreciation of Literary Texts: A Case Study of Multimodality. In The Canadian Modern Language Review, 64(3), 377–397. DOI: https://doi.org/10.3138/cmlr.64.3.377
Elf, N., Gilje, Ø., Olin-Scheller, C., & Slotte, A. (2018). Nordisk status og forskningsperspektiver: Multimodalitet i styredokumenter og klasserumsrumspraksis [Nordic research perspectives: multimodality in curricula and classroom practices]. In M. Rogne & L. Rune Waage (Eds.), Multimodalitet i skole- og fritidstekstar. Ein vitskapleg antologi (pp. 71–104). Oslo: Fagbokforlaget.
Elmfeldt, J., & Erixon, P.-O. (2004). Genrer i rörelse – estetiska textpraktiker i gymnasieskolan. [Genres in motion – esthetical text practices in upper secondary school]. In S. Thorson (Ed.), Andra nationella konferensen i svenska med didaktisk inriktning (pp. 37–46). Göteborg: Nationella nätverket för svenska med didaktisk inriktning.
Flewitt, R., & Wolfe, S. (2010). New technologies, new multimodal literacy practices and young children’s metacognitive development. In Cambridge Journal of Education, 40(4): 387–399. DOI: https://doi.org/10.1080/0305764X.2010.526589
Gilje, Ø. (2010). Multimodal Redesign in Filmmaking Practices: An Inquiry of Young Filmmakers’ Deployment of Semiotic Tools in Their Filmmaking Practice. In Written Communication, 27(4), 494–522. DOI: https://doi.org/10.1177/0741088310377874
Grapin, S. (2018). Multimodality in the New Content Standards Era: Implications for English Learners. In Tesol Quarterly, 53(1), 30–55. DOI: https://doi.org/10.1002/tesq.443
Godhe, A-L., & Magnusson, P. (2017). Multimodality in language education: exploring the boundaries of digital texts. In W. Chen, et al. (Eds.), Proceedings of the 25th International Conference on Computers in Education. New Zealand: Asia-Pacific Society for Computers in Education. Retrieved September, 2018 from http://hkr.diva-portal.org/smash/get/diva2:1163921/FULLTEXT02.pdf
Jewitt, C. (2007). A multimodal perspective on textuality and contexts. In Pedagogy, Culture and Society, 15(3), 275–289. DOI: https://doi.org/10.1080/14681360701601937
Jewitt, C. (2008). Multimodality and Literacy in School Classrooms. In Review of Research in Education, 32(1), 241–267. DOI: https://doi.org/10.3102/0091732X07310586
Kalantzis, M., & Cope, B. (2012). Literacies. Port Melbourne Vic: Cambridge University Press. DOI: https://doi.org/10.1017/CBO9781139196581
Kress, G. (2003). Literacy in the new media age. London: Routledge. DOI: https://doi.org/10.4324/9780203164754
Kress, G. (2010). Multimodality, A social semiotic approach to contemporary communication. London: Routledge. DOI: https://doi.org/10.4324/9780203970034
Kress, G., Bourne, J., Franks, A., Hardcastle, J., Jewitt, C., Jones, K., & Reid, J. (2004). English in urban classrooms: a multimodal perspective on teaching and learning. London: Routledge. DOI: https://doi.org/10.4324/9780203397305
Kress, G., & van Leeuwen, T. (2006). Reading images: the grammar of visual design (2nd ed.) London: Routledge. DOI: https://doi.org/10.4324/9780203619728
Lemke, J. (2002). Travels in hypermodality. In Visual Communication, 1(3), 299–325. DOI: https://doi.org/10.1177/147035720200100303
Lemke, J. (2006). Toward Critical Multimedia Literacy: Technology, Research and Politics. In M. C. McKenna (Ed.), International handbook of literacy and technology, 2, 3–14. Mahwah, N.J.: Lawrence Erlbaum Associates Publishers.
Macken-Horarik, M. (2016). Building a metalanguage for interpreting multimodal literature: insights from systemic functional semiotics in two case study classrooms. In English in Australia, 51(2), 85–99.
Mackenzie, N., & Veresov, N. (2013). How drawing can support writing acquisition: Text construction in early writing from a Vygotskian perspective. In Australasian Journal of Early Childhood, 38(4), 23–29. DOI: https://doi.org/10.1177/183693911303800404
NATE (National Association for the Teaching of English). (2009). Entitlement to ICT in Secondary English. Retrieved September, 2018 from www.nate.org.uk/cmsfiles/ict/Secondary_English-entitlement_document_2009.pdf
New London Group. (1996). A pedagogy of multiliteracies: designing social futures. In Harvard Educational Review, 66(1), 60–92. DOI: https://doi.org/10.17763/haer.66.1.17370n67v22j160u
New London Group. (2000). A pedagogy of multiliteracies designing social futures. In B. Cope & M. Kalantzis (Eds.), Multiliteracies: literacy learning and the design of social futures (pp. 9–37). London: Routledge.
Newfield, D., & D’Abdon, R. (2015). Reconceptualising Poetry as a Multimodal genre. In Tesol Quarterly, 49(3), 510–532. DOI: https://doi.org/10.1002/tesq.239
Oldham, J. (2005). Literacy and media in Secondary Schools in the United Kingdom. In B. Street (Ed.), Literacies Across Educational Contexts: Mediating Learning and Teaching (pp. 170–187). Philadelphia: Caslon.
Olin-Scheller, C. (2006). Mellan Dante och Big Brother: en studie om gymnasieelevers textvärldar. [Between Dante and Big Brother: a study on the text worlds of upper secondary school students]. Diss. Karlstad: Karlstads universitet.
Pahl, K. (2009). Interactions, intersections and improvisations: Studying the multimodal texts and classroom talk of six to seven year olds. In Journal of Early Childhood Literacy, 9(2): 188–210. DOI: https://doi.org/10.1177/1468798409105586
Pahl, K., & Rowsell, J. (Eds.) (2006). Travel Notes from the New Literacy Studies: Instances of Practice. Clevedon: Multilingual Matters Ltd. DOI: https://doi.org/10.21832/9781853598630
Pantaleo, S. (2012). Meaning-making with colour in multimodal texts: an 11-year-old student’s purposeful ‘doing’. In Literacy, 46(3), 147–155. DOI: https://doi.org/10.1111/j.1741-4369.2012.00664.x
Potts, D., & Moran, M. J. (2013). Mediating multilingual children’s language resources. In Language and Education, 27(5), 451–468. DOI: https://doi.org/10.1080/09500782.2012.720688
Rowsell, J., & Decoste, E. (2012). (Re)designing writing in English class: a multimodal approach to teaching writing. In Pedagogies: An International Journal, 7(3), 246–260. DOI: https://doi.org/10.1080/1554480X.2012.685226
Shipka, J. (2011). Toward a Composition Made Whole. Pittsburgh: University of Pittsburgh. DOI: https://doi.org/10.2307/j.ctt5hjqkk
Silseth, K., & Gilje, Ø. (2019). Multimodal composition and assessment: a sociocultural perspective. In Assessment in Education: Principles, Policy & Practice, 26(1), 26–42. DOI: https://doi.org/10.1080/0969594X.2017.1297292
Skolverket [The Swedish National Agency for Education]. (2016). It-användning och it-kompetens i skolan [It use and it competency in schools]. Retrieved March 2016 from http://www.skolverket.se/om-skolverket/publikationer/visa-enskild-publikation?_xurl_=http%3A%2F%2Fwww5.skolverket.se%2Fwtpub%2Fws%2Fskolbok%2Fwpubext%2Ftrycksak%2FRecord%3Fk%3D3617
Skolverket [The Swedish National Agency for Education]. (2017a). Läroplan för grundskolan, förskoleklassen och fritidshemmet 2011: reviderad 2017. [Curriculum for the compulsory school, preschool class and the recreation centre, 2011, revised 2017]. Stockholm: Skolverket. Retrieved July 2017 from https://www.skolverket.se/om-skolverket/publikationer/visa-enskild-publikation?_xurl_=http%3A%2F%2Fwww5.skolverket.se%2Fwtpub%2Fws%2Fskolbok%2Fwpubext%2Ftrycksak%2FBlob%2Fpdf3813.pdf%3Fk%3D3813
Skolverket [The Swedish National Agency for Education]. (2017b). Läroplan, examensmål och gymnasiegemensamma ämnen för gymnasieskola 2011, reviderad 2017. [Curriculum for upper secondary school 2011, revised 2017]. Stockholm: Skolverket. Retrieved July 2017 from: https://www.skolverket.se/polopoly_fs/1.266015!/Laroplan_gymnasieskolan.pdf
Smith, B. E. (2014). Beyond words: A review of research on adolescents and multimodal composition. In R. E. Ferdig & K. E. Pytash (Eds.), Exploring multimodal composition and digital writing (pp. 1–19). IGI Global: Hershey. DOI: https://doi.org/10.4018/978-1-4666-4345-1.ch001
Sofkova Hashemi, S. (2017). Socio-semiotic patterns in digital meaning-making: semiotic choices as indicator of communicative experience. In Language and Education, 31(5), 432–448. DOI: https://doi.org/10.1080/09500782.2017.1305396
Statens Medieråd [The Governmental Media Council]. (2017). Unga och medier 2015. [Young people and media 2015]. Stockholm: Statens Medieråd. Retrieved June 2018 from https://statensmedierad.se/download/18.7b0391dc15c38ffbccd9a238/1496243409783/Ungar%20och%20medier%202017.pdf
Svärdemo Åberg, E., & Åkerfeldt, A. (2017). Design and recognition of multimodal texts: selection of digital tools and modes on the basis of social and material premises? In Journal of Computers in Education, 4(3), 283–306. DOI: https://doi.org/10.1007/s40692-017-0088-3
Tønnessen, E. S. (2011). Multimodal textkompetense og dataspill, [Multimodal text competence and computer games]. In M. Ellvin, G. Skar & M. Tengberg (Eds.), Svenskämnet i förändring […] (pp. 82–99). Svensklärarserien 234.
Unsworth, L. (2008). Explicating inter-modal meaning-making in media and literary texts: Towards a metalanguage of image/language relations. In A. Burn & C. Durrant (Eds.), Media Teaching: Language, Audience, Production (pp. 48–80). Kent Town, South Australia: Wakefield Press and AATE-NATE.
Unsworth, L., & Thomas, A. (Eds.) (2014). English Teaching and New Literacies Pedagogy. Interpreting and authoring digital media narratives. New York: Peter Lang Publishing. DOI: https://doi.org/10.3726/978-1-4539-1311-6
Xerri, D. (2012). Poetry Teaching and Multimodality. Theory into practice. In Creative Education, 3(4), 507–512. DOI: https://doi.org/10.4236/ce.2012.34077