Creativity, communication and information literacy are among the competences often mentioned as required for students in the 21st century (‘ATC21S’, n.d.), and thus the competence to produce digital multimodal texts has become crucial. Furthermore, multimodal student production understood as a meaning-making activity can be viewed as valuable for learning (Selander and Kress, 2012). Accordingly, the national curriculum for the Danish school includes a description of the student as a ‘goal-oriented and creative producer’ in every school subject under the heading of ‘IT and media’ (Undervisningsministeriet, n.d.-b). Unfortunately, international studies on computer and information literacy (ICILS) show that multimedia production tools are rarely employed by teachers (Fraillon, Ainley, Schulz, Friedman and Gebhardt, 2014) and that Danish students do not carry out more advanced forms of digital communication as frequently as students of the same age in other countries (Bundsgaard, Pettersson and Puck, 2014). Furthermore, a national study on student products and assignments in Danish, Mathematics and Science show a tendency with teachers to encourage monomodal rather than multimodal productions, leaving potentials of meaning making through multimodality unrealised (Sloth, Hansen and Bremholm, 2016). In other words, it seems that learning potentials of digital multimodal student production are not fully utilised in the school subjects in Denmark.
From a social semiotic perspective, learning is an activity of sign-making, where the meaning of signs is socially and culturally defined (Selander and Kress, 2012). But in a school set up for more traditional literacy genres, teachers often fail to acknowledge how new multimodal genres communicate meaning (Jewitt, 2006; Burke, 2009; Sørensen, Levinsen, Skovbjerg, Ejsing-Duun, Henningsen, Tosca and Bremholm, 2016). Furthermore, teachers and students do not necessarily share expectations about what counts as quality which is unfortunate since shared expectations are needed to give relevant feedback, to assess student products and to ensure progression (Sadler, 1989; Black and William, 1998). In particular, explicit criteria used for formative assessment has proven to contribute to students’ learning (Hattie and Timperley, 2007; Wølner, 2015). Unfortunately, there is a tendency in Danish schools for formative assessment to be informal and not based on clear criteria (Danmarks Evalueringsinstitut, 2014). Thus, better use of explicit criteria for students’ multimodal products might improve digital multimodal student production in school.
Therefore, we conducted a research project in order to gain knowledge of how to support teachers’ drawing up and use of assessment criteria for digital multimodal products. The conjecture was that a digital tool, providing teachers with adequate assessment criteria, would improve teachers’ ability to assess multimodal products, to give feedback to students about multimodal products and to facilitate student production processes. The development of the tool was carried out as a design based research process: Through a series of seven interventions in Danish or History in 5th to 8th grade secondary classrooms in six different schools, involving seven teachers and about 200 students, a prototype was progressively extended, refined and improved based on interviews of teachers before, during and after student production processes, on observations of class sessions, on assignments and on students’ products. The aim of this article is to examine the significance of teachers’ conceptions of quality when assessing digital multimodal student productions. We show that an important obstacle for teachers considering assessment of digital multimodal student products is that teachers do not regard qualities of digital multimodal genres as legitimate in school.
Children in Denmark live in a highly digitalised society, and technology is an integrated part of the everyday life of children (Sørensen, Audon and Levinsen, 2010). They listen to music, watch videos, read news, play games and so forth (Bundsgaard, Pettersson and Puck, 2014). However, taking active part as citizens involves more than mere consumption, and the Partnership for 21st Century Learning (The Partnership for 21st Century Learning, 2015) includes not only media literacy and ICT literacy, but also communication skills and creativity as key competences in the 21st century. Similarly, UNESCO’s curriculum for teachers on media and information literacy emphasises the ability to understand how media functions and how one can engage with media as prerequisite for self-expression and democratic participation (UNESCO, 2011), and Danish studies show that empowering students to participate in a digitalised society presume expressive and critical communication skills obtained through production processes (Drotner, 2016). Thus, not only receptive, but also productive digital competences are essential for students’ digital literacy in the 21st century (Erstad, 2010; Undervisningsministeriet, n.d.-b).
In 2014 a new school reform was implemented in Denmark, requiring public schools to integrate IT and media as an interdisciplinary ‘theme’ in all taught subjects. The guidelines for the theme provide four ‘student positions’, of which the third specifically points to student production (Undervisningsministeriet, n.d.-c):
These four positions are seen as basically different and yet overlapping aspects of being IT and media competent, both providing new possibilities for teaching and learning and new relations between teachers and students. However, not only teaching and learning have changed: Subject content is influenced by IT and media. Accordingly, the aims for IT and media are incorporated in the objectives for the subjects; the idea being that issues of IT and media are an integral part of the subjects and not simply an add-on. For example, in the guidelines for History:
In the subject History, the students should […] also create and convey History. Here, IT and media hold a number of possibilities […] The integration of IT does not only change the teaching and the learning processes but also the understanding of the subject and its content. (Undervisningsministeriet, n.d.-d)
In the subject of Danish, an increased focus on multimodal reading and writing is evident throughout the whole curriculum. A central objective is the ability to ‘communicate through writing, speech, sound and picture’ (Undervisningsministeriet, n.d.-a), and ‘production and communication, […], sharing of knowledge and cooperation’ are seen as core themes within the subject in the guidelines (Undervisningsministeriet, n.d.-a). To sum up, digital student production is a part of the objectives for the Danish school in every subject, and the subjects themselves—its knowledge, methods and skills—have changed due to the integration of IT.
A study of the role of IT in two of the most used course materials in Danish concludes that none of them enables the students to engage in ‘complex multimodal productions’ (Carlsen, 2017). Similarly, a quantitative study of student production in Mathematics, Danish and Natural Sciences shows that assignments by teachers do not adequately encourage students to make use of multimodality for critical reflection or creative production (Slot et al., 2016). Furthermore, it shows that the framing of production processes is often loose and that students are left to their own devices and can be uncertain about expectations when producing multimodally (ibid). Other studies (Danmarks Evalueringsinstitut, 2014) show that feedback in Danish schools in general tends to be informal and not based on clear criteria. Thus, it seems that there is a need to support assessment practices of teachers and students to improve multimodal production processes.
Research on how teachers plan has shown that teaching materials are a decisive factor (Warren, 2000; Hodgson, Rønning, Skogvold and Tomlinson, 2010). Danish studies show the same tendency (Bremholm, Bundsgaard, Fougt and Skyggebjerg, 2017), pointing not least to the use of the increasing influence of websites with digital teaching materials (Skovmand and Hansen, 2011). This tendency is most likely intensified by the decrease of planning time for teachers, sometimes down to less than 15 minutes per lesson (Bräuner, 2014; Mainz, 2016). In light of this situation, teachers can be expected to be inclined to prefer teaching aids that are at hand digitally and easy to use.
To sum up, digital multimodal production is important in school, but teachers struggle with laying out clear assessment criteria for feedback. Furthermore, there is indication that suitable and feasible assistance could be in the shape of a digital teaching aid since teachers increasingly use digital resources. This project, therefore, rests on the conjecture that a digital tool, providing teachers with relevant assessment criteria, would improve teacher’s drawing up and use of assessment criteria for digital multimodal products.
To develop a tool with adequate assessment criteria and to analyse its relation to teachers’ conceptions of quality of multimodal products, we have drawn on theory on multimodal text practices in school. Furthermore, we have used theory on genre in digital multimodal texts in order to decide how to categorise texts. And finally, we have drawn on theory on explicit assessment criteria and their application in student production processes.
The study is governed by the social semiotic understanding that what is represented and how it is represented is deeply connected (Kress, 2003). Knowledge is always shaped, from the individual learner to society as a whole (Kress and Selander, 2012). From a learning perspective, it is thus important to acknowledge and be sensitive to new ways of shaping or ‘designing’ knowledge.
Today, the use and combination of different modes, especially the combination of written modes with that of still and living pictures or with music and sound, has become accessible for anyone with a digital device resulting in an increasing complexity of the ways textual meaning is composed (The New London Group, 1996; Kress, 2003). In a learning context, this calls for an increased interest in the important characteristics of multimodal texts, including an interest in communities significant for these texts (Burke, 2009). As previous studies have shown, this can be difficult for teachers in a school shaped by text and assessment practices that tend to be linguistic and print-centric (Jewitt, 2007; Wyatt-Smith and Kimber, 2009).
When communication shifts from primarily print-based to primarily screen-based, the traditional modes for expression are influenced by the logic of new modes (Kress, 2003). According to Kress and van Leeuwen (2001), modes such as image, writing, sound and gesture can be comprehended as a set of socially and culturally shaped resources for meaning-making. Each mode follows different logics and has different affordances for expression, actualised depending on the situation (Kress, 2003; Kress and van Leeuwen, 2006). Both teachers and students are influenced by the use of modes in other contexts, and this affects the potentials of a mode in a given situation (Løvland, 2009). Sensitivity towards the use of modes in different contexts therefore seem to be important, when producing and assessing multimodal texts in school.
Not only choices of mode, but also the interaction of modes is important in multimodal production (van Leeuwen, 2005; Maagerø and Tønnesen, 2014; Løvland, 2007). Often written text functions as anchoring of the many possible meanings of a picture, but modes can also complement each other and elaborate or extend an existing meaning (van Leeuwen, 2005; Barthes, 1994), or they might even contrast each other in order to express polyphonic or disharmonic messages, as is the case in many postmodern children’s books (Nikolajeva and Scott, 2000). When producing and assessing multimodal texts, an understanding of the possible interactions of modes is thus an important prerequisite.
To sum up, multimodal production calls for an in-depth awareness of the complexities of multimodal representations of knowledge, including choice of mode, affordances of modes and interaction of modes in the cultural context.
A key question regarding the assessment or value of a multimodal text is its ‘type’ since different types of texts might call for different assessment criteria: What makes a job application good may differ from important features of an excellent comic strip. We use the term genre to denote the categorisation of texts, and we understand genre as a ‘social agreement, a text norm, for which texts shall have status as text of a certain kind’ (Selander, 2008). Book vlog, radio reportage, stop-motion film, job application, VoiceThread, cartoon, satire are thus all examples of genres. The purpose of genre is to support—some might even say enable—communication from author to reader through conventions and expectations (Lüders, Prøitz and Rasmussen, 2010; Fowler 1989 in Chandler, 2000). Regarding classifications, they often come across as rather unsystematic: Sometimes the purpose is highlighted (job application), sometimes the tool (VoiceThread). Sometimes mode is essential (radio reportage), sometimes not (satire). Theoretically, genre has traditionally been understood with reference to ‘content (such as themes or settings) and/or form (including structure and style)’ (Chandler, 2000). But others have focused instead on a rhetorical dimension and have argued that genre must be ‘organized around situated actions (that is, pragmatic rather than syntactic or semantic)’ (Miller, 1984), and consequently the purpose of the text is emphasised (The New London Group, 1996). Whatever the case, ‘genres only exist in so far as a social group declares and enforces the rules that constitute them’ (Hodge and Kress, 1988 in Chandler, 2000), and therefore genre categorisation is determined empirically by the use in the specific social context in our project. As a consequence, we accept very different configurations of genre, employing the concept of genre only to identify a text as ‘a text of a certain kind’ in a social group, in this case teachers and students in Danish schools. Examples of genres include ‘book vlog’, ‘PowerPoint’, ‘website’, ‘news site’—some very generic genres, others rather specific subgenres.
Studies of evaluation and feedback practices have shown that learning is enhanced when explicit learning objectives and criteria, used for formative rather than summative assessment, are employed in the classroom (Black and William, 1998; Hattie, 2008; Hattie and Yates, 2013). Especially feedback about tasks based on explicit criteria has proven to contribute to students’ learning (Hattie and Timperley, 2007; Wølner, 2015). Furthermore, as shown in studies on writing instruction, feedback and revision based on criteria concerning text structure and communication purpose can improve students’ written products and their literacy skills in general (Philippakos and MacArthur, 2016; Kringstad and Kvithyld, 2013; Kvithyld and Aasen, 2011). Thus, this study is based on the assumption that explicit criteria can assist students and teachers in sharing expectations and that students thereby develop and qualify their multimodal productions.
In the field of multimodal text production in school, one can find different examples of criteria-based assessment. For instance, a large number of teacher-generated rubrics on the internet provide criteria for a wide range of different multimodal genres (e.g. Vandervelde, 2018). For the most part, they are rather specific which is problematic given the complexity and openness of multimodal texts. Instead, Kress and Selander argue that the introduction of multimodal texts in the classroom demands a new ‘ecology of evaluation’ (Kress and Selander, 2012) where students’ argument for their choices is the focal point of assessment. To support a vocabulary of arguments about the quality of multimodal texts in the classroom, some have tried to establish a framework on assessment of multimodal texts (Bearne, 2009) containing a set of ‘descriptors’ focusing among other things on ‘content and modes for specific purpose(s) and audience(s)’ and organisation of texts ‘for communicative purposes’. Some have drawn attention to the rhetorical dimension of multimodal design emphasising the student’s ‘ability to attend to the different purposes, audiences, and genres they will encounter in school and out’ (Baldwin, 2016). Others have tried to develop general theory-driven design rubrics to assess learners’ multimodal text production (e.g. Hung, Chiu and Yeh, 2013; Levy and Kimber, 2009). All these studies can point to important features of assessment of multimodal texts, though most do not acknowledge the importance of the specificities of the genre in question.
The aim of the project was to gain knowledge of how to improve teachers’ drawing up and use of assessment criteria for digital multimodal products and, as mentioned above, we concluded that a digital ‘tool’, providing teachers with relevant assessment criteria, might support teachers. Furthermore, we wanted to acknowledge the complexity of the problem in a real context, involving teachers in school doing projects with digital multimodal production. Therefore, we decided to develop a digital tool in a design-based research process (Kennedy-Clark, 2013; Christensen, Gynther and Petersen, 2012). The tool consisted of a simple website called godtprodukt.dk (‘good product’) that presented suggestions for assessment criteria for digital student products of different multimodal genres. It is important to stress that the tool in itself was not the only goal of the project: The development of the tool should be seen as a means for achieving knowledge about how to best support teachers’ drawing up and use of assessment criteria for digital multimodal products.
In accordance with the supposition that content and form are deeply connected (Kress, 2003), we decided to focus on products with a recognisable content, thus excluding on the one hand purely (or predominantly) aesthetical products—e.g. abstract music or sound art, or purely visual patterns. Furthermore, to draw attention to the learner’s shaping of knowledge, we decided to limit ourselves to student production where the content was not totally fixed in advance. Therefore, as a delimitation, we chose two subjects, Danish and History, where digital student production is part of the curriculum, where purely aesthetical products are rare or non-existent, and where there is an understanding that the way a story is told (form) affects what is being told (content).
The fundamental conjecture (Cobb and Gravemeijer, 2008) was that an intervention—introduction of the digital tool to the teacher—would improve considerations for quality of digital multimodal student products during production processes in the classroom through the employment of explicit quality criteria. Details of this overall conjecture was developed and refined throughout the project through the design of the tool. Furthermore, the development comprised suppositions concerning aspects such as the influence of the teacher’s knowledge of multimodality, the employment of explicit criteria and possible moderators influencing the intended impact.
As a preparation for the development of the tool, a preliminary research stage was carried out, consisting of four background studies, a brief literature review, an examination of the national curriculum and an analysis of educational materials about video to identify examples of assessment criteria in use. Furthermore, six semi-structured interviews with teachers with substantial experience with student video production were undertaken in order to explore experienced teachers’ view of important elements of quality of students’ products. These four studies formed the basis of the development of the first simple prototype that consisted of proposals for quality criteria for the genre in question.
The prototype was then progressively extended, refined and improved through a series of three iterations, each consisting of two or three interventions.
Two 5th grade History classes
An 8th grade Danish class
Two 8th grade Danish classes
An 8th grade History class
A 7th grade cross disciplinary course on globalisation.
A 7th grade History class
A 6th grade Danish class
The seven interventions were carried out at five different schools, involving seven different teachers and about 200 students. We concentrated on lower secondary classes to assure that students were old enough to understand and reflect on criteria for products and their own choices of design, avoiding upper secondary classes where the final exams potentially would have too much influence on the learning processes of multimodal production. Each iteration proceeded through the same general phases of preparation, experiment, analysis and redesign (Cobb and Gravemeijer, 2008; Christensen, Gynther and Petersen, 2012):
As preparation for each intervention, we updated the prototype to accommodate for the specific production process based on correspondence with the teachers. We also conducted brief interviews regarding the teachers’ knowledge of and attitudes to multimodal production in order to support the analysis of subsequent processes in the classroom.
The experiment phase of the intervention was carried out by the teachers in the classroom, and we carried out observations during the teachers’ introduction of the production task, during the production process itself (whenever possible) and during the students’ presentation of their products. The focus of the observations was how the teachers and the students communicated about assessment of product quality, including 1) the presented assessment criteria, 2) feedback from the teacher to the students and 3) students’ conversations with each other about their products. After the production process, we obtained a copy of the finished student products, and we conducted a second semi-structured interview with the teacher, again regarding the teacher’s knowledge of and attitudes to multimodal production. In particular, we asked about the drawing up of the actual criteria used in class, the relation to the suggested criteria of the tool and not least about the relevance of the criteria throughout the production process in the mind of the teacher. We also asked the teacher to comment on and assess the quality of the student’s products, including which (parts of) the finished student products the teacher valued the most, and why. Finally, we asked open questions regarding the relevance and value of the student production process in general (e.g. ‘what do you take with you from this production process?’).
After each production process, we carried out an analysis with the aim of closing gaps between the intended, the implemented and the attained design (McKenney, Nieveen and van den Akker, 2006; Christensen, Gynther and Petersen, 2012): In each iteration the overall intention of supporting teachers’ drawing up and use of assessment criteria was expressed as a specific implementation, an intervention in the shape of the tool, which was compared to the attained use of quality criteria in the production process. We were especially interested in the relations between the criteria of the tool, the presented criteria by the teacher, criteria used in feedback and criteria used in communication between the students. Furthermore, the used criteria were compared to expressions of quality manifest through the students’ products. Finally, in order to explain and understand discrepancies, we drew on the interviews with the teacher. The focus of the analysis was on how to improve the tool. How could gaps between the intended, the implemented and the attained design regarding the teacher’s drawing up and use of assessment criteria for digital multimodal products be closed? Should we adjust our conjectures regarding the mechanisms of the intended impact of the tool? Or was an absent impact rather due to the influence of detrimental moderators? (Dahler-Larsen, 2013) The results were then used to redesign the tool in an attempt to strengthen impacts and to respond to moderators.
All interviews and observations have been conducted in kind collaboration with the teachers who approved the aim of the project and that interviews could be used for quotation. All student products have been collected with permission from the schools. Students, teachers and schools are anonymised in the article.
This paper covers a specific part of the results concerning relations between the design of the tool and teachers’ conceptions of quality of digital multimodal products, leaving out other aspects such as e.g. user experience of the tool. The overall structure of the analysis is closely related to the design-based research process that was undertaken developing the tool. Firstly, we present the results of the preliminary research stage regarding teachers’ conceptions of quality. Subsequently we unfold each iteration. After each iteration, we present proposed design principles for supporting teachers’ drawing up and use of assessment criteria for digital multimodal products.
As part of the preparation for the first prototype of our tool, we conducted six interviews with expert teachers. We were particularly interested in how the teachers reviewed and appraised multimodal student products: What did they consider important features of a successful product? However, it proved surprisingly difficult to get the teachers to talk about product quality since they overwhelmingly prioritised other aspects.
First of all, multimodal student production often seemed to serve primarily as a means by which other goals were reached. For instance, when asked about assessing product quality, two teachers said:
You should get a product that you are proud of and pleased about.
Quality is making an effort.
Without any consideration or discussion, the focus was diverted from the question of product quality to questions of motivation. Put in a nutshell, a ‘good product’ is sometimes seen as a product that motivated the students.
Secondly, some teachers seemed more interested in the process than in the product. Thus, activities seemed more important to the teachers than product quality. Revealingly, answers to questions about assessing products included:
That is when the students work well.
Quality is to hit the right spot. Good, old Vygotsky. The zone of proximal development. […] if they enter flow. If you can watch them there, right there, there it hits. Then I think it becomes quality.
Apparently, the teachers conceptualised digital products primarily as a means for student activity that are not in themselves essential for students’ learning of the subject, and therefore questions of product quality again become less relevant.
A third aspect often overshadowing questions of assessing product quality was a focus on learning. In Denmark, there has been a strong emphasis from the Ministry of Education, influential researchers and local authorities on teaching governed by learning objectives. Therefore, it was not surprising to see a tendency throughout our project for questions of product quality to be superseded by a concern for learning goals. For instance, when asked about what constitutes product quality, a teacher answered:
That I experience that learning has taken place. And maybe not so much the product itself […] If learning has taken place, it might be that the product is not very good, but it is quality anyway because one has benefitted from it.
Of course, a crucial point of education is the students’ learning, and thus a strong focus on learning is perfectly reasonable. It is also well-known that one can learn from a ‘failed’ project, and an end-product of low quality does not necessarily imply a lack of competence but might just as well stem from a willingness to experiment and learn from mistakes. However, that does not entail that product quality is irrelevant: The ability to identify mistakes requires a notion of success, of a successful product. By the same token, when teaching a traditional school genre, e.g. the book report, characteristics of a ‘good’ book report are entirely customary. Nevertheless, even when asked specifically about product quality of student’s multimodal products, the discourse of assessing product quality was repeatedly trumped by the discourse of assessing learning goals. The focus on learning meant that questions of assessment or evaluation was thought of as exclusively aimed at the students, not the products. When asked about assessment of products, the teachers more often than not referred to assessment of student’s competences or skills.
To conclude, the preliminary study substantiated the assumption that product quality is given a very low priority or even absent when students produce multimodal texts. Thus, a suggestion for a design principle to be used in the development of the tool is that teachers need to be guided to draw in consideration for quality of student products. Furthermore, since the very term ‘assessment’ seemed to shift attention away from the products themselves, we decided to refer to ‘quality’ instead of ‘assessment’ when addressing questions of assessing or giving feedback to student products, or to rephrase questions, for instance avoiding a question like ‘how do you assess student products?’, instead asking ‘in your mind, what characterises a successful student product?’
During the first iteration, the digital tool itself was not developed. Instead, we applied ‘paper prototyping’, offering the teachers suggestions for quality criteria to be used as a starting point in a production process. Our formulation of quality criteria for the teacher was informed by theory on multimodal text (e.g. Kress, 2003; van Leeuwen, 2005) pointing at the representation of content through modes, cohesion between modes and constitutional organisation principles. Theory on criteria-based learning (Wille, 2012; Black and William, 1998) suggests that students benefit the most from criteria that are developed specifically for the class and together with the students. Thus, a premise for the criteria given to the teacher was that the teacher adapted them either by herself or together with the students when the assignment for the multimodal production was given.
The iteration consisted of two interventions: The first intervention in two 5th grade classes of History using Wix to create small websites about the Middle Ages, the second intervention in an 8th grade Danish class where the students were to produce a book vlog—a video blog where the vlogger presents and promotes a book.
In the two History classes, the teacher had formulated three ‘requirements for the website’ in the assignment focusing on 1) organisation of the website: ‘two pages’, 2) representation of knowledge: layout and content ‘must fit the subject’ and 3) mode: ‘pictures, video, sound, links, etc. must fit the subject’. In the Danish class, students developed criteria together with the teacher on the basis of different model texts. In both interventions, assessment through explicit criteria were at the core of the instructional processes, supporting the design principle of emphasising product quality (and avoiding the word ‘assessment’).
Nevertheless, the criteria were handled very differently in the two classrooms for reasons worthy of closer analysis. In the History classes, observations showed students being keen on designing the pages in specific ways, in accordance with the assignment criteria that ‘pictures, video, sound, links, etc. must fit the subject’. One group focused hard on choosing the right image of a sword or a beautiful castle, another expressed more poetic interpretations using black birds as a background for written stories about the rough life in the Middle Ages.
In general, students payed much attention throughout the production process to how different modes and their interaction could convey interpretations of important features of the Middle Ages. Nevertheless, communication about how and why backgrounds, colours and pictures suited the subject hardly took place throughout the presentations of the products in class. For instance, the students when asked why they had chosen a (ready-made) background with ferns for their page, simply answered: ‘Because we felt so’. When the same question about design was posed by the teacher to the group with the black birds as visual background, the answer was the same: ‘Because we felt so’. The reluctance to unfold their choices came across as highly contradictory: On the one hand, the students showed great passion and strong preferences for specific design choices; on the other hand, they did not engage in discussions about their multimodal design. When asked about the possible reasons for this, the teacher showed little confidence in the students’ ability to do more:
I think it is difficult to reach a higher level [of the discussion], […] where we can ask: is this picture appropriate for the text? I actually think, it is difficult to lift them up to a level where it is not just ‘You could have added an extra picture’.
Seen from a multimodal perspective, the students did communicate a lot about the Middle Ages through their products. Nevertheless, the students as well as the teacher were unable to or unwilling to express opinions about the value of the different choices of semantic resources, and the tool did not seem to support such conversation.
In the Danish class, discussions were much more explicit, and the teacher had a strong focus on the communication purpose of the book vlog. A week earlier, the students had handed in a written book review where the communication purpose had been to present, to analyse and to assess the book. Now it was time to perform the promotion of the book. To support this goal, the students watched examples of book vlogs and took hold of their production process with a table where they could check up on their own use of different modes such as scenography, gestures and speech to see if it served the purpose of the communication. This scaffolding of the students’ use of modes seemed to be very helpful for both teacher and students, especially because the use of modes was connected to the specific communication purpose.
In the History class, model texts had also been shown, but they exhibited very different communication purposes, in that two of them served commercial purposes (nike.com and boost.com) while only one—the ‘boring’ one, according to the students—served an information purpose (the website for the National Gallery). Not only the different purposes of the model texts, but also the communication in the classroom early on revealed that the teacher operated with an ambiguous communication purpose where the students had both to ‘inform’ (according to the assignment) about the Middle Ages and to do it in an ‘exciting’ way (according to what was said by the teacher in the classroom) so that students in the next classroom to whom the pages were to be presented would get interested in the pages. As far as this ambiguity of purposes was kept implicit, the students’ reluctance to explain their choices of colours, background and pictures seems to make sense.
All in all, communication purpose seemed to be a key component for teachers’ ability to give relevant feedback on multimodal products, both during production and evaluation processes. Therefore, we suggested as a design principle that communication purpose, including connections between modes and purpose, needed to be emphasised.
The second iteration took place with three different teachers at three different schools. One of them was a very experienced teacher of Danish who expressed interest in the tool hoping it would help her through a week with a focus on multimodal production in two 8th grade classes. She had little experience with multimodal production but expressed that she felt encouraged to work with book vlogs, through the scaffolding criteria provided by the tool. Another was a younger teacher of History who was to make picture presentations of historical paintings in his 8th grade class with the use of a digital tool named VoiceThread that was unfamiliar to him. His interest in the tool lay both in the possibility of helping us making up effective criteria and being helped to develop suitable criteria for the picture presentations. The third teacher with History and Social Science in two 7th grade classes led a course on globalisation with a theme on the American presidential campaign where students were to promote either Hillary Clinton or Trump through a website. Unfortunately, he was absent throughout the production process so students in his classroom ended up using criteria directly printed out from the tool.
In this iteration, we introduced a ‘tool’ in the shape of a website presenting suggestions for quality criteria for digital student products of different multimodal genres. In accordance with the design principle of the first iteration, we payed particular attention to giving prominence to communication purpose, including communicative affordances of the different modes. We therefore included criteria like:
Notes from our observations of the teachers’ first meeting with the website show that our intention to bring the purpose of communication to the teachers’ attention seemed to work. Presented to website and the purpose, the teacher with the book vlogs exclaimed: ‘Yes, the purpose is first and foremost, that the viewer gets interested and feel like reading the book.’ Similarly, the teacher of History: ‘It is a picture presentation, a primarily oral and visual presentation of an analysis and interpretation of a picture. Yes, check!’ Thus, the design principle of giving prominence to communication purpose seemed to improve teachers’ drawing up and use of assessment criteria for digital multimodal products.
Nevertheless, even though the suggested quality criteria of the tool of the second iteration clearly accentuated how the use of different modes can support the communication purpose, the teachers often failed to recognise connections between content and representation in a multiplicity of modes. Time and again, we were puzzled that certain qualities were not seen as part of the communicated content of the students’ products. For instance, the teacher in the course on book vlogs disclosed afterwards what she was really looking for as signs of quality of the book vlogs, namely certain ‘literary terms’:
What is lacking [in these productions] are the literary terms. They haven’t learned them as well as if they have had to make a written book review. […] They have had so much focus on that ambiance and on the performance that they simply forget about these things.
We had already seen similar conceptions in the first iteration, in the History class, where the students’ reason for not discussing multimodal aspects of the products might not only be due to a lack of competence to do so. A contributing reason might be that the students knew very well what counted as knowledge in the History classroom, even though it was not announced explicitly. In an interview held after the presentation of the websites, the teacher thus expressed uncertainty if the website project had improved learning about the Middle Ages:
If I had done it myself, there would have been a better opportunity to have a classroom discussion about what are this Middle Ages actually about.
What was deemed ‘proper’ history learning was not so much the students’ design of their understanding of the Middle Ages through the juxtaposition of pictures, writing and layout, but the ability to explain in words what the Middle Ages ‘were about’.
These teachers looked for linguistically represented knowledge; language in the traditional sense of the word—just as others have found that teachers in general do in multimodal products (Jewitt, 2006; Løvland, 2007). In both cases, the teachers seemed to have a notion of content unchanged by the modes in which it was presented—and in both cases, a lack of understanding about expressive possibilities of different modes resulted in a tendency to disregard the content not conveyed through words. The production of a website or a book vlog was simply seen as ‘new’ forms in which to put the same content. But if that were the case, why did the quality criteria of our tool not help, even though they emphatically pointed to the content conveyed through other means? We suggest that the predominance of language was not (only) due to a lack of awareness of—or even competence concerning—multimodal aspects of the products, but that language was seen—by teachers and students—as the ‘proper’ or ‘true’ mode of learning. In other words, language seemed to be viewed as the legitimate school mode and therefore was given predominance compared with other modes.
As we have argued, different multimodal genres—understood as empirically determined expectations and conventions—constitutes different configurations of not just mode, but of content, knowledge, structure, style and/or purpose (Chandler, 2000; Miller, 1984). Viewing language as the primary mode through which understanding can be communicated, not only overlooks the potentials of other modes, it overlooks the specific expectations and conventions of the genre in question. Thus, it seemed that choice of genre was often driven more by practicalities and somewhat random circumstances than a concern for expressive possibilities of different genres. Throughout the second (as well as the first) iteration, we encountered vague or contradictory conceptions of multimodal genres in different ways—not just related to giving predominance to the linguistic mode.
An example of opposing genre conceptions in the second iteration could be seen in the case of one of the students producing a book vlog. He had chosen to talk about the book, ‘Maze Runner 3’, while taking a shower as an illustration of the recurring rain in the book. What he said about the book and what happened in the picture was not directly connected, nor had the climax of the video when the foam of his shampoo ran down over his face. Nevertheless, the surprising visual setting managed to keep the attention of the viewer throughout most of his video. Thus, the video enacted a specific understanding of the book vlog genre: The purpose is to inform and entertain, and it combines promotion of a book and of the vlogger. Whereas facts about the book were communicated through language, the role of the visual setting was to highlight both aspects of the book and of the person—as well as connect the two. Unfortunately, our tool did not support the teacher sufficiently to appreciate the book vlog genre as expressed by the student, and the teacher thus ignored important communicated content in her feedback. Above, we suggested that this omission was due to the primacy of language, but another explanation might be that the teacher lacked a clear conception of the book vlog genre and therefore resorted to other, more well-known genres, namely book review or literary analysis in her assessment of the quality of the book vlog.
In the two other projects of the second iteration, we again saw that the tool failed to help the teachers to appreciate a comprehensive view of the genres in question, even though the teachers did adopt the suggested criteria. For instance, the layout of the promotional websites was often more the result of prefabricated templates than an understanding of the genre. In the latter project, the teacher had developed a detailed ‘checklist’ inspired by the criteria of the tool to assure that the students got around everything but failed to combine the components to a coherent appreciation of the genre.
Actually, we had already met similar struggles with the appreciation of the genre in question in the first iteration. Even the experienced teachers of the book vlog project of the first iteration said:
In fact, […] I would really love it if a professional TV journalist or something like that could help me. You know, I am just using my amateur teacher skills here. I make some success criteria from what I got from the subject of Danish, that is, what I know about cinematic effects and what I know about sound […]. But I am not a website expert or a production expert, you know.
To conclude, we suggested that a design principle to be used in the development of the tool was to convey an understanding of the specific genre in its entirety, not just the utilisation of many modes and not just communication purpose. Otherwise, traditional school genres might tend to take precedence and ‘overshadow’ the intended multimodal genre.
In our third iteration that took place in a 7th grade History class working with a theme on slavery and in a 6th grade Danish class working with book posters presented at a ‘book fair’ in the library, we adjusted the tool in order to draw attention to genre. Firstly, the teacher had to select from different options in a series of steps, thus emphasising the specificity of the selected genre. For instance, the teacher of History who said he wanted to work with ‘podcast’ was sent through the following steps:
Each option was described by a short characterisation that—depending on the choice—accentuated aspects concerning mode, target audience, purpose, style etc. The number of steps depended on the specificity of the genre in question. Finally, the genre in question was presented through a short characterisation before the suggested criteria. For instance, the characterisation for on-the-spot report was:
An on-the-spot report in sound is a first-hand account from a journalist on the place where an event takes place. The journalist determines the story angle. The purpose of the report is to inform about the event by giving the listener a sense of being present. Interviews can be included.
Preliminary results suggest that the design principle of conveying an understanding of the specific genre was a step in the right direction: Accentuating genre as much more than simply the coinciding use of many modes, or a communication purpose, or a set of structural traits meant that the teachers of the third iteration were much better able to take advantage of the tool. It also meant that further development of the tool could focus on other aspects than teachers’ conceptions of quality of digital multimodal student productions—the subject of this paper.
Throughout the project, we experienced that a clearer conception of what counts as quality of the products tended to improve teachers’ ability to facilitate student production processes. In particular, it proved useful to avoid talking about ‘assessment’ even when the subject was assessing quality since assessment was thought of as having to do with assessing students, not products. Furthermore, certain aspects seemed to be of value: Firstly, we were confirmed that explicit criteria seem to improve feedback and discussion of quality of multimodal products. If students are to become a ‘goal-oriented and creative producer’ in every school subject (Undervisningsministeriet, n.d.-b), explicitness about what counts as quality becomes an important way to support learning. Secondly, it proved vital to take students’ conceptions of quality seriously. Students’ eager and firm decisions about design indicate engagement and strong opinions that are worth taking seriously, and as noted, some even call for a new ‘ecology of evaluation’ (Kress and Selander, 2012) focusing on arguments for and discussions about choices. This might also call for new relations between teacher and students and new teacher roles. Nevertheless, discussions in themselves are not enough: In our project, we saw again and again how cultural norms and expectations regarding the particular product played a vital role, and sensibility to these norms and expectations seem to be an important basis for an understanding of quality. Thus, thirdly, conceptions of genre are of great importance, and not least the explicit verbalisation of genre as expressed through descriptions and criteria. On the other hand, a wish for hard-and-fast criteria is problematic if it means that the complex and ever-changing character of digital multimodal genres cannot be embraced (Lüders, Prøitz and Rasmussen, 2010; Wyatt-Smith and Kimber, 2009). Therefore, a tool providing teachers with criteria for multimodal genres should never be seen as a promise of stable norms, but rather as a proposal, open for discussion.
In our analysis, we point out some design principles that appeared to be useful in order to support teachers’ drawing up and use of assessment criteria for digital multimodal products, focusing on how to address teachers’ conceptions of multimodal genres. From the humble starting point that teachers need to be guided to draw in consideration for quality of student products, we accentuated communication purpose as a key aspect of multimodal production. Furthermore, it proved valuable to emphasize how meaning can be expressed by other modes than language. Finally, we came to the conclusion that teachers’ drawing up and use of assessment criteria for digital multimodal products are best supported by a complex, balanced and holistic view of the genre in question, sensitive to its specific combination of purpose, content, structure and style.
The authors have no competing interests to declare.
ATC21S. (n.d.). Retrieved 30 June 2017, from Assessment & Teaching of 21st Century Skills website: http://www.atc21s.org/.
Baldwin, K. M. (2016). Multimodal Assessment in Action: What We Really Value in New Media Texts (Dissertation, University of Massachusetts – Amherst). Retrieved 28 June 2019 from https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1778&context=dissertations_2
Black, P., & Wiliam, D. (1998). Inside the Black Box: Raising Standards Through Classroom Assessment. Phi Delta Kappan, 80(2), 139–144. DOI: https://doi.org/10.1177/003172171009200119
Bräuner, K. (2014). Forberedelsestid i reformfolkeskolen. [Preparation time in the school of the reform]. Retrieved 23 June 2019, from Folkeskolen.dk website: https://www.folkeskolen.dk/551496/forberedelsestid-i-reformfolkeskolen
Bremholm, J., Bundsgaard, J., Fougt, S. S., & Skyggebjerg, A. K. (2017). Danskfaget set gennem dets læremidler. [The subject of Danish seen through its teaching aids]. In Læremidlernes Danskfag. Århus: Aarhus Universitetsforlag.
Bundsgaard, J., Pettersson, M., & Puck, M. R. (2014). Digitale kompetencer: it i danske skoler i et internationalt perspektiv. [Digital competences: IT in Danish schools in an international perspective]. Aarhus: Åarhus Universitetsforlag.
Burke, A. (2009). Checkmarks on the Screen: Questions of Assessment and New Literacies in the Digital Age. In Assessing New Literacies. Perspectives from the Classroom. (pp. 35–54). New York: Peter Lang.
Chandler, D. (2000). An Introduction to Genre Theory. Retrieved from http://visual-memory.co.uk/daniel/Documents/intgenre/chandler_genre_theory.pdf
Christensen, O., Gynther, K., & Petersen, T. B. (2012). Design-Based Research – introduktion til en forskningsmetode i udvikling af nye E-læringskoncepter og didaktisk design medieret af digitale teknologier. [Design-Based Research – introduction to a method of research in the development of new e-learning concepts and designs for learning mediated by digital technologies]. Læring og Medier (LOM), Årg, 5(9), 20. DOI: https://doi.org/10.7146/lom.v5i9.6140
Cobb, P., & Gravemeijer, K. (2008). Experimenting to Support and Understand Learning Processes. In A. E. Kelly, R. A. Lesh & J. Y. Baek (Eds.), Handbook of Design Research Methods in Education: Innovations in Science, Technology, Engineering, and Mathematics Learning and Teaching (pp. 68–95). New York: Routledge.
Drotner, K. (2016). Børn og unges arbejde med film: Digital dannelse og produktiv læring for fremtiden. [Film making by children and youth: Digital literacy and productive learning for the future]. Det Danske Filminstitut.
Fraillon, J., Ainley, J., Schulz, W., Friedman, T., & Gebhardt, E. (2014). Preparing for Life in a Digital Age: The IEA International Computer and Information Literacy Study International Report. International Association for the Evaluation of Educational Achievement (IEA). DOI: https://doi.org/10.1007/978-3-319-14222-7
Hattie, J. (2008). Visible Learning: A Synthesis of Over 800 Meta-analyses Relating to Achievement. Routledge. DOI: https://doi.org/10.4324/9780203887332
Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81–112. DOI: https://doi.org/10.3102/003465430298487
Hattie, J., & Yates, G. C. R. (2013). Visible Learning and the Science of How We Learn. Routledge. DOI: https://doi.org/10.4324/9781315885025
Hodgson, J., Rønning, W., Skogvold, A. S., & Tomlinson, P. (2010). Paa vei fra læreplan til klasserom – Om læreres fortolkning, planlegging og syn paa LK06. [On the way from curriculum to classroom—On teacher’s interpretation, planning and view of LK06]. Nordland Research Institute.
Hung, H.-T., Chiu, Y.-C. J., & Yeh, H.-C. (2013). Multimodal Assessment of and for Learning: A Theory-Driven Design Rubric. British Journal of Educational Technology, 44(3), 400–409. DOI: https://doi.org/10.1111/j.1467-8535.2012.01337.x
Jewitt, C. (2007). A multimodal perspective on textuality and contexts. Pedagogy, Culture & Society, 15(3), 275–289. DOI: https://doi.org/10.1080/14681360701601937
Kennedy-Clark, S. (2013). Research by Design: Design-Based Research and the Higher Degree Research student. Journal of Learning Design, 6(2), 26–32. DOI: https://doi.org/10.5204/jld.v6i2.128
Kress, G. (2003). Literacy in the New Media Age. Routledge. DOI: https://doi.org/10.4324/9780203299234
Kress, G., & Selander, S. (2012). Multimodal Design, Learning and Cultures of Recognition. Internet and Higher Education, 15(4), 265–268. DOI: https://doi.org/10.1016/j.iheduc.2011.12.003
Kress, G., & van Leeuwen, T. (2006). Reading Images: The Grammar of Visual Design. Routledge. DOI: https://doi.org/10.4324/9780203619728
Levy, M., & Kimber, K. (2009). Developing an Approach for Comparing Students’ Multimodal Text Creations: A Case Study. Australasian Journal of Educational Technology, 25(4), 489–508. DOI: https://doi.org/10.14742/ajet.1125
Lüders, M., Prøitz, L., & Rasmussen, T. (2010). Emerging personal media genres. New Media & Society, 12(6), 947–963. DOI: https://doi.org/10.1177/1461444809352203
Mainz, P. (2016, May 31). Lærere har 14 minutters forberedelse til hver time. [Teachers have 14 minutes of preparation for each lesson]. Retrieved 23 June 2019, from Politiken website: https://politiken.dk/indland/uddannelse/art5623967/Lærer-har-14-minutters-forberedelse-til-hver-time
McKenney, S., Nieveen, N., & van den Akker, J. (2006). Design research from a curriculum perspective. Educational Design Research, 67–90. DOI: https://doi.org/10.4324/9780203088364
Miller, C. R. (1984). Genre as Social Action. Quarterly Journal of Speech, 70(2), 151–167. DOI: https://doi.org/10.1080/00335638409383686
Nikolajeva, M., & Scott, C. (2000). The Dynamics of Picturebook Communication. Children’s Literature in Education, 31(4), 225–239. DOI: https://doi.org/10.1023/A:1026426902123
Selander, S. (2008). Designs for Learning – A Theoretical Perspective. Designs for Learning, 1(1). DOI: https://doi.org/10.16993/dfl.5
Skovmand, K., & Hansen, T. I. (2011). Fælles mål og midler: læremidler og læreplaner i teori og praksis. [Common objectives and means: teaching aids and curriculum in theory and practice]. Århus: Klim.
Sloth, M. F., Hansen, R., & Bremholm, J. (2016). Elevopgaver og elevproduktion i det 21. århundrede – en kvantitativ og kvalitativ analyse af elevproduktion i matematik, dansk og naturfag. [Student assignments and student production in the 21st century—a quantitative and qualitative analysis of student production in Math, Danish and Science]. Retrieved from Læremiddel.dk website: https://www.google.com/search?q=sloth+elevopgaver&rlz=1C1GCEA_enDK797DK797&oq=sloth+elevopgaver&aqs=chrome..69i57.3644j0j7&sourceid=chrome&ie=UTF-8
Sørensen, B. H., Levinsen, K. T., Skovbjerg, H. M., Ejsing-Duun, S., Henningsen, B., Tosca, S., & Bremholm, J. (2016). Slutevalueringsrapport. December 2015: Projekt. Elevers egenproduktion og elevinddragelse. [Final evaluation report. December 2015: Project. Students’ production and student involvement]. Retrieved from Aalborg Universitet website: https://www.uvm.dk/-/media/filer/uvm/udd/folke/pdf16/mar/160314-evalueringsrapport--elevernes-egenproduktion-og-elevinddragelse.pdf
The New London Group. (1996). A Pedagogy of Multiliteracies: Designing Social Futures. Harvard Educational Review, 66(1), 60–93. DOI: https://doi.org/10.17763/haer.66.1.17370n67v22j160u
The Partnership for 21st Century Learning. (2015, May). P21 Framework Definitions. Retrieved from http://www.p21.org/storage/documents/docs/P21_Framework_Definitions_New_Logo_2015.pdf
Undervisningsministeriet. (n.d.-a). Dansk – Fælles Mål, læseplan og vejledning. [Danish—common objectives, instruction plans and guidelines]. Retrieved 28 March 2017, from EMU Danmarks læringsportal website: http://www.emu.dk/modul/dansk-m%C3%A5l-l%C3%A6seplan-og-vejledning%20
Undervisningsministeriet. (n.d.-b). It og medier. [IT and media]. Retrieved 28 March 2017, from EMU Danmarks læringsportal website: http://www.emu.dk/modul/it-og-medier-0
Undervisningsministeriet. (n.d.-c). It og medier - vejledning. [IT and media—guidelines]. Retrieved 29 May 2017, from EMU Danmarks læringsportal website: http://www.emu.dk/modul/it-og-medier-vejledning
Undervisningsministeriet. (n.d.-d). Vejledning for faget historie. [Guidelines for the subject of History]. Retrieved 28 March 2017, from EMU Danmarks læringsportal website: http://www.emu.dk/modul/vejledning-faget-historie
UNESCO. (2011). Media and Information Literacy Curriculum for Teachers. Retrieved from http://www.unesco.org/new/en/communication-and-information/resources/publications-and-communication-materials/publications/full-list/media-and-information-literacy-curriculum-for-teachers/
Vandervelde, J. (2018). Video Project Rubric. Retrieved 25 June 2019, from UW-Stout website: https://www2.uwstout.edu/content/profdev/rubrics/videorubric.html
van Leeuwen, T. (2005). Introducing Social Semiotics. London: Routledge. DOI: https://doi.org/10.4324/9780203647028