Mind the hand: A study on children’s embodied and multimodal collaborative learning around touchscreens

Apart from touching the screen, what is the role of the hands for children collaborating around touchscreens? Based on embodied and multimodal interaction analysis of 8- and 9-year old pairs collaborating around touchscreens, we conclude that children use their hands to constrain and control access, to construct and problem solve, and to show and imitate. The analyses show how a space emerges from the interaction between the children and the touchscreen, and how their hand movements reveal intelligence-as-action. Three situations with three different pairs were analysed to explore how children use their hands in activities around touchscreens, focusing in particular on how they collaborate. The analysis presented here is part of a research study on the use of touchscreens in children’s embodied and multimodal collaborative learning activities in their everyday classrooms. The general aim of the study is to contribute to the understanding of children’s multimodal collaborative learning activities around touchscreens.

Apart from touching the screen, what is the role of the hands for children collaborating around touchscreens?Based on embodied and multimodal interaction analysis of 8-and 9-year old pairs collaborating around touchscreens, we conclude that children use their hands to constrain and control access, to construct and problem solve, and to show and imitate.The analyses show how a space emerges from the interaction between the children and the touchscreen, and how their hand movements reveal intelligence-as-action.Three situations with three different pairs were analysed to explore how children use their hands in activities around touchscreens, focusing in particular on how they collaborate.The analysis presented here is part of a research study on the use of touchscreens in children's embodied and multimodal collaborative learning activities in their everyday classrooms.The general aim of the study is to contribute to the understanding of children's multimodal collaborative learning activities around touchscreens.

i n t r o d u c t i o n
Apart from touching the screen, what is the role of the hands in children's collaborative learning around touchscreens?This question seems especially relevant at a time when two quite different forces are becoming influential in public school systems across in Europe: the increasing use of touchscreen based learning tools in homes and schools and the extension of PISA to encompass testing of children's collaborative problem solving skills (OECD, 2013).The advent of touchscreens is apparent in a recent survey from Norway showing that 36% of children from 0-6 years had used tablets at home Mind the hand: A study on children's embodied and multimodal collaborative learning around touchscreens jACob DAViDSeN, Aalborg University, Denmark eLLeN ChRiStiANSeN, Aalborg University, Denmark (Guðmundsdóttir & Hardersen, 2012).Among other findings, the survey indicated that the children were primarily playing games, watching movies, or viewing photos.However, the 'activity categories' tell us little about how the activities were actually performed or what role the hands played in the multimodal and embodied assemblage of interaction.The benefits of collaborative learning supported by computers are well-established, but the embodied and multimodal interaction of children collaborating has so far been under-researched, even in recent studies of collaboration around touchscreens (Davidsen & Christiansen, 2013).Most studies reporting on collaboration around computers have been oriented towards what Wilson (1999) describe as book and language based knowledge, or intelligence-asinformation, in contrast to intelligence-as-action, which would include the use of hands in the multimodal and embodied assemblage of collaborative interaction.
This gap in the literature invites a detailed micro-level study of hand movements as part of an effort to understand children's embodied and multimodal collaborative learning around touchscreens.Specifically, by means of a detailed embodied and multimodal interaction analysis, the research question in the present study asked how 8-and 9-year old children use their hands in collaborative learning activities around touchscreens.We addressed this question by analysing video footage from a research project called "Move and Learn", in which children's use of touchscreens for collaborative learning activities in the classroom was explored over the course of one school year.In total, 150 hours of video footage was collected in this project and here we analyse three specific situations in which children's hand movements were part of the multimodal and embodied assemblage of interaction.The findings presented here contribute to a recent and more general interest in embodied and multimodal aspects of learning, collaboration and meaning making., Mercier, Burd, and Hatch (2011) argued that the introduction of new digital media such as tablets, touchscreens, and tabletops in classrooms requires a re-orientation towards studying the ways in which digital media can support children's face-to-face collaboration, which has been studied extensively in the past.During the 1980s, Papert (1980), one of the main proponents of computer support for children's (collaborative) learning, suggested that moving a programmable turtle on the floor instead of a cursor on a computer screen made mathematical reasoning more intuitive.However, as discussed by Nielsen (1987), Papert's description and conceptualisation of the children's work failed to capture the complexity of their interactions with the computer.Studying children working with LOGO, Nielsen emphasised the visual interface of the computer program, as well as empathy and dialogue between children, as important aspects of computer supported collaboration.Like Nielsen's work with children aged 14-15 around a computer using a mouse and keyboard, Roschelle and Teasley (1995) studied the collaborative processes of two 15-year-old boys working with the Envisioning Machine.Roschelle and Teasley noted that "…students are not wholly dependent on language to maintain shared understanding…(p.11)" and that "…actions and gestures can likewise serve as presentations of new ideas…(p.11)".However, in the end they concluded "…the most important resource for collaboration is talk…(p.26)".Likewise, Crook (1994) argued "…we need to be more specific about defining effective language use in this context …" (1994, p. 123).Several other studies of children's computer supported collaboration have also scrutinised the use of language or types of speech acts as indicators of the quality of the collaborative activity (e.g.Alant, Engan, Otnes, Sandvik, & Schwebs, 2003;Wegerif, Littleton, & Jones, 2003).In short, there has been a tendency to focus on talk in collaborative activities-intelligence-as-information-a fact also highlighted by Rowe (2012) in respect of classroom interaction research.We concur with Rowe's proposal to include multimodal aspects in the analysis of classroom interaction.It should be noted that researchers have shown increasing interest in embodied and multimodal aspects of collaboration, looking at how children negotiate meaning through their use of language and body in the material world while completing different tasks, with or without computer support (Greiffenhagen & Watson, 2009;Koschmann & LeBaron, 2002;Roth, 2001;Stahl, 2006).For example, Greiffenhagen and Watson (2009) studied how pairs of children visually repaired each other's contributions using a traditional computer.Nevertheless, none of these studies has in particular focused on understanding how hands are part of children's multimodal and embodied collaboration-the intelligence-as-action that is the primary interest of this paper.

Higgins
r e s e a r c h a p p r oac h Stahl, Koschmann, and Suthers (2006) identified three general research approaches: experimental and conditional studies, iterative design studies, and descriptive studies.In respect of the first of these approaches, Davidsen and Christiansen (2013) concluded that experimental studies of interactions with various touch interfaces focused on types of talk, number of touches, and layout of the shared workspace in the case of collaborative activities.In the second category, design-related studies have presented a mixture of intangible and tangible guidelines for the use of touch-technologies in collaborative learning activities.The third type of study has not been extensively dealt with in the research literature, but it is proposed here that descriptive studies can produce important insights into how touchscreens support embodied and multimodal collaborative learning.Consequently, the present study emphasises "exploring and understanding", in particular, the process of children's embodied and multimodal meaning making (Stahl, 2006) rather than "coding and counting".
The attention paid to types of language use and number of touches in the coding and counting studies also stands in contrast to recent theoretical perspectives from neuroscience (Wilson, 1999) and from embodied interaction analysis (Streeck, Goodwin, & LeBaron, 2011), where the relations between hand, activity, material, and mind are viewed as semiotic resources mutually elaborating each other.According to Wilson, "The most effective techniques for cultivating intelligence aim at uniting (not divorcing) mind and body" (Wilson, 1999, p. 289).Wilson (1999) studied the relation between hand and human development from a historical perspective, providing several examples along the continuum between intelligence-as-information and intelligence-as-action.In the case of PISA, children's collaborative problemsolving skills are currently understood as intelligence-as-information, e.g., "pooling their knowledge" (OECD, 2013, p. 6).On the contrary, proponents of intelligence-as-action have studied diverse activities such as archaeologists determining dirt using a munsell chart (Goodwin, 2000) or the skilled practice of wood workers (Sennett, 2008), convincingly demonstrating how actions and skills develop as a multifaceted interplay of physical and bodily engagement with situations, peers, and materials.
In similar fashion, Roth (2001) showed that children explain scientific concepts through gestures (intelligence-as-action) before they can articulate the concepts in language (intelligence-as-information).However, while in that research the hand is seen as an individual tool rather than as a social tool or as a tool for collaboration as a part of multimodal learning activities, Vygotsky (1978) had already stressed that the gestures of an infant are socially oriented: The simple gesture of pointing causes the mother or father to respond, relying on the surroundings to interpret the child's needs.Recently, researchers studying embodied interaction, including Streeck (2008Streeck ( , 2013)), have shown how car mechanics use hand movements, their bodies, and the material at hand in complex acts of meaning making.Instead of labelling their actions as "embodied", Streeck showed how peers' bodies-inmotion in the situation were crucial for their negotiation of meaning and building of the activity.In spite of these findings, researchers continue to study gestures and categories of gestures (e.g., Sakr, Jewitt, & Price, 2014) in a way that reduces gestures to "something to look for in the data".Just like the coding and counting studies of dialogue in collaborative activities, categories of gestures are abstracted from the situated context of action.Against this, the present analysis takes an inductive approach, studying the ways in which the hands and body move in a given situation, towards a situated and embodied practice of meaning making.Streeck (2013, p. 74) argued that "the meanings manufactured by human hand gestures are more than externalizations of mental structures and processes: they follow from the ways in which human hands in practical action make and experience the world."Thus, the present study does not assume that hand movements denote specific types/categories of gestures in interaction: rather, hand movements are understood as a semiotic resource in the embodied practices (Goodwin, 2000) of children's process of meaning making around touchscreens.
c o n t e x t , d ata , a n d a n a ly t i c a l m o d e l For one school year, a research team followed the everyday practices of teachers and children when using touchscreens in two second grade classrooms at Western State School in Denmark, in a project called "Move and Learn" (Davidsen & Georgsen, 2010).1In general, this project focused on exploring the ways in which touchscreens can support collaboration, interaction, and experimental forms of learning in classroom settings.The touchscreens were placed in the classrooms along the walls of the room and on a small island in the centre (see Figure 1).The main pedagogical/didactical idea was to let children work together in pairs around the touchscreens, with the teacher playing the role of guide/facilitator/designer of the children's collaboration.In this longitudinal study, ethnographic field data such as photos, learning materials, interviews, and observations were collected, and 150 hours of video footage was recorded.For the purpose of this paper, video footage has been selected from the work of three pairs in one of the classrooms, who worked over the course of one week on an assignment relating to the Christian tradition of Easter.At this time, the children and teachers had been using the touchscreens for the previous nine months.
During the assignment, nine hours and 32 minutes of video footage was recorded, which was transcribed using Transana (Woods & Fassnacht, 2013).At first, the transcripts were oriented towards what was said, but as interest shifted to the role that the children's hands play in collaborative learning activities, a micro-level embodied and multimodal transcription practice was developed (Davidsen & Christiansen, 2013;Davidsen & Vanderlinde, 2014a,b), focusing on language, hand movements, and the materials used in the activity.As already mentioned, the main goal of developing the multimodal transcripts was to overcome the limitations found in related work, in which researchers tended to focus on the "audio channel" while neglecting the embodied aspects of children's meaning making (Rowe, 2012).For this reason, the selection criteria emphasised the significance and meaning within a narrative account, as opposed to probabilistic concepts of frequency and representativeness (Derry et al., 2010).From the transcripts, the selected situations show the pairs engaging in embodied and multimodal interactions around the touchscreen.Figure 2 illustrates the analytical unit: the space between the children's hands when they interact with the screen to manipulate screen objects, and when they interact with each other in communication by hand movement.Figure 2 shows two instances of a pair sitting around a touchscreen, with learning material displayed on the screen.They could talk about the material, ask each other questions, or call for help.Likewise, both had their hands free; they could move towards the screen, point at something on the screen, or manipulate the learning material and interface on screen.In addition, they could interrupt each other with their hands or repair each other's work.They could also orient themselves towards the overall classroom environment, asking other children for help or commenting on what other children were doing.In summary, the unit of analysis comprises the way a pair used their hands, language, and supporting tools to organise their interaction, and how they defined, reconfigured, and accomplished assignments.In our analysis, we zoomed in on the children's meaning-making as a complex of what they said, what they did with their hands (and bodies), and how they oriented towards and made use of the learning material present on the touchscreen.In particular, we analysed the ways hands shape children's collaborative activities around touchscreens, as part of the embodied and multimodal interaction.
a n a ly s i s The analysis opens with an introduction to the overall activity, including a brief description of the learning material designed by the teacher.Afterwards, we present three situations which capture and exemplify the role hands play in children's activities around touchscreens.The primary goal of the analyses of the three different situations is to explore how hand movements shape children's embodied and multimodal collaborative learning activities.

What do you know about Easter, children? (L2)
The children and teacher had just returned from their Easter holidays, and the teacher, Anne, had planned for the children to work in pairs, for 1-2 hours each day for a week, on the topic of the Christian religious tradition of Easter.To begin, Anne introduced the overall topic and asked the children what they knew about Easter, writing all their suggestions on the interactive whiteboard (IWB).Only a few of the suggested words related to the religious and historical events of Easter: typical examples included "holidays", "Easter eggs", and "chickens".However, one child added "Jesus dies" to the collective brainstorm, enabling Anne to elaborate on the theme and the Christian religious tradition.She explained that the children were supposed to work with the events taking place on Maundy Thursday, Good Friday, Easter Day, and Easter Monday.After the brainstorm and the announcement of the subject, Anne read a story to the children about Easter.Finally, she introduced the assignment and the learning materials, which included one slide with the different tasks the children were to work with, a slide with text, a slide with a multiple choice quiz, and a slide with scenery and figures for the production of a multimodal story.The different slides were collected in one Smart Notebook™ file.To analyse the embodied and multimodal interaction of the 8-and 9-year olds collaborating around touchscreen in their classroom, three pairs were selected: Dean and Kimberly, Iris and Vince, and Nathalie and Peter.Rather than considering their academic competences or skills; the aim was to show how the hand shapes their collaborative activity around touchscreens.Each situation will be briefly introduced, and we will then analyse the role of hand movements as a part of the children's embodied and multimodal interaction in the situation.The three situations are also used to make visible that each pair contingently make relevant different semiotic resources collaborating around touchscreens, e.g. the children produced different situations with the present and the available resources in their activity.

Dean and Kimberly-"This takes forever"
Dean and Kimberly showed signs of frustration and impatience from the very beginning, finding that the computer was loading the webpage slowly.To troubleshoot, Dean opened the tool for diagnosing the network connection, but Kimberly was not completely satisfied and raised her voice saying, "DEAN".The central IWB then played a loud noise, which made Dean, Kimberly, and the rest of children laugh out loud.

The role of hands
The computer finally responded, and they opened the learning material on the screen.Dean continued to control the events on the screen, looking for CD-ORD (a text to speech program).Kimberly asked "why can I never?"(frames 5-6, Figure 3),2 to which Dean replied: "don't worry, you can try in a minute" (frames 7-8, Figure 3).Then, Dean opened up a window for Kimberly, allowing her to make this contribution: "where does it say language?"(frames 9-10, Figure 3).Kimberly immediately said "here" (frames 11, Figure 3), and pushed the menu button all programs.When we look at their bodies, and especially their hands, we see that Dean was controlling the screen with his left hand and that his right hand was placed on the frame of the screen.With this posture, he left little room for Kimberly to act in, pushing her hand away (in frames 4-6) when she approached the touchscreen.While Dean involved Kimberly verbally in some indirect ways (frames 9-10), he did not allow her to control the screen.We do not see their collaboration going beyond fighting over control of the touchscreen.Their collaboration seems to be a power struggle; their hands are their means of control, while their talk merely suggests an intention to engage in collaboration, with no signs of co-construction or other forms of complementary collaboration.On the contrary, Dean used his hands to constrain Kimberly's access to the screen, limiting the possibility that she could perform operations on the screen.In the follow-up interviews afterwards, unlike the two other pairs, Dean and Kimberly expressed negative attitudes towards being engaged in collaboration.For example, Kimberly said: "Dean is moving my hand away from the screen all the time".This statement further validates our interpretation of the way their hands are used in the collaborative multimodal activity around the touchscreen.
Iris and Vince -"Didn't it sound fine?" Iris and Vince turned on the computer and waited a couple of minutes while it was loading.Meanwhile, Vince asked Iris what she did on her holidays.Iris went through each of the days, but she never asked Vince the reciprocal question.As soon as Iris finished talking, Vince said, "I will find the program and such", taking the initiative to get their activity started by using talk before using hands.However, Iris and Vince both interacted with the screen and did not constrain each other's access.

The role of hands
When we look at how Iris and Vince negotiated the reconfiguration of the material provided by Anne on the final slide, we can see how Iris presented a new task by telling Vince that they needed a rock to cover Jesus in order to complete their production (Davidsen & Christiansen, 2013;Davidsen & Vanderlinde, 2014).At this point, they had just finished rehearsing their production of the multimodal story.In response to Iris' concern about the missing rock, Vince replied with a gesture in frames 2-3, snapping with his left hand twice.Next, Vince turned his head to the left, looking straight at Iris, and said, "I can draw it (.)NO WE CAN DRAW IT of course", he repaired his statement himself in frames 5, 7 and 8.We can see how Vince nominated himself at first, but then changed his division of agency and coordination of contributions to the dyad.In so doing, he referred to their mutual agreement about how to work together, a tacit rule that had developed through their collaboration.At the end of his turn, Vince said "of course".This can be understood in two ways: as a correction of himself, or as a way of saying "no problem, we can easily draw this rock together".Subsequently, Vince was sitting with his hands on top of his head while Iris moved her left hand towards the screen and then moved the scrollbar up.Afterwards, Iris retracted her hand from the screen and turned it around, palm upward, as she said, "Yes of course, we can (0.3), but ehmm" (frames 10-12), affirming Vince's suggestion to draw a rock.Vince accepted the gestural invitation from Iris prior to her verbal turn, and he then drew a rock to cover Jesus, using the freehand drawing tool.
In this case, hands were not used to control access to the screen but served two constructive purposes: as a means of creating the scene on the screen and as a means of communication in the process of configuring the new task.In their collaborative activity, the pair used the movement space to engage in a shared construction of a conception of a problem through action and information.In other words, the touchscreen augmented both their intelligence-as-information and their intelligence-as-action.Hands shaped their activity in a positive manner, and they constructed a space for mutual engagement in the problem solving activity.

Peter and Nathalie -"the wrong internet"
Peter and Nathalie faced a technical problem after they turned the touchscreen on.Then, they found that the printout of the text was stapled incorrectly, causing Nathalie to go to Anne, their teacher.Nathalie returned and told Peter that Anne said they should tear the papers apart and staple them correctly.Meanwhile, the touchscreen displayed an error message, and both of them turned towards this issue.The desktop showed several Internet icons; Nathalie said, "It is the wrong Internet", which we interpret to mean that they were expecting a particular screen while navigating between different windows.Finally, they found the file containing their learning material; they did not read the list of tasks aloud.Nathalie acted as if she thought they should use the text-to-speech software on the list of tasks.However, the content on this slide was locked, and they could not mark up the text.Nathalie called for the teacher's help.

The role of hands
After the pair figured out, with help from the teacher, how to mark up the text and use text-to-speech software, they started listening to the text.In this situation, we can see how Nathalie and Peter worked together with their hands, without talking.In this part of their activity, we see signs of the role of the hand in their embodied and multimodal collaborative activity.Nathalie had just marked up the text when she moved her right hand to a position above the keyboard, 10 centimeters from the screen (frames 1-2, Figure 5).Without saying a word, Peter moved his right hand forward while stretching his index finger to push the play icon (frames 3-4).Then, Nathalie laid her hand down on the table, and Peter moved his hand to the headset.
After Nathalie and Peter had listened to the text-to-speech reading, Nathalie marked up the text again (frames 1-2, Figure 6), but this time she pushed the play icon (frames 3-4).This time, she did not retract her hand, and she imitated Peter's action, pushing "play" as an addition to her turn.Between these two excerpts, Nathalie and Peter had listened to the text-to-speech reading for 1 minute and 11 seconds.Here, we see the learning together by following each other's hand movements without saying anything -they are in fact co-operating (Goodwin, 2013) the screen through their bodies, inhabiting and mutually monitoring the situation.Nathalie displayed intelligence-as-action by imitating Peter's action, and it is clear that the touchscreen setting offered room for imitation and the development of intelligence-as-action.The hands shaped the children's activity, providing a space for imitation, co-operation and learning.In summary, these three examples demonstrate how the hand can shape children's collaborative activities around a touchscreen.Essentially, the hand can be used to constrain and control access, to construct and problem solve, and to show and imitate.In addition, it is also clear that the three pairs, working with the same learning material, engaged differently in the collaborative assignment.In other words, each pair made relevant different semiotic resources in their configuration of the context (Goodwin, 2000) through their collaborative activities around the touchscreens.

m i n d t h e h a n d i n c o l l a b o r at i v e ac t i v i t i e s
Building on this analysis of the role of the hands in embodied and multimodal collaboration around touchscreens, we suggest that free hands open a venue for collaboration, not only in the execution of actions on the screen but also in negotiating which problems to solve, and in the division of labour when doing so.The space between the pair and the screen opens room for hand movements that enrich the embodied and multimodal relationship between the children, whether they are fighting about control or constructing a shared conception of a problem.It is also worth noting that, in this context, intelligence-as-action functions on two levels: on the screen and in the space between the screen and the pair.The obvious question is whether these findings can be generalised to other situations of children's embodied and multimodal collaborative learning activities, with and without interaction around touchscreens.We see great potential for further research in understanding how children use their hands as part of the embodied and multimodal assemblages of collaboration, as it may well enrich teacher's palette of pedagogical instruments, encouraging greater tolerance of children's free movement in the classroom.We are grateful for the openness and collaborative spirit of the teachers, children, and management at Western State School, who made this research possible.Thanks also to Rasmus Raun Paulsen and Camilla Hvass, students at Aalborg University, for making the illustrations.Thanks as well to the anonymous reviewer for valuable suggestions.
r e s e a r c h b a c kg r o u n d

Figure 7 :
Figure 7: Nathalie and Peter ac k n ow l e d g m e n t s a n n e k j ä l l a n d e r & fa r z a n e h moinian: Digital tablets and applications in preschool.Preschoolers' creative transformation of didactic design jac o b dav i d s e n & e l l e n c h r i s t i a n s e n: Mind the hand: A study on children's embodied and multimodal collaborative learning around touchscreens birgit te h o l m s ø r e n s e n & ka r i n t w e d d e l l l ev i n s e n : Digital Production and Students as Learning Designers s t i g t o k e g i s s e l : Talking Books in Reading Instruction and Student Behavior