Adaptive Learning Technologies (ALT) and Learning Analytics (LA) are expected to contribute to the customisation and personalisation of pupil learning by continually calibrating and adjusting pupils’ learning activities towards their skill and competence levels. The overall aim of the study presented in this paper was to obtain a comprehensive understanding of how a systematic implementation of ALT influenced the learning outcomes, learning environment and motivation of 10- to 12-year-old pupils (grades 5–7 in Norwegian education) in mathematics, and the paper explores the following research question: How do systematic use of adaptive learning technology influence pupils’ learning and motivation? In this small-scale, Mixed Methods Research (MMR) study, a real-life introduction of adaptive technology was initiated and explored. Fifteen minutes of ALT homework each day or a total amount of 60 minutes a week, was applied to streamline individual volume training and root learning and thus free up time for practical mathematics and deep learning at school. The pupils’ level of competence, learning, motivation and basic psychological needs were measured quantitatively before and after the four-week intervention, and the intervention was observed qualitatively. The findings of the study indicate that use of ALT can help streamline volume training and root learning, and thus free up time for practical mathematics and deep learning at the upper primary level (ES = 0.39, P = 0.001). However, the study also indicates a interwoven relationship between learning, motivation and volume training that teachers should be aware of when using ALT. Particular attention should be paid when pupils learn new mathematical concepts.
In the last couple of years, various types of learning analytics (LA) and adaptive learning technologies (ALT) have been made available for educational purposes (Lang et al., 2017). Such technologies have the potential to personalise and increase the volume of student activity and to provide continuous feedback. In addition, the technology provides teachers with empirically generated data about student activity, level of competence and progress in learning. Its inherent potential is promising and could support and improve metacognition and self-regulated learning (Knight & Buckingham Shum, 2017; Pardo et al., 2017; Winne, 2017) which is in line with requirements in the new Norwegian reform “Fagfornyelsen” (to be introduced into Norwegian education in the fall of 2020). We know a little about how LA and ALT technology is contextualised and introduced in post-secondary education (Krumsvik & Røkenes, 2016). Predicative models have been used in small-scale studies by some pilot universities and have produced encouraging results (Campbell et al., 2007). The ALT explored in this paper, Multi Smart Øving (MSØ) (Gyldendal, n.d.), is already used extensively in Norwegian primary schools. MSØ is developed in collaboration with Knewton and is thus built on the Knewton platform. However, we know little about how MSØ and similar ALT-technologies are implemented in primary educational practice in real life in Scandinavia (Norway). Knowledge about how adaptive technology influences pupils’ learning and motivation in real life is thus important.
This paper reports on findings from a Design-Based Research (DBR) project called Learning and Teaching with Adaptive Learning Technology (LaT-ALT). LaT-ALT was a partly planned and partly emergent mixed methods study (Schoonenboom & Johnson, 2017) in which each phase of the study informed the next. The study had an interdisciplinary profile at the intersection of pedagogy and media science, and its overall aim was to iteratively initiate, evaluate, adjust and improve the use of ALT within a local school context. Findings from the LaT-ALT project could contribute to the body of educational research by exploring how systematic use of ALT in homework and at school influenced pupils’ learning outcome, learning process, motivation and learning environment. During the four week intervention of the study, where systematic use of ALT was introduced in real-life practice, qualitative data was produced (during the intervention) and quantitative data was collected (pre-/post-intervention). These data form the empirical basis of this paper. The study was positioned within the logic of Mixed Methods Research (MMR) (Johnson & Christensen, 2017; Johnson et al., 2007) and Dialectical Pluralism (DP) (Johnson, 2017).
The following MMR question is investigated in this paper:
The MMR framework of this study builds on an ecological approach to teaching and learning, in which the introduction of the systematic use of ALT is considered to be a new element that could intentionally improve teaching and learning, but also could affect established learning environments and contexts in unforeseen ways. The study thus aim to address complexity, and the power of mixed methods research is its ability to deal with diversity and divergence (Schoonenboom & Johnson, 2017). In line with previous research on the educational use of digital tools and technology enhanced learning, this study assumes that the successful implementation of learning technology is a mutual interaction that includes, but is not necessarily limited to, 1) the inherent advantages and disadvantages of the specific technology being used, 2) the teacher’s ability and willingness to facilitate learning, and 3) the pupils’ motivation for learning. The actual operationalisation of these three aspects in the study are made concrete through the following concepts: Classroom Management, Self-Determination Theory and ALT (see Figure 1).
Classroom management (CM) is defined as “the actions teachers take to create an environment that supports and facilitates both academic and social-emotional learning” (Evertson & Weinstein, 2006, p. 4). CM is not an end in and of itself, but a means to create and maintain any given optimal learning environment (Brophy, 2006; Doyle, 2006). Behavioural approaches have been closely associated with CM and can be used clumsily (by forming a controlling and frustrating classroom environment) or skilfully (by supporting autonomy; Landrum & Kauffman, 2006). Awareness of the limitations regarding behavioural approaches has contributed to a paradigm shift in favour of approaches that emphasise self-regulation and trusting, caring relationships between teachers and pupils. Nevertheless, standardised and internalised understandings in CM are often adapted to a teacher-centred practice, associated with behaviour management and sets of rules (Emmer & Sabornie, 2015; Evertson & Weinstein, 2006). The effective use of technology in teaching and learning practices is, however, aligned with the general paradigm shift towards pupil-centred methods. Such approaches entail a paradigm shift from the teacher’s authority and control to shared control and responsibility (Schwab & Elias, 2015; Watson & Brattistich, 2006).
The shift from a (passive) teacher-centred classroom environment to an (active) pupil-centred classroom environment has instructional and managerial implications; it has become increasingly important to identify the intended pupil learning outcomes first, and to design learning activities and reflectively acknowledge what specific activities imply about desired pupil roles thereafter (Brophy, 2006). The learning process benefits from stating clear expectations and helping pupils understand what to do and why; in other words, to support autonomy. This paradigm shift makes some teachers fear a loss of control in their classrooms (Bolick & Barthels, 2015; Brophy, 2006; Hickey & Schafer, 2006). Some studies indicate that assumptions teachers hold about pupils shape their CM judgements and practices, and that teachers tend to focus on rule-setting, enforcement and “crime control” (Bullough & Richardson, 2015). CM provides the LaT-ALT-study with a framework for identifying and addressing the facilitation of learning in different contexts and learning ecologies. The operationalisation of CM in LaT-ALT is implemented in line with the overall mindset of Evertson and Weinstein (2006) and Emmer and Sabornie (2015). As illustrated in Figure 2, the core of CM is considered to be the facilitation of learning, maintaining or enhancing positive motivation and creating and maintaining an optimal learning environment. The overall aim of the LaT-ALT-project was to explore what happens when the logic of LA and ALT is introduced in the existing real-life (RL)-context.
The mapping of the pupils’ learning environment and motivation (Figure 2) uses self-determination theory (SDT) as a theoretical lens in the LaT-ALT study. SDT’s basic psychological needs (autonomy, relatedness and the feeling of competence) are generally seen as essential indicators of a productive learning environment (Evertson & Weinstein, 2006). In addition, motivation is considered to be the moving force of any action or behaviour (Ryan & Deci, 2000b; Ryan & Deci, 2000a). SDT suggests that a pupil can be motivated to a greater or lesser degree and be driven by various forms of motivation (Ryan & Deci, 2000a). SDT distinguishes between different types of motivation based on the reasons or goals that give rise to the action (Deci & Ryan, 2004; Deci & Ryan, 2016; Ryan & Deci, 2000a; Ryan & Deci, 2000b). The distinction between intrinsic and extrinsic motivations has for decades influenced motivational research both inside and outside of the educational field.
Intrinsic motivation is defined as the doing of an activity for its inherent satisfaction rather than for some other and separable consequence and refers to performing a task or an activity because one finds it enjoyable, interesting or fun in and of itself. Intrinsic motivation is linked to high-quality learning and creativity, and is considered a natural wellspring of achievement and learning that can be either catalysed or undermined by parent and teacher practices (Ryan & Deci, 2000a; Ryan et al., 1994).
Extrinsic motivation is defined as the doing of an activity for a reward or another separable consequence and seems to be a more complex and ambiguous term. Operant theory maintains that all behaviours are motivated by some sort of reward, and thus contradicts the very existence of intrinsic motivation (Ryan & Deci, 2000b). However, the SDT’s model in “A taxonomy of human motivation” (Ryan & Deci, 2000a) displays different types of extrinsic motivation on a continuum between the contrasting concepts of intrinsic motivation and amotivation, indicating that some extrinsic motivations are related to intrinsic motivation whilst others are related to amotivation. Amotivation is described as the lack of intentionality and sense of personal involvement, and results from not valuing an activity, not feeling competent to do it, or not believing it will yield a desired outcome (Ryan & Deci, 2000a; Deci & Ryan, 2004). The taxonomy additionally distinguishes between regulatory styles based on their associated processes and perceived locus of causality (IPLOC). SDT and the taxonomy of motivations thus provides the study with a framework for identifying and describing factors that undermine or enhance internalised (positive) forms of motivation.
“There is a pressing need to review the extent to which conventional theories are applicable to ICT-infused learning contexts” (Liu et al., 2016, p. 6). However, an underlying challenge is that technologies have spread so fast that formal research has trouble keeping up with real-life practices (Koh, 2016). In recent years, a new type of technological educational tool has been developed and thus gained attention. We know this technology as Learning Analytics (LA) and Adaptive Learning Technology (ALT). We find relatively few studies that combine ALT with homework as part of the intervention and that follow teachers’ everyday practices. However, Roschelle, Feng, Murphy, and Mason (2016) studied 2850 mathematics pupils who used adaptive learning software and homework as central parts of the intervention. They observed an increase in the pupils’ scores on an end-of-the-year standardised mathematics assessment as compared to a control group that continued with existing homework practices. Pupils with low prior mathematics achievement benefited most from this intervention. Compared to other areas of educational technology, there is still relatively little research on adaptive learning in elementary schools, both internationally and in Norway, and we need more research on how this can be attached to deeply entrenched structures in teachers’ everyday practices and to pupils’ homework. LA focuses on adaptive learning by tracing and analysing pupils’ learning activities to understand and optimise learning outcome in different learning environments and contexts (Blakelock & Smith, 2006; Lang et al., 2017). But what are LA and ALT?
Learning analytics is a term that refers to the use of digital data for analysis and feedback that generates actionable insights to improve learning. LA feedback can be used in two ways: 1) to improve the personal learning power of individuals and teams in self-regulating the flow of information and data in the process of value creation; and 2) to respond more accurately to the learning needs of others (Crick, 2017, p. 291). The use of data and models can predict pupil progress and performance, and thus provide pupils and/or learning facilitators with the ability to act on that information (Winne, 2017). Different ALT-technologies are of various qualities, have different affordances, and will thus have different impacts on pupils and teachers – and the interaction between them. In the LaT-ALT study ALT and LA are combined in the Multi Smart Øving (MSØ) software, as the software uses ALT for personalising/automating student activity tasks and has an inherent LA-access for teachers (Gyldendal, n.d.). MSØ is a practice program for root learning in basic mathematics, hence it aims to increase the volume of drills. It is not intended for practical mathematics and processes associated with deep learning. ALT could thus partly automate or support the teacher’s tasks of mapping the pupils’ activities, skill development and competencies and providing them with tasks and activities individually tailored to their needs. However, introducing learning analytics and adaptive learning into education has ethical and pedagogical implications (Bergner, 2017; Hoppe, 2017; Prinsloo & Slade, 2017). Winne (2017) points to the balance between accuracy and generalisation when describing a learner’s ipsative development as a challenge, noticing that two learning signatures will never match completely: “The field of learning analytics will benefit from frequent consideration of this challenge” (Winne, 2017, p. 248). In this article we aim to address the real-life pedagogical challenges and implications of introducing ALT systematically in primary education. Figure 3 roughly illustrates the process workflow of the ALT that was used during the intervention (MSØ).
The dark grey circles illustrate the active engagement of pupils and teachers in the program interface. The light grey boxes illustrate steps (processes) that are visible to pupils and/or teachers. The dark grey box represents the processes that are not visible to either pupils or teachers, but acts as the link between the two interfaces. The light grey box connected to the pupil is only viewable in the pupil interface during the activity process, and the light grey box connected to the teacher is only viewable in the teacher interface. The technology can help to facilitate learning on at least two levels, in line with Crick (2017). The first level is called the activity and program feedback loop. This is the automated process where the program selects tasks and activities from a database, tentatively adapted to the pupil’s competence level at any given time. The pupils are provided with immediate (summative) feedback on whether the answer is right or wrong and are given stars and/or diamonds by MSØ when they have reached certain levels within the program. Both pupils and their teachers are provided an overview of the amount of time the pupils have spent in the program, right and wrong answers and how many tasks they have given up on. The second level may be called the teacher feedback loop. This is a non-automated process (hence the dotted arrow) by which the teacher can actively use the empirical data from the dashboard to support their facilitation of learning as a supplement to the program feedback loop or outside the program. Teacher feedback could thus be summative, formative or both.
The data on which this article builds were produced and collected through a MMR study, conducted in the last half of 2017. The LaT-ALT project as a whole (Figure 4) was based on a principle associated with Design-Based Research (DBR): That practitioners and researchers work together to produce meaningful change in the context of real-life (Brown, 1992; The Design-Based Research Collective, 2003). The overall project design aimed to bridge the gap between theory and practice, by implementing an intervention in a real-life context, as opposed to a controlled laboratory context (Brown, 1992). Design-based research focus on advancing theory grounded in naturalistic contexts (Barab & Squire, 2004), and the design of the intervention is considered a key feature of the quality and result of the research project (Anderson & Shattuck, 2012). Putting a first version of the intervention design into the world (in this case a Norwegian upper primary context) to see how it works is the first step of a progressive refinement (Collins et al., 2004). The MMR-design of the study may be described as a partly planned and partly emergent design (Schoonenboom & Johnson, 2017), where some parts of the design was planned in advance whilst others emerged, informed by the teacher-researcher collaboration and preliminary (quan and qual) findings. The quan-qual integration was therfore both convergent and interactive (Fetters et al., 2013). Systematic observation of the intervention should enable the researchers to explore what consequences systematic use of ALT could have for 1) pupils’ learning, competence and motivation (pupil perspective), 2) teachers practices and professional role (teacher perspective), and thus be able to provide a DP approach to the 3) interaction between pupil learning and motivation and teacher practices when ALT is being used (Johnson, 2017; Schoonenboom & Johnson, 2017; Johnson & Christensen, 2017).
In line with the research questions that drives this paper we presents findings from phase 2 and 3, including pre-test data collected at the end of phase 1 and post-test data collected at the end of phase 3. The further description of the phases (as presented in Figure 4) thus emphasises the methodologies, data collection and data analyses related to learning and motivation during the real-life intervention.
The study was designed over a period of four months. The school leaders and teachers involved had begun preparing for the new content curricula reform “Fagfornyelsen”, which e.g. emphasize deep learning within and across subjects. Since time is generally a limited resource in school, they wanted to find good solutions to ensure time for deep learning processes while also providing pupils with basic mathematics knowledge. The study was thus designed to initiate, evaluate and adjust the first step of a desired change in the case school’s existing practice: To free up time for practical mathematics and deep learning in mathematics through effectively streamlining and personalising basic mathematical understanding using ALT. The case school’s teachers were (in line with the guidelines of the new reform) committed to safeguarding the pupils’ curiosity, creativity and need to explore; teachers were thus explicitly unwilling to compromise the pupils’ motivation for learning.
Long term commitment to interventions have a series of practical and ethical implications for pupils and teachers involved, and proposals for the design and intervention criteria were thus iteratively drafted by the researchers and adjusted in collaboration with the participating teachers (N = 3). The intervention period was set to 4 weeks and the intervention criteria were few, but real-life oriented: All pupils (N = 43) should have their own tablet with access to the MSØ software. Pupils should do volume training tasks in the software a minimum of 15 minutes per day or 60minutes per week as homework. The teachers were otherwise free to implement the use of the program in their own practice. However, the vendor advices against using the program more than 60 minutes pr. week, and also against helping the pupils because it will affect the adaptiveness of the program providing pupils with difficult tasks beyond their level of competence (Gyldendal, n.d.).
The pupils and teachers involved had used the program (unsystematically) earlier, and thus had a basic knowledge of its use. To establish a baseline start-up, all use of the program was stopped in the last 3 weeks prior to the intervention. At the beginning of the intervention during the transition between phase 1 and phase 2, a pre-test was conducted to collect data about the pupils’ motivation, basic psychological needs, perceived learning and perceived competence (the survey) and the pupils’ baseline knowledge about the theme (fraction and percentage) for the intervention period (the mathematics test). The survey questions were derived from validated items in self-determination (27 items), perceived learning (4 items) and competence (4 items) and adapted to the pupils’ context and age. Thus, experienced learning had only 4 items, but one of the questions was a reversed control question for comparison. Language and meaning content in the survey was developed over a period of several weeks. The participating teachers and experienced professionals in quantitative methodology was consulted during this period. The survey was also piloted during the development phase, and the researchers were actively conscious of monitoring for misconceptions during the pre-test.
The mathematics test consisted of 11 tasks and activities and was made by the teachers involved according to the following criteria: 1–3: Easy tasks far below national curriculum, 4–6: Towards national curriculum, 7–9: In line with national curriculum, 10–11: Tasks beyond national curriculum.
During the first two weeks, the intervention was observed qualitatively (Fangen, 2004; Merriam & Tisdell, 2016; Tjora, 2017) through participating in fieldwork (5 working days) and classroom observations (2 × 45–60 minutes in each class). The data from the survey and the mathematics test were preliminary analysed beforehand and formed an understanding of the pupils’ starting point. The aim was both to understand how the teachers implemented the intervention in their practice and how the pupils responded. The findings of this work also contributed to a revision of the interview guides. During the last two weeks, the intervention was observed through participating in fieldwork (4 working days) and through semi-structured interviews (Kvale & Brinkmann, 2009) and focus group interviews (Merriam & Tisdell, 2016). The aim of the interviews was to challenge and validate the preliminary understanding and to gain deeper insights into the complexity of implementing ALT.
The interview guides were based on both theoretical categories from phase 1 and preliminary findings and analyses from phase 2. Contextual and descriptive questions were deliberately asked in the beginning of the interviews, while more evaluative and validating/contrasting questions were asked later on. Active listening and second questions were also emphasised during the interviews (Kvale & Brinkmann, 2009).
DP is considered a process philosophy for dialoguing with difference, which entail an underlying assumption that much of reality is plural and dynamic rather than singular and static (Johnson, 2017; Johnson & Stefurak, 2013). The study aimed to go beyond narrow measures (Collins et al., 2004, p. 18), and the axiology of the study therefore imply pupils life-world experiences and perceptions as valuable perspectives. Since we know little about how upper primary pupils experience working with ALT (within a LA-system), the emic viewpoint of the pupils was considered to be an important epistemological contribution (Johnson & Christensen, 2017, p. 306). Pupil interviews would voice the pupils and bring their reasoning into the understanding how ALT affects their learning and motivation, from their point of view. The pupil interviews were carried out as focus group interviews. Three pupils from each of the participating classes (N1:3, N2:3 and N3:3) were asked questions about learning, their learning environment, the use of technology in general and the use of adaptive technology specifically. Group interviews were preferred over individual interviews in an effort to understand the lifeworld of the pupils better, to balance out the power asymmetry associated with the research interview, and to avoid a therapeutic turn of the interviews (in line with the informed consent) (Kvale & Brinkmann, 2009). The pupil interviews lasted a total of 132 minutes.
The analysis work in phase 4 was done step-by-step (Figure 5). The quantitative data were statistically analysed to comparatively explore if the pupils’ learning outcomes, perceived learning, competence and motivation had changed during the intervention. The qualitative analyses contributed complementary information (voiced by the pupils themselves) regarding how they experienced the use of ALT during the intervention. Integrating the two perspectives served two purposes: 1) to offer the practitioners at the case school multiple perspectives on how their pupils experienced the ALT and thus to enable them to improve their facilitation of learning, and 2) to contribute to further research on ALT by suggesting some theoretical implications for further research (e.g. Collins et al., 2004).
The study was designed in collaboration with the teachers and school leaders involved, and was approved by the Norwegian Centre for Research Data (NSD). Informed consent was obtained by parents and pupils prior to the intervention. In line with the assessment of the NSD, informed and active consent from the participants was repeatedly emphasised throughout the project (Johnson & Christensen, 2017, p. 135–136). However, real-life interventions and connected data collection have ethical implications beyond formal approvement and informed consent, as school is not an optional activity in itself (Merriam & Tisdell, 2016). Since some of the pupils should not attend (due to special needs or not consenting), it was important to additionally collaborate with the teachers to find good solutions that did not create a visible and stigmatizing distinction between the pupils who would participate and the pupils who would not participate. To avoid collecting personal data from pupils who were not to participate (without refusing those pupils to do their homework) researchers could e.g. not obtain direct prints of student activity in MSØ. Instead, the teachers and the researchers had updating conversations (weekly) about the pupils’ scope of activity and their academic progress based on the teacher dashboard view. It was also decided to hand out the test and survey on paper to ensure the anonymity of the participating pupils. The design of the encrypted data collection tools (which the researchers had access to) was not considered to be sufficiently adaptable to the pupils’ age and needs. Although the school and teachers participated in the design of the study, they should not have access to the pupils’ responses and personal information in line with the informed consent. Therefore, it was considered important to take precautions possible to avoid information leak (Johnson & Christensen, 2017).
We performed a statistical analysis with SPSS 19 (Statistical Package for Social Science, Chicago, USA). All table values are expressed as a mean ± standard deviation (SD). Changes within groups from pre- to post-test, were determined by the paired sample T-test. A two-tailed p < 0.05 was accepted as statistically significant for all tests. To investigate the magnitude of the effect in the case within each group and between groups, the effect size (ES) was calculated in the form of Cohen’s d (Cumming, 2012) for primary outcome variables. We used the web-based Practical Meta-analysis Effect Size Calculator1 (Lipsey & Wilson, 2001) to calculate Cohen’s d. An ES of 0.2 is regarded as small, 0.5 as medium and 0.8 as large (Cumming, 2012).
The registered learning improved moderately from pre-intervention to post-intervention (ES 0.39, P = 0.001; Table 1). However, there seems to be a decrease in perceived learning. The perceived learning (reversed control question) shows a moderate to large decrease (ES 0.86, P = 0.004; Table 2). Intrinsic motivation, identified and external regulation do not change, but there is a moderate increase in amotivation from pre- to post-intervention (ES0.4, P = 0.039; Table 3). No changes are revealed in basic psychological needs (Table 4).
|Total (N = 40)||6.13 (2.69)||7.18 (2.72)||1.05 (1.83)||0.466, 1.63||0.39||0.001**|
|Perceived competence||3.90 (0.63)||3.91 (0.61)||0.008 (0.56)||–0.19, 0.17||0.01||0.926|
|Perceived learning||3.62 (0.84)||3.34 (0.83)||–0.28 (1.08)||–0.7, 0.62||0.33||0.116|
|Perceived learning (reversed question)||2.15 (1.14)||2.93 (1.12)||0.78 (1.58)||–1.28, –0.27||0.68||0.004**|
|Intrinsic motivation||3.62 (0.88)||3.48 (0.89)||–0.14 (0.63)||–0.06, 0.33||0.15||0.173|
|Identified regulation||4.25 (0.58)||4.15 (0.61)||–0.10 (0.63)||–0.10, 0.30||0.16||0.298|
|External regulation||2.84 (0.61)||2.89 (0.59)||0.04 (0.46)||–0.19, 0.10||0.07||0.556|
|Amotivation||1.68 (0.59)||1.93 (0.65)||0.24 (0.72)||–0.47, –0.01||0.4||0.039*|
|Autonomy||3.69 (0.62)||3.75 (0.61)||0.06 (0.51)||–0.22, 0.10||0.09||0.450|
|Relatedness||4.46 (0.59)||4.43 (0.57)||–0.02 (0.52)||–0.14, 0.19||0.03||0.766|
|Competence||3.73 (0.52)||3.85 (0.61)||0.12 (0.57)||–0.30, 0.06||0.28||0.186|
The focus group interviews were transcribed in the original language (Norwegian). In order to preserve the pupils’ diverse and complex experiences and opinions, the category-based analysis had a step-by-step deductive-inductive approach (Schoonenboom & Johnson, 2017; Tjora, 2017). The transcribed interviews were first analysed by categories related to learning, competence and motivation to look for clear patterns. Second, they were compressed through a multitude of lifeworld-based subcategories derived from the pupils’ own perspectives. These subcategories were subsequently sorted into the following overall categories: 1)contextual factors, 2) the use of technology in general, 3)ALT and 4) learning, competence and motivation.
To provide an emic (inside out) insight into the pupils reasoning and voice about how learning with ALT relates to their everyday practice in school, we will first show some selected passages from the focus group interviews. Descriptive quotations were translated into English and slightly adjusted linguistically and grammatically to keep their original content meaning. The names of the pupils are fictional.
All the participating pupils (N = 9) described the ALT program in similar ways: as a task and activity-generator in which a right answer was supposed to provide them with harder tasks and a wrong answer was supposed to provide them with easier tasks.
The seventh-grade pupils thus described an individual volume training classroom practice, and framed the ALT homework as more or less an extension of this practice.
The fieldwork and classroom observation identified the sixth-grade classroom as varying between group and individual practice. The pupils also described the practice as a combined one, which varied between collaborative practical mathematics and individual volume training in the ALT program.
What do you typically do during math classes?
Even if the two classroom practices were different, a common trait shared by the sixth and seven graders was that they did not particularly distinguish between homework and schoolwork. Their answers throughout the interviews indicated an intertwined practice where some parts of the work were done at school whilst other parts were done at home. They did not fully enjoy all aspects of doing neither their homework or their schoolwork, but they seemed to accept and value both as part of their learning process. This point becomes particularly visible when compared to the fifth-grade pupils’ answers to the same question about their classroom practice.
Could you describe a typical mathematics class? What do you do?
This response is representative of how the fifth-grade pupils framed the use of the ALT program throughout the interview. Despite repeated attempts to keep the first part of the interview on a descriptive level, the pupils responded to most questions by attaching some sort of criticism about the ALT program. They were especially and explicitly critical towards using ALT as homework. The fieldwork and classroom observation had identified the fifth-grade classroom as mostly group-oriented, where collaborative problem-solving was a key ingredient. However, the fifth-grade pupils also did some ALT tasks at school when they had time to spare. The previous classroom observation carried no obvious indications of strong negative emotions such as frustration, and the pervasive critical rhetoric of the fifth-grade pupils’ interview was thus somewhat surprising.
It was an overall aim of the interviews to understand whether and how the ALT mediated volume training at home, and how this volume training corresponded to the general classroom practice during the intervention. Although the participating teachers shared common visions of active, motivated pupils who engaged in deep learning, these visions were operationalised through different practices in each of the classrooms. Informed by previous fieldwork and classroom observations, it was thus an aim for the researchers to understand how the logic of ALT corresponded to the dominant learning practices in each classroom and each learning environment (as previously illustrated in Figure 2). The main findings from analysing the student interviews in relation to the integration of MSØ homework and classroom practices during the intervention can be visually presented in the following way in Figure 6.
The figure aims to represent whether or not the pupils perceived an integration of ALT homework in their classroom practice. There seem to be contrasting views of the sixth- and seventh-grade pupils on the one hand and the fifth-grade pupils on the other. The fifth grade pupils were especially and explicitly critical towards using ALT as homework, beyond the (critical) viewpoints of the sixth and seven grade pupils. Even in short text passages, there were many overlapping codes in the fifth-grade interview. When they talked about classroom practice, they made a point of criticizing MSØ for not offering help and support as the teacher does in the classroom. And when they talked about the homework, they pointed out what they missed and wish the program could offer. This partially divergent perception between pupils thus inspired a new question: Why did the sixth- and seventh-grade pupils seem to accept and value volume training in MSØ more than the fifth graders did?
From a SDT perspective, the frustration of the fifth grade pupils might be associated with a lack of internally regulated motivation for learning (Ryan & Deci, 2000b). The motivational category in the analysis of the interviews may thus provide deeper insight regarding the contrasting views of the sixth- and seventh-grade pupils on the one hand and the fifth-grade pupils on the other. The motivational categories for analysing the qualitative data were derived from SDT’s taxonomy of human motivation (Deci & Ryan, 2014; Ryan & Deci, 2000a) as illustrated in Figure 7.
ALT factors that were interpreted as undermining or enhancing pupils’ internally regulated motivation are summarised in Table 5. To provide an emic insight into the pupils reasoning and voice about how learning with ALT influence their learning, motivation and feeling of competence, we will additionally show some selected passages from the focus group interviews below Table 5. The selected quotations exemplify how the pupils talked about mathematics, learning and ALT, and how their experiences were interpreted within the framework of SDT (Figure 7).
|PERCEIVED LEVEL OF COMPETENCE||REASONS WHY THEY LIKED ADAPTIVE LEARNING
Increasing motivational factors
|REASONS WHY THEY DID NOT LIKE ADAPTIVE LEARNING
Decreasing motivational factors
|Pupils who described themselves as “not very good” in mathematics||Explicitly did not like it.||Expressed greater resistance towards ALT than other tools and methods. Did not feel that the technology contributed what they wanted or needed (and expected). Expressed an unmet need for human support and help. Disliked the measurement, comparison and control aspect, especially regarding the time control and number of tries available.
AMOTIVATION and EXTERNAL REGULATION
|Pupils who described themselves as “ok” in mathematics||Experienced that the technology generally met their needs and provided variation and exciting activities. Provided clear and structured framework for activity.
|Disliked the measurement, comparison and control aspect, especially regarding time. Did not trust/understand the integrated clock. Expressed lack of human presence and judgement.
|Pupils who described themselves as “good/very good” in mathematics||Experienced that the technology generally met their needs and provided variation and exciting activities. Enjoyed math activities regardless of method and learning recourses.
INTERNAL REGULATION/INTRINSIC MOTIVATION
|Disliked the measurement, comparison and control aspect, especially regarding time control. Experienced the technology as limiting in a number of ways. Expressed lack of human presence and judgement. (NB: One pupil explicitly did not like ALT at all.)
When asked explicitly whether they liked working in the program the sixth- and seventh-graders were generally relatively positive. However, they also raised some objections:
Some of the pupils noted that the program did not reward slow work either. For example, David, a fifth-grade pupil, figured out how to solve a difficult task, but the time ran out before he was allowed a second try.
Issues regarding the time control were both initiated and elaborated on by all the interviewed pupils. Two perspectives were mainly identified as problematic: 1) counting the minutes instead of the tasks did not reward either effective or thorough work, and 2) they did not entirely trust the time control in itself.
Another issue brought up by the pupils was how long it would take for the program algorithms to understand the pupils’ level of competence, and thus provide them with the right activity.
Do you like to work in the program?
As previously mentioned, the fifth graders stated from the beginning of the interview that they did not like the program and the adaptive technology, especially as homework. However, they expressed their resistance differently. Kari emphasised that she was not good in math and that she didn’t like doing math tasks in general. Her descriptions bear some indications of the program reinforcing her feeling of amotivation and incompetence.
David, on the other hand, described himself as both liking math and being quite good at it, but he critiqued how the program continually displayed data about their working process.
The math pre-test further informed the observation that pupils demonstrated varied levels of skills and competence. It was thus important to gain insight into how pupils experienced working on math activities, both in general and in the program.
Every mixed methods research study has at least one point of integration (Schoonenboom & Johnson, 2017), and we have so far dealt with the emergent integration of methods (e.g. how one phase informed the next, and thus contributed to probes across methods Merriam & Tisdell, 2016; Creswell, 2019). Our quantitative findings show that learning in mathematics improved from pre- to post-intervention (ES = 0.39, P = 0.001; Table 1). This indicates that the pupils increased their competence in mathematics during the intervention. However, we have also shown qualitatively that pupils describe the integration of the ALT program in different ways (Figure 7 and Table 5) and that there was a quantitatively registered decrease in perceived learning (Table 2) and an increase in amotivation (Table 3). In this section we will further integrate results from the study and discuss them in relation to the research question: How do systematic use of adaptive learning technology influence pupils’ learning and motivation? In this part of the article we will further integrate quantitative and qualitative findings and discuss them in the light of the theoretical framework. The pupils participating in the LaT-ALT study were generally driven by internally regulated motivations whilst doing math activities, both before and after the intervention (Table 3). The case school’s learning environment was described by the participating teachers as supportive of autonomy (Landrum & Kauffman, 2006) and in line with the paradigm shift from teacher authority and control to shared control and responsibility (Schwab & Elias, 2015; Watson & Brattistich, 2006). At the end of the intervention, however, a moderate increase in amotivation was registered (Table 3). Amotivation was generally low both before and after the intervention, so this finding alone should not be too strongly emphasised. The qualitative analysis, however, shows a relatively large gap in the pupils’ motivation for learning when using ALT. Most pupils describe adaptive learning as a fun and varied way of learning mathematics, as long as they didn’t spend too much time in the program interface. The intervention criteria of 60 minutes a week corresponded with advice from the vendor, and seems to be a time frame that should not be exceeded. Even though all the pupils pointed to elements of the program they did not appreciate, most of them were still generally positive towards using it both at home and in school. For most pupils, the program seems to contribute to volume training in line with internally regulated (positive) motivation. They especially emphasised the varied activities as something they enjoyed, and said that the volume training was easier when they did not have to write full answers or draft their calculations.
We still find a contrasting pattern among the fifth-grade pupils. Their critical attitudes to the adaptive learning tool were pervasive, but the argumentation was not unambiguous. The pupils point out that the modelling of solution alternatives is poor and that technical solutions are weak. This does not necessarily mean that they are right in their evaluations, but it is interpreted as indicating that pupils have high expectations of what the program can contribute to their learning processes. The fifth-grade pupils also said that the program’s inherent summative feedback (stars and diamonds etc.) was being used by pupil peers in the classroom environment to compare themselves to others. Such easily comparable measures may, according to the taxonomy of human motivation (Ryan & Deci, 2000b) contribute to a more externally regulated (negative) motivation (or, ultimately, amotivation). The fifth-grade pupils generally indicated that the program controlled their homework and learning processes in a way they did not appreciate. This could be seen in light of ethical and pedagogical implications of ALT and LA (Prinsloo & Slade, 2017; Bergner, 2017; Hoppe, 2017). The technology can give a precise overview of the pupils activities and thus be used to facilitate learning, but the overview can also be perceived as controlling by the pupils. The fifth graders described the logic of the program as a controlling and frustrating one (Landrum & Kauffman, 2006). A key question in this context is why the fifth grade pupils expressed a more externally oriented and thus negative motivation when using ALT than the sixth and seventh grade pupils did? One piece of this complex puzzle may be found in the learning outcome data. Data transformation (Johnson & Christensen, 2017) allows us to view the learning outcomes from another point of view, and through transforming the (pre/post) mathematics results it can be shown that the average progress is more complex than Table 1 implies at first glance (as illustrated in Figure 8).
When the pupils’ level of competence before and after the intervention is color-coded (in line with the taxonomy of the mathematical test), we see that a large proportion of pupils (most prominently in seventh grade) demonstrated competence at or beyond the national curriculum level after the intervention. These pupils increased the average level of competence registered. The figure additionally shows that some pupils already had demonstrated a high level of competence before the intervention, and that even more pupils demonstrated a high level of competence after the intervention. However, in fifth grade there were more pupils who demonstrated a competence far below the curriculum level after the intervention than before. These pupils thus demonstrated lower competence after the intervention than before. According to SDT, lower registered competence might interact with other SDT-variables connected to learning (Deci & Ryan, 2004). As a single result, this decrease in learning outcomes can be explained in various ways. The same can be said about the criticism the pupils communicated through the qualitative interview. But together (and combined) the two results constitute an indicator that the intervention cannot be said to have been entirely successful in the fifth-grade learning environment. Systematic introduction of ALT seems to have had some sort of negative influence on the fifth-grade pupils.
Another piece of the puzzle might be found if we compare the class-level results (Figure 8). If the results are read horizontally (from fifth grade to seventh grade), we can see the contours of progress throughout the class levels (even if they are not strictly coherent). The circles become increasingly (dark) green. The national curriculum in Norway (KL06) is not specifically attached to a certain grade level, but identify competence aims after the second grade, fourth grade, seventh grade etc. The fifth graders in the LaT-ALT study can therefore be said to be beginners in a new mathematical learning cycle that the sixth-grade pupils are in the middle of, and the seventh-grade pupils are about to end. From this perspective, it makes sense that the fifth-grade pupils are calling for clearer modelling and explanations, and are criticising the program for giving them too little time to think. The teachers collaborated to adapt both their teaching and the math test to a level of competence appropriate to the pupils’ class level and age (known as the Knowledge Promotion Reform initiative), however, the individual pupil must still acquire a basic understanding of new words and key mathematical concepts. The acquisition of knowledge and understanding is an individual process that requires time and concentration and should maybe not be rushed through standardized time use. This is a central part of the learning process and the entrance to a learning cycle will thus be decisive for the pupil’s perceived competence and motivation for further learning (Liu et al., 2016). A pupil who strives to understand basic thematic concepts will have difficulty seeing how the same concepts can have practical utility in activities. According to the taxonomy of human motivation (Ryan & Deci, 2000a) IPLOC will influence pupils’ motivation for learning. Pupils who feel competent and able to do their assigned activities will most likely experience emotions associated with internally regulated motivation. On the other hand, pupils who feel less competent and unable to do their assigned activities will most likely experience emotions associated with externally regulated motivation or amotivation. This point does not only apply to ALT. However, the participating pupils were aware of the inherent potential of the technology, and thus expected more from ALT than from more traditional tools such as books.
According to Mathison (1988), the importance of triangulation is not limited to validating findings. She also emphasises the value of elaborating convergence, contradiction and inconsistency in mixed methods results and findings. An important methodological implication of the LaT-ALT study is that the quantitatively observed progress in learning outcomes mainly converges with, but also partly contradicts, the lifeworld experiences of the participating pupils. This inconsistency is visible in the transformed data (Figure 8), in the qualitative material (Figure 6 and Table 5) and in the statistical analysis (Tables 2 and 3). One can thus say that the use of ALT could potentially help streamline volume training and root learning, and thus free up time for practical mathematics and deep learning in line with Fagfornyelsen. However, the use of ALT also seems to have some associated challenges, especially related to pupils who struggle to grasp new mathematical concepts. This partially contradicts previous findings (Roschelle et al., 2016). Winne (2017, p. 248) notes that it is statistically very unlikely any two learners’ data signatures perfectly match when emphasising how ALT and LA must balance between accuracy and generalisation. This challenge applies particularly to primary school pupils who have to learn and understand basic principles while at the same time do volume training. The very intention of ALT-mediated volume training is to create personalised challenges in the pupils’ flow zone, between boredom and anxiety (Gallego-Durán et al., 2018). Feelings of competence and a sense of autonomy are important factors that generate variability in feelings of flow and intrinsic motivation. Individuals must experience both perceived competence (self-efficacy) and the activity to be autonomous and self-determined and for intrinsic motivation to be maintained or enhanced (Ryan & Deci, 2000b). If the volume training is not sufficiently personalised, or if the pupils are not sufficiently prepared to enter the ALT activity loop, it might lead to a volume of frustration (as opposed to a volume of training/learning), decreasing the pupils inherent motivation for learning. The fifth-grade pupils’ interview bore indications of the pupils wanting to progress in their learning process, but experiencing that MSØ did not help them understand as they had anticipated:
The LaT-ALT study shows that the use of ALT can help streamline volume training and root learning, and thus free up time for practical mathematics and deep learning. ALT can contribute to student learning outcomes at an average level (ES = 0,39, P = 0,001), across different classroom practices. ALT could also positively motivate pupils by offering varied and customised learning activities and tasks. However, the relationship between learning, motivation and volume training when ALT is systematically introduced seems to be intertwined. This relationship becomes especially apparent in the group of pupils who were meeting new mathematical concepts for the first time during the intervention. Pupils who had previous knowledge of the mathematical concepts expressed an aligned connection between school activities and volume training at home, while the pupils who were at the beginning of a new learning cycle expressed a greater degree of colliding logics between the volume training at home and the learning activities at school. We suggest that teachers should be aware of the difference between externally regulated motivation and internally regulated motivation when integrating ALT in their practice. The study thus concludes that ALT-mediated volume training should be carefully introduced if pupils do not have sufficient basic knowledge in key mathematical concepts. Although the program is supposed to be individualised and self-adjusting, it is crucial that teachers monitor the pupils’ use and mastery to avoid pupils being stuck in frustration spirals.
The LaT-ALT study is a small-scale study and more research is needed. We especially encourage research investigating the interplay between the learning of new concepts and volume training in adaptive learning programs.
The lack of a control-group and the real-life design of the study makes it hard to determine to what extent the tool itself directly affected the quantitative results.
We want to thank Sara Madeleine E. Kristensen and Ole-Johan Eikeland. We also want to thank the project “Digital competence in higher education” in the research group Digital Learning Communities (UiB) for support.
The authors have no competing interests to declare.
Anderson, T., & Shattuck, J. (2012). Design-Based Research: A Decade of Progress in Education Research? Educational Researcher, 41(1), 16–25. DOI: https://doi.org/10.3102/0013189X11428813
Barab, S., & Squire, K. (2004). Design-Based Research: Putting a Stake in the Ground. Journal of the Learning Sciences, 13(1), 1–14. DOI: https://doi.org/10.1207/s15327809jls1301_1
Bergner, Y. (2017). Measurement and its Uses in Learning Analytics. In C. Lang, G. Siemens, A. Wise & D. Gasevic (Eds.), Handbook of Learning Analytics (First, pp. 35–48). Society for Learning Analytics Research (SoLAR). DOI: https://doi.org/10.18608/hla17.003
Blakelock, J., & Smith, T. E. (2006). Distance learning: From multiple snapshots, a composite portrait. Computers and Composition, 23(1), 139–161. DOI: https://doi.org/10.1016/j.compcom.2005.12.008
Brophy, J. (2006). History of Research on Classroom Management. In C. M. Evertson & C. S. Weinstein (Eds.), Handbook of classroom management: Research, practice, and contemporary issues (pp. 17–43). Mahwah, NJ: Lawrence Erlbaum Associates.
Brown, A. L. (1992). Design Experiments: Theoretical and Methodological Challenges in Creating Complex Interventions in Classroom Settings. The Journal of the Learning Sciences, 2(2), 141–178. DOI: https://doi.org/10.1207/s15327809jls0202_2
Bullough, R. V., Jr., & Richardson, M. (2015). Teacher Perspectives on Classroom Management: Rules, Ethics, and “Crime Control”. In E. T. Emmer & E. J. Sabornie (Eds.), Handbook of classroom management (2nd ed.) (pp. 283–300). New York: Routledge.
Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design Research: Theoretical and Methodological Issues. Journal of the Learning Sciences, 13(1), 15–42. DOI: https://doi.org/10.1207/s15327809jls1301_2
Crick, R. (2017). Learning Analytics: Layers, Loops and Processes in a Virtual Learning Infrastructure. In C. Lang, G. Siemens, A. Wise & D. Gasevic (Eds.), Handbook of Learning Analytics (First, pp. 291–307). Society for Learning Analytics Research (SoLAR). DOI: https://doi.org/10.18608/hla17.025
Deci, E. L., & Ryan, R. M. (2016). Optimizing Students’ Motivation in the Era of Testing and Pressure: A Self-Determination Theory Perspective. In W. C. Liu, J. C. K. Wang & R. M. Ryan (Eds.), Building Autonomous Learners (pp. 9–29). Singapore: Springer. DOI: https://doi.org/10.1007/978-981-287-630-0_2
Doyle, W. (2006). Ecological Approaches to Classroom Management. In C. M. Evertson & C. S. Weinstein (Eds.), Handbook of classroom management: Research, practice, and contemporary issues (pp. 97–226). Mahwah, NJ: Lawrence Erlbaum Associates.
Emmer, E. T., & Sabornie, E. J. (Eds.) (2015). Handbook of classroom management (Second edition). Routledge, Taylor & Francis Group. DOI: https://doi.org/10.4324/9780203074114
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving Integration in Mixed Methods Designs-Principles and Practices. Health Services Research, 48(6, pt. 2), 2134–2156. DOI: https://doi.org/10.1111/1475-6773.12117
Gallego-Durán, F. J., Molina-Carmona, R., & Llorens-Largo, F. (2018). Measuring the difficulty of activities for adaptive learning. Universal Access in the Information Society, 17(2), 335–348. DOI: https://doi.org/10.1007/s10209-017-0552-x
Gyldendal. (n.d.). Multi Smart Øving. https://www.gyldendal.no/grs/Multi-Smart-OEving
Hickey, D. T., & Schafer, N. J. (2006). Design-Based, Participation-Centered Approaches to Classroom Management. In C. M. Evertson & C. S. Weinstein (Eds.), Handbook of classroom management: Research, practice, and contemporary issues (pp. 281–308). Mahwah, NJ: Lawrence Erlbaum Associates.
Hoppe, H. U. (2017). Computational Methods for the Analysis of Learning and Knowledge Building Communities. In C. Lang, G. Siemens, A. Wise & D. Gasevic (Eds.), Handbook of Learning Analytics (First, pp. 23–33). Society for Learning Analytics Research (SoLAR). DOI: https://doi.org/10.18608/hla17.002
Johnson, B., & Stefurak, T. (2013). Considering the Evidence-and-Credibility Discussion in Evaluation Through the Lens of Dialectical Pluralism. New Directions for Evaluation, 2013(138), 37–48. DOI: https://doi.org/10.1002/ev.20056
Johnson, R. B. (2017). Dialectical Pluralism: A Metaparadigm Whose Time Has Come. Journal of Mixed Methods Research, 11(2), 156–173. DOI: https://doi.org/10.1177/1558689815607692
Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a Definition of Mixed Methods Research. Journal of Mixed Methods Research, 1(2), 112–133. DOI: https://doi.org/10.1177/1558689806298224
Knight, S., & Buckingham Shum, S. (2017). Theory and Learning Analytics. In C. Lang, G. Siemens, A. Wise & D. Gasevic (Eds.), Handbook of Learning Analytics (First, pp. 17–22). Society for Learning Analytics Research (SoLAR). DOI: https://doi.org/10.18608/hla17.001
Koh, C. (2016). Translating Motivational Theory into Application of Information Technology in the Classroom. In W. C. Liu, J. C. K. Wang & R. M. Ryan (Eds.), Building Autonomous Learners (pp. 245–258). Singapore: Springer. DOI: https://doi.org/10.1007/978-981-287-630-0_13
Landrum, T. J., & Kauffman, J. M. (2006). Behavioural Approaches to Classroom Management. In C. M. Evertson & C. S. Weinstein (Eds.), Handbook of classroom management: Research, practice, and contemporary issues (pp. 47–71). Mahwah, NJ: Lawrence Erlbaum Associates.
Lang, C., Wise, A., & Gasevic, D. (Eds.) (2017). Handbook of Learning Analytics (First). Society for Learning Analytics Research (SoLAR). DOI: https://doi.org/10.18608/hla17
Liu, W. C., Wang, J. C. K., & Ryan, R. M. (2016). Understanding Motivation in Education: Theoretical and Practical Considerations. In W. C. Liu, J. C. K. Wang & R. M. Ryan (Eds.), Building Autonomous Learners (pp. 1–7). Singapore: Springer. DOI: https://doi.org/10.1007/978-981-287-630-0_1
Mathison, S. (1988). Why Triangulate? Educational Researcher, 17(2), 13–17. DOI: https://doi.org/10.3102/0013189X017002013
Pardo, A., Poquet, O., Martinez-Maldonado, R., & Dawson, S. (2017). Provision of Data-Driven Student Feedback in LA & EDM. In C. Lang, G. Siemens, A. Wise & D. Gasevic (Eds.), Handbook of Learning Analytics (First, pp. 163–174). Society for Learning Analytics Research (SoLAR). DOI: https://doi.org/10.18608/hla17.014
Prinsloo, P., & Slade, S. (2017). Ethics and Learning Analytics: Charting the (Un)Charted. In C. Lang, G. Siemens, A. Wise & D. Gasevic (Eds.), Handbook of Learning Analytics (First, pp. 49–57). Society for Learning Analytics Research (SoLAR). DOI: https://doi.org/10.18608/hla17.004
Roschelle, J., Feng, M., Murphy, R. F., & Mason, C. A. (2016). Online Mathematics Homework Increases Student Achievement. AERA Open, 2(4), 233285841667396. DOI: https://doi.org/10.1177/2332858416673968
Ryan, R. M., & Deci, E. L. (2000a). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78. DOI: https://doi.org/10.1037/0003-066X.55.1.68
Ryan, R. M., & Deci, E. L. (2000b). Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. Contemporary Educational Psychology, 25(1), 54–67. DOI: https://doi.org/10.1006/ceps.1999.1020
Ryan, R. M., Stiller, J. D., & Lynch, J. H. (1994). Representations of Relationships to Teachers, Parents, and Friends as Predictors of Academic Motivation and Self-Esteem. The Journal of Early Adolescence, 14(2), 226–249. DOI: https://doi.org/10.1177/027243169401400207
Schoonenboom, J., & Johnson, R. B. (2017). How to Construct a Mixed Methods Research Design. KZfSS Kölner Zeitschrift Für Soziologie Und Sozialpsychologie, 69(S2), 107–131. DOI: https://doi.org/10.1007/s11577-017-0454-1
Schwab, Y., & Elias, M. J. (2015). From Compliance to Responsibility: Social-Emotional Learning and Classroom Management. In E. T. Emmer & E. J. Sabornie (Eds.). Handbook of classroom management (2nd ed.) (pp. 94–115). New York: Routledge.
The Design-Based Research Collective. (2003). Design-Based Research: An Emerging Paradigm for Educational Inquiry. Educational Researcher, 32(1), 5–8. DOI: https://doi.org/10.3102/0013189X032001005
Watson, M., & Brattistich, V. (2006). Building and Sustaining Caring Communities. In C. M. Evertson & C. S. Weinstein (Eds.), Handbook of classroom management: Research, practice, and contemporary issues (pp. 253–280). Mahwah, NJ: Lawrence Erlbaum Associates.
Winne, P. H. (2017). Learning Analytics for Self-Regulated Learning. In C. Lang, G. Siemens, A. Wise, D. Gasevic & University of Edinburgh, UK (Eds.), Handbook of Learning Analytics (First, pp. 241–249). Society for Learning Analytics Research (SoLAR). DOI: https://doi.org/10.18608/hla17.021