Critical in all research studies is the reliability of data generation and every researcher needs to adhere to this in order to present valid and convincing results. This is true in any research study, but in a collaborative research project concerning toddlers’ (1–3 years of age) numerical development, this issue became distinct. The aim of the project was to investigate how numerical development can be facilitated and how the teaching of necessary aspects of numbers can be made meaningful for toddlers.1 The study was conducted in close collaboration between two researchers and three preschool teachers where the preschool teachers among other things conducted play-based interviews with 27 toddlers. These interviews were conducted five times with the toddlers during four semesters. The aim with the interviews was two-folded, partly to make possible taking the toddlers’ experiences, needs and interest as the starting point for developing educational interventions, and partly to follow the numerical development of the toddlers during the two years of the study. The focus of this paper is however not on toddlers’ numerical development per se, but methodological focusing on validity, or more specifically: on how unplanned divergence in the systematic interviews may have strengthened research quality, even though the opposite may be expected. In the paper, we provide empirical examples and argue that sensitive divergence in the interviews, divergence that makes the interviews hard to exactly replicate, may nevertheless have strengthened research quality. Of importance is that this sensitive divergence would not have been realized without the close collaboration between researchers and preschool teachers.
Interviews are among the most common tool used in studies with young children (Aubrey & Dahl, 2005), but there are few interview studies involving children younger than five years (Clark, 2005). Based on a review on 32 interview studies, Aubrey and Dahl (2005) conclude that open-ended questions are preferable for use in interviews with young children. However, interviews with young children are never a one-way event to extract information from children, it is rather a dynamic process whereby children and adults share and discuss meanings. To succeed in interviewing young children it is necessary to establish joint attention between child and researcher on a subject or question (Aubrey & Dahl, 2005; Johnson & Christensen, 2012; Siraj-Blatchford, 2010).
Reliability and validity are two often used notions in relation to quality in studies including interview studies (Bryman, 2016; Johnson & Christensen, 2012). Most often, the two notions are considered to complement each other in the sense that reliability influence validity, but not the other way around (Cohen et al., 2018, Wolming, 1998). It is often discussed whether the two notions are suitable for qualitative research or not. Sometimes the notions are adapted or redefined and sometimes other equivalent quality concepts are used (Bryman, 2016; Johnson & Christensen, 2012). One example of other quality concepts are Lincoln and Guba’s (1985) trustworthiness and authenticity. However, even if using different notions as Lincoln and Guba, the intention is the same, to guarantee that we can trust the results from research (Wolming, 1998). Thus, reliability and validity are important in all research but are acquired somewhat differently based on research design, which also can be seen in the difference in preferred notions (Cohen et al., 2018).
Reliability concerns precision and accuracy where reliable results are produced in a way where no uncontrolled temporary errors influence and where the measurement is free from bias by the person measuring (Cohen et al., 2018). When reliability is evaluated, how accurately and consistent a measurement is performed as well as replicability are often focused on. In qualitative research, inter-rater reliability can be used by letting several people measure and then comparing how the different people’s measurements match (Bryman, 2016).
Validity is about whether the study measures that which it is intended to measure (Bryman, 2016). To get valid results, the instruments used must be as sound as possible which can be hard in qualitative studies of abstract constructs where no natural measures or units of measurement exist. Thus, a method is valid if it measures what it purports to measure. Reliability, in terms of using standardized procedures in data generation, is one way to allege validity. Validity may be divided into different aspects such as internal, external and ecological validity. Internal validity is connected to the definitions of the included concepts that ought to be defined and operationalized (Cohen et al., 2018). External and ecological validity is connected to the degree to which the results from studies are valid in other situations as well (Cohen et al., 2018); thus, whether the knowledge shown by the children in the interview is valid in other situations. However, no study is valid in itself. Validity also refers to the conclusions that are drawn from the study.
In quantitative research, the involvement of the researchers in the process reduces validity. In qualitative studies, involvement of researchers is often a necessity. However, to maintain valid data the researcher must avoid personal views and perspectives to influence how data is collected and interpreted (Johnson & Christensen, 2012). In interviews, reliability can be obtained by using the same sequence of words and questions for each respondent. However, controlling an interview involves more than controlling the wordings and very structured interviews may miss out the complexity of social interaction. Since changes in wording, sequencing, context and emphasis may undermine reliability, training for interviewers is essential. Training of interview procedures may however decrease researcher bias (Cohen et al., 2018).
As mentioned, the interviews in the project were conducted five times with the toddlers during four semesters. Early on, several methodological challanges came through, for example how to generate valid data in these interviews when verbal skills among toddlers are limited. We addressed this methodological challage with designing a play-based interview (see also Björklund & Palmér, 2021). When designing this play-based interview, the preschool teachers stood for ‘an insider’s nuanced and reality-based perspective’ (Cai et al., 2018, p. 518). Their familiarity with the toddlers and the educational settings enabled them to anticipate the experiences, needs and interest of the children. The researchers stood for experiences of interview design and expertise in the area of numerical development to be investigated. The tasks in the interview were developed in accordance with the following theoretically driven (Björklund & Palmér, 2021) design principles: The principle of context: The context of the tasks ought to be familiar to the toddlers so that they can relate to and reason about the content based on their previous social and cultural experiences and understanding. The principle of numbers: The tasks ought to cover fundamental aspects of numbers. The principle of process: The tasks ought to be designed in a way in which the toddlers can express how they experience different aspects of numbers during the whole process of the interview. The principle of multiple-method activity approach: The tasks ought to enable the toddlers to express themselves both verbally and non-verbally, for example through expressions such as gestures and actions on manipulatives. The principle of differentiation: The tasks ought to allow the toddlers to express different ways of understanding, and allow a variety of experiences both between toddlers and within the same toddler over a prolonged period of time. Based on these five design principles, seven interview tasks were developed, each with five levels of difficulty. The context and narrative of the interview was the birthday of Kitty the cat, as even toddlers have experiences of birthdays. The tasks, framed as games or play, included manipulatives such as cookies that Kitty liked to eat and boxes where Kitty stored her cookies.
Since toddlers’ expressions are very subtle and thus demand exclusive knowledge of the individual child’s ways of expressing him/herself, we found it to be critical to involve the preschool teachers to conduct the play-based interviews. This was also an ethical consideration. Of course, we had written consent to conduct the interviews from the toddlers’ legal guardians (Swedish Research Council, 2017) but an ethical question of same importance is the agency of the child (Alderson & Morrow, 2011). Very young children may be anxious of unfamiliar people and have a harder time communicating to a stranger if they no longer consent to taking part in the interview. Thus, the preschool teachers were the most appropriate interviewers as they knew the toddlers and had the pedagogical skills to be sensitive to children’s needs and responses in the current situation. However, preschool teachers are not educated to conduct research interviews, which could decrease research quality in sense of reliable and valid data.
To promote research quality, we conducted a pilot study in which the preschool teachers were educated in conducting this specific interview and also educated in the area of numerical development to be investigated. The intended wordings and actions of the interviewer were documented in a script. In the pilot study, this script during six months was tried out in an iterative process where the preschool teachers conducted interviews with preschool children that were not included in the main study. These pilot interviews were video-recorded and jointly surveyed where the wordings and actions of the interviewer were elaborated on and revised. The intention of this iterative process was to ensure that the interviews were being conducted similarly and after six months we had jointly developed the script to be used.
In this section we present three examples from the first task in the interview conducted within the main study. Based on these examples, issues on reliability, validity and reserach bias will then be elaborated on. The examples are from the same teacher. The teachers had the script for the task on a card on the table while conducting the interviews. The script is as follows:
This is Kitty the cat. Yesterday was the cat’s birthday. How many years are you? [direct translation from Swedish ‘hur många år är du?’]
If the child answers with finger pattern: How many are there?
If the child does not answer: Can you show with your fingers?
In the study, this task had a twofold purpose, partly to arouse curiosity and interest by inviting the children to a joint conversation and partly to see how the children respond to this question, which they most likely have heard before. The toddlers in the examples below are about two years. The first example is Aisha.
Teacher Do you remember Kitty the cat? The teacher holds the toy cat. Aisha smiles, nods and looks the teacher in the eyes. Yesterday was the cat’s birthday. How many years are you Aisha?
Aisha Puts forward two fingers.
Teacher How many are there?
In this example, the teacher starts by referring to Aisha remembering Kitty the cat from previous interviews. After that, the teacher follows the script with the addition of the child’s name. The child has eye contact with the teacher more or less throughout the sequence. The next example is Elliot.
Teacher This is Kitty the cat. The teacher holds the toy cat. Elliot looks at the teacher and the toy cat.
Teacher Yes, Kitty.
Elliot Kitty’s tail. Points at the tail of the toy cat.
Teacher Yes, the tail of the kitty. Elliot, do you know. Yesterday was the cat’s birthday. How many years are you Elliot?
Teacher Two. Can you show with your fingers?
Elliot Here is two. Puts forward two fingers.
In this example, the teacher starts in line with the script. The child has eye contact with the teacher more or less throughout the sequence. When the child points at the tail and says ‘tail’ the teacher confirms. The teacher also confirms the child answering two. Otherwise, the teacher follows the script with the addition of the child’s name to the question. Finally, the third example is Amelia.
Teacher Yes, look, it was Kitty the cat.
Amelia Takes the toy cat and hold it in front of her and looks at it.
Teacher Yesterday was the cat’s birthday. Happy Birthday to you, happy birthday to you. The teacher sings the first phrase of a very common birthday song. Amelia starts to look at the teacher. How many years are you Amelia?
Teacher Yes. Can you show with your fingers?
Amelia Two. Put forward two fingers.
In this example, the teacher starts by confirming something that happened before the camera is turned on. When the child takes the toy cat she loses eye contact with the teacher. When the teacher starts to sing the child makes eye contact again. Then, the teacher follows the script with the addition of the child’s name to the question. The teacher confirms the child answering two.
In the introduction, we said that we would argue that sensitive divergence in interviews, divergence that make interviews hard to exactly replicate, may nevertheless strengthen research quality. The here exemplified divergence was however not planned, quite the opposite we used six months to educate the preschool teachers and to develop the script to be used. The teachers adapting the script to each toddler could be considered as researcher bias, allowing personal views and perspectives to influence how the data is collected and interpreted, making the interviews hard to replicate. However, we argue that this sensitive divergence in the interviews in fact strengthens the research quality.
Most often, reliable studies are described as studies that are replicable (Bryman, 2016; Johnson & Christensen, 2012), and this is why there was a script for the interviewer to follow. However, as shown in the examples above, the teacher made slight changes in the script. These changes are always additions; thus, sections from the script are not left out. The teacher in the examples is very sensitive to eye contact when asking the questions and one addition she makes is to add the names of the children to the questions, as a way to draw the child’s attention to what the interview intends to focus on. Another addition is the teacher confirming the answers of the children, correct or not. We argue that these additions do not decrease research quality but instead enable the necessary joint attention, highlighted by Siraj-Blatchford (2010) and thus increase research quality. Even though the interviews by these additions cannot be replicated in detail, all toddlers are given the best conditions to participate and respond in the interview. As mentioned, a method is valid if it measures what it purports to measure and the additions of the teachers increase the possibility of joint attention as the design principle of context is strengthened. Thus, the decrease in research reliability actually increase the research validity. By doing different, the teachers provided equal conditions for the toddlers to understand the interview why the additions increased the possibility to measure the intended, the numerical development of the toddlers.
This quality improving divergence was not planned and could not have been realized without the close collaboration between researchers and preschool teachers. As mentioned, the teachers made additions but no sections from the script were left out. In another collaborative study by Breive et al. (2018), teachers who were instructed to make applicable adaptations when implementing pre-planned lessons, did in fact no adaptations. Breive et al. concluded, one reason was a too detailed written instruction. The script used by the teachers in this study, was indeed very detailed but the teachers still made adaptions. We believe that the long-term collaboration between the teachers and the researchers was important, also the training in the pilot study. We also believe that the pilot study where the teachers were educated in the area of numerical development to be investigated in combination with their exclusive knowledge of the individual children’s ways of expressing made the divergence possible. From the joint surveys in the pilot study, the teachers knew the aim and objective embedded in each task and thus they knew what was possible to adapt without losing the aim. As such, the teachers’ insider and reality-based perspective (Cai et al., 2018) was necessary for making these adaptions possible.
To summarize, in this study the long-time collaboration between researchers and teachers made it possible for the teachers to adapt the tasks, offering each toddler the best conditions to show all their knowing, why more valid conclusions could be drawn from the study. Thus, a decrease in research reliability actually increased research validity.
The study was financed by the Swedish Institute for Educational Research (Grant no. 2018-00014).
The authors have no competing interests to declare.
Alderson, P., & Morrow, V. (2011). The ethics of research with children and young people. A practical handbook. SAGE. DOI: https://doi.org/10.4135/9781446268377
Björklund, C., & Palmér, H. (2021). Designing a tool for exploring toddlers’ numerical competencies in preschool. In Y. Liljekvist, L. Björklund Boistrup, J. Häggström, L. Mattsson, O. Olande, & H. Palmér (Eds.), Sustainable mathematics education in a digitalized world. Proceedings of MADIF12. SMDF
Breive, S., Carlsen, M., Erfjord, I., & Hundeland, P.-S. (2018). Designing playful inquiry-based mathematical learning activities for kindergarten. In C. Benz, A. Steinweg, H. Gasteiger, P. Schöner, H. Vollmuth & J. Zöllner (Eds.), Mathematics education in the early years. Results from the POEM3 conference, 2016. Springer. DOI: https://doi.org/10.1007/978-3-319-78220-1_10
Cai, J., Morris, A., Hohensee, C., Hwang, S., Robison, V., & Hiebert, J. (2018). Reconceptualizing the roles of researchers and teachers to bring research closer to teaching. Journal for Research in Mathematics Education, 49(5), 514–520. DOI: https://doi.org/10.5951/jresematheduc.49.5.0514
Clark, A. (2005). Listening to and involving young children: A review of research and practice. Early Child Development and Care, 175(6), 489–505. DOI: https://doi.org/10.1080/03004430500131288
Cohen, L., Manion, L., & Morrison, K. (2018). Research methods in education. Routledge. DOI: https://doi.org/10.4324/9781315456539
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Sage Publications. DOI: https://doi.org/10.1016/0147-1767(85)90062-8
Siraj-Blatchford, I. (2010). A focus on pedagogy: Case studies of effective practice. In K. Sylva, E. Melhuish, P. Sammons, I. Siraj-Blatchford & B. Taggart (Eds.), Early childhood matters. Evidence from the effective pre-school and primary education project (pp.149–165). Routledge.