CEPE Di Tella

Describe the product or program that is the focus of your proposed LEAP project.

The Center for Evaluation of Policies and Evidence-Based Policy-Making (CEPE Di Tella) specializes in conducting applied research that seeks to improve the quality of policy through the provision and promotion of evidence-based initiatives and rigorous evaluation methods. CEPE Di Tella’s Education and Behavioral Science Program studies contextual, behavioral, and neuroscientific determinants of both  learning and teaching for assessing and scaling cost-effective educational interventions that work. 

Our program focuses on researching learning and teaching abilities throughout the lifespan with a special interest in different populations of underserved children. Since 2017, we are working along with schools from Argentina oriented towards enhancing the education of Deaf and Hard of Hearing (DHoH) children in order to develop the Learning to Learn Program (LeLePro), an evidence-based intervention program to improve these children’s linguistic abilities and provide them better educational opportunities. Our studies  target  children between the ages 3 to 12.

Part of the design of our linguistic intervention program involves the evaluation of other cognitive abilities like metacognition and Theory of Mind (ToM). The development of these abilities during childhood are crucial for mental health, and the development of other cognitive skills.

While evidence from the field of experimental psychology strongly suggests that there is a tight relationship between metacognition, ToM and the linguistic performance in DHoH children, it is usually hard to disentangle whether metacognitive abilities and ToM are prerequisites for language development or the if linguistic abilities enable a satisfactory performance in ToM and metacognition task. Thus, a good diagnosis tool that relies as little as possible on language use is necessary to adequately assess metacognitive and ToM abilities in children with communication disorders.

Our team has already developed an assessment tool that measures both metacognition and ToM in children aged 6 to 9 with typical hearing and development, MetaTom. We would like to adapt MetaTom for the evaluation of DHoH children aged 3 to 12. In turn, we aim to convert the evaluation tool into a digital app capable of reliably collecting different types of data that in previous instances were manually analyzed.

Select the key characteristics of your target population. Select all that apply.

• Pre-primary age children (ages 1-5)
• Primary school children (ages 5-12)
• Peri-Urban
• Urban
• Low-Income
• Middle-Income
• Persons with Disabilities

In which countries do you currently operate?

• Argentina

In which countries do you plan to be operating within the next year?

• Argentina

How have you worked with affected communities to design your solution?

Since 2017, our team has committed to work with schools from Buenos Aires, Argentina oriented towards the education of DHoH children in order to provide evidence-based learning solutions for these children. Mainly, heads of three schools imparting exclusively oral education for these children required the creation of a diagnostic tool assessing their Spanish linguistic abilities. Since then, we have conducted different school-based research studies using methods from cognitive neuroscience in order to respond to questions like:  

• What communication strategies do DHoH children equipped with cochlear implants use during pedagogical events and how do they differentiate from those used by children who typically develop hearing? What comes first, the ostensive channel or the content of knowledge? 

• What has been described in the literature as typical comprehension patterns in orally educated children with prelingual hearing loss? 

• How do Spanish-speaking deaf children perform in sentence comprehension? And how do they perform in oral production?

Currently, we are expanding our research questions to the bimodal bilingual Deaf community and have engaged three bilingual (Argentine Sign Language - Spanish) schools to take part in our experimental studies. In this regard, we aim to understand the consequences of bilingualism for linguistic attainment in these children.

In 2022, the Inter American Development Bank awarded our team a grant to develop a project that seeks to apply techniques from Behavioral Science towards the improvement of Policy. Our project aims to understand how parents of DHoH children from Argentina choose these children’s linguistic education and to encourage bimodal bilingual education for these children and sign language learning in parents, take carers and close relatives.  During the past three months we have conducted interviews with different key figures from the Argentine Deaf Community, heads of schools and charities dedicated to this population in order to better understand the specific needs and perspectives of the community and design a cost-effective and high-impact intervention that provides better communication tools for these children and their families. The project involves the collaboration of the Ministry of Education of the cities of Buenos Aires and Mendoza, and is currently being implemented in all public schools that receive DHoH from those cities.

What is your theory of change?

Our theory of change identifies the possibility to build a minimum viable product (MVP) of a digital version of the MetaTom Diagnosis Tool adapted to DHOH children. We believe this research device will make a contribution to improve educational trajectories in this vulnerable population. Starting from already existing instruments, it seeks to expand innovation in two ways: i) technological development, and ii) diversification of use in DHOH children.

Essentially, our participation in the LEAP initiative would accelerate the Activities Phase of our theory of change by accessing enriched exchanges with academic and entrepreneurial fellows to embark on a necessary roadmap.

As we are able to come up with a tested tool, we would be able to open a relevant applied research agenda to test for short, medium and long term outcomes detailed below.

1 Activities

- 1.1 Elaborate a literature review on best practices for ToM and metacognitive abilities tools development and application techniques, including assessment of market placement opportunities in Argentina and Latin America.

- 1.2 Build a roadmap to adapt our paper-based assessment tool into a digital version, ensuring it remains relevant, user-friendly and reliable for school-based interventions

2 Outputs

- 2.1 Adapted Digital Version of MetaTom Diagnosis Tool for the evaluation of DHoH children aged 3 to 12, including application protocols.

3 Short-term outcomes 

- 3.1 Increased MetaTom Diagnosis Tool usage among DHoH children in Special Education Schools in Argentina.

- 3.2 Widened knowledge on mechanisms guiding the relationship between metacognition, theory of mind abilities and language skills through linguistic attainment within DHoH children.

4 Medium-term outcomes

- 4.1 Improved development of LeLePro curricula and strategies to serve DHoH children aged 3-12. 4.2 Increased participation rates of LeLePro beneficiaries in the country as more schools are assimilating program strategies.

5 Long-term outcomes

- 5.1 Improved educational attainment indicators within DHoH children aged 3-12.

- 5.2 Improved DHoH children' aged 3-12 well-being.

- 5.3 Increased satisfaction levels of DHoH children's parents with their kids performance and well-being.

How are you currently using evidence within your theory of change?

All our work developed at CEPE Di Tella’s Education and Behavioral Science Program is supported by rigorously collected data, following methods of experimental psychology and cognitive neuroscience. 

The key question we aim to answer with the development of an assessment tool for metacognition and ToM for deaf and hard of hearing children is: 

• How much of these children’s linguistic shortcomings is mediated by difficulties in metacognitive and ToM abilities?

Our research conducted in children with typical hearing introduces a valid adaptation of the ToM tasks developed by Wellman and Liu (2004) for children aged 6 to 8 years old. Furthermore, the tool designed by our team to assess metacognitive abilities (based on the work of (Kornell, Son, & Terrace, 2007) can independently detect differences regarding the child's metacognitive ability and tendency towards risk, two factors that are generally confounded in the standard treatment of confidence in the literature. Both works provide detailed explanation of how to replicate their experimental protocols. Yet, we don’t know if these tools will be sensitive indicators of ToM and metacognitive abilities in children with hearing impairment. 

Our pilot, currently at Stage 1 according to Nesta’s Standards of Evidence integration, aims at gathering the data that may support a change in the way linguistic attainment is assessed in these children. In doing so, our team needs to fulfill the following stages: 

1. Understanding what works for assessing ToM and metacognitive abilities in DHoH children and in children as young as 3 years old.

2. Providing a critical analysis of how language-dependent the diagnosis tools already available in the market are in order to adapt them to this underserved population.

3. Providing a reliable way to measure the tool’s impact. 

4. Building a roadmap to identify which steps we should take according to the possible outcomes of our study.

How are you currently tracking and measuring your solution’s impact?

We will use several metrics to measure the impact of our new tool:

First, in order to measure the validity of our tool, we will measure if it sensitively measures ToM and metacognitive abilities as compared to other tools in the psychology field in children with typical hearing. While we have already collected this data in children aged 6-8, we need to run this study in children as young as 3 and as old as 12 years old.

Second, in order to measure the impact of our new tool (MetaTom) in the deaf and hard of hearing sample, we will conduct a study including our experimental group (DHoH) and a control group (children with typical hearing). Data collected for the ToM tasks will include measures like accuracy rates, response time, error types and omission rates. Relevant measures of the metacognitive task include accuracy rates, error types and omission rates confidence in the accuracy of the response and response times. 

Third, cognitive and linguistic development are strong predictors of ToM and metacognitive skills, and during childhood, chronological age is a good proxy for cognitive  and linguistic development. However, when studying developmental trajectories in DHoH children, chronological age is not usually a good reflection of linguistic development, as most of these children are deprived of language for a period of time until they receive hearing aids, cochlear implants or learn sign language. Thus, we will use both chronological and verbal age (the earliest age at which the child received linguistic stimulation by means of hearing aids, cochlear implant or sign language) as covariables in our analysis to reliably measure the sensitivity of our tool in terms of how well it measures metacognitive and ToM skills independently of the children’s linguistic abilities. Matching samples according to verbal abilities is a standardized analysis procedure in the field of developmental neuropsychology, as shown in previous studies tackling the relationship between ToM and communication disorders in children with Specific Language Impairment (SLI) or Autism Spectrum Disorder (ASD), for instance. 

One-line project summary:

MetaTom is a diagnosis tool to measure metacognitive and Theory of Mind abilities in deaf and hard of hearing children.

What is your solution’s stage of development?

Pitch your LEAP project: How and where would integrating evidence (or stronger evidence) into your theory of change increase your organization’s impact?

Due to the tight relationship between education and language, learning imposes a great challenge for Deaf and Hard of Hearing (DHoH) children. In formal education, school programmes focused on this population put strong emphasis in teaching the linguistic abilities that are needed to thrive in life.

During the last three years, CEPE has worked along with schools of Argentina dedicated to DHoH children to develop a research and intervention program to improve the linguistic abilities of these children.

While the focus of this program has been language attainment, it is crucial that other cognitive skills are assessed as well for a more precise evaluation of its impact.

In particular, research has shown that there is a strong relation between language attainment and Theory of Mind and metacognitive abilities. 

Theory of mind refers to our capacity to understand others' mental states including beliefs, desires, and knowledge, and the ability to comprehend that these may differ from our own. 

Metacognition can be defined as that ability that allows us to think about content of knowledge and our own cognitive processes and, consequently, evaluate how to act to fulfill different purposes throughout life. The development of this skill is crucial for learning, since it allows us to guide, regulate and supervise our own learning activities; that is, knowing how to learn and how and when to use a series of strategies to regulate our behavior.

How do Theory of Mind and Metacognition relate to language skills?

These abilities are traits of the cognitive repertoire acquired during childhood. 

Regarding Theory of Mind, children develop an understanding of themselves and other people as psychological beings who think, know, want, feel, and believe very early in life, even before their first birthday. During development, they come to understand that what they think or believe may be different from what another person thinks and believes. They also learn that much of our behavior is motivated or caused by our knowledge and beliefs.

How do we shape our Theory of Mind? Through interaction with other agents, both in real life or for example, through books. It has been proposed that the lack of access to conversations in the environment causes DHoH children to miss important information about the world, causing a delay in the development of Theory of Mind. 

Crucially, many studies that assessed Theory of Mind in DHoH children have relied on verbal tasks to assess their understanding. It is then possible that deaf children have a Theory of Mind but don’t count on the linguistic skills necessary to understand the stories that are normally used to test them.

Importantly, evidence shows that profoundly DHoH children who show a good command and understanding of complex sentences usually exhibit good performance in tests of Theory of Mind irrespective of whether the tests required minimal verbal skills or not, suggesting that children who are not able to understand complex syntactic forms have difficulty understanding how their own thoughts and beliefs may differ from those around them.

As for metacognitive skills, different studies have shown that children as young as 12 months already show good abilities to optimally distinguish between what they know from what they don't, and that by 18 months of age children can already use spontaneous strategies to correct their mistakes during problem solving.

Evidence also shows that using metacognitive strategies is effective for the development of reading skills in DHoH children. Crucially for the development of our assessment program, our team has already designed  and implemented different tests that measure Theory of Mind and metacognitive skills in children aged 6 to 9 years old. However, these tools have only been applied at a small scale in lab-controlled studies with children with typical development and typical hearing.

The challenge we currently face is to adapt these materials for samples of DHoH children who access language visually through sign language or orally by means of hearing aids and cochlear implants. As we also expect to scale our pilot to several schools in the country, we also aim at designing a digital version of these screening tools, in order to improve data collection standards and results' reliability and replicability. 

Thus, some of the questions  LEAP Fellows participating in our project will help us respond are:

• What does the most recent and reliable scientific evidence say about the relationship between cognitive skills such as Theory of Mind and metacognition, and language attainment in DHoH children? 

• What is the most effective roadmap to adapt our paper-based assessment tool into a digital version so that it is user friendly and reliable for school-based interventions and for potential scaling?

• What are the best course of action, possible risks and caveats we need to take into account when applying assessment and intervention programs in children with communication disorders? 

• How can we set up uniform criteria and recommendations for future impact assessment?

Outputs expected as part of the LEAP sprint are:

• A thorough understanding of what tests and diagnosis tools are available for measuring ToM and metacognitive abilities in DHoH children, their strengths, pitfalls and reliability, with a comprehensive analysis of the samples of children in which they were used with. We expect that this initial approach to the problem will lead to the publication of a literature review on this issue.

• A roadmap on which steps we should follow to adapt our existing (paper-based) tool into:

  • a diagnosis tool that may be used with DHoH children and potentially with children with other communication disorders.

  • a tool that may be used across the span of 3 to 12 years old children.

  •  a digital and user friendly app that could potentially be used for scaling up our project to further schools and regions in the country.

By hosting a LEAP Project, our team will gain expertise in conducting qualitatively and quantitatively more reliable practices towards the design and implementation of evidence-based interventions targeted at DHoH children and underserved population in general.  We also expect that in the following five years, this pilot project will grow into a large-scale assessment and intervention program that will contribute to the improvement of learning experiences of these children across the country.

Pitch Video