Beyond Education

Who (first and last name) is the Team Lead for your application and LEAP Project?

Dr. Nathan Roberson

Describe the role the Team Lead plays in your organization. [100-200 words recommended]

Dr. Nathan Roberson is the Chief Research Officer for Beyond Education. He/They are responsible for managing the development of the BE research base, and to help ensure the products are consistent and aligned to our our present and new existing research evidence. Dr. Roberson is responsible for communicating the research needs and to coordinate them with the operations team and CTO in order to carry out and conduct new research. 

Explain how your Team Lead and supporting team members are well-positioned to effectively support the LEAP Project, given other priorities within your organization. [300-500 words - recommended]

As the Chief Research Officer, Dr. Roberson's primary responsibility is the development and communication of research tied to the vision of BE. As such, this is their primary duty and aligns closely to the LEAP project. Dr. Roberson completed their PhD in Measurement, Evaluation, and Research Evaluation from the University of British Columbia. As such, he comes to the project with extensive expertise already and can accordingly better partner and implement guidance with the other LEAP fellows. Dr. Roberson is the key decision maker for research based activities within the company, and so has the authority to make research-based decisions regarding priorities and implementation decisions. 

One-line solution summary: In 20 words or less, summarize your organization's product or program that is the focus of your LEAP Project.

Through our online platform, BE transforms education by measuring and empowering students globally with 21st-century competencies.

Define the problem that your solution seeks to solve. [300-500 words recommended]

The world is evolving rapidly, implying that the jobs of tomorrow, the socio-economic problems, and the technologies we will have to interact with, will no longer exist (WEC, 2020). However, faced with all these problems, we must continue to evolve our approach to teaching and learning. If education consists of a transmission of knowledge, the consequences of this rapid change implies that we must create new adapted ways of learning and teaching for the future (Fadel et, al., 2015 ; Trilling & Fadel, 2009). 

For this, a new set of 21st century skills and competencies will be necessary. We must rethink education in order to transmit our traditional knowledge but also to give us the necessary skills as individuals to not only apply this knowledge but to exploit it  (Fadel et al., 2015 ; Trilling & Fadel, 2009). The competencies for the 21st century are therefore defined as all the skills and competencies needed by individuals to face, with adaptability and consciousness, by themselves and together, the technological, societal and economic challenges that cannot be anticipated or thought of in the present because of their fast evolution and uncertain nature. Among the model to define the future of learning, the "Four-Dimensional Education" model  from the Center of Curriculum Redesign (CCR) developed by Charles Fadel et al. (2015) stands out by proposing a framework that organizes the competencies of the 21st century, defines them in a clear and usable way, and provides levers for action for all education stakeholders.

Beyond Education (BE) has developed the Competencies Compound Inventory for the 21st Century (CCI-21) through a series of research studies conducted with upper primary and secondary students from various English-speaking student populations around the world. Likewise, we have conducted a large study of the translation and application of the CCI-21 in Brazil using a Portuguese version of the tool. Currently we are both expanding our reach through partnership with Spanish-speaking students in Latin America, however it has become clear that we need to further develop and study the tool with primary students (under 13) in order to align our instruments into a cohesive set of measurements to monitor 21st Century Competencies throughout elementary education.

Describe your solution and how it works in simple terms. [300-500 words recommended]

We propose the first, scientifically validated measurement of twelve 21st competencies with a sensitive, reliable, and valid items. The results can help educational professionals and students to monitor and progress on these indispensable competencies. Students are assessed within three dimensions: Skills (Creativity, Communication, Collaboration, Critical Thinking), Character (Courage, Curiosity, Resilience, Ethics, Leadership, Mindfulness), and Meta-learning (Meta-cognition, Growth Mindset). Students receive an individualised report that is age-calibrated to help monitor their progress. 

Before taking the assessment, teachers and schools are trained on the theoretical foundation of the model and the importance of 21st Century Competencies. In addition, our team helps to oversee the technical integration to create the platform to deliver the assessments in a seamless way. 

Students take a 20-30 minute assessment that includes some questions on basic demographics, a social desirability scale, and the 36 questions to measure the competencies. At the end of the test, students immediately receive their customized reports. Once all the students have completed the assessment, our research team produces customized reports at the school and/or organizational level that provides a summary of the student results, statistical tests, interpretation of the data and possible recommendations.

Select the key characteristics of your target population. Select all that apply.

• Women & Girls
• LGBTQ+
• Primary school children (ages 5-12)
• Rural
• Peri-Urban
• Urban

Upload your solution's Theory of Change or Logic Model.

Where would you place your solution on Nesta's Standards of Evidence?

To date, what research/studies has your organization conducted or commissioned that have helped demonstrate the effectiveness of your solution? [300-500 words recommended]

To date, we have published three formal research pieces in academic journals. 

1. “A Case Study of 21st Century Cognitive, Social and Emotional Competencies Using Online-Learning” in the Journal of intelligence: https://www.mdpi.com/2079-3200...

2. “Psychometric Properties of the Competencies Compound Inventory for the Twenty-First Century” in Frontiers of Education: https://www.frontiersin.org/ar...

3. “Developing 21st Century Competencies among Youth through an Online Learning Program: BE a Global Citizen” in Education Sciences: https://www.mdpi.com/2227-7102...

These three papers give us a strong foundation on which we are continuing our research around the measurement and evaluation of 21st Century Competencies using our assessment and programs. These papers include formative and summative research. These studies have been conducted using principally older student samples (ages 13-18), and we are currently working to expand our research with upper primary-level students. Likewise,, we are also in the process of linking our current research to other academic outcomes in schools through research partnerships looking at dimensions such as math and literacy scores to understand correlation of cognitive domains to our non-cognitive measure (CCI-21). A fourth research study about the translation and use of the CCI-21 in Brazil is forthcoming, but demonstrates a similar level of strength to the English version.

What has the data collected from the research/studies revealed about your solution and how did it inform your work moving forward? [300-500 words recommended]

The research conducted so far has created a strong foundation to continue our work. First, the research around our programs has shown that students benefit from the program as evidenced through student and teacher-level testimonials. The programs are transformative, educational experiences that can expand the scope of education into students’ personal lives. This research has also shown some of the systemic constraints of our ability to engage in these programs, as the time-requirement for these programs is difficult for schools to do in addition to other requirements. Second, our psychometric study has demonstrated sound validity arguments for the use of the CCI-21 tool to measure our 12 psycho-social dimensions in Skills, Character, and Meta-Learning. The data show acceptable or strong levels of sensitivity, reliability, and validity, which included the use of discriminant analysis to other widely used measures. These data help give us confidence to expand and use the measure as a tool to monitor and measure change in other educational contexts beyond our programs. The instrument is scalable given the short time requirement. Findings from these studies and other feedback has emphasized our need to make certain modifications to our tool to reach upper primary students (10-12 years old) in order to adapt to their more nascent levels of literacy and normal human development cycle. Building out this research with younger students will expand our research and operations with a new age-demographic and allow us to contribute to a more cohesive cycle of evidence for change in connection with our existing instruments and programs.  

Describe your organization's need to strengthen the evidence base of your solution. [300-500 words recommended]

Since our inception, BE has reached nearly 100,000 students around the world through our programs and assessment. In this year alone, we will reach over an additional 100,000 students as we expand into new countries and school networks. Our rapid growth, while exciting, requires additional need to strengthen our evidence base. In particular, we need to demonstrate evidence about the cultural relevance of our products as we reach into diverse communities. That is to say, how and to what extent can we make cross-cultural comparisons using our existing evidence? Likewise, it is clear that school partners are looking for educational solutions that can be used as a part of a continuous process of change and improvement. This means that we are also asked to expand our solution to younger aged students that are nearly at the limit of intended development range. As such, we need to make modifications to our existing products that better engage with upper primary students (10-12), while also maintaining the cohesiveness to our other products and comparable scientific validity. Given our expansion in the last year, we now have new educational partners who are keen to work with us, which means we now have the opportunity to conduct new kinds of research to expand the relevance of our products. Similarly, as we are reaching greater levels of sample sizes from cumulative research activities, we now can also reconsider the types of measures of effectiveness to consider other forms of psychometric validity that require larger samples sizes (e.g. IRT models instead of CTT models). While our team has produced excellent results, our growth has also meant more limited capacity for research. As such, BE would benefit from external partnerships to help guide and advance our research base as we continue to advance an inclusive and transformative educational agenda.

What are 2-3 research questions that you would like your LEAP Project to help you answer? [100 words recommended]

How and to what extent is the best way to adapt our product to upper primary students while maintaining cohesiveness and scientific rigour with our existing products?

What are the best ways to make intercultural comparisons of our existing data?

What type of research methods do you think will help answer your stated questions? Select all that apply.

• Foundational research (literature reviews, desktop research)
• Formative research (e.g. usability studies; feasibility studies; case studies; user interviews; implementation studies; process evaluations; pre-post or multi-measure research; correlational studies)
• Summative research (e.g. impact evaluations; correlational studies; quasi-experimental studies; randomized control studies)

Please elaborate on your selection above by describing your desired outputs of the 12-week LEAP Project sprint. [300 - 500 words recommended]

I hope to produce a study framework to adapt and evaluate our tool for younger aged students that remains linked to our existing theoretical framework. This will likely include a pilot study design along with new monitoring tools to make linkages. 

For the intercultural comparison - this can likely be done with mostly desk research, and assistance for study designs around data analysis/comparisons. Summative research frameworks could already be put in place to make sense of this data. 

How will your organization put these outputs into action? [300-500 words recommended]

The extension of our products for primary students will be put into place using our existing sales-process. That is to say, the extension of a new product will extend the scope of our existing reach by being able to incorporate younger students into our professional relationships. In the beginning, I anticipate the outputs of the LEAP project will result in an initial piloting to collect evidence to make informed adaptations of our tool. 

Guidance on making intercultural comparisons using our existing data will be put into place in several ways. First, we are currently building a database that will enable us to be able to better organise and use the data we have. Once this is complete, we would like to be able to use our data in order to improve our existing products (school and network-level reports) so that we can include additional benchmarking comparison in order to help clients make use of their own data. Likewise, additional insights could be used to perhaps build out Professional Learning Communities (PLCs) among schools so that they can learn from each other based on our data.

Describe your desired long-term outcomes of the 12-week LEAP Project sprint for both your organization and solution. [300-500 words recommended]

The results from the LEAP project will have immediate short-term benefits of product development and extension of our company reach. However, the long-term outcomes include our ability to create greater system-level cohesiveness to integrate 21st Century Competencies into our educational programs. By expanding the reach of our products, and by better making use of the data we already are collecting, we can help contribute to an ecosystem change in education. An old adage in the Measurement community says, “What gets measured, gets treasured.” At BE, we believe this. As such, we believe that by focusing our efforts to measure and develop 21st Century Competencies we can help enable students and educators to foster those competencies that are not only truly human and irreplaceable even in the age of AI, but we can make education a more holistic experience. In this way, we can help shift the focus of our educational activities to put more emphasis on what we believe truly matters: transmutable, transversal, and human capacities for the future.