DeeperLearning thru LLM/ML/NLP

What makes your solution innovative?

Prior to an LLM like ChatGPT, existing commercial scoring of student writing had multiple severe limitations.  Big publisher and researcher auto essay scoring engines (e.g. Pearson, McGraw Hill, ETS) only worked with a defined publisher’s prompt and text, manually calibrated human scorers of ~200 student papers that were then used to calibrate the AI engine.  Change a word in the prompt or the text or have poor or limited teacher scorer accuracy and the engine did not work well.  Worse, the writing assessed typically was limited to 800 words or less and of low Depth of Knowledge: – e.g low cognitive level student writing or formulaic 5 paragraph essays.  They were not usable with “blind prompts” – prompts the engine hadn’t seen before – nor able to assess the deeper student learning expectations of national (or international) standards of learning (e.g. Common Core, Next Generation Science Standards, College, Career, and Civic Life) necessary for participating effectively in the non-routine work of the global future.  Scoring and feedback on extended student writing in response to authentic and complex disciplinary text also has not been available in the marketplace.  Although ChatGPT has created new opportunities, the shortcomings are (1) most emerging solutions driven by for profit companies driven are low quality bot responses to low cognitive level questions of highly uncertain response quality or even inaccuracy and bias.  (2) ChatGPT or other LLMs in and of themselves are not of sufficient quality or accuracy.  Using an LLM as a component rather than standalone, with SRI’s ML/NLP created and tested over decades in high profile contexts (e.g. national research laboratories, Dept of Defense and Homeland Security contracts) to date has proven far more effective.  Creating a teacher facing solution more user friendly than current ChatGPT interfaces.  Finally, over time the user learning engineering data generated will both create self reinforcing improvements in tool performance but equally importantly provide feedback on which instructional pedagogical practices and instructional materials most impactfully improve student deeper writing development and reading comprehension and – potentially – the most effective sequencing of those strategies and content in such student learning development.

Describe the core AI and other technology that powers your solution.

LDC will use both traditional innovations (analytic rubrics) and new technologies (Generative AI and SAS software development).  The SCALE analytic student rubric measures the student’s learning progress, signaling to both student and teacher the next step toward proficiency. For example, both the teacher and the student can easily understand the R2 learning progression and next proximal learning steps for the 3rd grade student scoring at 2nd grade level 2 with additional complex text based on the explicit SCALE analytic student rubric.  This will ensure the project builds upon the already established benefit of rubrics in that they are statistically more likely to mitigate bias in grading practices by standardizing scoring criteria, reducing the space for subjective assessment grading. This ultimately makes educators more effective in assessments and feedback built on years of research showing students learn best when they are exposed to relevant content and provided with opportunities to expand their critical literacy skills.  Even before Large Language Models, Machine Learning and Natural Language Processing demonstrated efficiency and accuracy to provide formative data and insights into student learning including reading comprehension and extended writing in response to complex text.  LDC began exploring ML/NLP solutions in 2013, ultimately beginning a research partnership with SRI International’s Artificial Intelligence Center.  From late 2016 to the end of 2019, LDC invested a half million in internal dollars to confirm the viability of using SRI’s proprietary AI to generate accurate and valid data to support student deeper learning goals.  Notwithstanding demonstrating the practical and conceptual benefit to the work, the absence of external funding stalled ongoing efforts.  However, with the release of ChatGPT in 2022, SRI reached out to suggest restarting our work as LLMs had significantly improved the potential for our collaborative work.  But while LLMs like ChatGPT are “out-of-the-box” remarkably capable of providing feedback to students, adapting them to the unique characteristics of student writing, and avoiding biases and errors that come with default models, requires tuning the existing model weights and the implementation of additional model components (e.g., model layers). By supplementing LLMs with additional model elements and training them on student data and validated scores, SRI will significantly improve upon foundation model accuracy, mitigate bias, and in tandem with LDC continuously improve its efficacy and impact on student writing over time.  The last innovation is standard software engineering sprint cycles coupled with parallel instructional resource sprint design cycles.  Throughout our SRI collaboration, LDC has continued to build CoreTools.LDC.Org to provide teacher-friendly online Open Educational Resources including online asynchronous learning microcredentials.  The asynchronous online learning includes a course that enables educators to calibrate their scoring of the SCALE analytic rubric.  However, the quantity of student work to calibrate against is small and does not include the more powerful inline discursive feedback.  CoreTools itself – which we would build upon – is a Ruby on Rails server, and Angular client running on Heroku hardware. We are currently in the process of converting from angular.js to the more modern Angular framework.

How do you know that this technology works?

Independent research (UCLA-CRESST) established that LDC doubled student learning in our USDOE i3 Validation grant in 100+ schools controlling for prior student, teacher, school state assessment performance, Free/Reduced Lunch, Sped, EL, ethnicity, etc.:  9.4 months in Los Angeles and 7.1 months in NYC.  See multiple independent research studies: https://ldc.org/impact/ For SRI Efficacy Research: Niekrasz, LDC - Automated Essay Scoring Using SRI’s ATSE Software, Project Interim Report #3 (SRI International – Advanced Analytics Lab)  https://ldc.org/aes-report/

After great teachers, research confirms that unbiased curriculum-embedded formative assessment drives more student improvement than any other resource. (Hattie 2009; Wiliams 2011)  However, no organization has created a scalable, accurate system providing feedback on deeper learning. Standardized selected response item banks designed by large-scale testing companies are too distant from, and lack contextualized connections to curriculum and instruction, nor to deeper student learning aligned to rigorous national standards. (Shepherd; Atkin) Multiple choice banks and low quality tasks do not respond to deeper learning needs. (Elmore 2010, Santelises 2015) Even putatively IRT-“validated” multiple choice and short constructed responses have little to do with the deep thinking skills contemplated by CCRS, NGSS, and C3 rigorous standards necessary to remain competitive in a global intellectual and economic market. The recent clamor around Science of Reading strategies only begins the student’s reading comprehension journey. Once students master phonics, phonemic awareness, etc. rigorous standards require developing student‘s extended writing in response to complex text – a deeper learning journey that requires connecting student data to detailed, scaffolded instructional strategies at the core of LDC’s statistically significant-confirmed framework.  National standards’ rigorous expectations and COVID have exacerbated this challenge, highlighting inequities in communities with high student-to-teacher ratios. An urban ELA teacher with 130 high needs ninth-grade students asking students to read informational text and write every month faces a student essay volume often too high for teachers to provide students with adequate feedback on even one round of their writing. Too many papers frustrate teacher efforts to collect and compile data to enable real-time grouping and differentiation. By the time students finish argumentative essays, the teacher has missed multiple opportunities to address skill deficits in the absence of usable formative data revealing student learning difficulties in their reading and writing. Likewise, students have also lost multiple opportunities to take ownership of their learning, build on teacher formative feedback, and practice the reading/writing skills essential for 21st-century learning. (Moreover scores alone are not as useful as inline discursive feedback on extended writing in response to complex text which this project would provide). More than ever, rigorous CCRS expectations for student achievement require teachers to provide frequent, direct, substantive feedback on students’ deep thinking work as manifested by frequent rigorous student writing. As Wiliam asserts, formative assessment is the “bridge between teaching and learning – only through some kind of assessment process can we decide whether instruction has had its intended effect.”  LDC has tested this SRI solution to achieve 85-87% nationally calibrated accuracy to date.

What is your approach to ensuring equity and combating bias in your implementation of AI?

As noted above, LDC will use both traditional innovations (analytic rubrics) and new technologies (Generative AI and SAS software development).  The SCALE analytic student rubric measures the student’s learning progress, signaling to both student and teacher the next step toward proficiency. For example, both the teacher and the student can easily understand the R2 learning progression and next proximal learning steps for the 3rd grade student scoring at 2nd grade level 2 with additional complex text based on the explicit SCALE analytic student rubric.  This will ensure the project builds upon the already established benefit of rubrics in that they are statistically more likely to mitigate bias in grading practices by standardizing scoring criteria, reducing the space for subjective assessment grading. This ultimately makes educators more effective in assessments and feedback built on years of research showing students learn best when they are exposed to relevant content and provided with opportunities to expand their critical literacy skills.  Even before Large Language Models, Machine Learning and Natural Language Processing demonstrated efficiency and accuracy to provide formative data and insights into student learning including reading comprehension and extended writing in response to complex text.  LDC began exploring ML/NLP solutions in 2013, ultimately beginning a research partnership with SRI International’s Artificial Intelligence Center.  From late 2016 to the end of 2019, LDC invested a half million in internal dollars to confirm the viability of using SRI’s proprietary AI to generate accurate and valid data to support student deeper learning goals.  Notwithstanding demonstrating the practical and conceptual benefit to the work, the absence of external funding stalled ongoing efforts.  However, with the release of ChatGPT in 2022, SRI reached out to suggest restarting our work as LLMs had significantly improved the potential for our collaborative work.  But while LLMs like ChatGPT are “out-of-the-box” remarkably capable of providing feedback to students, adapting them to the unique characteristics of student writing, and avoiding biases and errors that come with default models, requires tuning the existing model weights and the implementation of additional model components (e.g., model layers). By supplementing LLMs with additional model elements and training them on student data and validated scores, SRI will significantly improve upon foundation model accuracy, mitigate bias, and in tandem with LDC continuously improve its efficacy and impact on student writing over time.  The last innovation is standard software engineering sprint cycles coupled with parallel instructional resource sprint design cycles.  

How many people work on your solution team?

LDC has an internal team of 14 full time staff, but for our culturally responsive ELA, Science, and Social Studies (Civics) performance writing task development, we work with approximately 20 highly diverse (mostly not white or male) classroom educators across the country who provide consulting design service.  In addition, as noted above, we are currently collaborating on building both content and online culturally sustaining and relevant professional learning courses with WestEd and Common Good, among others.  We also work with Preva Learning to outsource some of our back end software engineering code writing.  Obviously, SCALE and SRI are collaborating partners.

How long have you been working on your solution?

December 2013 we began exploring Generative AI. We began collaborating with SRI in 2016

If your solution has a website or an app, provide the links here:

N.A. But see CoreTools.LDC.Org for our OER platform