Quill.org

What makes your solution innovative?

Quill differentiates itself from other AI-powered assessment tools through its extensive and culturally responsive AI customization process and its focus on assessing student writing through open-ended writing prompts, rather than multiple-choice questions, to provide a comprehensive understanding of students' knowledge, skills, and abilities. While many other chatbot tutoring tools simply deliver the output of a model as-is or charge large subscription fees as for-profit organizations, Quill is a nonprofit training its AI models using an extensive set of exemplar responses, custom directions, teacher feedback, and a grading rubric – all to ensure that our feedback is always relevant, culturally responsive, and rigorous for the priority learners in the communities we seek to serve.

Quill has also been building its own AI models for the past five years. While Generative AI-enabled assessment is a relatively new space, building AI-powered tools that prioritize Responsible AI principles and ensure data and privacy security has been core of Quill’s products for many years. As Quill builds new tools and improves existing tools using Generative AI, we are re-examining how to actualize these best practices within the context of this new technology. Another key goal of ours is to build a new framework outlining best practices for responsible Generative AI development that we then open source for the entire education community.

Describe the core AI and other technology that powers your solution.

Quill’s Reading for Evidence tool currently uses Predictive AI (Machine Learning) to analyze whether a student's written response used precise evidence and logical reasoning and then provides custom, targeted feedback on how to revise and strengthen their sentence. For each writing prompt, Quill first collects 1,000 sample responses, looking at all of the different ways a student can respond to a prompt. Quill collects this data consensually and then analyzes it. Based on these responses, a team of educators map out up to 50 guiding questions and suggestions that could help the student revise their response and improve their skills based on the different ways that students respond to the prompt. Each piece of feedback is linked to a series of student responses, and Quill uses that dataset to train a fine-tuned AI model to serve the appropriate feedback. Every time a student completes the writing prompt, the AI predicts which piece of feedback is most appropriate based on how similar the student’s responses are to the responses in the AI model.

Quill will be migrating from using Predictive AI to Generative AI by the end of 2024 for its Quill Reading for Evidence tool. As a nonprofit, Quill has always been specifically dedicated to designing AI customized for schools serving primarily Black students, Latino students, and all students experiencing poverty. To continue doing so, Quill is creating custom Generative AI prompts for every student writing activity rather than simply delivering the output of the AI as-is. The Quill team collects 50-100 examples of real student writing from priority learners (anonymized and aggregated), and a team of educators writes custom feedback for each response. Quill then feeds all of the examples of the real student data, paired with teacher responses, to the AI model. Quill’s approach is a “thick wrapper” approach of highly-customized systems rather than a “thin wrapper” one used by many other organizations so that Quill can provide effective feedback for priority learners while addressing algorithmic bias. In using Generative AI, it is meant to do the job of the formative assessments we already provide and set the stage for future AI-powered product development. For Generative AI, Quill is developing its own LLM prompts and is currently testing different models to generate feedback based on these prompts. These LLMs under evaluation include Google’s Gemini Ultra and OpenAI’s GPT-4. Quill is also evaluating open-source models that Quill would self-host, including Llama 3 and Mistral.

How do you know that this technology works?

Quill’s tools are designed around a claim-testing pedagogical approach and are classified as a type of formative assessment. Quill’s approach is rooted in giving individualized writing feedback to learners in the moment rather than assessing achievement through multiple-choice tests, with a targeted focus on supporting learners facing the biggest barriers to opportunity. Asking students to complete “but,” “because,” and “so” statements to demonstrate and enhance their understanding of a text is central to our approach, and Quill Reading for Evidence’s ability to incorporate informational writing across the school day supports students’ writing, reading, and critical thinking skills across a wide range of subjects. While Quill’s first five tools focus on improving student writing in the ELA classroom through grammar, syntax, and punctuation activities, Quill’s AI-powered Quill Reading for Evidence tool focuses on getting learners to demonstrate their understanding of non-fiction texts across subject areas through writing claims about the text. Because Reading For Evidence uses non-fiction texts across a wide array of topics, its implementation will stretch beyond ELA and into social studies and science classrooms.

While Quill is not a summative assessment platform, we use psychometrically validated assessments to measure the effectiveness of our own program. These tests enable us to identify how students are learning on the platform. Working with the College Board in 2022, Quill used a paragraph revision task and an SAT essay prompt to measure the effect of a Quill.org intervention on paragraph revision over the course of an 11-week RCT. Pre-test and post-test measures included 1) paragraph revision task, 2) an SAT essay, and 3) a scale measuring writing motivation. The intervention lasted four weeks, totaling 4 hours, and students completed Quill activities daily. A posttest was administered immediately at the close of the intervention period, two weeks later, and for a final time two months later. After controlling for pretest differences, the treatment group scored significantly higher at immediate, two-week, and two-month posttest compared to the control condition (all p’s < .05, Cohen’s d’s > 0.80). The gains that students made through using Quill were significant and persisted over a period of two months to showcase sustained writing growth.

Similarly, Mathematica did a study looking at the transference of student work on Quill to essay writing outside the tool for the Bill & Melinda Gates Foundation. We would be happy to share more information on that study with program officers directly.

What is your approach to ensuring equity and combating bias in your implementation of AI?

Quill’s AI is specifically developed for students attending Title-1 schools and those facing the biggest barriers to opportunity. We customize our AI by anticipating responses from priority learners and then map their responses to teacher feedback in our system – this ensures that we both understand the diverse needs of the learners we seek to serve and actively develop our tools to best respond to those needs. We are training our AI models using real teacher feedback responses and mapping those to anticipated responses from students attending low-income, Title-1 schools. These pairs of student responses and teacher feedback serve as a rubric that tells the AI how it should assess the student’s writing and what feedback is appropriate for this particular activity. At the 2024 ASU-GSV conference and AIR Show, Quill did not find any other organizations that specifically use real student writing from Title-1 schools to customize their AI-generated feedback. 

To achieve its mission of serving priority learners specifically, Quill also continually conducts data analyses to ensure its writing tools are meeting student needs. Through Quill’s current approach to AI development, Quill has already evaluated 250,000 sentences and counting — a team of six former educators, full-time employees at Quill, spend more than a thousand hours each per year evaluating student writing to see where students have the most success and where they are getting stuck. Quill’s AI systems are then retrained based on that data.

This approach extends beyond mere analysis – it encompasses a continuous cycle of feedback, retraining, and improvement of AI models to best support priority learners. By continually evaluating where students excel and where they encounter difficulties while using the program, Quill ensures that its AI tools are not static but evolve to address common student errors and misconceptions. This commitment is further underscored by the fact that 90% of Quill's engineering efforts are directed toward developing sophisticated evaluation tools. Beyond merely assessing student performance, these assessment tools are about understanding the nuances of learning and retraining the AI to mitigate bias and best meet the needs of Quill’s learners. To train the AI, every Quill LLM prompt is developed with a baseline of at least 50 examples of student writing, providing a rich dataset for refining the AI’s feedback mechanisms and then continually reevaluating from that starting point. This approach improves the accuracy of feedback and ensures that the feedback delivered is meaningful and relevant to students.

Quill’s commitment to promoting equity and combatting bias also extends to its work with experts in culturally responsive pedagogy. Quill works with an educational consultant specializing in DEI best practices who evaluates every text the team develops for the Reading for Evidence tool. Every text is sent to this external reviewer, who gives Quill multiple rounds of feedback to ensure that the texts are culturally relevant and appropriate. By incorporating teacher, student, and DEI expert perspectives into the custom content development process, Quill ensures its AI feedback is accurate and inclusive.

How many people work on your solution team?

Full time - 24

Part time - 0

Contractors - 5

How long have you been working on your solution?

Quill has built its own AI models for the last 5 years. Reading for Evidence launched in 2022.

If your solution has a website or an app, provide the links here:

https://www.quill.org/

What is your plan for being pilot ready (if not already) within the next year, and what evidence can you provide that you are on track to meet your goals?

N/A - the Quill team is ready for this next phase of scale to bring its AI-powered assessment tools to even more priority learners across the US.

What are your plans to ensure your solution is available, accessible, and affordable to priority learners at scale?

As a nonprofit on a mission to support students facing the biggest barriers to opportunity, Quill’s tools have always been available, accessible, and affordable to priority learners. Quill’s tools are, and will always be, available to any teacher and student free of charge. Districts and schools may choose to purchase Quill Premium if they would like access to advanced reporting and professional development for their educators, but our core tools and products are always available at no cost to individual teachers. 

As for-profit organizations tend to have paywalls that make full access to their materials difficult for all educators to access, Quill, as a non-profit, provides a free, high-quality alternative. To provide our learning activities for free, we create our own AI models that cost 10 to 20 cents per student per year in server hosting fees. With the launch of GPT-4, other EdTech organizations have started building AI-powered learning tools. While these new technologies are impressive, they often come at a server hosting cost of $200 to $300 per student per year. As a result, many new AI-powered tools are limited to schools that can afford the high subscription fees. For Quill, it is not enough to merely create powerful tools – we must be able to build powerful and cost-effective tools so that all learners can benefit from them.

With accessibility and supporting differentiation in mind, Quill also has a variety of supports available for students with diverse learning needs. Through the Quill Diagnostic, teachers can easily determine which of 700+ Quill activities are most appropriate for each student and quickly assign them. Quill has also developed materials to support Multilingual Learners specifically –  our diagnostics include translations for 16 languages, including Spanish, Mandarin, French, Vietnamese, Arabic, and Hindi.

Why are you applying to the Learner//Meets//Future Challenge?

Quill Reading for Evidence enables the formative assessment of open-ended writing responses and provides students with continuous feedback during the learning process so they can build their argumentation skills. With the rapid acceleration of Generative AI’s accuracy and coherency over the past six months, there is an opportunity to move Quill’s current and future product development to Generative AI with the support of mission-driven partners in order to reach our goal of serving 10 million priority learners per year in the next five years. 

First, while Quill has built a large library of Quill Reading for Evidence activities for English teachers, we have limited science and social studies offerings. Research is clear that students should read and write in every subject – to support this goal, Quill needs further investment support in order to build out new activities across curricular subjects. Secondly, Quill’s AI is limited to providing feedback on individual sentences. The idea of deepening Quill’s impact to support paragraph and essay-level writing is particularly exciting to us and will require financial and technical support from like-minded partners to support the development and scale of new tools in the future.

In which of the following areas do you most need partners or support?

• Monitoring & Evaluation (e.g. collecting.using data, measuring impact)
• Public Relations (e.g. branding/marketing strategy, social and global media)
• Technology (e.g. software or hardware, web development/design)