The Ultimate Guide to Evaluating AI Tools in Your School
Set the tone for AI tools evaluations in your school. Explore practical steps that guide you through this evaluation process.
Purvee Chauhan
13min read
Share
The enthusiasm with AI in classrooms is almost tangible—it promises to revolutionize education with personalized learning, less paperwork for teachers, and a boost in student engagement. So far, we have seen great success stories, like a student who, despite struggling with communication, used an AI tutor to create a deeply thoughtful and engaging project. Or the school that became four times more productive thanks to detailed analytics on student reports. But — not all of them are wins.
In one case, a class tried using an AI chatbot for homework help, only to discover that it provided outdated or incorrect information, causing a student to base an entire science project on faulty data. Another school introduced AI facial recognition for attendance tracking, only to face immediate privacy concerns from parents, leading to its removal.
These hiccups tell us that while they have great potential to improve teaching and learning, we cannot simply hand AI tools over to the users and hope for the best.
Your role as a technology and school leader has become far more complex — you are responsible for not only shortlisting and implementing [1], but also meaningfully assessing the impact and effectiveness of these tools.
You don’t all have to be AI experts, but you do need a foundation of knowledge that can help you get started with evaluating these tools for your students and teachers.
But starting out can seem overwhelming. Where do you begin? This resource will provide leaders with the right tools that can help them draw purposeful next steps. You’ll find
Understanding what should you be evaluating, &
A 10-step process to conduct these evaluations
A Comprehensive Guide to Conducting an AI Tools Evaluation in Your School
Make a copy now!
Please rectify the errors in your form
+
You’re just one step away!
Fill in your information and hit that download button to get free access to this resource.
Hello Lifelong Learner!
We’re thankful for your interest in our Learn resources.
Please check your email for the information you requested.
Also, don’t forget to check out the best of Toddle:
Access the ultimate library of resources
Collaborate with your team to create your own units.
What’s more exciting is a fillable step-by-step comprehensive guide that will walk you through the entire process, from forming an evaluation team to analyzing feedback and making improvements. The hope is that you walk away feeling more confident and prepared to conduct these evaluations in your school. Let’s get started.
Understanding What Should You Be Evaluating
In this section, we’ll understand what objectives we need to consider when assessing the impact of AI tools in schools — basically how we set the intention of the evaluation. Later in the resource, we’ll detail the process of evaluation and share a step-by-step guide to make the evaluation a reality.
As you set the tone for the evaluation and start thinking about the objectives, there are a few key things every school leader should consider. One, it’s important to make sure that any AI tool aligns with the school’s vision and goals. Next, safety and privacy should be top priorities to keep everyone secure. And let’s not forget about inclusivity and accessibility—AI tools should work for all students and stakeholders.
We’ll unpack all of these in detail using UNICEF’s 3 Ps framework [2] and the 3 essential questions—
Keep in mind that you can also consider other factors like its usability and functionality or the environmental impact of the AI tool (Did you know? Writing a 100-word email using ChatGPT consumes 500 ml of water. Global AI water use by 2027 is predicted to be 4.2-6.6bn cubic meters. AI data centers have greater cooling demands as they are processing huge amounts of data [3]). The three factors mentioned here are just the foundational elements to start your evaluation journey.
Provision – Do Good
Essential Question #1: Are the AI tools improving education at my school and align with our academic vision?
When adopting any AI tool into your school, you might have checked whether it aligns with your educational goals and enhances the learning experience. Some time after its implementation, it is imperative to ask — Is the AI tool still aligned with our academic vision and delivering the educational outcomes we intended?
One of the ways to evaluate this effectiveness is the use of the SAMR framework —Substitution, Augmentation, Modification, and Redefinition [4]. The popular framework guides educators in using technology meaningfully, progressing from basic enhancement of tasks to creating transformative learning experiences, all with the intent to improve student engagement and outcomes.
While the SAMR framework is popular in shortlisting an AI tool, it can equally serve as an effective lens for evaluating the efficacy of AI tools in education. By assessing where an AI tool falls within the SAMR model, we can gauge how transformative the tool is in relation to teaching and learning practices [5]. This can help determine if the tool is meeting their pedagogical needs or if it requires further refinement to fully support the school vision.
Substitution: At this level, AI tools simply replace traditional methods without any significant changes. For instance, students might use AI to present research through a slideshow, but the AI doesn’t alter how they complete the task compared to using traditional tools.
Augmentation: Here, AI tools improve functionality but don’t change the core task. Students might use AI to brainstorm ideas or get feedback, while still creating all the content themselves. This approach enhances the process but keeps the essence of the task intact.
Modification: AI tools allow for significant changes in how tasks are performed. For example, students might use AI to analyze data for a science project and generate detailed reports. While AI contributes to the task, students need to ensure accuracy and properly cite sources, integrating AI’s capabilities to modify their approach.
Redefinition: At this transformative level, AI enables entirely new tasks that weren’t previously possible. Students could use AI-powered design software to create 3D models or simulations, fundamentally changing how they engage with and present their work. This level of integration redefines the learning experience, leveraging AI to transform the educational process.
PAUSE TO PONDER
As you reflect on the use of AI tool in your school, consider thinking about the following questions and including them as a part of your evaluation:
Does this tool align with my school’s academic vision? How does the AI tool fit into our long-term goals for technology integration?
At what level (Substitution, Augmentation, Modification, Redefinition) do we want to use the AI tool most frequently in our classrooms, and why?
What specific features of the AI tool are augmenting the learning experience, and are they useful for both teachers and students?
How does this AI tool modify the way students engage with the content? Does it allow for new forms of collaboration, creativity, or critical thinking?
Does the tool empower learners to become users, creators, and evaluators of AI?
Protection – Do No Harm
Essential Question #2: Are these AI tools safe for our schools and students?
Safety is as important a factor to consider as the alignment with school’s visions. We know that AI systems often process personal information such as student names, birth dates, and grades. While laws are in place to protect this data, breaches can still occur, posing risks to the safety of students, staff, and families. Ensuring robust data privacy and security measures is crucial and must be evaluated for each AI tool in use.
Here are three key things to consider as you prepare for an evaluation.
Data and privacy: As a school leader, it’s important to understand privacy laws and ensure that these regulations are in place to protect students’ rights and keep sensitive information secure.
FERPA (Family Educational Rights and Privacy Act) – AI tools must safeguard the privacy of student educational records and adhere to parental consent regulations. The educational institution must retain full control over the data at all times [6].
CIPA (Children’s Internet Protection Act) – Schools must ensure that AI content filters comply with the CIPA by blocking access to harmful or inappropriate content [7].
COPPA (Children’s Online Privacy Protection Act) – AI chatbots, personalized learning platforms, and other technologies that collect personal information and user data from children under 13 are required to obtain parental consent [8].
PPRA (Protection of Pupil Rights Amendment) – AI tools must comply with the Protection of Pupil Rights Amendment (PPRA) by obtaining parental consent before collecting personal information from students for surveys and research activities [9].
Apart from understanding the regulations mentioned above, administrations must take these additional steps:
ロ Update vendor contracts to reflect data privacy standards for AI tools, including consequences for policy violations.
ロ Track compliance with data privacy policies, processes, and procedures.
ロ De-identify or block the sharing of legally protected data with non-compliant AI tools.
ロ Flag the sharing of legally protected data with non-compliant AI tools.
ロ Ensure protections to prevent school information from being used to train non-compliant AI tools.
Age restrictions and parental consent: When integrating AI tools into educational settings, it’s essential to understand the legal age restrictions and consent requirements that govern their use.
ロ Most AI tools are designed for students aged 13-18. AI tools cannot be used by students under 13 in the US legally, so you’ll need to manage classroom technology accordingly, especially in mixed-age groups.
ロ If you’re using AI tools with students 13-18, explicit parental consent is required. Make sure your school has clear procedures for gathering this consent before using AI in the classroom. Find a sample parental consent form [11].
Understanding the small print: The terms of use for AI tools can be complicated, but reviewing them is essential.
ロ Since AI models are trained on vast amounts of data, some of which may be biased or inappropriate, it’s crucial to know what you’re agreeing to before bringing these tools into your school. For example, most AI tools are built on others such as OpenAI or Anthropic – it’s important to verify what the tool and its parent company/ LLM terms are.
ロ Many companies don’t accept liability for issues that arise from using their tools. Be aware of what protections are in place—and what aren’t—so your school can handle any problems that come up. Example: “… in no event will we be liable for any incidental, special, consequential, or indirect damages arising out of or relating to your use or inability to use the services, including, without limitation, damages for loss or corruption of data or programs,… [Tech Tool Company 2] assumes no liability or responsibility for any errors, mistakes, or inaccuracies, personal injury or property damage resulting from access to or use of our services…” (Retrieved on April 20, 2024)
ロ Some AI companies include indemnification clauses in their agreements, meaning the user (in this case, your school) is responsible for any claims or damages. You can’t use the tools without accepting these terms, so read carefully.
PAUSE TO PONDER
As you reflect on the safety of AI tool in your school, consider thinking about the following questions and including them as a part of your evaluation:
How is sensitive data, like student information, collected, stored, and protected by the AI tool?
Does the AI tool comply with privacy laws such as FERPA, COPPA, CIPA, and other relevant regulations?
What are the terms of use and privacy policies, and do they clearly outline how data is used, shared, or retained? Does the AI tool provide transparency in how it processes data and generates outputs?
What liability protections are in place, and does the tool include any indemnification clauses that shift risks to the school?
How frequently is the tool updated or monitored to address security vulnerabilities and new threats?
Participation – Include All
Essential Question #3: Do the AI tools consider the needs of all stakeholders?
Apart from its alignment with educational goals and safety, it’s crucial to consider how the AI tools affect everyone involved—students, teachers, and the broader school community. Bias, discrimination, and lack of access can significantly impact all members of a school community, making it crucial to address these concerns. Research has shown that children can absorb biases embedded in AI tools, which can have long-lasting consequences [12].
Administrators and school leaders should be aware that many AI usage policies place the responsibility for addressing bias and discrimination on the users—teachers, students, and schools—rather than on the companies that develop the tools. As such, schools must exercise heightened caution, as many AI providers do not accept responsibility for the data or potential harm caused by biases in their models.
To address this, administrators should:
ロ ensure that teachers have access to comprehensive guidance, training, and ongoing support to effectively and ethically use AI tools in the classroom. For example, professional development opportunities should be provided to equip educators with the skills needed to use AI responsibly.
ロ confirm that the AI tools being adopted are accessible to a diverse range of learners and comply with established accessibility guidelines.
ロ assess whether the tool can be adapted to various learning environments and is scalable for different class sizes, ensuring flexibility across different teaching settings.
ロ determine when and how students might develop AI literacy.
ロ determine whether the AI tools can seamlessly integrate with existing teaching practices and learning management systems to enhance, rather than disrupt, the school’s current infrastructure.
ロ are well-versed with the HIPAA (Health Insurance Portability and Accountability Act), IDEA, and Section 504 policies. – AI tools must not deny students with disabilities access to education opportunities [13, 14, 15].
As you begin the process of evaluating AI tools for your school, it’s essential to start with three fundamental factors and their guiding questions: Provision – Are the tools enhancing education and aligning with our academic vision?Protection – Are the tools safe for our schools and students?Participation – Do they address the needs of all stakeholders? With these in mind, the next step is understanding the key processes involved in conducting the evaluation and ensuring you are fully prepared to begin.
PAUSE TO PONDER
As you reflect on the inclusivity and accessibility of AI tool in your school, consider thinking about the following questions and including them as a part of your evaluation:
Do teachers receive adequate training on how to use the AI tool effectively and ethically?
Is the AI tool accessible to all students, including those with disabilities or special needs?
Does the tool comply with established accessibility guidelines and standards?
Can the AI tool be adapted to different learning environments and teaching styles?
How well does the AI tool integrate with current teaching practices and learning management systems?
10 Steps to Conduct an AI Tools Evaluation in Your School
Did you know? 25% of schools struggle to figure out if EdTech tools actually improve student outcomes, leading to challenges with long-term adoption [16]. This challenge often stems from lack of knowledge of robust, clear, and simple evaluation processes — without this evaluation, we risk getting stuck in a cycle of inefficiency with yet another tool that doesn’t quite hit the mark.
But creating a simple, clear yet effective evaluation framework can be challenging with all that’s going on at schools. Where do you even start with all that information?
To help out, we’ve put together 10 steps to guide you through this AI tools evaluation process [17-20]. We understand that school leaders have a lot on their plates and the evaluation process could seem like a tall order.
These steps have been organized to simplify the process, making it more manageable, accessible, and time-bound for K-12 environments. This format is based on the author’s insights from her graduate class, T-523 “Formative Evaluation of Educational Products by Dr. Christine Reich at the Harvard Graduate School of Education [21].
Note: This evaluation should be seen as a cyclical, iterative process. Upon completing one cycle, you may discover new questions that require further exploration in another round of investigation [22].
Step 1: Form the Evaluation Committee
Gather a team of teachers from different departments, administrators, IT staff, students and families to ensure a variety of perspectives – try keeping this process inclusive. Give each person clear responsibilities to make the process smooth and efficient.
Step 2: Define the Objectives and Scope
Define – Why are you conducting this evaluation? Think about 1) alignment with the school’s mission and educational goals, 2) safety, 3) inclusivity. Set a timeline to keep everything on track.
Step 3: Choose Evaluation Methods
Pick methods that gather both qualitative and quantitative data. Interviews, surveys, focus groups, and classroom observations can offer a range of insights. You might also compare students using the AI tool to those who aren’t and review artifacts like student work or teacher notes to dig deeper into its impact.
Step 4: Identify and Prepare Participants
Make sure you have a good mix of participants to run your evaluation with —teachers, students, and parents. Clearly explain the process and time commitment so everyone knows what to expect and feels prepared. The more diverse the group, the better the outcomes of the evaluation study would be. This group is different from the evaluation committee.
Step 5: Develop Evaluation Instruments
Design interview and survey questions that cover key topics and make sure they’re easy to understand. Plan how you’ll record responses—digitally or manually—and use simple, user-friendly tools.
Step 6: Collect Data
Now, it’s time to gather the data! Conduct interviews, distribute surveys, hold focus groups, and make your observations. Keep everything organized so it’s easy to analyze later.
Step 7: Analyze Data
Dive into qualitative data by coding and categorizing responses to spot themes and patterns. For quantitative data, look at trends and statistics. Analyzing artifacts like student work will help highlight any recurring outcomes.
Step 8: Prepare the Report
Bring your findings together in a clear report, using charts and graphs to make the data easy to understand. Highlight key insights and make recommendations for the next steps.
Step 9: Share Findings
Present your findings to the school community. Share the report with staff, students, and even the AI tool vendors, encouraging discussions and feedback on the results.
Step 10: Implement Recommendations
Now it’s time to act! Make any necessary improvements based on the evaluation and continue monitoring the AI tool to ensure it remains effective for your school.
Conduct an AI tools evaluation at your school with this comprehensive resource. This guide is straightforward and practical, walking you through each step of the evaluation with clear instructions. It also includes 30 evaluation questions you could consider as you assess your EdTech tools.
Make a copy and get started—everything you need to streamline the process is right here.
A Comprehensive Guide to Conducting an AI Tools Evaluation in Your School
Make a copy now!
Please rectify the errors in your form
+
You’re just one step away!
Fill in your information and hit that download button to get free access to this resource.
Hello Lifelong Learner!
We’re thankful for your interest in our Learn resources.
Please check your email for the information you requested.
Also, don’t forget to check out the best of Toddle:
Access the ultimate library of resources
Collaborate with your team to create your own units.
2. UNICEF. (2005). The Convention on the Rights of the Child: Three P’s framework – Provision, protection, participation. Retrieved from https://www.unicef.org
13. U.S. Department of Health and Human Services. HIPAA (Health Insurance Portability and Accountability Act). Retrieved from https://www.hhs.gov/hipaa/index.html
14. U.S. Department of Education. IDEA (Individuals with Disabilities Education Act). Retrieved from https://sites.ed.gov/idea/
17. Rieger, R. H., A. Sturgill (1999). Evaluating on-line environments: Tools for observing users and gathering feedback. New Directions for Evaluation, 1999(84):45-58.
18. Patton, M. Q. (2002). Qualitative research and evaluation methods. Thousand Oaks, CA, Sage Publications, Inc. (pgs. 339-391)
19. Patton, M. Q. (2008). Utilization-focused evaluation. Thousand Oaks, CA, Sage Publications, Inc. (Chapter 13)
20. Evaluation Association (2004). Guiding Principles for Evaluators. Fairhaven, MA, American Evaluation Association. http://www.eval.org/p/cm/ld/fid=51
21. Reich, C. (2019). T-523: Formative Evaluation of Educational Products. Harvard Graduate School of Education.
22. Pattison, S., Cohn, S., & Kollmann, L. (2014). Team-based inquiry: A practical guide for using evaluation to improve informal education experiences. Portland, OR: Oregon Museum of Science & Industry for the NISE Network. https://www.nisenet.org/catalog/team-based-inquiry-guide
Purvee Chauhan
Purvee Chauhan
Purvee brings over a decade of experience across teaching, instructional design, and product strategy. She has developed digital tools for educator growth and conducted STEM research in educational robotics, resulting in a book and a chapter. Her early work as a teacher and pedagogy specialist impacted diverse classrooms. Purvee holds a Master’s in Education from Harvard, specializing in Technology and Innovation, and an undergraduate engineering degree. She is passionate about creating meaningful and joyful learning experiences for all.
Disclaimer : The ideas and resources presented in this blog have been developed independently from and are not endorsed by the International Baccalaureate (IB)
