EducAid & Fab Data

Learning by Doing: Pilot project

Overview

Enhancing teachers’ potential is crucial for shaping the future of our children and society. However, many countries, Sierra Leone included, struggle to provide adequate and sufficient training for teachers.

In Sierra Leone, financial constraints mean that only four in ten teachers are employed by the government, leading the system to rely on communities to hire volunteer teachers and contribute towards their salaries. Beyond teacher shortages, over 25% of teachers lack formal education training. As a result, foundational literacy and numeracy is a challenge for most pupils.

Fab Data, a tech start-up, developed TheTeacher.AI, an AI-driven chatbot designed to support teachers. TheTeacher.AI integrates with WhatsApp, offering teachers with immediate access to subject matter clarifications and on-demand opportunities to develop their teaching skills. By offering this support at a low cost, the potential impact is huge!

Aims of the pilot

Fab Data and EducAid (a Sierra Leone-based NGO) partnered to explore how this tool can be scaled effectively and affordably.

To explore this, they’ve planned to offer varying degrees of training support on using the chatbot to groups of teachers with similar characteristics who have signed up for using the chatbot, ranging from basic access with no training to in-school coaching, supported by a series of nudges along the way. Their main focus is to shed light on the optimal level of support teachers require to effectively use the chatbot for their professional development.

From their experience with previous, smaller-scale deployments, they’ve observed that much of the usage (48%) involved seeking clarification of concepts and a smaller proportion around questions about lesson planning (21%).

Building on this, they are now keen to explore different levels of support influence the usage and quality of interactions with the chatbot. A second key aim is assessing how the chatbot answers improve when it is augmented with Sierra Leone’s knowledge base, serving as additional resource for information retrieval.

How the team will determine the level of support needed by teachers?

Around 195 teachers and headteachers from approximately 55 schools signed-up for the chatbot in Port Loko district. They’ve organised these participants into four distinct groups, ensuring a balanced representation in terms of gender, the grade levels they teach, their qualifications, and number of headteachers. Starting February 17th, each group (except Group 4) will receive access to the chatbot and varying levels of support:

Group 1: immediate access to chatbot + prompt training.
Group 2: immediate access to chatbot + prompt training + lesson planning and pedagogy training.
Group 3: immediate access to chatbot + prompt training + lesson planning and pedagogy training + in-school coaching.
Group 4: late-access to chatbot

In addition to analysing usage patterns and quality of interactions with our chatbot, they’ll be closely monitoring teachers’ perceptions and awareness of AI technology, their attitudes towards teaching, their satisfaction on the job, and their views on barriers to continuous professional development.

Group 4 is their control group, which serves two purposes for the study. First, because they will receive access to the chatbot later than the other groups, meaning this early period allows them to assess the impact of having access versus not having access to the chatbot on the measures described above. Second, because they will receive no training when they do receive access, this late period allows them to assess the impact of no training against the different types of training received by groups 1, 2, and 3 on usage patterns and quality of interactions with the chatbot.

How will the team compare the performance of the chatbot?

The team will be conducting A/B testing with a select group of teachers in a controlled ‘lab’ environment. This part of the pilot will involve a blind comparison between two iterations of the chatbot: the current ‘live’ version powered by GPT-3.5-turbo and an advanced prototype.

This prototype version enhances the chatbot with a Retrieval Augmented Generation (RAG) system. This system integrates the chatbot with a vector-based database containing educational materials from the MBSSE Knowledge Platform.

The intention is to determine if enriching the chatbot’s responses with more relevant and contextual information improves the answers teachers receive.

Project timeline

7 months (January to July 2024)

January: Inception phase
FebruaryJune: Implementation phase
July: Analysis and report writing

Current progress

On 27th January, the team conducted the training of trainers in Port Loko on using TheTeacher.AI chatbot.

On 17th February, Groups 1 to 3 teachers who attended the training sessions were given access to the chatbot.

Meet the team

Dr Oliver Garrod is the Chief Data Scientist of Fab Data.

Erin Northey is the Chief Executive of EducAid.

What are you most excited about the potential of AI?

Erin Northey: We are hopeful that AI tools can help bridge the poverty gap between those who have and those who do not have. We hope to see how AI enhanced tools can improve the quality of teaching and learning in all classrooms across the country, overcoming barriers of remoteness, poverty and education levels of the teachers. We hope technology (such as that generated using voice-to-voice technology) contributes to reduce barriers to those experiencing disabilities and increase inclusion.

Through more accurate data use and AI-enhanced professional development, we believe that disadvantaged communities can leapfrog over many of the development stages that other more privileged communities have had to go through.

We are also eager to learn about the potential spill-over effect to see how AI promotes digital literacy, an asset for everyone. AI can also lead to cost-efficiency in areas such as curriculum development, assessment and even administrative tasks in the future.

What are you hoping to share with the community?

Dr Oliver Garrod: We’ve already made public some of our code on AI-for-Education.org, such as Fab Data’s LLM library. We are looking forward to sharing our conversation analysis code when we reach that phase. This will include analysis on:

– Types of interactions
– Content of interactions
– Usage patterns

We’ll also be sharing our training materials used in this pilot so they can become widely available for public use.

What are you hoping to get from the community?

Erin Northey: We look forward to learning and sharing how people are overcoming infrastructure challenges. We would like our peers to share ideas about how they are using AI to improve education and efforts on the way to make localised content available for AI tools to use.

It would also be beneficial to engage communities in a dialogue about the ethical concerns related to AI and collaborate to develop ethical guidelines and safeguarding procedures for AI projects.

Assessing level of support needed by teachers to use AI chatbots successfully.

Pilot information
Large Language Models
NLP: Generation
NLP: Question

Based in Sierra Leone (EducAid) and UK (Fab Data)

U

Developing products for Sierra Leone

Contact – EducAid

Erin Northey

i

Chief Executive

Contact – Fab Data

Dr Oliver Garrod

i

Chief Data Scientist

We’ve already made public some of our codes on AI-for-Education.org, such as Fab Data’s LLM library. We are looking forward to sharing our conversation analysis codes when we reach that phase. 

Dr Oliver Garrod

Chief Data Scientist, Fab Data

AI-driven support to teachers in the most remote areas – TheTeacher.AI provides instant pedagogical advice, lesson plans and learning assessments.

Learning By Doing

We are providing small grants to support the development of AI products & components in LMICs. We know that innovation investment is high-risk. Our aim is that our community can benefit from the lessons learned in these pilots – what works and what doesn’t.

Learn more about our pilot projects here. We will be following each project and reporting on key learnings.

Sign Up

Join our mailing list to keep up to date with news and events.

Community          Knowledge          FAQ
          Privacy Policy

AI-for-Education.org was set up by Fab Inc. in partnership with Team4Tech. We are grateful to the Bill & Melinda Gates Foundation and the Jacobs Foundation for their support.

Powered by FabData.IO