Case study: how AI helps to evaluate online lessons in an IT academy for children

Learning is not an easy process, even for adults. Let’s not talk about children and teenagers who are already mastering “computer” specialties. To make learning as enjoyable and fruitful as possible, the professionalism of teachers and consultants must be carefully monitored. Read how GoITeens IT Academy implemented a well-thought-out and comprehensive supervision system, reducing the time spent, and how Ringostat AI Supervisor helped.

Background

GoITeens is an international online academy that trains children and teenagers in various IT specialties. Students can take 12 full-fledged courses and another nine mini-courses. The curriculum covers various topics and specialties, from Python and GameDev to drawing to Leadership Workshop. Over 50,000 kids have gone through the academy in 8 years of operation. GoITeens’ approach to learning is genuinely large-scale — there are about 700 teachers who have taught more than 90,000 classes in just one year.

It is very important for the company to provide quality education and do everything possible to ensure that the future student chooses the right direction of study. To do this, it is necessary to control how the call center, the department of individual consultations, and the teachers work. However, with “manual” control of such a volume of data, the company inevitably encountered problems.

1. Large time commitment. The first phone conversation with a prospective student or his/her parents lasts about 10 minutes. Counseling and trial sessions — about an hour, classes — up to 1.5 hours, and sometimes 3. The team spent about two hours just listening to 10 conversations daily, including notes on the employee’s work.
2. The need for additional staff. For example, four assessors and one other specialist analyzed the work of the individual counseling department alone.
3. There is a risk of missing something important or, on the contrary, analyzing conversations that do not require attention. Not all dialogs need to be monitored — after all, they may be error-free. However, there is no way to understand this at once. Therefore, assessors had to listen to dialogues and analyze lessons selectively. So, some time was wasted on controlling something already good. And vice versa, there was always the possibility of missing cases that required intervention.

GoITeens was looking for a tool that would reduce the time spent on supervision and allow automation of this process. The solution was artificial intelligence — Ringostat AI Supervisor. As GoITeens management notes: “AI technology is exciting and promising, and it can be used now. It saves employees’ time, speeds up work processes by time, and helps to develop and improve all company departments”.

Based on the Ringostat AI Supervisor GoITeens results, we wrote “Case study: artificial intelligence instead of a supervisor in an IT academy for children” in April 2024. However, since then, the company has taken a big step forward by improving the control and customization of Ringostat AI Supervisor for its project.

What is Ringostat AI Supervisor, and the work of which GoITeens departments is analyzed with its help

Ringostat AI Supervisor is a service where artificial intelligence is customized for a specific business. The user provides our developers with ToRs, conversation scripts, and company internal rules. Or, if this is not available, the user describes its processes and explains what kind of analysis he needs. As a result, the company gets a digital assistant that is maximally adapted to the needs of the business and has a good understanding of its peculiarities.

GoITeens requires this kind of customization because there are many nuances to the academy’s work that are important to consider when evaluating conversations and lessons. First and foremost is the diversity of departments and their conversation scenarios.

1. The call center receives customer calls from the website, advertisements, etc. Its task is to provide an initial consultation to the client and interest him/her enough to sign up for the first trial lesson. If a person already knows GoITeens well and does not need a trial lesson, the operator should switch them to a sales manager.
2. The Individual Counseling Department communicates more thoroughly with the prospective student and his/her parents during an online meeting. The methodologist carefully asks about experience, preferences, and skills and helps choose an appropriate course if the child is undecided. He also conducts a trial lesson.
3. The sales department communicates with customers by phone and calls back users who leave requests on the GoITeens website. In addition, the sales department contacts customers who, for whatever reason, did not enroll in the course immediately after the trial session.
4. Instructors teach online classes to students via video.

Conversation scenarios differ not only by department but also by topic of discussion during the dialog. For example, a lesson on project preparation has a very different structure than, say, a design class. Group and individual lessons also need to be evaluated differently. Therefore, the AI must understand which department’s dialog and topic should be analyzed. Accordingly, it will use the necessary evaluation criteria.

How is the assessment carried out

One of the pros of Ringostat AI Supervisor is that it analyzes 100% of customer conversations. This applies not only to call center and sales conversations but also to consultations and lessons during online meetings in Google Meet. For this purpose, videos uploaded to Google Drive are uploaded to Ringostat, and then analyzed by artificial intelligence. We will focus on how the learning process is evaluated.

Dialogue transcription

Artificial intelligence turns all conversations and video meetings into text. This means accessors don’t have to listen to or watch them. It is enough to view a text transcript containing the dialog analysis quickly. The convenience is also that the specialist controlling the department’s work can search for the necessary word in the dialog and read the lines where it appears. For example, it will be helpful if an employee needs to analyze how the teacher checked the homework. It is enough to enter this word combination into the search and read the necessary dialog fragment.

If an employee clicks on the text of a particular line, the audio of the dialog will start playing from the moment it is spoken. This can help to understand the tone of voice of the teacher.

Overall grade for the lesson and a brief sense of the conversation

GoITeens has an elaborate system for evaluating conversations, depending on which department communicates with the student and on what topic. In particular, it consists of criteria the conversation must meet — specific actions and replicas of the employee. For example, conducting a lesson on application development is evaluated according to almost 20 criteria.

The analysis also contains a summary that can be read instead of analyzing the text of the dialogue in its entirety. From it, the accessor can learn:

• what the lesson was about;
• what was discussed during it;
• what the teacher did right during the lesson;
• what points could be improved etc.

Assessment of adherence to the lesson structure

The AI “knows” the ideal lesson structure and easily notices if the teacher deviates from it. If the specialist has performed all the mandatory actions at each stage of the lesson, it will get 5 points. For example, AI makes sure that the teacher:

• first actualized the knowledge and checked the homework;
• made an introduction to the lesson and summarized it at the end;
• took apart the homework assignment and briefed him on it;
• made an interesting announcement of the next lesson to make the students want to attend;
• adhered to the sequence of lesson elements approved in the online academy.

Adherence to communication criteria

Communicating with children requires special skills and the ability to keep them always interested in the subject of the lesson. Therefore, GoITeens pays great attention to how the teacher behaves while communicating with students. In particular, the artificial intelligence gives a grade for:

• the teacher’s disposition to the lesson;
• organization and discipline;
• individualized approach to students;
• quality of explanation;
• knowledge of the language — the teacher should not take too long to choose words, use jargon, etc.

In addition, the teacher should address each student by name throughout the lesson, answer questions thoroughly, and engage students in discussion through open-ended questions. The teacher should also praise students when appropriate, and praise should be personalized and varied.

If a specialist fails to adhere to the criteria, he/she will not receive a score. The list of teacher’s mistakes is collected in a separate block. So the team that monitors the conversations can instantly understand what went wrong in the lesson.

Detailed assessment of the teacher’s knowledge, professional preparation, and other related criteria

The lesson is not only rated according to the criteria we mentioned above. The call analysis contains a detailed description of the teacher’s work, described through the prism of a specific criterion.

1. Knowledge and experience. Artificial Intelligence describes how knowledgeable the teacher is about the topic he or she was teaching during the lesson. And whether his or her explanations were understandable and aligned with industry standards.
2. Relevance and teaching methods. AI checks whether the examples are relevant and practical, and whether the content aligns with current practices. How was the learning process organized, and did the teacher prepare learning materials in advance.
3. Dynamism and interactivity of the lesson. It is crucial that the learning experience is engaging and that the student does not get bored. Therefore, AI evaluates how much balance is maintained between theory and practice during the lesson. The analysis also considers how actively the teacher interacted with the children.
4. Professional preparation and behavior of the teacher. In the analysis results, accessor read how well the teacher was prepared for the lesson and whether the teacher gave practical examples.

Lesson Details and Details

Artificial intelligence does not just give an assessment and summarize the situation but also justifies its conclusions. Ringostat AI Supervisor, for example, shows in which fragment of an online meeting the teacher summarizes the lesson. For each step, a time code is given — the time when it started in a given dialog. Below is an explanation of what was happening at that particular stage. If you click on the time code, the audio of the conversation automatically scrolls to the desired fragment. This is very convenient if a supervisor sees a teacher with a low grade for a particular stage and wants to analyze it.

Assessors also separately assessed the artificial intelligence, which shows which students are present in class and how often the instructor has addressed them.

How the GoITeens team assesses the quality of conversations and how the process has gotten easier over time

To get such an analysis, GoITeens specialists responsible for monitoring first select the proper dialogs. For example, conversations that received a low rating are taken into account. Or a certain number of dialogs from a specific department that needs to be monitored regularly.

Previously, accessors had to copy a link to the desired lesson or consultation to analyze each dialog and paste it into the interface for analysis. The employee also had to specify information about the lesson and many additional parameters. This required a lot of time, so our developers made a separate interface that allows you to analyze dialogs in bulk. The GoITeens manager now has enough:

• collect dialogs that require control in one table;
• specify lesson topics, scenarios, and teachers’ or methodologists’ data — so that AI applies the proper criteria;
• download the table to the interface for analysis.

“On the way out,” the online academy gets the results of analyzing all the right conversations without any unnecessary actions.

GoITeens also has access to a tool that allows them to edit their own criteria for evaluating conversations. For example, here, employees can describe what actions a teacher or facilitator should perform at each stage of a dialog or lesson on a specific topic. The accessor can also upload a script of the conversation and set the evaluation of each item. Thus, the online academy does not have to contact tech support whenever they need to edit existing settings or add additional ones when a new course appears. All of this can be done independently.

How artificial intelligence data is helping to improve training and counseling

It is crucial for the company that the conversations of each department are analyzed by specialists who are well-versed in specific department’s processes. That is why the work of teachers is analyzed by some assessors, the work of individual counseling methodologists by others, etc. Quality control specialists must analyze about 300 consultations and 650 lessons every month. That is, one assessor analyzes 100–140 lessons. And that’s not to mention that call center and sales calls are also monitored. In this respect, artificial intelligence greatly simplifies the work. After all, it gives a grade or indicates in which conversations or lessons there are problematic moments.

Thanks to this, the assessor may not spend a lot of time on okay conversations or milestones and focus only on problems. When the specialist notices and analyzes the worker’s mistakes, he or she composes feedback for the department’s team leader. In this way, academy workers can improve their skills and professionalism. If the individual counseling methodologist or the teacher does not agree with the feedback, he or she can file an appeal. In this case, the specialist analyzes the conversation once again, paying attention to the employee’s objections.

The GoITeens assessors also meet weekly to discuss the points they noticed during the analysis and ways to improve the work. In this way, representatives of different departments can share good practices and improve the work of online academy.

Customer Testimonial: How Ringostat AI Supervisor saved time and reduced the team that is involved in supervision

“I used to have a team of four people working with me who only evaluated the counseling sessions. Thanks to AI, the work of the individual counseling department can now be supervised by me alone. Colleagues who used to be assessors have now moved to other departments, strengthening them.

For example, it takes me 3 hours to prepare 95 videos of hours-long counseling sessions, and it also takes 3 hours for AI to analyze these recordings. Before, without AI, this work used to take a whole month. Plus, we use the Claude Sonnet language model, and it’s even faster. If you compare it to how we worked before AI, it used to take more than 30 minutes to analyze just one video. “

Maxim VoytenkoAI manager GoITeens

“Thanks to Ringostat AI Supervisor, we have ready-made reports detailing how the training took place. This helps both assessors check the lesson and team leaders check appeals.

We also flexibly customize the artificial intelligence to meet our needs. We added additional program categories: teacher knowledge and experience, relevance and teaching methods, lesson dynamism and interactivity, teacher’s professional preparation, and behavior. This allowed us to check the technical component of the lesson as well. The supervisors who check the teachers’ work have accelerated their work by 10%. Thus, they can evaluate teachers with low scores twice.”

Anastasia DudkinaTeamLead Quality Control GoITeens