This article was originally published here on RTI’s SHARED blog.
Before the coronavirus 2019 (COVID-19) pandemic struck, the USAID-funded Malawi Early Grade Reading Improvement Activity (MERIT) had a plan to train 17,600 instructional leaders in a 2-day refresher course on how to facilitate teacher learning circles. When the pandemic’s travel restrictions prevented us from proceeding as planned, the MERIT team quickly began thinking of ways to provide training remotely. MERIT chose interactive voice response (IVR) as the best option, combined with small group discussions. With IVR, the system automatically calls training participants, who listen to a pre-recorded message about the training content. The pre-recorded messages can include questions about the content or quizzes, which the participant can respond to using the keypad on their phones. Simple? In theory. In practice, this was an ambitious undertaking, and we learned a lot along the way. Below we provide our top 10 lessons learned for others to consider.
Introduction
Malawi, like many other countries in sub-Saharan Africa, does not have reliable, widespread access to high-speed internet. However, MERIT knew program educators had access to cell phones as they had provided their phone numbers during registration for a face-to-face training held earlier in the year. What MERIT did not know was how many had smartphones that could access messaging apps like WhatsApp. To answer that question, we conducted a quick survey of instructional leaders in a nationally representative sample of 86 schools, which revealed that at least 86% of schools had at least one instructional leader with a smartphone.
Armed with these survey results, MERIT decided to use multiple modalities—IVR, SMS, and WhatsApp —to reach as many teachers as possible. IVR was attractive because it required only a basic cell phone and allows for different audio content (e.g., dialogues, role-plays, games, etc.). What IVR does not provide are visual supports, or opportunities for discussion and interaction with the content and with peers—for that we used WhatsApp.
After MERIT decided on a specific technology, the next step was to convert the materials we had developed for the face-to-face training into messages suited for IVR. In concert with the IVR messages, we also developed discussion questions for WhatsApp. Since this was our first remote training, we needed to monitor the implementation—another area where IVR shines. We could get real-time information on pick-up rates, completion rates, and answers to questions. Because we had some demographic data attached to phone numbers (gender, region, position), we could also analyze responses by these characteristics. Outside of the real-time reporting from IVR, we conducted a telephone survey of targeted participants to learn about their experience with the training.
On June 7, four weeks after MERIT signed the contract with our partner Viamo(link is external) (link is external), which provided the IVR platform and technical help on content design, we launched a 2-week remote training for instructional leaders. The first week of training included an initial SMS message to all 17,600 targeted participants about the upcoming training, an introductory phone call (Sunday), three content messages (Monday, Tuesday, Wednesday), WhatsApp Discussion (Thursday and Friday), and a call reviewing the week’s content (Saturday). We continued this schedule into the second week.
Lessons Learned
Participation
1. IVR is a viable way to provide virtual training to large numbers of participants. Using IVR, MERIT reached over 15,000 instructional leaders during the two-week long training. The average pick-up rate per lesson was 68%, with a range of 62% to 76%. Of those who picked up the calls, 85% listened to the end of the call, and 52% completed the key questions. The percentage of participants who responded to questions in Week 1 started very low, at 39%, but after providing participants information via WhatsApp on how to use their keypads to answer questions, the response rate increased to over 50% by Week 2. While the medium might not be suitable for all kinds of training content, for this target group, for whom the training was largely a refresher, the IVR format provided the minimum input and interactivity necessary to reinforce the concepts.
2. There was little difference in participation rates by gender, but urban residents had higher participation rates than did rural residents. Although we worried about gender bias in the technology’s use, the data did not show a significant gender difference: participation rates were slightly higher for women (87%) than men (85%). However, there were differences in participation rates by location. The highest participation rates were in the three most urban locales: Blantyre Urban (98%), Blantyre City (95%), and Lilongwe City (94%). This is likely a result of better and more reliable network coverage in these areas.
3. Call-backs helped. Participants who missed the scheduled call could call back at a time that was more convenient for them, and many made use of the opportunity. This was important because teachers who missed a call because of a poor phone network, a dead battery, or conflicting tasks could call back—still at no cost to them—to listen to the message before the release of the next scheduled message. That said, another lesson was that the pace of the Monday–Tuesday–Wednesday schedule of calls might have been too rapid. Allowing at least 1 or 2 days between calls would have allowed more time for call-backs.
4. Government officials are important advocates for increasing participation. Besides sending SMS messages to teachers about the training, Primary Education Advisors (PEAs)–ministry officials who support teachers–used WhatsApp to communicate to teachers within their school cluster about participating in the program. Training participants needed a lot of advance notice to engage with the training (see “Logistics” section, below).
5. Use multiple communication channels to increase engagement. Though the primary vehicle for training was IVR, MERIT used several channels to communicate with training participants. At the start of the training, we used SMS messages to inform participants of the upcoming training. Those participants who were part of the zonal WhatsApp groups received supplementary information, like PDF infographics, to reinforce the content, and took part in discussions based on reflection questions we shared. And while MERIT did not plan this, we learned of small groups of teachers who came together to reflect on the discussion questions. We learned that participants were also recording the phone messages and sharing them with other teachers. Finally, for participants who, for whatever reason, could not take part in the IVR program, we made the content available on-demand, by building the raw materials into the Tangerine™ platform. Users can now access this content online, make an offline copy on their mobile device, and revisit any time. (See also this blog post about “hacking” the Tangerine data collection system for other purposes, including content delivery.)
Logistics
6. Work with telecom companies beforehand to ensure that their networks can handle the task. MERIT had originally planned to have all 17,600 calls made to participants at 5:00 pm on each of the training days. However, the network infrastructure could not handle the volume of calls, and calls were being queued; for example, some calls we scheduled for delivery on Monday did not go through until Tuesday. This caused some disruption during Week 1 as teachers were not receiving calls on the scheduled date. We solved this problem by staggering the time of message delivery by education divisions. Malawi has six education divisions, and in Week 2, each division was assigned a time window within which it would receive the calls, and we communicated this schedule via WhatsApp. Because of this change, Week 2 calls went out according to schedule.
7. Prepare to spend significant time and effort to prepare for the launch of the IVR. To ensure a successful launch, MERIT invested significant time in validating phone numbers and setting up zonal WhatsApp groups. Malawi has over 500 PEAs, who are each responsible for a zone (or cluster) of about 12 schools. Our team worked with PEAs to ensure that they established zonal WhatsApp groups that included all instructional leaders in their schools and at least one of our MERIT staff (district coordinator or monitoring and evaluation officer). We derived our list from phone numbers provided about 6 months before the virtual training, but even then, many calls were unsuccessful. The phone survey that we conducted at the end of the IVR training suggested that calls were unsuccessful for several reasons: phone was switched off, number was invalid, phone was out of the network coverage area, phone was unanswered or answered by someone other than the intended participant (e.g., a spouse). Additional advance time spent generating awareness of the program, disseminating instructions for answering calls, and reminding participants of the training schedule may help improve pick-up rates.
Design
8. Vary the audio content to make the material more engaging. IVR allows the program to provide information to staff via audio in several ways. MERIT provided the information in the form of dialogues (e.g., between two teachers or between teacher and student), scenarios (“here is a situation… what would you do?”), and quizzes. Using a variety of formats is important to keep the user engaged and make the content interesting.
9. Good question design is essential. The survey questions and quizzes that test whether participants have grasped the content are as important as the content itself, and serve not just as an external validation of learning, but as a form of content reinforcement. Questions need to be simple and clear for the audio format. After the participants have made their selection, it is important to validate their responses and provide the right answer if their choice was incorrect. We found, unsurprisingly, that questions that allowed for multiple tries to reach the right answer yielded more correct answers.
10. Pilot and monitor! We conducted both a pretest with MERIT staff and a pilot with about 100 beneficiaries. The purpose of the pilot was to test the system functionality, test the questions, ensure the audio recording was clear, and learn the best time of the days to send calls. We also had two pilot groups—one that received an advance SMS message notifying them of the call and another that did not get the pre-SMS. This design was to test whether the advance SMS increased the pick-up rates. MERIT would recommend that there be enough time between pretest, pilot, and launch to allow for changes to the messages or recordings. Having a monitoring plan will help make the most of the experience. Throughout this process MERIT got feedback from beneficiaries that helped us improve the training on the fly. Our staff were in constant communication with the PEAs and some intended participants via the WhatsApp discussion groups. From this they received real-time information regarding what was working—or not. For example, we knew that in the first week calls were not going out as intended, that trainees were struggling with responding to questions, and that some messages were not clear. We could address most of these challenges as they arose. We also conducted phone surveys with a small group of leaders after each week of the training to get feedback on their experience. All our monitoring efforts contributed to learning what worked and whether this is a viable method for refresher training or additional behavior change support in the future—and we conclude that it is!
We hope those of you considering using IVR for training will find this to be a useful resource. Send questions or provide insights about your experiences with large-scale remote training for teachers in the discussion.
This article was written by Bidemi Carrol with input from Patience Sowa, Sarah Pouezevara, Barbara Greenwood, Grace Chiuye, and Cosnat Ntenje.
More information about the Malawi MERIT program here.
Cover Photo: Thandie Mangani, Teacher at Kazengo School, Kaphuta Zone (c) 2020 Ishmael Zabuloni for USAID/Malawi MERIT
