By Osman Siddiqi
Our previous post on predictive analytics shared the incredible story of Zachary Ndegwa about how he used Arifu to multiply his income streams over the years. This story makes us ask every day how we can make Arifu a lifelong learning companion for people particularly to inform day-to-day decisions at scale.
To achieve this vision, it becomes important to understand how to credibly and frequently know if an Arifu learner is learning, applying knowledge, and how to respond with information or service linkages across time, space, and needs in order to build the kind of agency, decision-making, and growth Zachary exhibited with a little help from Arifu.
This is not straightforward work. But we have some serious momentum.
Arifu’s Theory of Change is complicated. It’s complicated because we’re sector and problem-agnostic. What this means is that our operations cut across agriculture, financial services, health, employment, business growth, across countries, learner segments, and their intersections. Even so, two aspects of our Theory of Change drive our strategy. First, is a deep focus on us being able to inform day to day decision-making and activities through accessible learning content. The second is about leveraging market forces from the private sector, government, and more to showcase our value-proposition to partners to drive their growth and spur further creation, iteration, and scaling of Arifu’s content.
In 2020, however, we have laid strong foundations for honing in on these two big questions across multiple learner segments. This includes a recently concluded pilot study in partnership with AgriFin Accelerate and with researchers from the World Bank and Harvard Business School. This study showed linkages between our internal Learning Scores (our Knowledge Score and Skills Score) and with self-reported farm revenues and profits. It further showed linkages with learning retention.
Figure 1: Locations of all surveyed farmers for this study
The study categorized a random set of learners into four groups ranging from very low engagement with our content to very high engagement. Arifu’s backend Learning Scores were compared with findings from an in-person survey implemented with the World Bank. The survey asked learners about their demographics, farm-level outcomes, and replicated questions from within our content in person. The sample size in the pilot size was 153 learners. The learners had engaged with Arifu’s content at least three months before the survey was implemented. The study taught us two main things:
- Our Learning Scores are predictive of learning retention i.e. farmers’ responses to our quiz questions were highly correlated with their responses to questions in the content.
- Our Learning Scores were predictive of financial outcomes such as annual revenue and profits.
Importantly, the results showed strong associations between the Learning Scores and financial outcomes and resolved an open question on learning retention. Specifically, a one standard deviation increase in the Learning Score is predictive of an increase in annual revenue of approximately KSH 9,065 (USD 84) and in annual profits of around KSH 1865 (USD 17).
Figure 2: Arifu’s Learning Score associations with total revenue and profits. Given the low resolution of the image above, the metrics from top to bottom on the y-axis are Respondent’s earnings past 30 days, Total HH income past 30 days, Total revenue past year, Total profits past year.
The findings tell us the Arifu Learning Score has strong potential as a possible Credit Score. In addition, it is possible that the Learning Scores are good predictors of behavior change and adoption of practices, purchase of inputs, and more.
We have seen through several independent studies that providing actionable and relevant information can improve the quality of life of learners, show learners how to use products and services in beneficial and non-harmful ways, and for building resilience during lean periods. This includes a 60 Decibels implemented impact study on potato farmers which showed that between 47-71% of farmers saw their crop production improve and between 16-49% of farmers saw an improved ability to save attributable to Arifu [1].
So what’s next? We have launched and are part of over four large scale impact evaluations, including at least two randomized controlled trials, across three countries in Sub-Saharan Africa, covering merchants, farmers, and youth [2]. The projects will reach well over 600,000 learners to further inform the precision of our Learning Scores prediction strength towards behavior change, product and service utilization, generation of leads for partners, and understand whether outcomes may have improved for learners.
We are continuously looking for new partnerships to refine our Learning Scores and to build research projects that can help inform and further our vision for how we can leverage predictive analytics, as one tool, to drive our impact and drive the goals of our partners.
Footnotes
- A significant portion of farmers who responded to this survey had not seen improvements in their production since they had not harvested their crop yet. To unpack this we are conducting a follow-up survey one year since the original study.
- These projects are in partnership with Google.org, the World Bank, Government of Kenya, Mastercard’s Center for Inclusive Growth, and UC Berkeley among other renowned organizations.