Case StudyEvaluation

Case study: Assessment of impact of online courses on digital finance services practitioners

By September 15, 2020No Comments

From October 2018 to March 2019, Development Works Changemakers completed their assessment of the Impact of Digital Frontiers Institute (DFI) Online Courses on Practitioners in the Digital Financial Services (DFS) Sector.

The project was funded by FSD Africa, Bill & Melinda Gates Foundation, and Omidyar Network, covering a geographic scope of Mozambique, Malawi, Zambia, Rwanda, and Uganda.

DFI assessment

Project Outline

DFI aims to develop the capacity and enhance the professional development of DFS professionals working in the private, public, or development sectors. By closing the DFS capacity gaps currently experienced in developing markets, the organisation, in the long term, aims to accelerate financial inclusion.

To achieve this DFI provides online training and education courses. Consisting of seminars through a built-for-purpose online campus. These primarily focus on foundational DFS knowledge and skills, but also include areas of leadership development and change management.

Additionally, DFI facilitates a network of professionals, or communities of practice (CoP) which include in-country face-to-face meetings of DFI students and DFI-affiliated professionals, as well as the moderation of online meetings through DFI’s built-for-purpose digital network and series of global seminars. DFI’s first full year of courses was in 2016.

In 2017, DFI undertook focus group research to understand the impact one of its foundational training courses, the certificate in digital money (CIDM), was having on alumni and their organisations. Data was collected from Zambia, Rwanda and Uganda. In 2018, Development Works Changemakers (DWC) was commissioned to undertake follow-up data collection.

Unlike the 2017 data collection, DWC’s assessment considered the impact of all DFI’s online training courses and specifically focused on the extent to which DFI funders’ M&E indicators were being achieved.

The primary purpose of this assessment was to

1) assess the extent to which DFI funders’ M&E indicators are being met; and

2) assess the impact DFI training courses have had on participants, their organisations and the industry to date.

In-country visits for primary data was collected from five DFI markets in Sub-Saharan Africa (SSA), namely Mozambique (Maputo), Malawi (Blantyre), Zambia (Lusaka), Rwanda (Kigali), and Uganda (Kampala).

Project Deliverables

  • Development of primary data collection tools including a survey for practitioners, and interview and focus group discussion (FGDs) guides for practitioners, CoP facilitators, line managers and HR managers, and institution representatives.
  • Final reports including 1) an overall executive summary; 2) an overall introduction, method, a summary of secondary survey data (collected by DFI at six and 18-month follow-up) and recommendations; and 3) individual country reports for the five aforementioned countries, reporting the achievement of indicators and the perceived impact of the courses.
  • Infographic per country depicting the number of students trained, number of training attended, number of participants in this study and findings per indicators and in terms of overall impact.

Our Approach

The assessment responded to a select list of key indicators of interest. The extent to which these indicators were being achieved in the five markets was explored by a combination of cross-cutting data sources and data collection instruments.

The combination of data collection sources and tools aided methodological and data triangulation, which further allowed for the verification of data and a more textured, comprehensive account of DFI’s impact. A mixed-method approach was utilised. This incorporated both qualitative and quantitative data collection and analysis methods that were inclusive and complementary.

This approach allowed for data gathering in multiple ways and the team was able to elicit a variety of perspectives on DFI courses and impact.

Secondary data was collected via

1) a document review of previous DFI reports; and

2) analysis of programme monitoring data, namely six-month and 18-month ex-post survey which are administered to CIDM graduates only.

Primary data was collected from evaluation participants from four target groups namely

1) practitioners (who completed DFI training courses/in the process of completion);

2) line managers and/or HR managers (individuals who oversee/manage the practitioners or are involved in recruitment and/or development within their companies);

3) CoP facilitators (individuals who facilitate the in-country CoP meetings); and

4) institution representatives (Individuals who work for key institutions within the DFS sector and could provide broad insight into the DFS market/sector within their country).

Data was collected from these participants using an online survey for the practitioner and line manager/HR managers respectively, and FGD/interview guide for practitioners, and an interview guide for line managers/HR manager, CoP representatives and institution representatives. Surveys were administered online on Survey Monkey (and administered in-person in-country to gather more responses) and participants were incentivized to participate with the offer of discounted courses with DFI.

FGDs and interviews were conducted primarily face-to-face in each of the country capitals. The practitioners were invited via email to a CoP meeting, where DWC team members conducted the FGDs. Practitioners in FGDs and in surveys provided the contact details of their line managers and/or HR managers. DWC followed-up with for interviews and CoP facilitators made themselves readily available. And also assisted in arranging interviews with individuals in major institutions, including government ministries, banks, and interbanks.

The evaluation team analysed both primary and secondary data that was collected using ATLAS.ti for thematic analysis of the qualitative data and Microsoft Excel for descriptive statistical analysis of quantitative data.

Value

The assessment provided a valuable opportunity for DFI to take stock of its achievements since 2016. The assessment provided DFI with insight on the extent to which their set indicators were being met. As well as where gaps exist, areas for improvement or best practices that could possibly be expanded upon in different countries.

It also provided insight into the value of and impact that the course may be having on individuals and in their companies. Based on the findings, several recommendations were made, that if implemented, could improve the DFI courses going forward.

Recommendations focused on improving CoP attendance and morale, amendments to and additional DFI courses, identifying local partnerships to reduce costs and increase reach, course support, and monitoring and evaluation (M&E). M&E recommendations noted the challenges experienced with data collection and made suggestions for follow-up data collection in 2019 and 2020.

DWC also suggested that the report be used as a tool for learning. Not only for DFI’s internal planning but for each country’s alumni and CoPs. Specifically, the initiatives being developed in different countries and the achievements that they had should be shared with alumni and CoP facilitators, who may be able to learn how to implement initiatives themselves. This led to the development of infographics per country. Showing how each country fared against the indicators and what overall impact was reported per country.

The infographics can serve as marketing tools of how participation in a course can add value to one’s personal and career development. And as a learning tool for other countries. Especially in terms of launching their own formalized CoPs.

We’d love to work with you

We hope that by showcasing our case studies with you, that it shares insight into the areas of expertise that we have experience in. If you have any questions about the research, evaluation, monitoring and development industry, feel free to contact us.