Our results go
with real progress
Baseline, Endline and Impact Assessment of the Line Up Live Up (LULU) Programme in South Africa
Client: United Nations Office on Drugs and Crime (UNODC)
The Line Up Live Up (LULU) programme is a sport-based life skills training curriculum developed to improve youths’ knowledge/awareness, perceptions, attitudes, life skills and behaviours to build resilience to violence, crime and drug use. The programme is designed to be delivered over 10 sessions to male and female youth, between the ages of 13-18 years. Each session includes interactive sports-based activities, interspersed with reflective debriefing spaces in which life skills are imparted. These sessions are envisaged to lead to various outcomes, which in the long-term include youth engaging less in risk and antisocial behaviours and demonstrating resilient behaviour. The LULU programme is being piloted in Brazil, Colombia, the Dominican Republic, Kyrgyzstan, Lebanon, Peru, Palestine, Tajikistan, Uganda, Uzbekistan and in 2019, it was piloted in South Africa. In South Africa, the programme is run in cooperation with the Western Cape DCAS as part of its flagship MOD afterschool programme.
In 2019, DWC was commissioned to conduct a baseline, endline and impact assessment of the LULU programme in nine schools in Khayelitsha and Mitchells Plain, two high-crime areas in the Western Cape, South Africa. The purpose was to assess only the short-term outcomes (knowledge and perceptions) and selected medium-term outcomes (attitudes and behaviours) of the LULU programme. The findings of this study are intended be to used for cross-country comparisons, and for informing programme improvements.
Final Impact Evaluation of the Mandela Washington Fellowship Programme of the Young African Leadership Initiative (YALI)
The Mandela Washington Fellowship for the Young African Leaders Initiative aim to equip young African leaders under 35 with skills, networks and support needed to lead positive personal and community change in themselves. It offers academic exposure, opportunities to participate in follow-on programmes and alumni networks. This will build skills of young African leaders to improve the accountability and transparency of government, start businesses and serve their communities.
DWC carried out a final impact evaluation of the follow-on activities of the programme on male and female participants. The study determined and portrayed the emerging results of the programme and informed current and future youth leadership programming, specifically women’s leadership in Africa. A mixed-method approach, incorporating a quasi-experimental design which compared male and female fellows who participated in follow-on activities and those who did not participate, was used (disaggregated per cohort and other variables including academic track). The comparison of the two groups drew on both primary and secondary data, as well as quantitative and qualitative data. Data was collected through online surveys, as well as KIIs and FGDs collected through visits to Zimbabwe, Namibia, Nigeria, Ghana, Kenya and South Africa.
External M&E Audit of the Impact Bond Innovation Fund (IBIF)
This Early Childhood Development programme is the first Social Impact Bond to launch in Africa. Mothers2mothers (m2m), Volta Capital and the Western Cape Foundation for Community Work (FCW) are jointly implementing this flagship initiative. Since November 2017, child development services have been provided in Delft and Atlantis, two under-resourced Western Cape communities. The main objective of the programme is to equip children aged 3 to 4 years – who don’t have access to ECD – with language, motor, cognitive and socio-emotional skills as preparation for school readiness before enrolment in Grade R.
DWC is the M&E auditor responsible for a review of the M&E system and reporting on the programme implementation and outcomes with a focus on three programme indicators:
- recruitment and retention
- early learning outcomes measurement
Year on year, DWCs audit has been conducted assessing the accuracy of data and reporting against the above three indicators, reviewing key documents, conducting KIIs and FGDs, speaking to beneficiaries and conducting home visits.
Assessment of the Impact of Digital Frontiers Institute (DFI) Online Courses on Practitioners in the Digital Financial Services (DFS) Sector
Client: Digital Frontiers Institute (DFI)
Year: 2018 – present (on-going M&E partner)
Digital Frontiers Institute provides a variety of online training and education courses for digital financial service professionals which aim to enhance their professional capacities, leadership skills and financial inclusion. In 2018 and 2019, DWC conducted a rapid assessment of the impact of these training courses on practitioners, organisations and country markets. It also assessed if funders’ M&E indicators were being met. In 2018, DWC gathered data in-country from Mozambique, Malawi, Zambia, Uganda and Rwanda. In 2019, all data was gathered through remote methods. Additionally in 2019, five case studies looking at the systemic impact of the DFI courses at a country and industry level were conducted. Industry case studies include the Bank of Kigali, MTN, Accion, while country case studies were undertaken for Zambia and Uganda. The DWC approach has allowed for multi-pronged data gathering and the team has been able to elicit a variety of perspectives on DFI courses and their impact. To enhance the confidence and reliability of findings, DWC triangulates the data between various sources and kinds of data. During 2020, DWC continues to support the M&E priorities of DFI, including a continued rapid assessment of the impact of the DFI training courses, as well as a case study for Bangladesh Bank which looks at the course’s systemic impact.
Assessment of After School Programmes in 112 Western Cape low- and no-fee schools
Client: Western Cape Department of the Premier (DotP)
The After School Game Changer programme is implemented in 160 no- and low-fee schools in the Western Cape province. This flagship programme aims to increase participation of learners in after school programmes by improving their quality, and making them more attractive. It is a joint programme in partnership with local government bodies, provincial government departments, and a number of nonprofits. The programme is a transversal programme managed out of the Department of Cultural Affairs and Sport.
DWC was commissioned by the Western Cape Provincial Government to conduct the last of three assessments of the programme. The aim of this rapid assessment was to measure the status of facilities/centres providing school aftercare services as part of the After School Programmes against the baseline assessments conducted in 2018.
The DWC team identified 112 schools by using a sampling strategy that facilitated targeted data collection sufficient to form a comprehensive understanding of the situation in each of the eight districts. A team experienced in conducting research in schools and proficient in Afrikaans and isiXhosa visited all 112 schools across the Western Cape over three weeks. DWC then produced a total of 112 assessment reports based on school visits, as well as an overview report showing major trends and recommendations.
TB in the Mining Sector (TIMS) Monitoring & Evaluation Systems Strengthening Assessment
Client: Wits Health Consortium (WHC)
The Global Fund grant “TB in Mining Sector in Southern Africa” (TIMS) is a ten-country regional grant, supporting a programme which aims to help to reduce the burden of Tuberculosis (TB) in the mining sector in Southern Africa. The grant is being implemented in Botswana, Eswatini, Lesotho, Malawi, Mozambique, Namibia, South Africa, Tanzania, Zambia and Zimbabwe, in selected Districts with a high burden of TB, large mining populations, and/or high levels of artisanal and small-scale mining. The Wits Health Consortium (WHC) is the Principal Recipient (PR) of the grant, while implementation at country-level is undertaken through two Sub Recipients (SR), with each SR managing five countries. These SRs work closely with the National TB Programmes in each of the countries in which they work.
The TIMS programme gathers data from a range of healthcare facilities and occupational health centres in participating countries, specifically data relating to TB prevalence among mining Key Populations (KPs). WHC relies on the national health system workers (e.g nurses, District officials) to collect, capture and report on these indicators to the PR, with the SRs acting as a conduit of this data, and playing a key monitoring, verification and support role. The PR reports to the Global Fund on the results, which are aggregated across all 10 countries. To help manage this data flow and reporting, WHC established data management systems and processes intended to aid them to consolidate all programme data to submit as part of their semi-annual reports to the Global Fund. WHC, however, has experienced a number of issues with these data management systems and processes and it thus contracted DWC, as M&E experts, to conduct a technical review of its systems, processes and tools and advise on appropriate measures to ensure these challenges are overcome.
Design and Implementation Evaluation of the Cash Plus Care Programme
Client: Western Cape Department of Health (WCG: Health)
The Cash plus Care programme is a component of the Western Cape Department of Health’s (WCG Health) Young Women and Girls Programme (YW&G), funded by the Global Fund (GF). This programme was implemented in two neighbouring sub-districts in Cape Town, Mitchells Plain and Klipfontein. The aims of the YW&G programme were to: decrease new HIV infections in girls and young women (aged 15-24); decrease teenage pregnancies; keep girls in school until completion of grade 12; and increase economic opportunities for young people through empowerment. The objectives of the intervention were to: enhance the sexual and reproductive health of participants through preventative and promotive health and psychosocial interventions whilst enhancing their meaningful transition to adulthood; and to reduce HIV and STI incidence, unintended pregnancy and gender-based violence amongst Cape Town women in late adolescence. The programme, also called “Women of Worth” (WoW), provided 12 weekly empowerment sessions and access to youth friendly healthcare, seeking to address a range of biomedical, socio-behavioural, structural and economic vulnerabilities amongst participating young women. Cash incentives of R300 per session were provided to the young women for participation in empowerment sessions. Implementation of the programme was sub-contracted to the Desmond Tutu HIV Foundation (DTHF) by the Western Cape Department of Health in November 2016, and implementation of the programme began in April 2017.
DWC was contracted by the WCG: Health to conduct an evaluation, the purpose of which was to assess the Cash plus Care programme’s design and implementation. Specifically, the evaluation focussed on the appropriateness of the programme design, incentives, and recruitment processes; beneficiary and stakeholder satisfaction; and the quality of implementation. The evaluation also identified successes and challenges / barriers, and made recommendations to the Global Fund and WCG: Health to inform the design and implementation of future programmes.
Design Evaluation of the Software@Schools Programme
Client: Western Cape, Department of Economic Development and Tourism (DEDAT)
The Software@Schools Programme (SWD@Schools) was conceptualised in recognition of the challenges and opportunities to address critical skills shortages within the IT sector and youth unemployment. The programme takes the form of a strategic partnership between the Western Cape Government Department of Economic Development and Tourism (WCG DEDAT), Western Cape Department of Education and Oracle SA. Learners in Grades 10, 11, and 12 from eleven schools in the Western Cape are trained in Java fundamental development skills as part of an in-school extracurricular activity. The cohort moves to an advanced training after finishing school, gets internship placements and is trained and prepared to sit for the Oracle Certified Associate Exam.
WCG DEDAT commissioned DWC to conduct an independent design evaluation of the SWD@Schools programme; looking at its Theory of Change (ToC) and Theory of Action (ToA), assumptions, indicators and data systems. The stakeholders sought to understand how the programme was and is designed to work, and whether the design was robust and likely to yield the anticipated outcomes and impact. A further requirement of the evaluation was that it needed to assist the Department to understand if further evaluations are viable, under what conditions, and what programme fundamentals should be in place for future evaluations.
Rapid Baseline, Midterm and Endline Assessment of the International Enterprise Pilot Programme (IEP) in South Africa
Client: British Council
The UK Premier League Enterprise programme, which is funded by the Premier League and Comic Relief, uses the context and business models of professional football clubs to deliver education focused on enterprise. The IEP is funded by Premier Skills, a partnership between the British Council and Premier League, and Comic Relief in the UK. The IEP uses Premier Soccer League (PSL) clubs as a model to introduce learners to business, the working world, and the daily operations of a professional football club. The British Council, Premier League and Sports Relief have introduced the IEP programme to South Africa, building on previous work in this area, with the aim of unlocking learners’ potential, and helping them develop eight essential enterprise and employability skills using the power of football.
The British Council contracted DWC to conduct a baseline, midline and endline assessment to investigate how the programme has influenced learners’ enterprise knowledge and skills, self-esteem and self-efficacy, awareness of job opportunities and motivation to learn. A formative approach was used throughout this assessment to find out what was working and what could be improved in future programming. Additionally, a mixed methods approach was used, which incorporates quantitative data as well as qualitative data collection and analysis. Secondary data collection included reviewing programme documentation and writing a high-level literature review.
The value of the assessments was that these gave the British Council insight into whether the programme was playing a role in the learners achieving the desired outcomes. The assessments also shed light on some additional unexpected positive outcomes gained by the learners, which included exposure to a reputable football club, development of social cohesion and possible programme expansion opportunities.
Creating a Monitoring and Evaluation Framework for Innovative Childhood Development Programmes
Year: 2018 -2020
Penreach is a non-profit and social impact organisation working towards educational excellence in disadvantaged communities in the Ehlanzeni district of Mpumalanga province, through innovative childhood development, literacy and numeracy, maths and science, leadership, cultural and bursaries projects.
DWC has assisted Penreach in four phases. Phase one involved the development of a high-level Theory of Change, as well as cascaded ToCs for each project. Following the initial engagement, phase two involved the development of a gender-sensitive M&E Strategy and Framework was developed. Phase three undertook to finalise the M&E frameworks for each project and data collection instruments and process flows for reporting were developed. Phase four consisted of the provision of further support and capacity development with reporting and learning, to enable Penreach to shift from mainly compliance-based M&E to a Monitoring, Evaluation, Learning and Reporting (MERL) system, aimed at enabling Penreach to integrate learning and reporting in its M&E work.
The Penreach project was characterised by intensive engagement with project staff and integration of M&E capacity development throughout the process. DWC continues to act as an enabler, as and when Penreach requires assistance, and in this sense, DWC walks beside Penreach in their MERL journey.
Evaluation of the Youth Work Programme (YWP) in Relation to the Expanded Public Works Programme (EPWP)
Client: Western Cape Government Department of Community Safety (DoCS)
The Western Cape Government Department of Community Safety commissioned Development Works Changemakers to conduct an independent evaluation of the YWP in 2018 to 1) assess programme implementation 2) assess outputs and outcomes achieved and 3) provide recommendations for improvement. The evaluation was primarily qualitative in nature, supported by quantitative monitoring data. At the initial stages, a document and literature review was conducted, as well as a clarificatory workshop to produce a Theory of Change and Theory of Action for the programme. Primary data was collected from 69 beneficiary telephonic interviews and staff focus group discussions. DWC then provided short term and long-term recommendations for the programme.
Rapid Baseline and Endline Assessment of the Small, Medium and Micro Enterprises Training and Support Programme
Client: Green Building Council South Africa (GBCSA)
The Green Building Council South Africa is a non-profit which aims to lead the greening of the country’s commercial property sector. GBCSA partnered with JP Morgan to host their first training and support programme for 100 participants from 50 black-owned small, medium and micro enterprises operating in construction, building maintenance, material supply and building interiors. DWC conducted a rapid assessment with the aim of assessing:
- the short-term impact or change in participants’ knowledge as a result of the SMME training intervention
- the short-term outcomes of the knowledge sharing and networking events
- emerging outcomes or results that can be attributed to the full set of programme interventions, specifically noting changes in business statistics.
Using a mixed-method approach, primary data was gathered at five stages: baseline, pre-training, post-training, end of programme and finally an assessment of SMMEs’ business development post-programme.
Peer Review of Department of Cultural Affairs and Sports Drama Evaluation Report 2019/2020
Client: Western Cape Department of Cultural Affairs and Sports (DCAS)
DCAS has a vision to create an inclusive, creative, active and connected Western Cape. The Department encourages excellence and inclusiveness in sport and culture through the effective, efficient and sustainable use of resources and through effective partnerships. In moving to excellence and making the Western Cape the sport and cultural centre of South Africa, the Department aims to create the conditions for access and mass participation, talent identification and skills development. As part of its effort to achieve this vision, the Department established the Drama Development Programme in 2011. The programme aims to uncover artistic and creative theatre talent in youth participants from rural communities and focuses on talent identification and employment for youth. The programme provides training in production development, script development, character development and marketing.
In 2019, the Department conducted an internal evaluation of the Drama Development Programme. DWC was contracted by the Department to conduct a peer review of the report; reviewing and commenting on 52-page report.
Evaluation of the Youth Work Programme (YWP) in Relation to the Expanded Public Works Programme (EPWP)
Client: Western Cape Government Department of Community Safety (DoCS)
The Youth Work Programme is a cornerstone of efforts by the Western Cape Government Department of Community Safety to reduce the harms experienced by people living in poor, insecure and under-resourced communities in the Western Cape. The YWP creates temporary work opportunities for youth (aged 18-25) by providing 3-month training to enhance their life, employability skills and placement opportunities. WCG DoCS commissioned DWC to conduct an independent evaluation of the YWP in 2018 to:
- assess programme implementation;
- assess outputs and outcomes achieved; and
- provide recommendations for improvement.
The evaluation was primarily qualitative in nature, and where relevant, supported by quantitative monitoring data. DWC conducted a document and literature review, as well as a clarificatory workshop and produced a programme Theory of Change and Theory of Action. DWC then collected primary data from 69 beneficiaries through telephonic interviews and staff focus group discussions and provided short- and long-term recommendations for the programme.