Case StudyEvaluation

Case Study: UNODC – Baseline, endline and impact evaluation of the LULU programme

By August 20, 2020No Comments

At Development Works Changemakers, our passion for change can be seen in our several case studies. The Baseline, Endline and Impact Assessment of the Line Up Live Up (LULU) Programme in South Africa began in May 2019 and was recently completed in January 2020.

The client, United Nations Office on Drugs and Crime (UNODC), worked with the Western Cape Government Department of Cultural Affairs and Sports (DCAS) to provide baseline, endline and impact assessment. Focusing on the area of Khayelitsha and Mitchells Plain in the Western Cape, the following findings were recorded.

children in a classroom

Project Outline

The  Line Up Live Up (LULU) programme is a sport-based life skills training curriculum developed to improve youths’ knowledge/awareness, perceptions, attitudes, life skills and behaviours to build resilience to violence, crime and drug use. The programme is designed to be delivered over 10 sessions to male and female youth, between the ages of 13-18 years.

Each session includes interactive sports-based activities, interspersed with reflective debriefing spaces in which life skills are imparted. These sessions are envisaged to lead to various outcomes, which in the long-term include youth engaging less in risk and antisocial behaviours and demonstrating resilient behaviour.

The LULU programme is being piloted in Brazil, Colombia, the Dominican Republic, Kyrgyzstan, Lebanon, Peru, Palestine, Tajikistan, Uganda, Uzbekistan and in 2019, it was piloted in South Africa. In South Africa, the programme is run in cooperation with the Western Cape DCAS as part of its flagship MOD afterschool programme.

In 2019, DWC was commissioned to conduct a baseline, endline and impact assessment of the LULU programme in nine schools in Khayelitsha and Mitchells Plain, two high-crime areas in the Western Cape, South Africa. The purpose was to assess only the short-term outcomes (knowledge and perceptions) and selected medium-term outcomes (attitudes and behaviours) of the LULU programme. The findings of this study are intended to be used for cross-country comparisons, and for informing programme improvements.

Project Deliverables

As part of the assessment, DWC produced:

  • Adjusted data collection tools that were provided by UNODC and adapted to the South African context and made more youth-friendly; these included a baseline/endline survey for youth, a self-reporting survey for youth participating in LULU, and focus group discussion (FGD) guides for youth, coaches, area managers and DCAS management.
  • A literature review focused on the context of crime in South Africa and the Western Cape province specifically, Khayelitsha and Mitchells Plain profile; policy and other approaches to tackling crime in South Africa; and the sports-based life skills approach international and local examples.
  • A baseline report outlining participating learners’ profile (including demographics and experiences of family/home life, school and community) and outcomes of interest prior to launching the LULU programme in schools
  • An endline report comparing baseline data to endline data to assess changes in the outcomes of interest following the completion of the LULU programme; and
  • An impact report, building on the endline report by additional discussing lessons learned and recommendations.

An executive summary report and summary report of the final impact report.

Our Approach

The assessment followed a mixed-method approach, which combined qualitative and quantitative data analysis in order to bring a robust and credible set of findings to the report.

A non-equivalent, multiple group time-series design was employed, whereby data was collected at baseline before the programme commenced and at endline once the programme concluded. Data was collected from learners from 9 schools across Khayelitsha and Mitchells Plain and collected from:

  • Learners who participated in the LULU programme (treatment group)
  • Learners involved in the afterschool MOD programme (control group I); and
  • Learners who do not participate in any afterschool activities (control group II).

While the initial design of this evaluation assumed all LULU learners would have attended all 10 sessions, this was not the case. Due to the high proportion of learners who did not attend all sessions, all learners were still included, but additional analyses were incorporated to assess the extent to which attendance at 1-6 vs 7-10 sessions influenced outcome indicators.

Secondary data was also collected through a literature and programme document review. Primary data was collected using surveys and focus group discussions (FGDs) provided by UNODC, which were adapted by DWC to be more child-friendly, include colloquial language, ensure that all outcomes were adequately measured by adding additional questions and to shorten the surveys to keep learners interested. In terms of primary data collection:

  • Baseline survey data was collected from 724 learners (313 LULU learners; 204 MOD learners; and 207 non-intervention learners);
  • Follow-up endline survey data was collected from 658 learners (262 LULU learners; 195 MOD learners; and 201 non-intervention learners);
  • Endline self-administered survey data was collected from 210 LULU learners; and
  • FGDs were conducted with a) 8-10 learners from five schools, respectively; b) 16 coaches from all nine schools; c) four Area Managers covering the two Metros and d) two DCAS programme management staff.

Ethical approval from a research ethics committee was granted for this evaluation. The programme and the study itself were constrained by a highly limited timeline, which impacted the implementation of the programme. The study period and school timelines forced the programme to be implemented within a five-week period rather than 10-weeks, which limited the programme’s dosage and duration.

The study period also did not allow sufficient time for LULU participants’ learnings to be fully absorbed and advanced. There were also issues with programme fidelity, and most learners did not attend all 10 LULU sessions as required. These issues made it challenging for outcomes, and especially the more medium-term outcomes of attitude and behaviour change difficult to achieve. These challenges were highlighted for consideration for when findings of the study were interpreted.

Data from primary and secondary data collection were analysed using Atlas.ti for thematic analysis for qualitative data, and Microsoft Excel and IBM SPSS for quantitative data to conduct both descriptive and inferential statistics.

Value

The evaluation produced valuable information, including significant lessons learned and recommendations on the LULU programme that may help inform the improvement of the programme going forward in South Africa; lessons and recommendations included the need for key stakeholder buy-in, longer and more intensified coach training; support to coaches and area managers, and the need for psychosocial support for both learners and coaches.

Further, those short-term outcomes that were achieved can provide evidence to potentially support funding and buy-in for the ongoing implementation of the programme in the future. Finally, the data can be used for comparison with the other programme implementation pilot countries, and lessons learned from this assessment can guide programme implementation and the study thereof in these other countries going forward.

Overall, given the difficulties faced, the programme and its implementers/managers should also be commended on the outcomes realised; what was achieved suggests that had the programme been implemented as intended (in terms of dosage, duration and fidelity) and under the right conditions (with full attendance by all learners and enough time for change to manifest within the study period), further outcomes could have been achieved.

Development Works Changemakers Evaluation

Over the next couple of months, we’ll be showcasing more of our case studies and highlighting the various methods of our approach.

To stay up to date with industry news and happenings, you can sign up to our newsletter and follow us on social media.

Facebook

LinkedIn

Twitter