Case StudyEvaluation

Case study: TIMS M&E Systems Strengthening Assessment

By October 16, 2020No Comments
aerial view of mining ground

The Development Works Changemakers’ portfolio is expansive, with several case studies in a variety of development niches. Between January and March 2020, we worked for Wits Health Consortium (WHC) to provide an M&E systems review.

Funded by the Global Fund to Fight AIDS, Tuberculosis, and Malaria, we conducted Monitoring and Evaluation systems strengthening assessment on TB in the mining sector across Southern Africa.

Find out more about the project outline, our approach, the deliverables, values and more.

Project Outline

The Global Fund grant “TB in Mining Sector in Southern Africa” (TIMS) is a ten-country regional grant, supporting a programme which aims to help to reduce the burden of Tuberculosis (TB) in the mining sector in Southern Africa. The grant is being implemented in Botswana, Eswatini, Lesotho, Malawi, Mozambique, Namibia, South Africa, Tanzania, Zambia, and Zimbabwe, in selected Districts with a high burden of TB, large mining populations, and/or high levels of artisanal and small-scale mining.

The Wits Health Consortium (WHC) is the Principal Recipient (PR) of the grant, while implementation at the country-level is undertaken through two Sub Recipients (SR), with each SR managing five countries. These SRs work closely with the National TB Programmes in each of the countries in which they work.

The TIMS programme gathers data from a range of healthcare facilities and occupational health centres in participating countries, specifically data relating to TB prevalence among mining Key Populations (KPs). WHC relies on the national health system workers (e.g nurses, District officials) to collect, capture and report on these indicators to the PR, with the SRs acting as a conduit of this data, and playing a key monitoring, verification and support role.

The PR reports to the Global Fund on the results, which are aggregated across all 10 countries. To help manage this data flow and reporting, WHC established data management systems and processes intended to aid them to consolidate all programme data to submit as part of their semi-annual reports to the Global Fund. WHC, however, has experienced a number of issues with these data management systems and processes and it thus contracted DWC, as M&E experts, to conduct a technical review of its systems, processes and tools and advise on appropriate measures to ensure these challenges are overcome.

Project Deliverables

Project deliverables were:

  • Reviewing the systems, processes and tools at the PR level, SR level, and NTP level to identify areas of weakness and make recommendations to improve the systems. An M&E System Review Report and recommendations were produced, as well as two country mission reports after in-country assessment visits. A customised M&E good practice checklist was also developed and provided.
  • Reviewing the progress update report and supporting documents prior to submission to the Global Fund. A report was produced on the review of all data submitted to the GF.
  • Conducting training of M&E staff at PR and SR levels on data quality assurance aligned to PUDR reporting and GF guidelines. A training workshop was conducted, and a workshop report was produced.

miners

Our Approach:

This was a rapid technical review of the TIMS M&E systems, conducted over a 25 day period in early 2020. The project took place in four stages:

Phase I: Rapid Review of Systems, Processes and Tools at PR Level

The following activities and methods were used in Phase I:

  • An inception meeting was held with the M&E team from the WHC, which led to the development of an inception report and work plan.
  • A document review was then conducted of all relevant documentation on the TIMS M&E systems and tools. In excess of 20 documents were reviewed.
  • An M&E checklist, based on criteria for a good M&E system was drawn up by the evaluation team. This checklist was used as a guiding reference and standard for reviewing the systems in Phases I and II. It also informed the development of all interview tools.
  • A physical inspection of the M&E system at a PR level was conducted.
  • The initial meetings, document review and M&E checklist were used to generate structured interview/survey tools, which were used for interviews with key staff and other role-players at a PR level. These interviews were conducted via Skype/WhatsApp in the week of the 20th of January and focussed on key questions around how the PR is experiencing the system, challenges, system gaps and failures, user-friendliness and good practices. In all, six key individuals were interviewed.

Phase II: Rapid Review of Systems, Processes and Tools at SR and NTP Level

Following the review at PR level, DWC undertook two in-country visits to review the systems, processes and tools at SR and National TB Programme (NTP) level. Country visits were conducted in South Africa and Botswana from 27-29 January 2020.

  • DWC undertook a review of key documents at SR and NTP level (e.g. tools, SOPs, quarterly reports) in cases where these were not included in the PR system review. A total of 11 additional documents were reviewed in this phase, including SOPs, and M&E tools and protocols.
  • DWC then undertook a physical inspection of the M&E systems, processes and tools in two of the grant countries. Based on discussions with the PR, South Africa and Botswana were identified, given that these are both more cost-effective and where the two SRs are based. Of the two days spent in each country, one day was spent with the SR learning about the systems and issues faced at the management site, and one day was planned to be spent in the field learning how data is gathered and what the ground-level dynamics and constraints of data collection are.
  • A structured interview/survey was developed based on the learnings from Phase I and the additional SR- and NTP-related document review. The evaluators conducted the interviews with those involved in the data flow at the country level, including SR staff and available NTP staff. In all, 11 individuals were interviewed during country visits.

Phase III: Rapid Review of Draft Progress Update Report and Supporting Documents

In the third stage of the consultancy, DWC undertook a rapid remote review of the draft progress update report (PUDR) and supporting documents ahead of WHC’s submission to the Global Fund.

This review served to assist and ensure that there are no discrepancies between source documents and data, and the final reported results. Where variances were identified, these were documented in a rapid report, and thereafter amended in the PUDR report.

Phase IV: One-day Training Session

The fourth phase of the assignment involved the design and delivery of a one-day training session for PR and SR M&E staff.  The training session was informed by the findings of the systems and PUDR reviews, as well as GF M&E protocols.

Value

This technical review of M&E systems was highly valuable to the WHC, as well as all other role players within TIMS. It allowed for the independent verification and documentation of the many challenges faced in gathering, monitoring, verifying, collating and reporting on TB data from key mining populations in 10 Southern African countries. This was not a normal M&E systems review. The TIMS M&E Unit was well trained and had set up reasonably good systems, but the complexity of managing a 10-country data gathering and management system caused challenges.

One of the main challenges related to the sheer complexity of trying to manage a health systems intervention with 10 different countries in one region. In this case, TIMS was trying to get each country to gather disaggregated data on mining key populations and their families with the data they already gathered in their TB Registers. Although all part of the SADC region, and using WHO TB tools and processes, each country’s health system differs in many respects.

testimonial quoteAlthough each of these 10 countries signed on to the TIMS programme at a high level, the challenge was in getting standardised data gathering, monitoring and reporting tools, processes and systems across all of these countries. At the onset of TIMS, none of the countries was gathering TB data specifically on mining key populations, although they all used the WHO TB Register system, albeit in slightly different formats, or even languages.

The TIMS programme, therefore, worked with each National TB Programme to introduce a sticker system which would allow those being recorded in TB Registers at primary health facilities to also be identified as a miner, ex-miner, family of these two groups, or living in a mining community.

Each month, health staff were supposed to record and report on the details of those marked by the sticker system. These data were then meant to be captured and reported up to measure against certain targets which TIMS had set itself. However, there were many problems with the entire data chain, from the use of the sticker system to the recording of the data at the facility level to the capturing at the District level, and then to the collation of data at country level, and finally, reporting to the GF.

Problems included the fact that the TIMS M&E Unit was understaffed, and there was no way that two full-time staff could build relationships, train staff, verify data and ensure capturing was accurate in all 10 countries. Their capacity was highly stretched. The NTPs also reported their statistics late, which meant that there was little time for the M&E Unit to ensure it was accurate and to enter it correctly into the reporting tools for the GF. The use of MS Excel spreadsheets also caused many problems with data accuracy, which were only later resolved. A new Management Information System was then introduced to improve on this system and iron out the mistakes.

DWC was able to explore the causes of these challenges and complexities in detail, which assisted not only in documenting the TIMS experience but also in the identification of a number of recommendations for how the systems could be improved. The PUDR review also identified a number of discrepancies and mistakes in the data, which WHC could then correct in time to submit to the GF. Finally, the M&E Systems training was highly useful for PR and SR staff in getting them to understand how to address the challenges they faced and improve their systems.

To stay up to date on all of our projects and industry news, you can follow us on Facebook, Twitter and Linkedin.