C19RM evaluation results show the importance of the investment
Article Number: 3
But more data needed to analyze if it was sufficient to mitigate COVID’s impact on the three diseases and health systems
This article reports the highlights of the recently published evaluation of the COVID-19 Response Mechanism commissioned by the Technical Evaluation Reference Group. It was not the usual type of TERG-commissioned evaluation due to the particular nature of C19RM as an emergency response to the COVID pandemic and thus organized, funded, implemented, and monitored in a very different way from grants under the regular funding cycles. The review’s findings suggest that C19RM 1.0 was an important investment in 2020/2021 that contributed to mitigating COVID’s impact on HIV, TB, and malaria programs. It delivered some valued commodities and services to fight COVID directly in low- and middle-income countries. However, the lessons for improved mitigation and optimization of the Global Fund’s investment in this area are not yet clear.
In response to a request from the Global Fund Board, the Technical Evaluation Reference Group (TERG) commissioned an evaluation of COVID-19 Response Mechanism (C19RM). The independent evaluation, conducted by Pharos, covered C19RM 1.0 (between April 2020 and June 2021) but not the second period of C19RM 2.0. It was designed to provide inputs for the Strategy Committee (SC) in the form of lessons learnt as well as the Global Fund’s response to these and was published in June 2023.
The report highlights recommendations for improvement and lessons for the Fund and other organizations channeling financing to lower-middle income countries (LMICs) to prepare for and fight pandemics. This remains highly relevant as the new Pandemic Fund is launched and WHO and others strive to mobilize more support for Health Emergencies Preparedness, Response and Resilience (HEPR).
The report does suggest that, overall, C19RM 1.0 was an important investment in 2020/2021. Whether the response was exactly what was needed to mitigate impact of COVID on HIV, TB and malaria (HTM) programs will need to be further analyzed when data become available. That said, various innovations during the first year of the pandemic were initiated, a large quantity of commodities were purchased and distributed and significant steps were taken towards developing skills at the international level to help manage responses to a global epidemic.
We look at the findings and recommendations in more detail below, based on the detailed TERG Position Paper, Management Response and Final Report available here.
To respond to the COVID-19 pandemic, the Global Fund committed to protect the gains made in HTM programs from the impacts of the COVID pandemic while assisting countries to fight COVID- 19 directly and build stronger, more resilient health and community systems. In March 2020 the Global Fund decided to allow reprogramming of existing grants ($232 million in “flexibilities” were ultimately approved), and in April 2020 mounted a new financing facility, the C19RM. Under the C19RM, $757 million was granted in 2020 and $3.2 billion in 2021 to support 129 countries and regions. This totalled almost $4 billion in addition to grant “flexibilities”.
Evaluation objectives, scope and inputs
The evaluation’s key objectives were:
- Determine the relevance and appropriateness of C19RM investments;
- Analyze whether, how well, and why the C19RM was effective in mitigating the impact of the COVID-19 pandemic on HTM and broader public health, with special attention to the protection of human rights and key and vulnerable populations;
- Assess how effectively C19RM assisted low- and middle-income countries to fight COVID-19 directly and to strengthen health and community systems to prepare these countries to prevent, detect, and respond to future pandemics;
- Examine the results and merits of global coordination to roll out the C19RM grants through the Access to COVID Technology Accelerator (ACT-A) partnership, with attention to possible duplications and important gaps at the interface with other financiers; and
- Provide lessons learned to inform enhancements to future C19RM grants and in the Global Fund’s capacity to respond to future pandemics, especially in the light of the major ongoing global conversation regarding a new architecture and financing for pandemics.
The scope of work and the evaluation questions were developed through extensive consultations with the Secretariat and the SC. The Secretariat and stakeholders at the global and country levels provided substantial contributions to the evaluation work. The Office of Inspector General was also consulted to ensure that the evaluation would add value and avoid duplication.
As well as quantitative and qualitative data analysis, the evaluation team conducted eight country case studies (in Angola, El Salvador, Malawi, Peru, Rwanda, South Africa, Ukraine and Viet Nam), comprising over 100 key informant interviews (KIIs) with government officials, the Country Coordinating Mechanisms, Principal Recipients (PRs), civil society organizations and international non-government organizations, and development partners. In addition to the KIIs, the team reviewed numerous country documents (e.g., C19RM grant proposals, performance reports, and letters); policies of national government institutions (e.g., national strategies and plans for the COVID-19 response, plans to mitigate HTM service delivery disruptions); and global and national documents from partners (e.g., the African Development Bank, World Bank, WHO and PAHO).
The TERG acknowledged that this evaluation is, in many ways, different to the ones it usually commissions. First, C19RM was set up very rapidly and not within the usual business model, even though there was considerable overlap. Second, it came into being due to the crises triggered by the new and little-understood epidemic. Third, the fact that so many Secretariat staff and others were busy dealing with the pandemic and its impact on the three diseases meant the evaluation operated under unusually challenging conditions.
Despite these constraints, the evaluation team found some interesting discoveries that can be translated into lessons and recommendations to inform the role of the Global Fund in future pandemics. The evaluation team identified ten main findings grouped around five themes: M&E; Global Fund and Country Processes Governance; Mitigation; Procurement/Direct Covid Response; and Health and Community Systems. These findings are grouped under 10 priority “first tier” recommendations (although there are another six recommendations in the final report).
The 10 first-tier recommendations are presented below with a progress column showing progress already made by the Secretariat in implementing each recommendation (Table 1).
Table 1: Findings and recommendations by thematic area and progress on recommendations
|Monitoring and Evaluation
I.0 monitoring system nascent, weak ability to systematically track inputs, outputs, outcomes and impact which affected implementation, budgeting and prioritization.
Complete the implementation of the C19RM M&O framework developed for 2.0 (April 2021), with a focus on downstream implementation, impact and quality.
|Monitoring and Evaluation
|Time to design and develop an M&O system is limited during a crisis and takes a backseat to rapid execution.
|Develop a basic turnkey M&O system for any type of pandemic that the GF might be called upon to respond to in the future.
Global Fund and Country Processes Governance
|Limited toolkit for C19RM 1.0 costing, budgeting, and priority-setting among competing demands inhibited investment optimization and reduced downstream ability to assess efficiency of grants.
Develop and disseminate tools and technical assistance for C19RM grant costing, budgeting, optimization, and expenditure tracking and reporting .
Global Fund and Country Processes Governance
While C19RM 2020 guidance envisioned flexible reallocation of grants to respond agilely during an emergency, this option of continuous reprogramming was not used.
Develop processes to allow for PRs and CTs to make more frequent adjustments to activities and budgets, on a quarterly basis or more often if required.
No clear evidence or ability to understand whether C19RM 1.0 investments have had an impact on mitigation; surveys and analysis fragmented among multiple agencies.
Create a stronger, more coherent, and coordinated system for monitoring HTM services and disruption/recovery, both within the GF and with countries and other leading organizations.
Ecological evidence of investments in innovative service delivery suggests that some of innovations/adaptations may have contributed to mitigation, but there has been no systematic effort to capture learnings.
Sponsor and establish a knowledge repository and learning hub for good practices in HTM innovation, adaptation, and mitigation in the face of COVID-19.
Procurement/ Direct Covid Response
Price and quality reports (PQRs) not required from PRs for COVID-19 products in C19RM 1.0. Reports from country informants of limited stocks, price fluctuations, and inconsistent quality have not been documented, compromising the GF’s ability to track non-PPM procurement.
Invest in an integrated health product demand forecasting and planning management system.
|Procurement/ Direct Covid Response
While the GF did a commendable job on procuring COVID-19 tests and on volume commitments, the lack of stockpiles of personal protective equipment (PPE) cost the GF and its client countries time in completing in-country deliveries for urgently requested commodities.
Develop and implement agile instruments for pandemic procurement including stockpiles and hold limited buffer stocks in suitable LMIC hubs for health products that do not have short shelf lives.
|Procurement/ Direct Covid Response
The ACT-A consortium did not set up clear decision rules for procurement supply management (PSM) roles under COVID- 19. Several actors established parallel structures rather than optimizing existing ones such as the pooled procurement mechanism (PPM). This fragmented health products procurement caused confusion for countries and suppliers and delayed efficient pooled procurement of PPE and oxygen equipment.
ACT-A partners, global health security key stakeholders, and agencies with emergency response mandates should develop clear decision rules for PSM roles in a pandemic/ emergency context.
|Health and Community Systems
C19RM 1.0 health system strengthening (HSS)/civil society strengthening (CSS) investments were only 10% of total grant awards, in part because of bias in favor of short-term emergency actions plus unclear guidelines and more deliberate processes to design HSS activities and to fully engage KVPs and Civil Society.
Consider a set-aside or earmark for HSS and CSS in future C19RM and PPR grants, including special incentives and separate timelines that encourage and enable countries to submit strong HSS/CSS proposals.
*Progress is measured by the four quadrants of the circle with all four quadrants filled equaling full implementation of recommendation and no filling meaning no progress.
The TERG provided responses to each of the ten recommendations above, which can be read in the evaluation report.
Overall, the TERG considered the evaluation to be rigorous and completed under considerable time constraints. It generally agreed with the following overarching statements on findings: “overall, the GF showed through C19RM 1.0 that it could leverage many of its existing strengths (technical, operational, partnerships) and adopt new ways of doing business to respond rapidly and effectively to a global pandemic like COVID-19. At the same time, the GF struggled in several areas to utilize its pre-existing model to act effectively during a fast-moving pandemic.”
The TERG acknowledged the evaluation report’s reference to country case selection as sub- optimal with opportunities for improvement in selecting a more balanced sample in the future. Although country selection was agreed with the Secretariat’s Grant Management Division, the time taken to get consensus on countries available to participate in the evaluation was unduly long and delays were experienced. The ultimate selection was not as balanced as the TERG would have desired with consequences to the evidence available to be collected. One mitigating factor was that countries were not selected because Country Teams were challenged by operational realities and were focusing on delivery against the environment of COVID-related disruptions. The TERG recognized that data sharing could have occurred more rapidly, but most documents that were available were shared with the evaluators, albeit some of these very late in the evaluation process, which made it difficult for them to be properly analyzed.
General comments on recommendations
The TERG noted that C19RM 1.0 was an emergency situation and the Global Fund managed to get resources to countries swiftly, in a relevant and appropriate manner, when they needed these the most; and other organizations were not able to do this.
The TERG encouraged the Secretariat to consider future pandemic preparedness and response (PPR) strategies and to ramp up emergency pandemic systems, based on the experience of the C19RM and this evaluation’s findings and recommendations. Additionally, the TERG noted that there are global strategies (e.g., around International Health Regulations) which would assist the Global Fund in coordinating and synergizing its role in relation to PPR together with other global partners such as the Pandemic Fund
The TERG was pleased to note that the Independent Panel for Pandemic Preparedness & Response has called for a political declaration and a “roadmap for coherent and transformative reform” and has also recommended an independent evaluation of ACT-A.
However, according to TERG, recommendations related to PSM are partial and only marginally respond to challenges outlined in findings.
The TERG also commented that the evaluation said little about how well C19RM performed in relation to CSS and HSS strengthening. The reality is that in 2020 the world was in chaos in relation to COVID-19 with little knowledge and understanding of the epidemic. The bulk of the funding went to mitigating the impact through the purchase of commodities required for this. However, HSS is essential to the new Global Fund Strategy going forward and PPR is a component of this. The TERG felt that in order for the Global Fund to impact significantly on HSS it will need to put more funding into this and will need to be able to be transparent about what this funding is for, through better classification and standardization of the elements of HSS.
Similarly, the TERG also noted that there is insufficient evidence for this evaluation to make definitive findings on the impact of C19RM 1.0 grants on the mitigation of HTM diseases, or on country investments on HTM.
The review’s findings suggest that C19RM 1.0 was an important investment in 2020/2021 that contributed to mitigation of COVID’s impact on HIV, TB, and malaria programs and delivered some valued commodities and services to fight COVID directly in low- and middle-income countries. However, the lessons for improved mitigation and optimization of the Global Fund’s investment in this area are not yet clear. Speed of grant approvals was not fully matched by timely implementation, in part because of limits to the GF’s existing operating model and its adaptation to a rapidly emerging and unpredictable pandemic. Monitoring systems in 2020/21 struggled to keep up and report on financial, procurement, and programmatic results, something that improved in year 2. C19RM investments in national health systems to help LMIC countries prepare for and prevent future pandemic outbreaks were modest in 2020/21 but have grown significantly over the past two years. Overall, LMIC countries benefitted from the Fund’s C19RM efforts, and the Fund demonstrated its ability to leverage its existing systems, know-how, and partnerships to address a global pandemic.
Robert Hecht, Founder and President of Pharos Global Health, said, “We found that the Global Fund was responsive and creative in using its resources to help countries protect their HIV, TB, and Malaria programs during COVID-19 and to fight the pandemic outbreak directly. There were notable successes. But not everything worked, there were gaps and shortfalls as you would expect in the midst of a new and unpredictable global outbreak. The key thing is for the Global Fund to learn from these early ups and downs, and to build stronger systems for the future. There is a lot of potential for the Fund to do that.”