On the cover of its Results Report 2018, the Global Fund loudly proclaims: “27 million lives saved.” The top-line figure is the result of a new methodology used to estimate the Fund’s impact. The results report provides a crucial input to the Global Fund Investment Case, which is the primary reference document for the Sixth Replenishment drive now underway.

The new methodology is described in a Note on Methodology prepared by the Global Fund at the time of the report’s publication. (For more on the report and the methodology, see articles in GFO 341 and GFO 342.)

In a 25 January article in The Lancet, Dr. Rocco Friebel of the London School of Economics and Political Science and three other health economists presented a critique of the new methodology. “It is our judgement that the Global Fund’s results reporting is insufficiently rigorous to inform the allocation of scarce resources,” they stated. “The method for calculating lives saved is obscure and risks overstating the Global Fund’s effects,” the authors further cautioned.

Co-authoring the article with Friebel were Rachel Silverman and Amanda Glassman, both of the Center for Global Development in Washington, D.C.; and Dr. Kalipso Chalkidou of the Center for Global Development and the School of Public Health, Imperial College London.

The 2018 results report provides results to the end of 2017. Under the new methodology, national results figure more prominently than they did before. Previously, the Fund’s global results were derived from a mixture of individual program and national results, although over the years national results assumed more and more importance.

(National results are the combined outcomes achieved by Global Fund–supported programs, programs funded by other donors and domestic investments –– for a given disease in a given country. Individual program results are the outcomes achieved by programs directly supported by the Global Fund.)

In a response emailed to supporters on the same day (25 January) and posted on the Fund’s website, the Fund said: “For the Global Fund, what matters most is […] the aggregate impact of all funding, whether provided by us, other external assistance or from domestic sources…. We don’t want to celebrate the results from an individual project when a country’s overall health strategy is off-track.”

Seth Faison, Head of Communications for the Global Fund, told Aidspan that “after studying this issue with our partners long and hard, we are convinced that a partner-based approach to results reporting makes the most sense. Attempts to separate results by organization are not only outdated, they also undermine what global health advocates are trying to achieve.”

Faison said that in the process of developing its 2017–2022 strategy, the Global Fund undertook extensive consultations with partner organizations (such as the World Health Organization, UNAIDS, PEPFAR and other technical and academic institutions) and key experts in the fields of data analysis and results reporting. Those partners and experts were engaged to agree on an approach for performance targets and results reporting that was both useful and regarded by all partners to be scientifically sound, Faison said.

(Some of the more granular findings and recommendations emanating from that consultative process appear in the 2014 “Report of the First Meeting of the expert panel on health impact of Global Fund investments.”)

Contributive vs. attributive results

The main crux of The Lancet critique is that the “contributive” approach to results reporting now used by the Fund is less accurate and less useful to donors and observers than an “attributive” approach would be. The contributive approach to results recognizes that many actors (various donors, domestic funders and implementers) contribute to the interventions and systems that ultimately save lives. The Fund considers all of the “saved lives” achieved by national responses in which it is involved to be part of its 27 million lives saved figure. So, PEPFAR, Gavi, the Vaccine Alliance, other donor mechanisms and domestic resources may also be involved in saving some of the same lives –– thus, Friebel et al say, “making it difficult to distinguish comparative advantage of the investment case for each funding or technical assistance instrument.”

An attributive approach is narrower and would likely produce a number lower than 27 million. Such an approach to results would rely more on Global Fund grant and program reporting than on national results.

Friebel argues that an attributive approach is more appropriate, and that the Global Fund cannot, on the one hand, claim credit as “a stand-alone funding instrument” and on the other, state that all results are achieved through partnerships. To illustrate his point, in an email to Aidspan Friebel asked: “If results achieved in partnership are being credited, then would the Global Fund be willing to share funds raised as a funding instrument at the replenishment?”

To Friebel, the Fund is using an attributive approach to fundraising while it remains, he asserts, contributive in its results reporting.

In its email of 25 January, the Fund stated that while the specific impact of programs funded by the Global Fund and overall national results are both important, “the primary focus of our results reporting should be on national results. This reflects the Global Fund’s mandate. While we want to ensure that our specific interventions deliver the desired impact, our overarching goal is to accelerate countries’ progress towards ending the epidemics of AIDS, TB and malaria and in building stronger health systems…”

Faison added: “It is just not right to say that the Global Fund is ‘a stand-alone funding instrument.’ We are in fact a partnership organization that is composed of many parts – governments, civil society, the private sector and people affected by the diseases. And of course, we do share our funding with many organizations since 100% of our grants go to partner organizations. There is no such thing as a Global Fund program. So, by definition we are sharing all funding. We fund programs together with national governments and other international organizations in virtually every country where we fund grants.”

In its Note on Methodology, the Fund addressed the question of whether it was exaggerating its impact: “The Global Fund is not claiming credit for national results. As agreed with our partners, we are acknowledging the catalytic effect of international funders.”

Transparency

Another critique raised in The Lancet article concerns what the authors perceive to be a lack of transparency. “The Global Fund’s methods and resulting estimates…cannot be verified or reproduced by external researchers,” the authors said.

In an email to Aidspan, Friebel said, “We are calling for the Global Fund to make data and models publicly available, and to allow for peer-reviewed versions of the analyses. Organizations that are funded through public monies should be held accountable for their claims, which must be backed up with robust evidence.”

Friebel said he is concerned that a climate of skepticism pervades political processes today, especially in the U.S. and U.K., two of the largest donors to the Fund. He fears that what he considers to be a questionable approach to results reporting could give ammunition to those who would undermine the political will to contribute to the Global Fund and other aid mechanisms.

“With the replenishment conference forthcoming,” Friebel et al stated, “we need robust evidence of impact and due process, or we risk empowering the sceptics and compromising the good work of the Global Fund and aid in general.”

Friebel told Aidspan about a December 2018 select committee meeting of the British Parliament, in which members peppered representatives of the Department for International Development (DFID) with questions about how they measure impact. (A recording of that meeting can be viewed on parliamentlive.tv.) Friebel suggested that the intensity of such scrutiny should be expected to grow and, therefore, that results reporting must be strong enough to withstand brutal interrogation.

The Fund believes that it has been transparent. Its email of 25 January described “extensive consultations with partners including the WHO, UNAIDS, Stop TB and the RBM Partnership to End Malaria” which informed and validated the Fund’s methodology for results reporting… “These well-established and readily accessible methodologies have been developed in consultation with countries, use state-of-the-art modelling techniques, and draw on widely accepted data sources.”

Reactions

Aidspan approached several organizations for a comment on the Global Fund’s lives saved methodology.

Aidspan has learned that many of the concerns mentioned in The Lancet article were raised by several civil society organizations that are part of the Global Fund Advocates Network (GFAN) and that participated in a telecon GFAN organized on 29 August 2018 to discuss the new methodology. As far as we know, GFAN itself has not taken a position on this.

The Friends of the Global Fight advocates for support for the Global Fund among U.S. lawmakers and so is very close to the process of presenting the investment case to the Fund’s largest donors. The president of Friends, Chris Collins, said:

“All the partners agreed to this approach of measuring country results which is, after all, the goal. Multiple external reviews of the Global Fund attest to its success in focusing on results, carefully managing grant performance, and operating in a transparent fashion. We’ve seen policy makers consistently impressed by Global Fund results and the clear accounting of what each dollar invested buys in terms of bed nets, ARVs and other interventions. The results focus and transparency [are] what has built lasting bipartisan support for the Global Fund.”

Timothy Hallett and Katharina Hauck, of Imperial College London, provided the following feedback:

“First, it is reasonable for a funder to report that [it has] contributed to the impact that has been achieved by the countries with which it works rather than trying to claim exclusive credit for a particular fraction of it. Given that multiple funders are intentionally coordinating their efforts towards a common goal and, in so doing, reaping the considerable synergies of providing complementary services, analyses that aim to dissect out those various efforts would have little meaning. For example, in the case raised by Friebel et al., if PEPFAR funds medical staff, Global Fund the drugs and the government the running costs of clinics, which is responsible for the persons receiving treatment?

“Second, we do not agree with Friebel et al. that modelling is an ‘obscure’ method that ‘risks overstating the Global Fund’s effects.’ The simplicity of instead relying on direct output metrics may appeal but this would neglect many factors, including the substantial ‘spillover’ effects of interventions against infectious diseases at a population-level: one more person on ART or sleeping under a bed net can result in more (or less) than one life being saved. Modelling is a reasonable and common means by which an estimate of impact against a counterfactual can be derived. Whilst we agree that the results of modelling are bound to be uncertain and subject to assumptions and that these issues must be communicated carefully and clearly, the authors offer no justification for their assertion that a bias will necessarily be in the direction of overstatement.”

Hallett and Hauck noted that they have both received contracts in both a personal and institutional capacity from the Global Fund for work related to the Fund’s investment case and strategy modelling.

With respect to what Hallett and Hauck said above about modelling, Friebel told Aidspan that he is not saying that modelling is an obscure method; rather, he is saying that “models (executable versions with their underpinning assumptions and data sources) ought to be made available to allow scrutiny from groups not directly funded by the Global Fund.”

Friebel added:

“Whether other groups have agreed to a contributive approach does not answer my question of whether the Global Fund would then be willing to share monies with the other partners… And if not, how can one calculate a return of investment for the Fund? And how can one protect against development partners paying twice (or thrice –– the Global Financing Facility, the WHO and PEPFAR all make similar contribution cases)? The most important point here is that we are NOT against modelling, we are against any form of research and analysis which resists (or whose producers resist) open scrutiny –– the foundation of academic review and science.”

Peter Denis Ghys, head of Strategic Information and Evaluation at UNAIDS, commented as follows:

“UNAIDS welcomes the recent discussion on the transparency of the methods used by the Global Fund to report on impact. UNAIDS highly values transparency around modelling methods. UNAIDS understands that the methods used by the Global Fund include UNAIDS-compiled data (available on http://aidsinfo.unaids.org ) in addition to counterfactual scenarios.

“UNAIDS also believes that limiting the analysis to the impact that can be directly attributed to the Global Fund misses the larger goal of the Global Fund to leverage additional efforts including national actions toward the three diseases. Also, such attributable impact would be very hard to estimate, including because of the changes over time in the Global Fund’s investment portfolio in any given country.”

+++++++++++++

One point that the Fund, the authors of The Lancet article and many others agree on is that the reliability of health data in low- and middle-income countries is not, on the whole, where it sorely needs to be to ensure the scientific accuracy of results reported or to adequately assess program performance. In The Lancet, Friebel et al note that “uncertainty is unavoidable, driven by incomplete death-registration data, weak or no evidence of comparative effectiveness of alternative treatments, and scarcity of reliable, routinely collected billing information to estimate health-care resource use….” For them, this is reason enough to nuance and qualify impact claims by the Fund and other donors.

More details on the Global Fund’s results methodology can be found here: “Estimating Lives Saved and Case/Infection Averted by Global Fund-Supported Programs.”