In a follow-up audit of the Global Fund’s Key Performance Indicator (KPI) Framework, the Office of the Inspector General (OIG) found that reporting, governance and oversight arrangements are effective but that the Global Fund Board and the Secretariat have room for improvement in their use of KPI data for their operations.

The audit report was published on 27 March 2019. The Global Fund Board approved the 2017-2022 KPI Framework in June 2016 as a tool for monitoring progress and improving the organization’s accountability.

This audit focused on:

• Adequacy and effectiveness of KPI data collection, aggregation and reporting
• The utilization of KPI data by the Global Fund Board and the Secretariat in performance-based decision
• The established governance and oversight arrangements for the KPI Framework.

The audit covered all Strategic and Implementation KPIs, except for KPI 12a and KPI 12b, which relate to the availability of affordable health technologies, and iKPI-h about the Pooled Procurement Mechanism (PPM). KPI 12a, KPI 12b and iKPI-h were reviewed in an OIG audit report published in September 2018 (Procurement processes: Follow-Up Audit).

Context

Key Performance Indicator Framework 2017-2022

The Framework contains 12 Strategic KPIs and nine Implementation KPIs (iKPIs). The Strategic KPIs measure the Global Fund’s strategic objectives progress. The iKPIs, which were approved in January 2017 by the Global Fund’s Management Executive Committee, track inputs, outputs and outcomes required to achieve the Strategic KPIs and operational objectives.

Figure: Strategic Key Performance Indicators

Changes in the management of the KPI Framework

As of mid-2017, the handling of the KPI monitoring and reporting process was transferred from the Global Fund Finance department to the Strategy and Policy department. This was done to enhance the alignment of the KPI Framework with strategy monitoring.

Main Findings

The OIG had two main findings:

• KPI data collection, aggregation and reporting significantly improved but control weaknesses were noted
• Limitations in KPIs’ design hinder their integration into daily operations

KPI data collection, aggregation and reporting significantly improved but control weaknesses noted

The OIG report noted that generally there has been a significant improvement in KPI data collection, aggregation and reporting. This is following the implementation of the KPI Accountability Framework, since early 2017, which details data collection and calculation methodologies. The roles and responsibilities for strategic and implementation KPIs were clarified in the KPI Accountability Framework. Moreover, the Secretariat established focal points in different departments that specifically deal with KPI data collection, processing and validation and this has improved KPI results reporting. The use of internal information systems like the Global Fund System (GFS) and the Grants Operating System (GOS) has enhanced data collection processes, the OIG said. The OIG noted that there are plans to fully automate data collection processes soon.

Despite these improvements, the audit noted gaps in data collection and aggregation, specifically in quality control. For instance, data for KPI 7  – Fund Utilization: Absorptive Capacity – is obtained from two different information systems – GMS and GOS – without a proper reconciliation mechanism to ensure the quality and accuracy of the information obtained. This lack of reconciliation results in duplicated data to the Global Fund Board. Methods for collecting documentation to aggregate KPI 11 data – Domestic Investment – are not standardized and routine, and lack a mechanism to trace how each country’s results are put together. This lack of documentation means that only the staff who were involved in the original data gathering are sources of information on the methodology of data aggregation.

For some of the KPIs, data collection and aggregation are manual and thus prone to human error during data ‘cleaning’. An example is KPI 9b on Human Rights where data is extracted from detailed grant budgets and analyzed manually. Considering the volume of grant budgets received by the Global Fund, this is a tedious and error-prone process. Another gap the OIG identified was that some KPIs had incomplete process maps (implementation arrangement maps) at the time of the audit. This was noted for half of the iKPIs and four of the 22 KPIs. However, the Secretariat had completed all but one of the KPI process maps by the end of 2018. For some of the KPIs, their data is not readily available. For instance, the Global Fund does not have the primary data for KPI 11 on Domestic Investment and so the data is obtained from external sources. The Global Fund has limited control on availability and consistency of parameters obtained from such data.

KPI data reporting is done annually or semi-annually with exception of KPI 9b, which is reported on a quarterly basis. The KPI data is submitted to the KPI focal points for calculation and reporting. The audit noted significant delays in KPI data submission. A case in point: only one KPI met the KPI team’s internal submission deadline during the November KPI reporting process. Another challenge the OIG noted is the lack of secondary review of the data collected, calculated and reported by the focal points in different departments. The lack of validation of data, and of the process involved in the KPI focal points’ report writing, leaves inaccuracies undetected, the OIG said, as KPI team may not be well versed on the underying data.

Limitations in KPI design hinder their integration into daily operations

The Global Fund Board approved the current Strategic KPI Framework eight months before the beginning of the 2017-2022 strategy implementation. This is a significant improvement, considering that it took the Board 18 months after the implementation of the strategy had begun to approve the last framework. The KPIs are aligned to the strategy, and the strengths and weaknesses of each indicator have been identified and documented in the KPI framework.

However, the audit noted areas that require futher improvement. The methodologies for measurement and processes related to KPI 6a (Strengthen Systems for Health: Procurement systems) remain undefined despite the KPI Framework having been in use for over a year. This was also the case for three of the ten implementation KPIs. The Global Fund Secretariat had not defined indicators, set targets and agreed on data collection tools for three iKPIs, namely the ‘forecast accuracy – commodity demand’, ‘commodity procurement under management’ and ‘roll-out of innovative products’. However, the OIG noted that major restructuring of the departments responsible for these KPIs was underway at the time of the audit.

The current KPI Framework lacks performance tracking for three of the seven sub-objectives that make up the lion’s share (65%) of Global Fund’s RSSH investments. The three sub-objectives are for ‘Human Resources for Health’, ‘Communities’, and ‘Integrated Service Delivery’. However, the Secretariat monitors the progress of these sub-objectives through the Global Fund Strategy Implementation Plan (SIP) reporting.

The audit noted the use of interim indicators for two KPIs – 5 and 9c – to monitor performance due to unavailability of data to measure their progress. In other instances, availability of data was limited for KPI 2. Data for one of the sub-objectives under KPI 2 – Indoor Residual Spraying: service delivery target – was only available for 30% countries implementing the intervention.

Entrenching KPIs into the Global Fund’s daily operations is hindered by limitations in their design, the OIG said. For instance, the Global Fund Board has evolved from discussing methodology or targets for each KPI to discussing data being reported. However, limitations in obtaining additional data and analysis prevents the Board from carrying out a detailed assessment of organizational performance. At the Secretariat level, some KPIs are integrated into day-to-day activities, and some are not. For example, KPI 11, KPI 2 and KPI 7 are embedded within Grant Management operations, which is not the case for KPI 4 on Investment Efficiency.

Agreed Management Actions

The Strategy and Policy Hub will work on two Agreed Management Actions (AMAs) by 31 December 2019:

• Review and fine-tune the KPI process maps and Accountability Framework to include KPI data quality controls
• Develop and implement new standards for measuring, reporting and monitor organizational performance

Further reading:

• This audit report, The Global Fund’s Key Performance Indicator Framework Follow-up Audit, 27 March 2019 (GF-OIG-19-008)
• Earlier KPI audit report, The Global Fund Key Performance Indicator Framework Audit Report, 26 March 2016 (GF-OIG-16-009)
• Earlier strategy audit report, The Global Fund Strategy Planning, Implementation and Monitoring Processes (GF-OIG-16-008)
• Current strategic KPI Framework, 2017-2022 Strategic Key Performance Framework, (GF-B35-07a).