In-Country Data and Data Systems

Author:

Oliver Campbell White

Article Number: 5

Will countries respond accordingly and sustain improvements?

The Office of the Inspector General’s audit report on in-country data and data systems presents findings and agreed management actions that cut across the entire Global Fund portfolio. It reports on inadequate programmatic data quality that reduces performance monitoring and capacity to adapt programs to changing contexts; deficient procurement and supply chain data that negatively impact the ability to mitigate stock-outs and expiries; health information systems that are not effectively used to support data quality; and supervision and assurance on data that remain weak, thus limiting the reliability of results. This is an important report but the extent to which the agreed management actions will resolve these issues will, as this article explains, depend on factors outside the control of the Global Fund.

On 3 April 2023 the Office of the Inspector General (OIG) issued its report on the audit of in-country data and data systems.

Background

Each country is responsible for implementing its response to the three diseases including the use of health management information systems (HMIS) that depend on quality and timely programmatic data to be effective. The Global Fund’s 2023-2028 Strategy emphasizes data-driven decision-making and calls for investing in systems and capabilities to enable the rapid generation, analysis, and use of complete, timely, and accurate context-relevant disaggregated data.

Programmatic and procurement and supply management (PSM) health information is collected through various types of systems or subsystems at national, regional and facility levels, which can be manual and/or electronic. They are either parallel to or integrated with the aggregate national health information. The Global Fund aims to strengthen and support national data systems where possible.

Typically, the national HMIS and disease-specific reporting systems collect data on routine health services that are reported from health facilities (HFs). From HFs to the national level, data undergo several collection, reporting and aggregation processes.

While in-country data and data systems are country-owned, investments and joint efforts from multiple stakeholders (governments, donors, partners, and implementers) are required to be able to develop and strengthen them. Progress in this area is contingent on the cross-cutting enablers that affect the entire data life cycle such as:

• The availability of domestic and donor funding to ensure sufficient suitably trained staff, infrastructure, equipment, etc.;
• Robust governance and implementation arrangements (e.g., laws, policies and procedures, leadership, administrative structure, etc.) to ensure sustained results; and
• Well-coordinated in-country and international level stakeholder partnerships to steer investments in the prioritized areas.

The Global Fund has supported countries to improve the quality of their data and data systems with country grants supplemented by Strategic Initiatives (SI). Due to the complexity of national health systems and limited resources, the Global Fund has leveraged partnerships with domestic, technical and other donor partners to facilitate this work. Given the complex nature of data and data systems and broader health sector challenges, improvements in health systems often require longer term investments to generate improvements.

First Finding

The Global Fund has supported improved deployment and integration of HIV, TB and malaria data into national electronic HMIS. While the ability of funded programs to report complete and disaggregated data has improved, challenges in timeliness and data accuracy remain and are impacting effective decision-making.

The Global Fund considers reporting timely when at least 80% of reports from health facilities are submitted to the electronic HMIS within the deadline. Progress has been made in improving reporting timeliness at the country level, with average timeliness across high-impact and core countries improving by 22% between 2017 (54%) and 2021 (76%). However, performance on this metric has stagnated since 2019 and remains below target as of December 2021. In addition, the average timeliness across high-impact and core countries is skewed by high performing portfolios, with 59% of high-impact and core countries having timely reporting from HFs.

Across the eight countries sampled by OIG, there were significant variances in the level of data accuracy. All sampled countries that had a national or targeted data quality review (DQR) during the audit period (since 2018) had rated data accuracy of selected indicators as poor or very poor for at least one of the three diseases:

In addition, 18 OIG country audits since 2018 (45%) identified material gaps in data accuracy relating to discrepancies between: (a) data reported in the HMIS and the primary source documents at HFs; and (b) consumption and patient data. The OIG attributes these data inaccuracies to:

At the sub-national level and delivery points:

• • Issues with fragmented information systems causing increased workload for staff and decreasing the time and ability to ensure accurate data are reported;
  • Lack of data reporting tools and registers, resulting in non-standard recording that increases risk of errors;
  • Staff capacity issues (vacancies, lack of training and knowledge gaps), resulting in non-adherence to guidelines and procedures; and
  • Gaps in programmatic data monitoring and oversight that cause data errors to go unidentified.

At the national level:

• • Challenges using DHIS2 to analyze data and improve data quality. DHIS2 has built-in functionality to support data accuracy checks, but it is not being effectively used to identify and correct data errors. Data quality validation rules for individual disease components were either not set up or data quality apps functionality were not used at national or sub-national levels. In seven of the sampled countries that use DHIS2, data quality apps were not adequately deployed or used.

At the Secretariat level:

• • Limited portfolio-wide monitoring of data accuracy by the Secretariat. In-country data accuracy is also not systematically tracked: there are no strategic or operational key performance indicators (KPIs) at the Secretariat level to track data accuracy, which has allowed data accuracy issues to persist.

Second Finding

There is limited end-to-end supply chain data visibility and there are significant data quality issues at the country level. Limited investment and lack of a detailed approach to strengthening PSM data and systems undermines the effectiveness of the Global Fund to address these issues. Limited availability, quality and triangulation of consumption data negatively impacts supply chain management leading to increased risk of stock-outs and expiries at HF level.

Over 55% of the Global Fund’s total grant allocations relate to health product procurement and supply chain-related activities. However, the availability of country-level data for PSM activities is limited and the quality of related consumption data is poor. Fragmented Logistics Management Information Systems (LMIS) limit the availability of aggregate consumption data at the central level. There are also material variances between health product consumption and patient/programmatic data. These variances can be attributed to low-quality reporting for patient data in DHIS2 and/or consumption data in LMIS and the failure to routinely triangulate patient and health product consumption data.

There is limited direct investment in routine data quality, reporting and verification to support a robust PSM process. Grant funding for health product management systems is mainly applied to infrastructure (e.g., construction, maintenance and renovation of warehouses) and other enablers (e.g., technical assistance) for storage and distribution of health commodities; but only 16% of grant funding for health product management systems relates to investment in PSM data and data systems.

At the Secretariat level, the Global Fund’s ability to address data gaps is undermined by:

• The absence of an implementation framework focused on supply chain data and systems;
• The lack of appropriate measures to track availability and quality of in-country supply chain data and systems limits visibility of country-level PSM data maturity;
• Fragmented roles and responsibilities over programmatic and PSM data in the Secretariat which undermine implementation of cross-cutting interventions; and
• Limited central visibility of funding for country PSM data systems which impacts the ability to measure, monitor and evaluate the impact of Global Fund investments.

Third Finding

Investment in programmatic data systems at the country level and country ownership are priorities for the Global Fund.  Although the Global Fund has a strategic framework for programmatic data in place, gaps have been identified in performance measurement and tracking which have limited the Secretariat’s understanding of, and response to, the residual challenges for data quality.

The Global Fund developed the Global Fund Strategic Framework for Data Use for Action and Improvement at Country Level, 2017-2022 (DUFAI) to support its objectives and achieve the KPIs. For the five components in the DUFAI, there are a total of 20 related indicators. However, these indicators are not adequately tracked and monitored, with only six of the 20 indicators formally reported to senior management. This is despite the DUFAI requiring continuous evaluation and annual reporting to senior management. There are also operational gaps in tracking data completeness since the Secretariat does not monitor completeness of indicator data at the facility level. This is different from overall reporting completeness, which only assesses the receipt of monthly or quarterly reporting forms.

Tracking and measurement of performance against the DUFAI indicators is hindered by:

• The Global Fund’s M&E system profile platform which is not working as planned; and
• Fragmented roles and responsibilities at Secretariat level over programmatic data.

Fourth Finding

The Global Fund has developed detailed guidance and tools to monitor data availability and quality, but inadequate supervision and oversight at the facility level undermines the ability to detect and remediate material data inaccuracies.

At the grant level, there are HMIS and M&E performance indicators within grant performance frameworks for at least one grant in all eight sampled countries. All have at least one national level indicator for reporting completeness or timeliness within grants with significant funding for HMIS and M&E interventions. This enables the Global Fund to measure, monitor and evaluate the impact of Global Fund investments and track the trajectory of routine data quality at country level. For NFM3/GC6 grants, four of the eight sampled countries have an outcome indicator to track national aggregate HMIS deployment and functionality.

Despite this, the OIG identified two challenges in the effective monitoring of programmatic data at the health facility level and for in-country assurance:

With respect to the second bullet, there is limited depth in the verification of reported data and a lack of focus on core M&E specific reviews by the LFA in relation to programmatic data. This is due to:

• • Limited core programmatic data assurance for high M&E risk portfolios;
  • A lack of in-depth verification of reported data at the primary source; and
  • The significant proportion of M&E expert time spent on cross-functional reviews.

OIG points out that the failure to obtain adequate assurances against material data inaccuracies in reported data may lead to inadequate decision-making, program planning and performance measurement at country and Secretariat levels.

Agreed Management Actions

To address the above findings, the following three management actions have been agreed with the Global Fund Secretariat:

1. The Secretariat will: (a) develop by 30 June 2024; and (b) implement by 30 June 2025 an end-to-end operational framework to enhance country programmatic data quality (including the dimensions on completeness, timeliness, and accuracy), data use, data analytics (including triangulation) and assurance and oversight. The framework will include:

▪   Processes of monitoring and reporting on data quality

▪   Roles and responsibilities across the Secretariat; and

▪   Enhanced assurance on data quality.

To support the embedding and implementation of the operational framework, relevant updates to Secretariat systems, tools and processes will be completed.

2. The Secretariat will: (a) develop by 30 June 2024; and (b) implement by 30 June 2025 a detailed implementation framework to support the operationalization of the Supply Chain Road Map with the objective of enhancing digitalization, data availability, data analytics (including triangulation) and use of in-country supply chain data. This framework shall include key metrics to monitor and assess the availability and quality of in-country supply chain data, processes to report on these metrics and guidelines on coordination with global partners.

Commentary

This report does not make for easy reading and one needs a good knowledge of the topic to understand the importance of the findings. It is an important report because the subject applies not just to the Global Fund’s portfolio but other donors like the US President’s Emergency Fund for AIDS Relief (PEPFAR) and the President’s Malaria Initiative (PMI), as all of them use the same reporting systems and report on the same data. The improvements highlighted by the OIG are essential to achieve grant performance objectives.

The difficulty that the Secretariat is likely to face is that, despite developing and implementing the operational framework and the framework to support the operationalization of the Supply Chain Road Map, there is no guarantee that in-country data and systems will improve. There has to be the will in country to improve health data recording, transmission, collation and dissemination, with a focus on improving data recording at primary health facilities. That raises two issues.  First, the necessary knowledge, resources and motivation have to extend from the centre, through regional and area bodies, to primary health clinics; and that is a tall order. Second, some key populations (KPs) have to be considered.

While important for tracking performance, data can also be a danger for some KP individuals. In order to account for the consumption of antiretroviral treatment products, a health clinic has to report the number of patients services and quantities dispensed.  However, members of some KPs – particularly those threatened by social or legal repercussions – are reluctant to access health services unless they are confident that their identities and social classification remain confidential, preferably unrecorded. The concern for those persons is that their identities may also be reported or that a clinic’s record of their identity and other details may be accessed by regional and central authorities. Hence persons may be tempted to give false names or to use different identities at different clinics; and, when that occurs, data on the numbers of persons treated and on adherence are distorted.

We live at a time of increasing corruption globally. In that context, data systems are attractive to potential hackers because they offer opportunities to copy and/or delete data, whether for financial gain, political reasons or other personal reasons. Those systems must therefore be secure. Surprisingly, security is not mentioned at all in this OIG report. However, when we shared this article with the OIG, its response was that security was not the focus of the audit. Yet it seems to us that security is a fundamental issue when it comes to data systems and data management and at the very least this should have been acknowledged.