Contextual factors and health service performance from the perspective of the provincial health administrators in Papua New Guinea

Introduction

The Rural Primary Health Services Delivery Project is an initiative of the Government of Papua New Guinea (PNG). The project aims to improve the coverage and quality of rural primary healthcare services through a range of initiatives at national, provincial, district, health facility and community levels¹. The project is being piloted in two districts in eight provinces, with the intention of further roll-out if effective. The project is expected to improve health system performance according to a range of access, quality, and maternal and child health indicators.

While health programs are largely measured through quantitative indicators, it is acknowledged by WHO that ‘Health systems are complex and their performance and impact are difficult to capture using only quantitative indicators’². The context within which a program is implemented can have an effect on the success or otherwise of the program. In order to measure progress and impact of a program, or to scale up an intervention, it is important to understand the contextual factors that may influence performance^3,4. In PNG, several factors have been identified to negatively impact health facility performance, including remoteness; access to funds; and cultural beliefs of the community, particularly around family planning, pregnancy and birth^5-7. These factors have largely been identified through quantitative studies where data on predefined variables were used but may not have captured the full range of factors that could influence performance. A key challenge for this evaluation was to measure changes in coverage and quality of primary health care in eight very different provinces.

Qualitative data are regularly used in monitoring and evaluation of health programs⁴. Health administrators can offer a useful perspective on the reasons for high or low performance at the health facility and district level due to their contextual knowledge. In order to understand the contextual factors that may affect performance of the project, the performance of key indicators were analyses and a contextual analysis was conducted through interviews with senior provincial health administrators to document perceived factors affecting performance at baseline. The authors propose that this sequential explanatory mixed methods approach to the contextual analysis could be beneficial for evaluating complex programs in low-resource settings.

Methods

Setting

The project commenced in 2012 and operates in two districts in each of eight provinces of PNG: Central and Southern Region in the Autonomous Region of Bougainville, Wewak and Maprik in East Sepik, Okapa and Kainantu in Eastern Highlands, Laiagap-Porgera and Kompian-Ambum in Enga, Alotau and Kiriwina-Goodenough in Milne Bay, Bulolo and Menyamya in Morobe, Talasia and Kandrian-Gloucester in West New Britain and Tambul Nebilyer and Mul-Baiyer in Western Highlands¹. These eight provinces are spread across the four regions of PNG (Highlands, Islands, Momase and Papua regions). The project supports the review or development of national policies and standards, establishment of partnerships at provincial level with state and non-state partners in health; health worker training, infrastructure development through construction of two community health posts in each district and community-level health promotion activities to improve demand for services¹.

Study design

Two health system consultants were engaged to conduct a formative evaluation during implementation from 2013 to 2017. A performance assessment framework was developed for the project. In line with best practice, and to avoid duplicate data collection systems, the indicators were aligned with the National Health Plan Monitoring and Evaluation Framework and data for the indicators were taken from the existing National Health Information System (NHIS) at the facility and district level⁸. A sequential explanatory mixed-methods study design was employed involving analyses of the indicators in the performance assessment framework for the baseline year (2013) and subsequent key informant interviews with provincial health administrators from participating provinces to help understand the context of the performance indicator⁹.

Indicator analysis

The NHIS is used to record data on outpatient and other services provided at health facilities. Data were transcribed from automated reports from the NHIS for 2013 into Microsoft Excel. The indicators calculated were outpatients per person per year, outreach clinics per 1000 children aged less than 5 years, measles immunisation coverage at 9–11 months, family planning couple years protection per 1000 women of reproductive age, proportion of pregnant women who had at least one antenatal care visit (ANC1) and proportion of pregnant women who had a supervised delivery using the methods defined in the National Health Plan Monitoring and Evaluation Plan⁸. All indicators were adjusted for missing monthly summary reports by dividing by the percentage of reports submitted for the year. Health facilities with no catchment population provided were excluded because rates or percentage coverage could not be calculated. Indicators at the facility and district level for each indicator were compared to national targets⁸.

Key informant interviews

Face-to-face key informant interviews were conducted with all provincial health advisors or provincial health authority chief executive officers (interviewees) from the project provinces. The interviews were conducted in English by three trained monitoring and evaluation staff. The interviewees were considered key informants, having in-depth knowledge of the health system and more than 10 years of experience.

The interviews were semi-structured with questions about factors affecting access, utilisation and quality of health services. The interviewees were provided with a table that included the indicators from the NHIS data at the facility level for districts within their provinces. Some of the probing questions were derived from these tables, specifically asking about the context in which facilities with exceptionally high or low performance in indicators operate. Audio recordings were transcribed and data were coded using NVivo v10 (QSR International; http: www.qsrinternational.com/nvivo). First, the data were coded according to the question. Second, inductive thematic analysis was conducted for three of the interviews, which means that themes were identified from the data¹⁰. From this initial analysis, the main themes identified were geography, cultural and religious beliefs, human resources, financing and health information, with further themes identified within these main themes. A coding framework with these themes was developed and applied to all interviews.

Ethics approval

The key informant interviews were conducted only after informed consent was obtained. The audio recording was used for transcription purposes only and the transcripts were de-identified. A diversity of quotes was sought for presentation in this article, and interviewee numbers have been removed to protect anonymity. The study was approved by the PNG Medical Research Advisory Committee (15.10) and the University of New South Wales (HC15208).

Results

NHIS analysis

For brevity, only four of the most relevant indicators from the performance assessment framework are presented, covering health system access, and maternal and child health. The indicators presented are outreach clinics per 100 children aged less than 5 years, measles vaccination coverage at 12 months of age, ANC1 coverage and supervised delivery coverage. The performance indicators varied widely across the 16 districts (Tables 1,2). Both districts in Milne Bay met all four targets for the indicators presented. Conversely, no targets were met in Central Region in the Autonomous Region of Bougainville, Kandrian-Gloucester in West New Britain and both Bulolo and Menyamya in Morobe.

Six districts met the outreach clinic rate coverage target. In these, 100% (7/7) of facilities in Kiriwina-Goodenough and 29% of facilities (2/7) in Kompian-Ambum met the target. Seven districts met the target for measles vaccination with three districts notably exceeding 100%. Similar to results for the outreach clinic rate, a variable proportion of facilities within those districts met the target for measles vaccination: from 29% (2/7) in Maprik to 100% (3/3) in Kainantu. Seven districts met the ANC1 target. In Kainantu, the ANC1 target was not met despite all facilities listed in the NHIS having met the target. Six districts met the target for supervised deliveries and of these districts Laiagap-Porgera and Wewak had only one facility that met the target (1/6 and 1/9 respectively) while in Kiriwina-Goodenough 86% of facilities (6/7) met the target.

Key informant interviews

All provincial health advisors or provincial health authority chief executive officers (interviewees) from provinces covered by the project participated in the interviews. The interviewees (n=8) had been working in health for 12–43 years with a majority of that time working in the province in PNG where they were currently employed. Together they represented all of the eight provinces. The responses presented here relate to discussion of all indicators presented: outpatients per person per year, outreach clinics per 1000 children aged less than 5 years, measles immunisation coverage at 9–11 months, family planning couple years protection per 1000 women of reproductive age, proportion of pregnant women who had at least one antenatal care visit and proportion of pregnant women who had a supervised delivery.

Geography

Difficult terrain for travel, either mountainous regions or locations only accessible by sea, was a reason given by many interviewees for poor performance of health facilities (Table 3). The negative impacts cited from challenging geography include people presenting to the health facility only if they are seriously ill; restriction of health workers’ ability to conduct outreach in the catchment communities of their facilities; limited support and distribution of essential supplies from the district and provincial level to remote facilities; and increased costs for providing health services in remote locations, particularly if air travel is required. By contrast, high performance was attributed to the health facility location being accessible by road and in urban areas, where people frequently travel for other purposes.

Cultural and religious beliefs

When asked if cultural beliefs affect access and utilisation of health services, interviewees often cited beliefs in sorcery, the preference of women to deliver babies in the village and religious beliefs around the use of family planning. For example, in some communities people believe that illness is caused by sorcery rather than believing in traditional Western beliefs about causation, which influences their health-seeking behaviour, including late presentation to health facilities. Several cultural issues were attributed to women preferring to deliver babies in the village, noted by two of the interviewees. These were the expectation by male health workers that they do not deliver babies and that women prefer to deliver in the village with a known person rather than delivering at a facility with the possibility of a male health worker in attendance. The low uptake of family planning was attributed to the community beliefs around family planning and some church-run health facilities not providing contraceptives.

Human resources

Human resources was the theme mentioned most frequently as impacting people’s access and utilisation of health services. Having an adequate number and distribution of skilled staff was seen as a vital component for improving health services and access by the community. Half of the interviewees noted that the existing health workers and support staff did not have adequate skills to do their jobs.

In remote locations, some interviewees said health workers face issues with accommodation and limited support. Health workers in remote locations may frequently travel away from their facility to go to urban areas, and when staff are not at their health facility people will bypass the facility and go to a higher level facility such as a hospital. The corollary is that health workers were perceived to be the reason for high performance of some facilities. For two interviewees, the availability of staff was a reason for high performance and they reported that people will seek out these facilities even if they have to travel further. In addition to the presence of formal health workers, high performance was attributed to village health volunteers by one respondent, particularly in encouraging women to attend antenatal care and have supervised deliveries.

Financing

The majority of interviewees mentioned funding as a factor contributing to low performance. In particular, delays in funds were mentioned as a factor affecting the delivery or services. One interviewee stated there were sufficient funds but the main issue was with the delay in receiving the funds. Another interviewee stated that the issue was not insufficient funding but the use of the funds available and the capacity of staff. The experience of the recently introduced free healthcare policy demonstrates the possible impact of the misuse and poor management of funding, with the reintroduction of user fees to ensure facilities can remain open after funds have been depleted.

Health information

Several interviewees stated that the NHIS data were not accurate either because of inaccurate reporting from facilities or inaccurate population estimates for calculating percentages and rates. Regardless of questions around data quality, two interviewees mentioned using data for reviewing performance at the facility level and creating targeted interventions to improve performance on a specific indicator.

Discussion

Provincial health administrators perceive that multiple factors from the health system and the context in which health facilities operate affect performance. Health facilities accessible by road in urban areas with competent and/or higher level staff and health services operated by churches or private companies were cited as contributors to high performance. Conversely, inadequate numbers of staff, poorly skilled staff, delays in funding and challenging geography were the major contributors noted for poor performance. The authors identified a large variation in performance indicators between and within the project districts: while a district may meet the national target, a high proportion of facilities within the district may not, suggesting that coverage of services is not optimal.

Several factors affecting performance identified in this study have been identified previously: workforce of appropriate number, cadre mix and skills¹¹; challenging geographical access to health facilities^6,12,13; delays in receiving funds for health services⁵; and cultural factors, particularly for maternal and reproductive health indicators¹⁴. This contextual analysis offered new information on factors affecting performance such as the availability of health workers at a facility and use of data to develop targeted interventions to improve performance. Further, the contextual analysis aided understanding of quantitative data used for the indicator analysis.

The interviewees noted issues with data quality and district and health facility catchment populations used within the NHIS to calculate coverage and rates. Issues with the NHIS data quality and completeness have been documented^15,16. The population catchments used for calculating percentage coverage were estimates based on projections from the 2000 population census. In this study there were many coverage rates well above 100%, which could be due to inaccurate numerator data from the NHIS, inaccurate population estimates, or people from outside the catchment area accessing the facility, who would not be included in the population denominator. According to the interviewees, some facilities with coverage rates well above 100% were accessed by people outside the catchment area either due to the location in urban areas or perceived quality of services.

The indicators used in this study are usually only presented at the national, provincial or district level^12,16,17. As seen in this study, individual health facility performance is highly variable, and reporting at a district level may mask underperformance in a number of facilities. While previous studies have suggested district-level benchmarking is important, it is also important to review data at each facility. The Reaching Every District strategy encourages monitoring of data and plan development at the health facility and district level to improve immunisation coverage, and has been adapted for prevention of mother-to-child transmission of HIV^18,19. The Reaching Every District strategy is supported by the National Department of Health for immunisation^20,21. However, most districts had multiple facilities well below the target coverage for measles immunisation, suggesting that the strategy has not been effectively implemented. Two interviewees did talk of using a strategy similar to Reaching Every District, whereby they reviewed indicators, developed interventions and observed improvements in indicators.

There are several limitations to this study. As noted by some of the interviewees, the validity of the NHIS data may be questionable due not only to inaccurate reports but also inaccurate populations for coverage and rate calculations. This limitation highlights the benefit using mixed methods for this evaluation, particularly sequential explanatory mixed methods where qualitative data can explain the quantitative data⁹. A second limitation is that the key informant interviews provided qualitative information that is the perception of a limited number of people; it does not necessarily mean that these are the only views. These interviews form just one perspective of the health system in PNG.

Conclusion

Undertaking a contextual analysis using a sequential explanatory mixed methods design for the baseline of the formative evaluation of the Rural Primary Health Services Delivery Project proved useful for understanding the context in which the health facilities operate and potential issues with the data for use in the evaluation. Further, analysis of indicators at both district and health facility level enabled a richer understanding of the heterogeneity of performance. Subsequent to this study, the focus of the evaluation methods shifted from predominantly quantitative to predominantly qualitative methods to document the contextual factors that enabled or prevented project progress. Further, analyses of performance in indicators were undertaken at the district and health facility level rather than just the district level as planned. Such ongoing contextual analyses and facility-level analysis of indicators may be useful for health service delivery programs in similar contexts where health information data quality is poor and the context is not adequately documented elsewhere.

Acknowledgements

The authors sincerely thank the health administrators who participated in this study. This work was carried out as part of the formative evaluation of the Rural Primary Health Service Delivery Project. The Rural Primary Health Services Delivery Project is a joint initiative of the Government of PNG, the Asian Development Bank, Australian Department of Foreign Affairs and Trade, OPEC Fund for International Development, World Health Organization, United Nations Children’s Fund and the Japanese International Cooperation Agency.