Problems measuring community health status at a local level : Papua New Guinea ' s health information system

Introduction: The Papua New Guinea Department of Health monitors the performance of the health system using a computerised national health information system. This article draws on the recent evaluation of a national-wide donor-project community development initiative to highlight the problems of the lack of and disaggregated village health data. This data could be used to monitor health status, health worker performance and intervention impact. Methods: An extensive outcome evaluation conducted in 2006 used qualitative and quantitative data. The in-depth study covered 10 provinces (50%) and 19 districts (21%), obtaining data from 175 health personnel informal interviews and 77 community focus group discussions. Quantitative data from the health information system were examined for validation of the qualitative findings over a 7 year period (1998-2004). Results: Healthier lifestyle and enhanced social and economic wellbeing were claimed by the community to be the result of the project intervention. The evaluation found village claims of post-project improved physical health, increased use of health services and reduced maternal and child mortality could not be substantiated statistically. Health-centre data failed to provide a complete and accurate assessment of community health status within the national health information system.


Introduction
Health care in Papua New Guinea (PNG) is provided through a unified system of community aid posts, rural health centres and provincial hospitals 1 .A network of over 2400 aid posts, 500 health centres and 45 urban clinics are supported by 18 provincial hospitals and one national hospital 2 .Those who provide health care include community health workers, nursing officers, health extension officers and doctors 3 .
At the village level, the aid post serves a population of between 500 and 3000 4,5 .As a traditional village community has less than 500 people, each aid post serves between one and 6 villages 5 .Aid-post health workers provide basic primary health care and provide information on healthy lifestyle practices to prevent illness 6 .
A health centre and its staff serve a population of between 2000 and 20 000 6 .Each health centre provides curative and preventive health services and acts as a referral centre for between 3 and 5 aid posts (Fig1) 6 .District health centres and hospitals coordinate health services within the district and act as a referral point for acute conditions for their subdistrict level health centres 6 .
The provincial health office coordinates health services within the province, while the National Department of Health provides an advisory role to the provinces 6 .Staff within the National Department of Health have overall responsibility for maintenance of policy and standards, providing technical advice, coordination of the health information system, health planning and data systems 6 .
The PNG Department of Health monitors the performance of the health system using a computerised national health information system.Health information systems are considered vital for assessing population health needs, managing health service delivery and evaluating program coverage and effectiveness 7,8 .The computerised health information system was established in PNG in 1989 9 .By 2002 the system was centrally managed and regarded as providing quality data for monitoring and planning, with links across all health system levels 9 .The system uses a well-defined set of performance indicators for monitoring performance of health facility, district, province and nationally 6 .
Each aid post forwards their monthly statistics to the supervisory health centre 10 .Information gathered monthly from each health centre is sent via the district health office to the provincial health office where it is entered into a computerised database 11 .The system calculates percentages against census population data for immediate analysis 11 .The data is then sent to national level where it is transferred into the national computerised health information system 11 .
.The study found there was an absence of village-level health data.The health information system does not allow reporting of villages separately or the tracking of changes in health status over time according to identifiable villages 12 .Because policy dictates exclusion of village data from the national health information system, qualitative data could not confirm village claims of improved health 12 .
National policy dictates that aid-post data is not included in health centre statistics 6 .This means aid-post data is not forwarded on from health centre to national level, thus there is a failure to show true and accurate health status.This policy is not always adhered to because some health centres, within the same province or district, include aid-post data in their monthly statistics.Findings from the evaluation indicated this was so in numerous settings 10 .The reason given is that aid-post data is considered inaccurate and inappropriate for inclusion in the national system.Aid-post health workers are considered not to have the skills required to competently diagnose illness.However, the ageing

Methods
The 2006 outcome evaluation of a Donor Aid Project community health intervention throughout PNG used both qualitative and quantitative methods.The collection of qualitative data, its synthesis and analysis, used the rapid assessment, response and evaluation (RARE) process 13 .
Qualitative data were obtained through 175 informal interviews with health personnel at national (n = 18), provincial (n = 34) and district health facilities (n = 58), as well as from 77 community focus group discussions.Data were collected from 10 provinces (50%) and 19 districts (21%) in PNG in communities that were recipients of the donor project input 10 .Communities and health facilities were selected based on the likely variability of uptake of the donor project community intervention.Quantitative data were obtained from a review of national census population figures for communities and selected health-centre statistics over a 7 year period (1999 to 2005, inclusive).Analysis of community use of health services included antenatal, childbirth and immunization indicators, as well as the environmentally induced illnesses of malaria, pneumonia and diarrhoea.

Results
The qualitative data showed improved physical health had enhanced social and economic wellbeing as a result of the donor-aid project 10 .The changes initiated led to a healthier lifestyle and an improved hygienic living environment with improved behaviour in maternal and child health.
Communities claimed to be positively influenced by these changes and there were obvious physical differences between villages that received successful intervention and those that did not.Local ownership of changes and expressed pride in community achievements helped sustain changes over time.Communities that partnered with the local health system increased their use of health services and were helped to reduce maternal and child morbidity and mortality.
In areas with no successful project outcome or interventions, it was observed by the researcher that the workload of the aid-post worker was excessive.These health workers spent as much as 10 hours per day providing curative treatments 10 .
In areas where behavioural and environmental health changes had been made following the project, decreased illness meant health workers spent only one or 2 hours per day providing curative treatments.This freed the health worker's time for health education and to support community health initiatives, which were observed in action.Numerical data on key maternal health, child health and illness trend indicators over 7 years (1999-2005), however, provided little evidence of improvement in the health of women and children nationally.The relatively small increases in aggregated selected maternal and child health indicators were insufficient to claim a statistically significant impact on improved maternal and child health.Where aggregated health-centre data showed significant increase in maternal or child health indicators or decrease in illness trend in one year, this did not appear to have been maintained in succeeding years.Some communities reported having active village-level health volunteers educating pregnant women on safe childbirth practices, and these were apparently making a difference in qualitative measures or observation.However, the statistical impact was diminished by the large proportion of communities without health volunteers when these village statistics were aggregated at the health centre.

Anomalies in Papua New Guinea health information system
Qualitative data indicated information supplied by the national health information system had poor interpretive value in terms of monitoring key maternal and child health indicators at village level.Field work suggested problems with the quality of data overall.Direct observations suggested under-reporting, over-reporting, and errors with calculation and transcription.Frequent discrepancies were observed when provincial and national level records were compared with health-centre records 10 .
Calculations frequently produced percentages over 100% and sometimes as high as 200% or 300%, especially for antenatal visits and supervised births.Data validity and reliability is questionable when three National Department of Health documents reporting five-year performances did not match for any year 10 .The authors were told the variability in these documents was due to the practice of correcting data errors when identified, even if this was from years already reported.
Human error in transcribing data from daily tally sheets to monthly health information forms occurred frequently at health centre level, with calculation errors also being common.At provincial level there was risk of human error occurring during computerised data entry.This risk was reduced considerably because the computerised form mimics the hard copy of the health information system forms.Errors at national level were less likely to occur because data is electronically transferred from the province.The health information system records the total number of births; for example all known births occurring within the health centre catchment area 2 .Confusion in interpretation arises over Department of Health definition of 'supervised birth'.A supervised birth is classified as a birth taking place under the supervision of a qualified health worker at a health centre or hospital 2 .This excludes village births assisted by a qualified health worker or village-level health volunteer.
Observation and interviews from this study indicated health workers interpret this policy differently.Therefore, supervised birth data may or may not include village-level births supervised by health volunteers.
Village-level deaths are not required to be included in health-centre statistics.However, there may be overreporting when a woman dies en-route from an aid post or health centre to hospital because both the health centre and hospital may report the death.This appeared to have occurred in at least one province visited during this study.
Confusion can easily occur when filling in the monthly health information reports.The national health information system form is supplied to health facilities by the PNG Department of Health.The form clearly instructs to 'include … transferred patients in total' (p1).Further confusion occurs when completing the section on page 3 that instructs: '...report all maternal deaths here, even if they occur in the village…'.The inclusion of provincial hospital data with rural health-centre data impacts significantly on maternal mortality rates and reflects poorly, and possibly inaccurately, on district performance 10 .

Absence of village-level data
The reliability of data collected at health centre level was questionable.The almost random inclusion or exclusion of aid-post statistics with health centre figures made it impossible to compare health centres over time, let alone villages.This research showed the situation varied between health centres, districts and provinces.In one district, government health-centre statistics included aid-post data while church health centres did not, despite being the supervisory health centre for those aid posts 10 .In another

Recommendations
The following changes are recommended: • Change national policy to include aid-post data in health-centre statistics.
• Differentiate data by village at aid-post and health centre level.Aid-post and health-centre data already collected could record village identity next to client diagnosis/ service provided.This would allow data to be disaggregated by village and for communities to be compared within the aid-post and/or health centre catchment area.
• Collect selected data relating to village-level health volunteer activities at aid-post and health centre level.Compare these across villages over time to monitor health volunteer performance and community response to their input.
• Train health centre staff to scrutinise aid-post data and monitor health services and illness trends and assess community level health worker performance.
• Baseline data should be collected prior to and after interventions to measure their effect.

Conclusion
Problems evaluating community interventions or local service performance in PNG occur because there is an absence of reliable village-level data.The data collected through the health information system is crude and needs improvement for accuracy as well as monitoring and planning at local level.While aid-post data is available it is not always included in the health information system nor is it disaggregated according to village to show changes in health status.This has resulted in a serious information shortfall that can result in difficulties assessing changes in physical health status without village-level baseline data to measure illness trends and improvements in health in identifiable villages.
Qualitative data from a recent extensive evaluation showed community health interventions were effective in improving health behaviour and the health of women and children.However, statistical verification of improvement in health was not possible using the health information system and consistency in data collection.Health-centre data failed to provide a complete and accurate assessment of community health status because it does not disaggregate by village.
Policy changes must occur centrally to prevent the loss of aid-post data to the system.Aid-post and health-centre data needs to be disaggregated according to village to allow for changes in community health status to be monitored over time.Future planning for community health intervention strategies should take account of the need for baseline data against which to measure change over time.

Figure 1 :
Figure 1: Health system in Papua New Guinea.
10lness trends cannot be monitored among villages nor can improvements in village health be determined10.Similarly, data collected at health centre level cannot be disaggregated by aid posts, so it is not possible to compare performance over time.Any evaluation data from community interventions or trend data from villages is diminished in value if aggregated totals include the large proportion of communities within the health centre's catchment area, which could mean combining 15 to 30 or more villages.
villages is not tracked over time or reported separately.Therefore, without village-level baseline data the system cannot monitor change among villages or aid posts, evaluate community health activities or review performance.