Original Research

Development of the Nursing Community Apgar Questionnaire (NCAQ): a rural nurse recruitment and retention tool


name here
Molly Prengaman1
MS/FNP, Assistant professor *

name here
Jeri Bigbee2
PhD, Adjunct professor

name here
Ed Baker3
PhD, Director/ Professor

name here
David Schmitz4
MD, Associate director/Affiliate faculty


* Molly Prengaman


1 Boise State University School of Nursing/Center for Health Policy, Boise, Idaho, USA

2 UC Davis The Betty Irene Moore School of Nursing, Sacramento, California, USA

3 Boise State University Center for Health Policy, Boise, Idaho, USA

4 Family Medicine Residency of Idaho/Center for Health Policy, Boise State University, Boise, Idaho, USA


4 March 2014 Volume 14 Issue 1


RECEIVED: 17 April 2013

REVISED: 25 July 2013

ACCEPTED: 12 August 2013


Prengaman M, Bigbee J, Baker E, Schmitz D.  Development of the Nursing Community Apgar Questionnaire (NCAQ): a rural nurse recruitment and retention tool. Rural and Remote Health 2014; 14: 2633. Available: www.rrh.org.au/journal/article/2633

© Molly Prengaman, Jeri Bigbee, Ed Baker, David Schmitz 2014 A licence to publish this material has been given to James Cook University, jcu.edu.au


Introduction: Health professional shortages are a significant issue throughout the USA, particularly in rural communities. Filling nurse vacancies is a costly concern for many critical access hospitals (CAH), which serve as the primary source of health care for rural communities. CAHs and rural communities have strengths and weaknesses that affect their recruitment and retention of rural nurses. The purpose of this study was to develop a tool that rural communities and CAHs can utilize to assess their strengths and weaknesses related to nurse recruitment and retention.
Methods: The Nursing Community Apgar Questionnaire (NCAQ) was developed based on an extensive literature review, visits to multiple rural sites, and consultations with rural nurses, rural nurse administrators and content experts.
Results: A quantitative interview tool consisting of 50 factors that affect rural nurse recruitment and retention was developed. The tool allows participants to rate each factor in terms of advantage and importance level. The tool also includes three open-ended questions for qualitative analysis.
Conclusions: The NCAQ was designed to identify rural communities' and CAHs' strengths and challenges related to rural nurse recruitment and retention. The NCAQ will be piloted and a database developed for CAHs to compare their results with those in the database. Furthermore, the NCAQ results may be utilized to prioritize resource allocation and tailor rural nurse recruitment and retention efforts to highlight a community's strengths. The NCAQ will function as a useful real-time tool for CAHs looking to assess and improve their rural nurse recruitment and retention practices and compare their results with those of their peers. Longitudinal results will allow CAHs and their communities to evaluate their progress over time. As the database grows in size, state, regional, and national results can be compared, trends may be discovered and best practices identified.

Key words: instrument, nursing, recruitment, retention, workforce.

full article:


According to the Health Resources and Service Administration (HRSA), the current US nursing shortage is anticipated to grow to 20% by 2015. This increase is due to several factors including an increase in demand from the aging population and an aging nursing workforce1. Furthermore, the HRSA report does not include the impact of the recently enacted Patient Protection and Affordable Care Act, which is expected to add 28 million more Americans to those with health insurance2.

The nursing shortage is even more acute in rural settings, where nurse-to-population ratios are significantly lower, nursing vacancies are more common, nursing recruitment is a continual struggle, and critical access hospitals (CAHs) indicate that filling nurse vacancies is their number one concern3. Compounding the situation are the significant health disparities experienced by rural populations. Rural populations suffer higher rates of diabetes, obesity, hypertension, stroke, and suicide than their urban counterparts4. Rural residents are more likely than urban residents to describe their health status as fair or poor. In addition, rural adults seek preventative care such as mammographies, pap smears, and colonoscopies less often than do urban adults. Rural adults are also more apt to smoke and more commonly forgo regular exercise or the use of seatbelts. Rural youth have higher incidence of tobacco and alcohol use than do urban youth4-6. These factors all affect the health of rural populations and contribute to the demand for rural nursing services.

In addition to being a significant health issue, rural nurse recruitment and retention is an economic concern. Improving retention by even 1% has been noted to save employers a minimum of $250,0007. In Idaho, a predominantly rural state in the Pacific Northwest region of the USA with 90% of the state designated as a health professional shortage area, the nursing shortage looms large. Idaho's Nursing Workforce Center reports that the state's number of licensed nurses per 100 000 people ranks 48th in the USA, and Idaho's rural hospitals' nurse vacancy rates are up to 50% higher than that of their urban counterparts8. Nurse vacancies are anticipated to accelerate in the near future as more of the nursing workforce retires. According to a recent Idaho Department of Labor report, 60% of Idaho's registered nurses are 45 and older9.

The purpose of this study was to develop a tool with which rural communities and their CAH partners could evaluate the factors impacting their rural nurse recruitment and retention. The development of this tool will enable rural communities to better identify their strengths and weaknesses. With such knowledge rural nurse recruitment and retention efforts could be tailored to highlight the community's and hospital's strengths, maximize the effective use of scarce resources, and enhance their rural nurse recruitment and retention outcomes.

This project expanded on the previous work that assessed rural physician recruitment and retention challenges experienced by rural communities and hospitals10. Suggesting the concept of a 'community Apgar score' for rural Idaho medical staff environments, the authors hypothesized that identifiable parameters impact a rural community's ease of physician staffing. Schmitz et al's work resulted in the development and application of the Community Apgar Questionnaire (CAQ), which has been utilized to enhance rural communities' physician recruitment and retention efforts11. The CAQ has been well received by rural communities throughout Idaho and beyond. Thus, development of a similar tool for rural nurse recruitment and retention was sought.

Conceptually, the project draws upon Bushy's model and its application of the core nursing concepts of person, environment, health, and nursing to the unique context of rural nursing12. Bushy's model includes aspects such as the generalist orientation of rural nursing, the orientation to the natural environment, and newcomer/old timer dynamics. Although the methodology parallels the previous work done on the physician CAQ, Bushy's model guided the factor identification process and aspects of the NCAQ development unique to nursing.

Literature review

An extensive literature review was performed in order to identify those factors that impact rural nurse recruitment and retention. The vast majority of literature focusing on nurse recruitment or retention concentrated on urban settings. Much of the research addressing rural nurse recruitment or retention was completed in Canada or Australia; very little literature specifically examining rural nurse recruitment and retention in the USA was identified.

The available research related to the recruitment and retention of rural nurses suggested that rural nurses are generally less educated, less apt to be employed in a hospital, and more likely to be employed full time than urban nurses. In addition, a more rural area of residence was found to correlate with a lower nurse's salary13. The literature further revealed that nursing students with a rural background were more apt to choose a rural work setting upon graduation14. Clinical exposure to rural practice settings and course work regarding rural nursing theory were also cited as potential contributing factors when graduating students choose to practice in a rural setting12. Organizational characteristics that reflect rural values, rural lifestyle, job diversity, and patient variety were noted in the rural nurse retention literature as possibly having an impact on nurses' decisions to remain in a rural practice setting3. Earlier research on rural nursing retention demonstrated that work environment and contentment with the community had significant impact on whether or not a nurse remained in her rural practice setting15. A literature search did not reveal a research instrument to assess and assist in the recruitment or retention of rural nurses, thus proceeding with the development of such a tool was deemed appropriate.


Content validation

One hundred and twenty-eight factors were originally identified from the literature as potentially affecting nursing recruitment and retention. Because the literature specific to rural nurse recruitment and retention is scarce, the majority of the factors were gleaned from the urban nurse recruitment and retention literature. The primary investigator performed informal one-on-one interviews with content experts to review the factors for content validity. Rural nurse executives, practicing rural nurses, students participating in a rural clinical placement elective, personnel from the Idaho Office of Rural Health and Primary Care, and the board of directors from Rural Connection, a group of statewide nursing leaders, were each provided with the list of factors and asked to indicate which of the factors they deemed most seminal. Seventy-five factors were deemed most seminal by the content experts. Many of the factors discarded by the content experts were ones more applicable to urban settings, such as preceptorships for student nurses, dedicated pool of relief staff, or percentage of travel nurses utilized. In order to be consistent with formatting of the physician's CAQ and the infant Apgar tool for which it is named, the research team narrowed the list of 75 factors down to 50. This was accomplished by combining factors, such as flexible scheduling, optimal shift availability, and 12-hour shifts into one factor. Thus, the 75 factors deemed most seminal by the content experts were all included in the final 50-factor tool.

Nursing Community Apgar Questionnaire format

The 50 factors were grouped into five classifications of 10 factors each, consistent with the format of the physician CAQ (Table 1). Schmitz and Baker provided consultation to enhance consistency with the physician factor classification process. The classifications utilized in the physician CAQ were geographic, economic, medical support, scope of practice, and hospital and community support. The classifications identified for the Nursing Community Apgar Questionnaire (NCAQ) varied slightly; they were geographic, economic/resources, management/decision making, practice environment and scope, and community practice support.

Sixteen of the 50 factors in the NCAQ were also noted to be factors in the physician CAQ. In the geographic classification, access to a larger community, patient mix, social networking, recreational opportunities, spousal/partner satisfaction, schools, and climate were the factors common to both the CAQ and the NCAQ. In the economic classification, moving allowance and internet/technology access were the factors identified as seminal in both rural physician and nurse recruitment and retention. In the management/decision-making classification, hospital leadership/management and teaching/mentoring/administrative role opportunities were common to both the CAQ and NCAQ. Within the practice environment and scope classification, clinical variety and electronic medical record availability were the factors shared by both the CAQ and the NCAQ. There were three factors in the community practice support classification that the NCAQ had in common with the CAQ: perception of quality of care, emergency medical services availability, and a welcome and recruitment program.

Table 1: Factors and groupings in the Nursing Community Apgar Questionnaire

Nursing Community Apgar Questionnaire administration

A glossary of terms was compiled to enhance a common understanding of the factors by all participants (Appendix I). Dr Dave Schmitz provided training on administering the NCAQ to ensure consistency with that of the physician CAQ. The physician CAQ was completed with the CEO and the practicing physician most involved in physician recruitment and retention at each participating CAH. The NCAQ will be completed with the nursing administrator and a practicing rural nurse. The rural nurse participant will be one identified by the administrator as familiar with the community and knowledgeable about the CAH's retention and recruitment practices and history. Following completion of informed consent, the NCAQ will be administered by the primary investigator in a private face-to-face interview with each participant.

Each factor within the five classifications will be reviewed with each participant. The participant will rate each factor in terms of whether it is an advantage or challenge, and on its level of importance. Each rating is on a four-point Likert scale and assigned a corresponding score. In the advantage or challenge ratings, 'major advantage' = 2, 'minor advantage' = 1, 'minor challenge' = -1, and 'major challenge' = -2. The importance ratings scores were 'very important' = 4, 'important' = 3, 'unimportant' = 2, and 'very unimportant' = 1.

In addition to rating the 50 factors, participants will be asked to respond to three open-ended questions regarding rural nurse recruitment and retention at their CAH. The questions will address participants' perception of the greatest barriers to recruitment and retention of nurses, potential solutions to these perceived barriers, and the reasons given by nurse candidates for not accepting employment in rural communities. These were:

  • 'What are your greatest barriers to recruitment and retention of nurses (registered nurses and licenced practical nurses)?'
  • 'What can be done to overcome these barriers?'
  • 'What reasons has a successful nurse candidate given for not accepting a position in the hospital? What did that person ultimately do instead (if you know)?'

Ethics approval

This research was approved by the Boise State University Human Subjects Institutional Review Board; ethics approval number IRB # EX 187-MED10-018


The numerical rankings for the NCAQ will be utilized to calculate a weighted 'Apgar' score. The rating for each factor regarding advantage/challenge will be multiplied with the rating for importance to calculate the weighted Apgar score. Thus, the Apgar scores for each factor can range from 8 to -8.

Statistical analysis of these scores will demonstrate strengths and weaknesses for each participating CAH and community, at both the factor and class levels, as well as the factors deemed most important by participants. These scores will also be analyzed for any differences between administrators' and staff nurses' responses. In addition, examination of NCAQ scores from a variety of CAHs may demonstrate if rural nurses have unique reactions or share common responses to the various factors that affect rural nurse recruitment and retention.

The qualitative data will be analyzed to determine support of those factors deemed most salient. The data will also be used to identify any factors overlooked in the development of the NCAQ.

Next steps

The NCAQ will be administered at 12 CAH sites, six 'alphas', which have historically done well with nurse recruitment and retention, and six 'betas', which have struggled with nurse recruitment and retention. The results of the 'alpha' CAHs will be compared with those of the 'beta' CAHs. The aggregate results will serve as a database with which subsequent participating CAHs can compare their results. Pilot administration of the NCAQ will also allow for reliability and validity testing of the tool.


After the NCAQ has been piloted and a database of results compiled, CAHs will be able to see how their facility and community compare with those in the database. The NCAQ is designed to function as a dynamic, real-time tool for guidance on the most helpful timely interventions. The NCAQ will allow CAHs to assess and improve their rural nurse recruitment and retention practices. Longitudinal NCAQ results can be used to track CAHs and their community's progress over time, similar to the clinical use of Apgar scores in newborns, as well as compare their results with those of their peers. Utilizing the CAQ and the NCAQ together will enable rural CAHs and their communities to address their unique workforce needs.

As the database grows in size, state, regional, and national results can be compared, trends may be discovered, and best practices identified.


This research was funded by the Office of External Funding, Boise State University College of Health Sciences and the Idaho Department of Health and Welfare, State Office of Rural Health and Primary Care. The authors thank Mary Sheridan, Supervisor of the State Office of Rural Health and Primary Care, for her assistance with this research. The authors also wish to thank Jessica Nelson, RN, for her contributions to the literature review portion of this research; and Lisa MacKenzie, Elisa Howard and Bradley Morris, research assistants in the Boise State University Center for Health Policy, for their assistance.


1. Health Resources and Services Administration. Method for identifying facilities and communities with shortages of nurses, summary. (Online) 2007. Available: http://bhpr.hrsa.gov/healthworkforce/nursingshortage/default.htm (Accessed 10 March 2009).

2. Buttgens M, Holahan J, Carroll C. Timely analysis of immediate health policy issues. Washington, DC: The Urban Institute 2011.

3. Molinari D, Monserud M. Rural nurse job satisfaction. Rural and Remote Health 8: 1055. (Online) 2008. Available: www.rrh.org.au (Accessed 12 March 2009).

4. National Rural Health Association. What's different about rural health care? (Online) 2007. Available: http://www.ruralhealthweb.org (Accessed 10 March 2009).

5. Bennett K, Olatosi B, Probst J. Health disparities: a rural-urban chartbook. Columbia, SC: South Carolina Rural Health Research Center, 2008.

6. Bigbee J. Rural nursing: another Idaho gem. RN Idaho 2007; Fall: 17-19.

7. Russell J. How technology solutions can impact nursing retention. Nursing Economics 2008; 26(3): 188-190.

8. Idaho Nursing Workforce Center. The acute care nursing workforce in Idaho: results of a survey of Idaho hospitals. (Online) 2006. Available: http://www.inwc.org/hosp%20demand%20final%20report.pdf (Accessed 11 March 2009).

9. Idaho Department of Labor. Education investments, recession help close some nursing gaps. (Online) 2011. Available: http://labor.idaho.gov (Accessed 14 January 2012).

10. Baker ET, Schmitz DF, Wasden SA, MacKenzie LA, Epperly T. Assessing Community Health Center (CHC) assets and capabilities for recruiting physicians: the CHC Community Apgar Questionnaire. Rural and Remote Health 12: 2179. (Online) 2012. Available: www.rrh.org.au (Accessed 10 January 2013).

11. Schmitz DF, Baker E, Nukui A, Epperly T. Idaho Rural Family Physician Workforce Study: the Community Apgar Questionnaire. Rural and Remote Health 11: 1769. (Online) 2011. Available: www.rrh.org.au (Accessed 10 January 2013).

12. Bushy A, Leipert B. Factors that influence students in choosing rural nursing practice: a pilot study. Rural and Remote Health 5: 387. (Online) 2005. Available: www.rrh.org.au (Accessed 10 March 2009).

13. Skillman N, Palazzo L, Keepnews D, Hart L. Characteristics of registered nurses in rural versus urban areas: implications for strategies to alleviate nursing shortages in the United States. The Journal of Rural Health 2006; 22(2): 151-157.

14. Manahan C, Lavoie J. Who stays in rural practice? An international review of the literature on factors influencing rural nurse retention. Online Journal of Rural Nursing and Health Care 2008; 8(2): 42-53.

15. Pan A, Dunkin J, Muus K, Harris R, Geller J. A logit analysis of the likelihood of leaving rural settings for registered nurses. The Journal of Rural Health 1995; 11(2): 106-113.


Appendix I: Nursing Community Apgar Questionnaire glossary of terms