Original Research

The Pharmacy Community Apgar Questionnaire: a modified Delphi technique to develop a rural pharmacist recruitment and retention tool

AUTHORS

name here
Daniel R Terry
1 PhD, Senior Lecturer * ORCID logo

name here
Blake Peck
2 PhD, Senior Lecturer, Nursing ORCID logo

name here
Danny Hills
3 PhD, Associate Professor, Nursing ORCID logo

name here
Jaclyn Bishop
4 PhD, Research Translation Coordinator ORCID logo

name here
Mark Kirschbaum
5 MPharmSc, Lecturer ORCID logo

name here
Kehinde O Obamiro
6 PhD, Lecturer ORCID logo

name here
Hoang Phan
7 PhD, Research Assistant ORCID logo

name here
Ed Baker
8 PhD, Professor

name here
David Schmitz
9 MD, Chair ORCID logo

CORRESPONDENCE

*Assoc Prof Daniel R Terry

AFFILIATIONS

1, 2, 3 Institute of Health and Wellbeing, Federation University, PO Box 663, Ballarat, Vic. 3353, Australia

4 Western Alliance, Warrnambool, Vic., Australia

5, 6 Centre for Rural Health, University of Tasmania, Hobart, Hobart, Tas. 7000, Australia

7 Menzies Institute for Medical Research, University of Tasmania, Hobart, Tas. 7001, Australia

8 Center for Health Policy, Boise State University, Boise, ID 83725, USA

9 Family and Community Medicine, School of Medicine and Health Sciences, University of North Dakota, Grand Forks, ND 58203, USA

PUBLISHED

29 November 2022 Volume 22 Issue 4

HISTORY

RECEIVED: 10 December 2021

REVISED: 2 September 2022

ACCEPTED: 11 September 2022

CITATION

Terry DR, Peck B, Hills D, Bishop J, Kirschbaum M, Obamiro KO, Phan H, Baker E, Schmitz D.  The Pharmacy Community Apgar Questionnaire: a modified Delphi technique to develop a rural pharmacist recruitment and retention tool. Rural and Remote Health 2022; 22: 7347. https://doi.org/10.22605/RRH7347

AUTHOR CONTRIBUTIONSgo to url

This work is licensed under a Creative Commons Attribution 4.0 International Licence


abstract:

Introduction:  An adequate healthcare workforce remains essential for the health of rural communities. Strategies to address rural health workforce challenges have often centred on the medical and nursing workforce; however, addressing the rural pharmacist workforce also remains critical as they are often the first point of contact for health advice. Initiatives have increased pharmacist supply; however, key issues such as poor attraction, recruitment, and retention to rural areas remain. The aim of this study was to support the recruitment and retention of pharmacists in rural areas of Australia through the development of the Pharmacy Community Apgar Questionnaire (PharmCAQ).
Methods:  A modified Delphi technique was employed to develop the PharmCAQ. A panel of experts were purposively selected. Eight representatives were from organisations with rural experience relevant to the study including the Society of Hospital Pharmacists of Australia, the Pharmaceutical Society of Australia, the Pharmacy Guild of Australia, the Pharmacy Board of Australia, and a representative of a government health agency, who also leads a hospital pharmacy. Three additional participants included local and international academics with health policy and rural health workforce expertise. All participants participated in three separate focus groups of 45–60 minutes duration, where the review and refinement of factors that drive recruitment and retention of pharmacist were discussed. Face and content validity was achieved through the representatives, while internal consistency was achieved when the tool was piloted among 10 rural pharmacists in rural Victoria.
Results:  Fifty key factors that impact the recruitment and retention of pharmacists were identified, developed and succinctly described. All factors were grouped into five classifications: (1) geographic, (2) economic and resources, (3) practice and scope of practice, (4) practice environment and (5) community practice support. After final consensus, the factors and their definitions formed the final questionnaire. Lastly, the reliability of PharmCAQ was determined, with a Cronbach’s alpha coefficient of 0.852.
Conclusion:  While the development and use of the Apgar questionnaire for the recruitment and retention of health professionals is not a novel idea, seeking to specifically focus on pharmacists is unique. However, 10 factors were similar to factors associated with rural recruitment and retention of both physicians and nurses; they encompassed geographic, community support, and economic and resource factors. Regardless of similarities or differences between health professions in terms of recruitment and retention, as a mechanism for addressing the worsening health professional shortage currently experienced in rural areas, the PharmCAQ was developed to support the recruitment and retention of the pharmacist workforce in rural areas.

Keywords:

Australia, Community Apgar, Delphi, pharmacist, recruitment, retention, rural community.

full article:

Introduction

Healthcare services are necessary to the functioning, vibrancy, and both individual and public health of rural communities. Providing these services requires healthcare professionals such as medical practitioners, nurses and allied health professionals (eg occupational therapists, physiotherapists and pharmacists). Maintaining an adequate and stable healthcare workforce in rural communities remains essential for the continuity of care and improved health outcomes of these populations1-5. Although fundamental, rural communities continue to struggle to have these essential professionals living and working in these areas despite increased training, funding and programs to support these goals1,6-9.

Strategies implemented to address health workforce shortfalls in rural communities have often centred on addressing medical and nursing workforce deficits, thereby overlooking another critical workforce, pharmacists10. Pharmacists are often the first point of contact in rural areas and play a critical role in providing primary care as well as triage and referral of community members to other health professionals within the healthcare system11-13. In some cases, a pharmacist may be the only health professional that is easily accessible in a rural or remote community14.

Healthcare professional workforce initiatives over the past two decades have resulted in an increase in the number of pharmacists in Australia14,15, while maldistribution has resulted in a persistent undersupply across rural communities. Although increasing the pharmacist supply partially mitigates these longstanding challenges, key issues remain with poor attraction, recruitment and retention of pharmacists to rural areas3,10,15,16. To date, there has been limited research focused specifically on the recruitment and retention of pharmacists in rural communities3,14-16.

Literature reviews have investigated the drivers and challenges of rural pharmacist recruitment, retention and related decision-making10,14,15. The findings indicate that pharmacists seek job satisfaction, good working conditions, a desire for responsibility, training and an adequate but not overbearing workload15. Additionally, beyond the workplace-identified geographical and family related themes that influence the choice to practice rurality, there are other key drivers of rural employment. These include economic drivers of rural employment such as wages and incentives, an expanded scope of practice afforded to pharmacists in rural areas, and the rural practice environment. Characteristics of the community itself and its support for pharmacists who join and become part of the town or area are also important factors10,14,17-19. Although there are many pull factors that may bring pharmacists to contemplate, consider and undertake rural practice, there currently remains a need to define how these factors may be uniquely and practically utilised among individual rural communities to recruit and retain their much-needed workforce10,18,19.

An approach to addressing these challenges among physicians and nurses has been use of the Community Apgar Questionnaire (CAQ) and the Nursing Community Apgar Questionnaire (NCAQ), respectively7,20,21. These instruments, each consisting of 50 individual factors or key elements that influence practice location decision-making, have been used as objective measurement tools to assess the assets and challenges of individual rural communities to successfully recruit and retain doctors and nurses. Just as an Apgar test is designed to assess a newborn’s health and indications for action, the CAQ and NCAQ were developed to quantitatively scale the important resources and capabilities of rural communities that impact recruitment and retention1,7,20. The CAQ and NCAQ assist rural communities to be more effectively equipped to recruit medical and nursing staff that are the ‘best fit’ with their unique service demands and community, while also identifying longer term strategies that services can undertake to improve recruitment and retention7,20.

Research aim

The aim of this study was to develop the Pharmacy CAQ (PharmCAQ) and examine its reliability, while also understanding the similarities with the CAQ and NCAQ.

Methods

In this exploratory study, a modified Delphi technique was employed to formulate and develop the PharmCAQ, informed by the development process undertaken with the CAQ and NCAQ7,20. The ‘Delphi’ is an established method employed to achieve clear consensus regarding healthcare policies and guidelines, can be applied qualitatively, and is regularly utilised to create solutions in the healthcare environment22-25.

Modified Delphi technique

A Delphi remains fluid and flexible in its application due to the absence of a concrete theoretical framework. To mitigate this lack of a framework, an adaptation of the Conducting and Reporting of Delphi studies (CREDES) guideline was utilised to inform this study26. Overall, three key elements are essential, with at least 8–12 participants who (1) are experts within the field, (2) come from a variety of backgrounds and (3) are heterogeneous in terms of their specialities, insights and understandings regarding the subject matter24,25,27.

Along with these three participation elements, the Delphi process is achieved through four distinct steps28. Initially an exploration of the subject matter occurs, where each expert contributes information that they consider important. In the second step, the group collectively works together to gain an understanding of the issue and discuss disagreements. It is these differences that need to be embraced, commended and explored in greater detail. This informs the third step, where insights and the development of creative solutions or alternatives occur. The final step in the process encompasses the gathering and analysis of the data, which is fed back to the group for further clarification and understanding. This process is then repeated several times until an artifact is created or solution is developed and agreed upon. The process undertaken is summarised in Figure 1. Although anonymity within the Delphi process is ideal to reduce influence or social bias, in some circumstances this may not always be possible given experts may already know each other, or it may be essential for a group of experts to seek consensus on key matters25.

table image Figure 1:  Flow chart of Delphi technique to develop the Pharmacy Community Apgar Questionnaire.

Participants

In this study, a panel of experts were purposively selected through known contacts of the various professional organisations both across the study area as well as nationally, and included eight stakeholders. These known contacts comprised representatives from professional organisations with comprehensive rural experience relevant to the study. They were in contact with many individuals and organisations across the country in their current and past roles, while some had or were also working clinically in rural contexts. In addition, they either represented or regularly met with rural individuals and organisations, which included areas classified as Modified Monash MM4 to MM7. These representatives were from the Society of Hospital Pharmacists of Australia (n=1), the Pharmaceutical Society of Australia (n=1), the Pharmacy Guild of Australia (n=1), the Pharmacy Board of Australia (n=1) and a government health agency and public hospital (n=1). Other participants included local and international academics with decades of health policy and rural health workforce expertise (n=3). No demographic data are reported here, to maintain the confidentiality of all participants.

Procedure for questionnaire development and refinement

Open discussion, as with previous CAQ and NCAQ development processes1,20, was used rather than survey-type questioning. This was essential to understand and define the key factors vital to rural pharmacist recruitment and the nuances regarding how each factor was defined24,25. As such, the Delphi technique facilitated the capture of vital information regarding those factors considered essential and as defined within the literature10. The group leader, an independent researcher (DS) and other researchers (DT and EB) met collectively with all participants for three separate focus groups of 45–60 minutes duration.

Initially, a draft set of factors and their description, based on a systematic literature review by Terry et al10, was introduced to the group prior to the first meeting. Throughout each group discussion, individual responses were written and summarised in real time and later analysed, with recommended changes made directly to the draft of the PharmCAQ. The revised questionnaire was then redistributed for the next meeting round. This exploratory process was repeated until consensus was achieved regarding the key factors that were to be included in the PharmCAQ and how each was defined. As a part of the process, each meeting session was recorded to facilitate ease of recall regarding key suggestions and points made by each participant.

Three rounds of discussions were anticipated to be achieved22,23, with the process being completed over a 5-week period. Additional review and refinement of each of the factors occurred via email in the subsequent weeks to ensure accuracy of description as well as the ordering of the factors, and to support the flow of conversation when the PharmCAQ was being implemented in practice. Overall, the process enabled an in-depth discussion of each factor associated with the PharmCAQ, its definitional development, and fine-tuning as it was prepared to be piloted in the field.

In addition to the ordering to support conversation flow, each factor was assigned values using a four-point scale according to a community’s advantages or challenges (major advantage = 2, minor advantage = 1, minor challenge = –1, major challenge = –2) and then its level of importance based on a four-point scale (very important = 4, important = 3, unimportant = 2, very unimportant = 1). This was in line with the CAQ and NCAQ previously developed and allows each factor to be examined and provided a weighted Apgar score between +8 and –8 according to the following algorithm:

advantage or challenge score × importance score = Pharmacist Community Apgar Score1,7,20

Reliability testing

An essential element of the PharmCAQ development process was its initial face and content validity, which was achieved through the participation of representatives with rural experience from professional organisations. In addition, internal consistency was achieved when the tool was initially piloted among 10 rural pharmacists in rural Victoria (MM4 to MM5), where numerical rankings for each factor within the tool were utilised to calculate a weighted Apgar score. Each rural pharmacist was recruited through known contacts, informed of the study, and invited to participate in the testing of the PharmCAQ. This also provided further face and content validation of the tool. The process highlighted the important unique features identified as advantages and challenges of a community that may be modifiable or require additional support to mitigate their potential for a negative impact.

Four of the ten interviews were conducted face-to-face, while the remaining occurred by video-conferencing technology. Each interview was 45–90 minutes in duration and was audio- or video-recorded. After interviews were conducted, data were cleaned, checked and analysed using Microsoft Excel and the Statistical Package for the Social Sciences v25.0 (IBM; http://www.spss.com). As outlined through the procedure in Schmitz et al20, data were scored accordingly, and the internal consistency of the PharmCAQ was assessed by determining the Cronbach’s alpha coefficient, a standard measurement of reliability, where alpha scores greater than 0.7 are considered acceptable29.

Ethics approval 

Ethics approval was provided by the Federation University Australia Human Research Ethics Committee (#A21-023) and the University of Tasmania Human Research Ethics Committee (#26068), with all elements of human research being conducted in line with the National Statement on Ethical Conduct in Human Research 2007 (updated 2018)30.

Results

All eight experts participated in rounds 1 and 2 using video-conference software. A third round was conducted through a series of email exchanges that occurred between the lead researcher (DT) and each participant. The bulk of the tool’s development was achieved by the second-round discussion, with the third round aimed at refining minor elements of the PharmCAQ. The outcome of the process is detailed below, including all three rounds of development leading to the final development of PharmCAQ.

Round 1: discussing and defining factors

Prior to the first meeting with the various content experts, a draft list of 90 factors, based on the systematic literature review conducted by Terry et al10, was sent to each participant. Each factor was grouped according to five classifications, consistent with the format of the CAQ and NCAQ7,20. All participants were then invited to read and review all identified factors considered most important to recruitment and retention of rural pharmacists for discussion at the initial meeting.

The outcome of the first meeting highlighted that, out of the initial 90 factors, many were similar and should be combined and then further clarified among participants. All factors were then rated by level of importance for recruitment and retention. This led to the initial list of factors being reduced to 35 considered very important, with an additional 25 being considered important. All other factors were excluded as they were considered less important or unimportant: they pertained to attributes of the pharmacist rather than the community, which is what the tool seeks to measure. For example, starting a business with friends and being married with children are unimportant when assessing a community.

In addition to discussing, defining and ranking key factors drawn from the literature, the expert group also identified several factors that were missing or not well defined within the literature yet were also known to have an impact on recruitment and retention of pharmacists in rural areas. These included opportunities to participate in research, the obligation or ability to provide discounted or value-added pharmacy services, and the health and wellbeing of the pharmacist themselves.

After the list of factors was identified, the nuances regarding each factor were discussed with the experts, providing greater context or details that may have been missing from the literature. For example, the factor associated with the availability of support staff was considered very important, however, it was indicated that ‘availability’ was only one facet of the factor itself. Specifically, it was suggested that ‘competency’ of both technical and other support staff, such as front of house or retail assistants, was also essential. In this way, definitional details were provided by each expert regarding each factor identified.

Round 2: further discussion and refining

Two days prior to the second meeting, an updated set of factors was sent to all experts. At the second meeting, there was a greater and increased level of discussion regarding key factors and their relevance, while seeking better ways to differentiate the various factors. In so doing, the factors were reduced from 60 to 50 through further clarification and combining of similar factors. This process also ensured the meaning of factors was clarified and more succinctly described. For example, multiculturalism identified from the literature did not accurately outline what this meant for recruitment and retention of pharmacists to a rural area. As such, the team of experts was able to more carefully define this as acceptance of cultural diversity within a community, specifically pertaining to a pharmacist who may be from a diverse cultural background.

Beyond carefully defining what each factor meant and how it could be assessed when examining a rural community, each factor was again scrutinised to ensure it was grouped according to the most appropriate classification and that the classifications were adequately described. As such, minor adjustments were made and each factors was grouped into one of the following five classifications: (1) geographic, (2) economic and resources, (3) practice and scope of practice, (4) practice environment and (5) community practice support factors.

Round 3: final revision and approval

As with previous rounds, each expert suggested nuances and further adjustments, which resulted in minor wording changes for several factors. This discussion and feedback were used to develop the final definitions along with key tag lines, used to briefly describe each of the factors that were to be present in the final questionnaire. The final group of factors, tag lines and definitions were sent to all experts prior to the next meeting. However, given the smaller number of participants at the final meeting, such an endeavour provided a limited number of suggestions. To overcome this, any additional information or modifications that were provided from the various experts via email or conversation were further incorporated into the final list of factors. In most cases, any modifications suggested were minor in nature, and focused on wording changes and clarifying statements where ambiguity may have remained. After final consensus, the factors, tag lines and definitions were used to develop the final questionnaire along with a glossary of terms. Additionally, three open-ended questions used to identify specific factors for an individual community, identical to those on the CAQ and NCAQ7,20, were included (see Appendix I and Appendix II).

Reliability testing – final revisions and acceptability

As the PharmCAQ was piloted among the 10 pharmacists, numerical rankings for each factor within the tool were utilised to calculate a weighted Apgar score. In this case, the process further informed the internal consistency of the tool in that it measured accurately what it intended to measure, while feedback on the PharmCAQ was sought from each pharmacist in relation to how well the questions were measuring each classification and associated factor. In addition, the reliability of the PharmCAQ was assessed using Cronbach’s alpha coefficient, resulting in an overall Cronbach’s alpha of 0.852, which is above the minimum threshold of acceptability (0.7), and further increasing the reliability was not achieved if any of the individual items were removed (Appendix III).

Discussion

The outcome of the modified Delphi technique was the development of a PharmCAQ that has been suitably refined, assessed for reliability and validated for use across rural communities in Australia, to facilitate the identification of key factors considered most vital to the recruitment and retention of pharmacists. While the development and use of the Apgar questionnaire for the recruitment and retention of health professionals is not a novel idea7,18,20, seeking to specifically focus on pharmacists is unique10,14. The discussion, deliberation and collaboration between organisations with varied perspectives provided an opportunity for nuanced insights regarding the many recruitment and retention factors during the questionnaire development. In addition, the dialogue that occurred as part of the Delphi process allowed the sharing of challenges and methods used to address these mutual concerns, all of which provided a deeper level of engagement. All participants in the study identified a need for a sharper focus upon the pharmacist as a health professional who also encounters related but unique challenges compared to their medicine and nursing counterparts in coming to and working in a rural community1,6-9.

This finding was particularly evident in the development of the PharmCAQ. For example, it is noted that, among the 50 factors identified and developed, 10 factors were similar to those found in both the CAQ and NCAQ. These included five geographic factors: communication infrastructure, social and cultural opportunities, recreation and adventure opportunities, spouse or partner opportunities, and availability or accessibility of schools7,20 (Fig2).

Three factors similar to the CAQ and NCAQ were economic and resource factors: financial income, contract flexibility and support for continued professional development. The remaining two factors that were similar were associated with workplace diversity (the demographic mix among clients or patients) and community recognition (appreciation for and community support of a new pharmacist)7,20.

Although 10 factors were shared between all health professions, pharmacist and physicians shared eight factors that were not shared with nurses, including moving allowance, support staff, teaching and having a sense of purpose in the community20. Further, pharmacists and nurses shared six additional factors not found among physicians in their respective questionnaires, including day care, housing and community size7. Although these similarities and differences did not always have identical wording, the consistency demonstrates some factors remain important for considering a rural career for any health profession7,20.

Despite the similarities and differences, various factors are considered specifically important to pharmacists. These include, but are not limited to, pharmacy ownership, mutual support and collaboration with other health professions, the presence of a multipurpose service or local health hub, and the obligation or ability to provide discounted or value-added pharmacy services. Other unique factors include the availability and competency of technical and ancillary staff, and separate credentialing that enables the pharmacist to deliver diabetes education, immunisations, and alcohol and other drug services10.

Regardless of similarities or differences between health professions in terms of recruitment and retention, improving access to primary health care is an essential yet complex conundrum for rural communities in Australia and internationally31. People in rural communities, globally, typically experience higher levels of complex and chronic disease, have lower levels of health literacy and reduced access to health services31,32. The combination of these factors further emphasises how vital rural pharmacists are in providing health care, the need for an expanding or a more rural generalist scope of practice, while ensuring improved recruitment and retention strategies are in place33,34.

table image Figure 2:  Similarities and differences between medicine, nursing and pharmacy Community Apgar Questionnaire responses.

Limitations

The Delphi technique was insightful and used several heterogeneous experts from key organisations across Australia and partners in the USA. However, the findings of the Delphi and the developed PharmCAQ may not be representative of the nuanced differences found in other countries that experience challenges in recruiting and remaining pharmacists in rural areas. Due to this recognition, the PharmCAQ may require slight wording changes and further adaptations to meet local needs. Further, pre-testing to verify the 50 factors may be vital to ensure they are also relevant to country-specific issues and challenges. Additional testing may also be essential in more rural communities (MM7), where key drivers of what makes a rural or remote community differ to less remote communities. However, the open-ended questions within the tool are designed to capture key nuances of specific communities, including those who may be considered remote.

Despite these limitations, the study has resulted in the development of a tool that can assist in understanding the fundamental factors associated with pharmacist recruitment and retention within the Australian context, and how these may be addressed within rural community settings. Lastly, the authors have shown that the factors that influence the pharmacy workforce in a rural area are proposed here as being unique to the discipline of pharmacy. However, a close discrimination of those factors more important to the differing settings of pharmacy practice, such as hospital pharmacy compared to community pharmacy, have not been provided. While there are clear differences between the roles of the pharmacist in these diverse settings33,35, development of additional tools that are more setting specific could be a benefit in those particular circumstances.

Conclusion

Pharmacists represent a cornerstone of the healthcare system and provide a first port of call into this system for many of those residing in rural areas of Australia. As a mechanism for addressing the worsening health professional shortage currently experienced in rural areas, the opportunities for the pharmacist to play an expanded role are compelling. Despite the pivotal role of pharmacists, recruitment and retention have proven troublesome for rural communities, despite an increased number of graduates from pharmacy education programs. Using a Delphi technique, the PharmCAQ was developed, refined, assessed for reliability and validated as a tool for supporting the recruitment and retention of the pharmacist workforce in rural areas. The authors believe the key factors contained within the PharmCAQ may offer – for the first time – an opportunity to better support communities to identify and address the key factors considered most vital to the recruitment and retention of a pharmacy workforce. More research is required to further test and demonstrate the overall impact of the tool across various rural communities in their capacity to recruit and retain pharmacists to further support the healthcare needs of the community.

references:

1 Prengaman M, Terry D, Schmitz D, Baker E. The Nursing Community Apgar Questionnaire in rural Australia: An evidence based approach to recruiting and retaining nurses. Online Journal of Rural Nursing and Health Care 2017; 17(2): 148-171. DOI link
2 Smith T, Sutton K, Pit S, Muyambi K, Terry D, et al. Health professional students' rural placement satisfaction and rural practice intentions: A national cross‐sectional survey. Australian Journal of Rural Health 2018; 26(1): 26-32. DOI link
3 Kirschbaum M, Khalil H, Taylor S, Page AT. Pharmacy students' rural career intentions: perspectives on rural background and placements. Currents in Pharmacy Teaching & Learning 2016; 8(5): 615-621. DOI link
4 Fleming CA, Spark MJ. Factors influencing the selection of rural practice locations for early career pharmacists in Victoria. Australian Journal of Rural Health 2011; 19(6): 290-297. DOI link, PMid:22098212
5 Daniels ZM, Vanleit BJ, Skipper BJ, Sanders ML, Rhyne RL. Factors in recruiting and retaining health professionals for rural practice. The Journal of Rural Health 2007; 23(1): 62-71. DOI link, PMid:17300480
6 Humphreys J, Jones J, Jones M, Hugo G, Bamford E, Taylor D. A critical review of rural medical workforce retention in Australia. Australian Health Review 2001; 24(4): 91-102. DOI link, PMid:11842721
7 Prengaman M, Bigbee J, Baker E, Schmitz D, Baker E. Development of the Nursing Community Apgar Questionnaire (NCAQ): a rural nurse recruitment and retention tool. Rural and Remote Health 2014; 14: 2633. DOI link, PMid:24588333
8 Cramer M, Duncan K, Megel M, Pitkin S. Partnering with rural communities to meet the demand for a qualified nursing workforce. Nursing Outlook 2009; 57(3): 148-157. DOI link, PMid:19447235
9 Terry D, Lê Q. Social capital among migrating doctors: the "Bridge" over troubled water. Journal of Health Organization and Management 2014; 28(3): 315-326. DOI link, PMid:25080647
10 Terry D, Phan H, Peck B, Hills D, Kirschbaum M, Bishop J, et al. Factors contributing to the recruitment and retention of rural pharmacist workforce: a systematic review. BMC Health Service Research 2021; 21: 1052. DOI link, PMid:34610827
11 Melton BL, Lai Z. Review of community pharmacy services: what is being performed, and where are the opportunities for improvement? Integrated Pharmacy Research and Practice 2017; 6: 79-89. DOI link, PMid:29354554
12 Hoang H, Barnett T, Kirschbaum M, Dunbar S, Wong R. 'The public rely on me a lot': rural pharmacists’ perspectives on their roles in oral health care. Australian Journal of Rural Health 2019; 27(1): 57-63. DOI link, PMid:30719795
13 Hays C, Taylor SM, Glass BD. Pharmacists’ ‘Full scope of practice’: knowledge, attitudes and practices of rural and remote Australian pharmacists. Journal of Multidisciplinary Healthcare 2020; 13: 1781-1789. DOI link, PMid:33293821
14 Obamiro KO, Tesfaye WH, Barnett T. Strategies to increase the pharmacist workforce in rural and remote Australia: a scoping review. Rural and Remote Health 2020; 20: 5741. DOI link, PMid:33249850
15 Hawthorne N, Anderson C. The global pharmacy workforce: a systematic review of the literature. Human Resources for Health 2009; 7: 48. DOI link, PMid:19545377
16 Scott MA, Kiser S, Park I, Grandy R, Joyner P. Creating a new rural pharmacy workforce: development and implementation of the Rural Pharmacy Health Initiative. American Journal of Health System Pharmacy 2017; 74(23): 2005-2012. DOI link, PMid:29167142
17 Carvajal MJ. A theoretical framework for the interpretation of pharmacist workforce studies throughout the world: the labor supply curve. Research in Social and Administrative Pharmacy 2018; 14(11): 999-1006. DOI link, PMid:29567087
18 Cosgrave C. The Whole-of-Person retention improvement framework: a guide for addressing health workforce challenges in the rural context. The International Journal of Environmental Research and Public Health 2020; 17(8): 2698. DOI link, PMid:32295246
19 Cosgrave C, Malatzky C, Gillespie J. Social determinants of rural health workforce retention: a scoping review. The International Journal of Environmental Research and Public Health 2019; 16(3): 314. DOI link, PMid:30678350
20 Schmitz D, Baker E, Nukui A, Epperly T. Idaho rural family physician workforce study: the Community Apgar Questionnaire. Rural and Remote Health 2011; 11(3): 1769. DOI link, PMid:21790262
21 Baker E, Schmitz D, Wasden SA, Mackenzie L, Epperly T. Assessing Community Health Center (CHC) assets and capabilities for recruiting physicians: the CHC Community Apgar Questionnaire. Rural and Remote Health 2012; 12: 2179. DOI link, PMid:23259832
22 Mozuni M, Jonas W, Ji S. An introduction to the morphological Delphi method for design: a tool for future-oriented design research. The Journal of Design, Economics, and Innovation 2017; 3(4): 303-318. DOI link
23 Taylor E. We agree, don't we? The Delphi method for health environments research. Health Environments Research Design Journal 2020; 13(1): 11-23. DOI link, PMid:31887097
24 Brown J. Interviews, focus groups, and Delphi techniques. In: P Brough (Ed.). Advanced Research Methods for Applied Psychology. London: Routledge, 2018; 95-106. DOI link
25 Lilja KK, Laakso K, Palomäki J (Eds). Using the Delphi method. 2011 Proceedings of PICMET'11: Technology Management in the Energy Smart World (PICMET). Portland, OR: Institute of Electrical and Electronics Engineers, 2011.
26 Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliative Medicine 2017; 31(8): 684-706. DOI link, PMid:28190381
27 Habibi A, Sarafrazi A, Izadyar S. Delphi technique theoretical framework in qualitative research. The International Journal of Engineering and Science 2014; 3(4): 8-13.
28 Linstone HA, Turoff M. The Delphi method. Reading, MA: Addison-Wesley, 1975.
29 Pallant J. SPSS survival manual: a step by step guide to data analysis using SPSS for Windows. Sydney: Allen & Unwin, 2011.
30 Commonwealth of Australia. National Statement on Ethical Conduct in Human Research 2007 (Updated 2018). Canberra: National Health and Medical Research Council, Australian Research Council and Universities Australia, 2018.
31 World Health Organization. Imbalances in rural primary care: a scoping literature review with an emphasis on the WHO European Region. New York: WHO Regional Office for Europe, 2018.
32 Wakerman J, Humphreys JS. ‘Better health in the bush’: why we urgently need a national rural and remote health strategy. Medical Journal of Australia 2019; 210(5): 202-203. DOI link, PMid:30801723
33 Taylor S, Cairns A, Glass B. Expanded practice in rural community pharmacy in Australia: pharmacists' perspectives. Journal of Pharmacy Practice and Research 2021; 51(1): 43-53. DOI link
34 Taylor S, Cairns A, Glass B. Expanded practice in rural community pharmacy: a macro-, meso and micro-level perspective. Rural and Remote Health 2021; 21(3): 6158. DOI link, PMid:34304577
35 Smith JD, White C, Roufeil L, et al. A national study into the rural and remote pharmacist workforce. Rural and Remote Health 2013; 13: 2214. DOI link, PMid:23683357

appendix I:

Table 1: Pharmacy Community Apgar Questionnaire.table image

appendix II:

Pharmacy Community Apgar Questionnaire glossary of terms.table imagetable image

appendix III:

Table 1: Reliability analysis results for each factortable image