Original Research

Developing conceptually sound items for a clinical courage questionnaire


name here
Robert Brooks1
PhD, Associate Professor Rural Medicine

name here
Isabella A White2
Academic GP Trainee

name here
Lucie Walters3
FACRRM, Director *

name here
Susan Williams4
PhD, Researcher

name here
Ian Couper5
MFamMed, FCFP(SA), Director

name here
Jill Konkin6
FRRMS, Interim Coordinator Rural Integrated Community Clerkship

name here
David G Campbell7
MBBS, FACRRM, Censor-in-Chief


*Prof Lucie Walters


1 Department of Rural Health, Broken Hill University, Broken Hill, NSW 2880, Australia

2 Adelaide Rural Clinical School, The University of Adelaide, Nairne, SA 5252, Australia

3 Adelaide Rural Clinical School, Faculty of Health and Medical Sciences, The University of Adelaide, Mount Gambier, SA 5290, Australia

4 Department of Rural Health, University of Sydney, Broken Hill, NSW 2880, Australia

5 Ukwanda Centre for Rural Health, Department of Global Health, Stellenbosch University, Cape Town, South Africa

6 Office of Rural & Regional Health, Edmonton Clinic Health Academy, University of Alberta, Edmonton, AB T6G 1C9, Canada

7 Australian College of Rural and Remote Medicine, Cunninghame Arm Medical Centre, Lakes Entrance, Vic. 3909, Australia


7 May 2023 Volume 23 Issue 2


RECEIVED: 19 April 2022

ACCEPTED: 12 January 2023


Brooks R, White IA, Walters L, Williams S, Couper I, Konkin J, Campbell DG.  Developing conceptually sound items for a clinical courage questionnaire. Rural and Remote Health 2023; 23: 7592. https://doi.org/10.22605/RRH7592


This work is licensed under a Creative Commons Attribution 4.0 International Licence


Introduction:  Clinical courage can be described as a rural doctor’s adaptability and willingness to undertake clinical work at the limits of their training and experience to meet the needs of their patients. This article describes the in-house development of survey items to include in a quantitative measure of clinical courage.
Methods:  The questionnaire development involved two key concepts: a second-order latent factor model structure and a nominal group technique, used to develop consensus among the research team members.
Results:  The steps taken to develop a sound clinical courage questionnaire are described in detail. The resulting initial questionnaire is presented, ready for testing with rural clinicians and refinement.
Conclusion:  This article outlines the psychometric process of questionnaire design and presents the resultant clinical courage questionnaire.


courage, survey development, psychometric test, rural doctors.

full article:


With the worldwide shortage of physicians living and practising in rural and remote areas, it is important to understand better the nature of rural practice. A recent qualitative study examined the lived experience of rural doctors when their patients’ needs and the extent of their capabilities intersect1. This study identified an overarching construct called clinical courage. This phenomenon resonates strongly with rural clinicians, who have found it difficult to explain the nature of rural practice where they seek to balance quality and safety of health care with patients’ access to services. However, the clinical courage literature prior to this study was limited to conceptual pieces2-4

The six attributes of clinical courage experienced by rural doctors have recently been described further in the literature:

  1. ‘Standing up to serve anybody and everybody in the community’1. Rural doctors often have a deep commitment to providing inclusive health care in the communities they identify as belonging to5-7. Interconnected relationships enable doctors to have place-based knowledge of their patients1,8. However, managing intersecting professional and personal relationships, while commonplace in rural practice, is considered problematic by urban-centric codes of ethics and practice standards8-10.
  2. ‘Accepting uncertainty and persistently seeking to prepare’. Current skills and knowledge are challenged by the broad range and complexity of cases seen11. Rural doctors seek to maintain, extend and expand their clinical skills, seeking to develop adaptive expertise through integrative wisdom7,11.
  3. ‘Deliberately understanding and marshalling resources in the context’5. Working in resource-limited environments, rural doctors describe requiring an understanding of the physical and human resources available and the adaptability of these in serving patients’ needs7,8. Thin resources, which can shift in substantial ways over short periods of time, mean constantly balancing individual needs with broader health service and community needs8.
  4. ‘Humbly seeking to know one’s own limits’. Self-awareness of what one can and cannot do enables a rural doctor to articulate her capability to herself, her local colleagues and tertiary referral services8.
  5. ‘Clearing the cognitive hurdle when something needs to be done for your patient’. When no action is not an option, and identifying that other resources are not available and no one else is better qualified or experienced, the rural physician makes the choice to act.
  6. ‘Collegial support to stand up again’. Rural physicians consciously seek out peer support from both those in their community of practice and others who are familiar with rural practice to maintain their resilience and capacity to practice5,7.

Some attributes of clinical courage have been discussed in rural health professionals other than doctors, including rural paramedics and rural physiotherapists9,12. An exemplar of clinical courage is shown in Box 1.

A related concept, rural practice self-efficacy, has been found to be associated with rural and remote clinical practice13,14. Self-efficacy is having the belief that you can achieve what you set out to do within a defined context – in this case, rural practice. Clinical courage and self-efficacy overlap to some extent, with clinical courage covering a broader range of specific concepts. In a similar research vein, a study in Australia identified the motivations of existing practitioners who worked in remote communities for 3 years or more15. The quantitative study found 3 years or more of remote clinical practice more likely in males, in nurses (compared with other health professionals) and those with high clinical self-containment and intercultural interest scores and lower needs to accommodate relationships or have balanced lifestyles15. Thus, there is a growing body of knowledge that indicates a key role for psychosocial behavioural concepts contributing to ongoing rural practice. These do not however describe the praxis of rural medicine. Clinical courage has been identified as a potentially relevant concept that rural clinicians seem to identify. Potentially the attributes described above and the overall concept of clinical courage could be applied to support the engagement, development, support and retention of rural doctors. Being a relatively new concept there is no existing questionnaire that can be used to measure or gauge its role. A focused survey that is psychometrically sound and relatively short will enable the rural health sector to measure the six attributes consistent with the lived experience of clinical courage.

The overarching aim of this research is to develop a quantitative measure of clinical courage, based on a theoretical understanding of the previously described six aspects of clinical courage, combined with psychometric principles to produce a refined initial questionnaire. The purpose of this article is to detail the initial in-house development of survey items used to create a clinical courage questionnaire using a nominal group technique, which facilitated consensus building regarding questionnaire items among members of the research team16. Subsequent work will undertake psychometric evaluation and development of the questionnaire.

table imageBox 1:  An exemplar of clinical courage.


The questionnaire development involved two key concepts: a second-order latent factor model structure17 and the nominal group method of creating survey items18,19.

Latent factor model

The questionnaire development here focuses solely on the first-order factors, identified as aspects of clinical courage in the qualitative research. Evidence indicates that, for initial stability, five questions are required for each of the six identified domains, called first-order factors20,21. Three rather than five or more questions are shown in the clinical courage questionnaire in Figure 1.

table image Figure 1:  Theoretical model clinical courage questionnaire.

Nominal group method

Nominal groups usually involve a structured face-to-face interaction of 5–12 participants16. For the in-house development of survey questions, based on the researchers’ previous qualitative research, the nominal group consisted of four of the five researchers who undertook the original qualitative study of clinical courage, all of whom are rural doctors (DC, IC, JK, LuW)1,5. Also included in the group were an academic general practice registrar (IW) and an experienced qualitative researcher, who had both been involved in further research on clinical courage within the context of COVID22,23. The team are from three countries: Australia (DC, LuW, IW, SW), South Africa (IC) and Canada (JK), bringing a range of cultural perspectives to the process of survey item design.


The moderator (RB) is a remotely based academic researcher with a strong background in questionnaire development and psychometrics. As a research psychologist he is familiar with working with and understanding latent constructs. RB has a history of small-group work, clinically, educationally and in research. He ensured all participants were fully engaged and that their voices were heard. The group accepted RB as an appropriate facilitator.

The group facilitation process iteratively consolidated the knowledge of the group and built agreement around all elements of the questionnaire. All meetings were conducted via Zoom to allow for the real-time, virtual, face-to-face interaction of the group members. This is a suitable approach for complex tasks when the interaction of experts is likely to be productive14.  

Consensus process

To ensure methodological rigour, the steps in the nominal group method rely on drawing on expertise to generate ideas, confidentiality and consensus formation16. This was achieved by the moderator managing the flow of information in the meetings and between each meeting, as outlined in Table 1. Usual practice with consensus research is for a predetermined definition of consensus24. In this study, consensus was defined as the highest ranking questions, provided that no member of the nominal group membership objected to the inclusion of the item. The three to six highest ranked items for each domain were then returned to the group, where group members were invited to object to the inclusion; if this did not occur, a final review and wordsmithing of the item were completed.

Table 1:  Independent work and group work in survey developmenttable image

Project team tasks

Six main activities were undertaken during this nominal group process.

1. Developing and agreeing on the definitions of six attributes of clinical courage:

  • Expertise  The group followed standard brainstorming processes to ensure all contributed and no-one was censored.
  • Confidentiality  All the ideas were worked with independently by the moderator, who constructed draft definitions distributed prior to the next meeting.
  • Consensus formation  Open discussion of each definition occurred, challenging meaning and structure; this was followed by rewriting each definition as a group. The final definitions were then written by the moderator and distributed for final approval by the group.

2. Generation of a pool of items for each of the domains:

  • Expertise  All group members contributed to generation of items.
  • Confidentiality  Group members returned suggested questions directly to the moderator, who then presented the group with items in a random order within each domain in turn.

3. Decisions about inclusion of items:

  • Consensus formation  The items were discussed by the group and consensus reached about whether any questions should be excluded or modified.
  • Expertise: All group members were asked to choose their top five questions and provide this information to the moderator. The highest ranked items for the domain were calculated by the moderator and the top five for each domain provided to the group.
  • Confidentiality  The process of ranking was undertaken as a group activity. Although choosing was independent, reporting to the moderator was not confidential.

4. Decisions about item construct:

  • Expertise  Questions for each domain were presented to the group and assessed as being good questions for the domain or not.
  • Confidentiality  This was an open process relying on group dynamics to generate discussion.
  • Consensus formation  Each question was examined and modified as needed on the basis of discussion, and the final questions were then put into a draft questionnaire by the moderator.

5. Construction and review of the initial questionnaire items

  • Expertise  The clinical courage research group examined the questionnaire and moved items or modified items as too close to other items, or too similar to another construct.
  • Confidentiality  This was an open process, using the group process to generate diverse contribution.
  • Consensus formation  The final questionnaire was drafted by the moderator.

6. Initial piloting and item revision

  • Expertise  This was based on lived experience from rural doctors in three countries who were colleagues of the clinical courage research group. Each clinician in the group approached rural colleagues to complete and review the questionnaire.
  • Confidentiality  Comments were returned to the colleagues, who asked them to comment, and then de-identified and sent to the moderator. The comments were used to modify the questionnaire as necessary.
  • Consensus formation  This modified questionnaire was distributed to the participants for sign-off, which all did, and this was the version used in the next stage of the development of the clinical courage questionnaire.

These activities were undertaken across a series of weekly meetings across a 3-month period with the facilitator (RB), providing a summary of consensus work and next steps prior to each meeting, as outlined in Table 1. 

Ethics approval

This research project received low-risk ethics approval from The University of Adelaide Human Research Ethics Committee (project number H-2022-086). 


The question generation process resulted in 73 questions across the six domains (Table 2). For each domain, between 7 and 19 draft questions were reviewed by the group to eliminate questions that were (1) not a measure of the domain or (2) overlapped too much with each other within the domain. After exclusion and refinement, individual ranking and review after post-pilot feedback, 30 domain-specific questions remained.

After considerable discussion, the introductory statement for each question was modified to the form of the affective judgement (Table 3, option 3): ‘In my current practice how like me is it …’ The consensus was that clinical courage was most likely a praxis associated with contemporary work (‘in my current practice’), rather than a description of an individual’s characteristics (option 1) or values (option 2).

A trial questionnaire was constructed from the final questions in each domain. This trial questionnaire was modified in view of feedback from eight rural clinicians, who completed and commented on it. At the penultimate group meeting, the nominal group discussed the feedback and through consensus decided to split one question and delete a question that was perceived as very similar to another question. Based on experience with rural and remote clinicians, the primary author suggested a global construct item: ‘to experience personal pleasure from meeting the significant clinical challenges of rural and remote patients’. A global item is one that is not directly related to any single underlying construct and may be found to serve as a proxy for the concept (global construct) of clinical courage. If it is determined to be a proxy, it can be used in situations when survey space is critically short in place of the questionnaire. Further, a global item may be useful in developing the scoring structure of the six factors. The process resulted in the initial clinical courage questionnaire (Table 4), which consists of 30 items covering the six domains and one global item.

Table 2:  Developing and refining questions for the clinical courage questionnairetable image

Table 3:  Style of questions considered for the clinical courage questionnairetable image

Table 4:  Definitions and final questions for the clinical courage questionnairetable image


Clinical courage has been identified through a number of qualitative articles as one way of experiencing work as a rural doctor1,5. Developing a clinical courage survey tool will allow researchers to better understand the frequency, intensity and stability of this praxis in rural medicine and potentially explore clinical courage in other professional groups. Recruiting and retaining clinicians to work in rural and remote areas is a multifaceted problem with factors internal and external to the clinician that influence this decision25. Many of the factors, particularly the external ones, are not amenable to change – such as urban background, partner preference and location of amenities26. Thus, efforts to understand intrinsic psychosocial emotional factors that attract and retain physicians working in rural and remote communities may provide a means to increase the number of clinicians in rural practice.

The steps outlined in this article provide an example of the internal processes a research team can undertake to improve construct validity of an initial questionnaire. Building valid questionnaires requires a methodical approach based on existing literature, expert knowledge and psychometric principles27. In consensus research, a small group well known to each other may be at risk of persuasion bias due to the social influence of stronger personalities of individuals within the group28. The authors recognise this is a study limitation; however, this group has spent several years undertaking critical discourse for meaning-making during qualitative data analysis29.

Questionnaires are a ubiquitous research tool and it is clear that they have been widely misused30. The nominal group approach, using a small number of knowledgeable participants, provided a solid foundation to achieve construct validity in the questionnaire design. Careful development of agreed definitions of the domains involved provided a strong base on which to write and rewrite survey items, as there was a definition against which to test each question. The selection and ranking processes provided a reasonably quick method of identifying the preferred items. However, this voting process was not confidential. Confidential voting is an expectation of nominal group processes16,28. The research group has worked well together over several years and our familiarity with working together resulted in this intentional omission.

Another potential weakness to this study is that consensus does not equate to correctness28. The nominal group members are all researchers who bring their previous common research history to this process. This limitation will be compensated for in the second phase of this research, when the clinical courage questionnaire is released for testing by rural doctors. The purpose of sharing this questionnaire at this time is, first, to inform other rural researchers about the questionnaire development process and, second, to show the questionnaire at this stage to other rural researchers and rural clinicians for further discussion and feedback.

Future developments

Physicians working in rural and remote communities around the world will be invited to complete the initial questionnaire. Physicians will be recruited through a number of rural physician organisations, such as the Australian College of Rural and Remote Medicine, Rural WONCA (World Organization of Family Doctors), and the Society of Rural Physicians of Canada. When sufficient numbers of surveys have been returned, the questionnaire will be modified based on psychometric principles. The phenomenological and psychometric approaches will not have a 1–1 correspondence and it is anticipated there may be substantial differences. Ultimately, modifications will be made to include other specified populations such as students undertaking rural/remote placements, nurses undertaking similar placements, and for non-physician clinician groups.

The authors hope the results of this questionnaire will ultimately help to inform health professional education for rural and remote practice as well as recruitment and retention strategies. It may also be a useful, relevant tool that will provide valuable information to guide program evaluation and support future research.


1 Konkin J, Grave L, Cockburn E, Couper I, Stewart R, Campbell D, et al. Exploration of rural physicians' lived experience of practicing outside their usual scope of practice to provide access to essential medical care (clinical courage): an international phenomenological study. BMJ Open 2020; 10(8): e037705. DOI link, PMid:32847915
2 Dunlop P. Clinical courage. Canadian Journal of Rural Medicine 2011; 16: 107.
3 Maclellan K. Clinical courage. Canadian Journal of Rural Medicine 2011; 16: 107.
4 Wootton J. President's message. Clinical courage. Canadian Journal of Rural Medicine 2011; 16: 45-46.
5 Walters L, Couper I, Stewart R, Campbell D, Konkin J. The impact of interpersonal relationships on rural doctor's clinical courage. Rural and Remote Health 2021; 21(3): 6668. DOI link, PMid:34560821
6 Mandal A, Phillips S. To stay or not to stay: the role of sense of belonging in the retention of physicians in rural areas. International Journal of Circumpolar Health 2022; 81(1): 2076977. DOI link, PMid:35575172
7 Chipp C, Dewane S, Brems C, Johnson ME, Warner TD, Roberts LW. ‘If only someone had told me…’: lessons from rural providers. The Journal of Rural Health 2011; 27(1): 122-130. DOI link, PMid:21204979
8 Wilson MM, Devasahayam AJ, Pollock NJ, Dubrowski A, Renouf T. Rural family physician perspectives on communication with urban specialists: a qualitative study. BMJ Open 2021; 11(5): e043470. DOI link, PMid:33986048
9 Gingerich A, Van Volkenburg K, Maurice S, Simpson C, Roots R. Urban ideals and rural realities: physiotherapists navigating paradox in overlapping roles. Medical Education 2021; 55(10): 1183-1193. DOI link, PMid:33617663
10 Leung J, Smith A, Atherton I, McLaughlin D. ‘Everybody knows everybody else's business’ – privacy in rural communities. Journal of Cancer Education 2016; 31(4): 811-812. DOI link, PMid:26007636
11 Lynch JM, Driel M, Meredith P, Stange KC, Getz L, Reeve J, et al. The craft of generalism clinical skills and attitudes for whole person care. Journal of Evaluation in Clinical Practice 2022; 28(6): 1187-1194. DOI link, PMid:34652051
12 Mallinson T. Clinical courage. Journal of Paramedic Practice 12(11): 9 November. DOI link
13 Bentley M, Dummond N, Isaac V, Hodge H, Walters L. Doctors' rural practice self-efficacy is associated with current and intended small rural locations of practice. Australian Journal of Rural Health 2019; 12486. DOI link, PMid:30957340
14 Isaac V, Walters L, McLachlan C. Association between self-efficacy, career interest and rural career intent in Australian medical students with rural clinical school experience. BMJ Open 2015; 5: e009574. DOI link, PMid:26671960
15 Tyrrell MS, Carey TA, Wakerman J. The work motivations of the health practitioner who stays for a substantial time in the very remote Indigenous community workplace. Australian Journal of Psychology 2018; 70: 318-329. DOI link
16 Humphrey-Murto S, Varpio L, Gonsalves C, Wood TJ. Using consensus group methods such as Delphi and Nominal Group in medical education research. Medical Teacher 2017; 39(1): 14-19. DOI link, PMid:27841062
17 Chen FF, Sousa KH, West SG. Testing measurement invariance of second-order factor models. Structural Equation Modeling: A Multidisciplinary Journal 2005; 123: 471-492. DOI link
18 Block ZA, Brinkman JT, Gard SA. The utilization of consensus techniques in education and research in medical professions. Journal of Prosthetics and Orthotics 2021; 33(3): 175-183. DOI link
19 Tran C, Wood TJ, Humphrey-Murto S. Consensus group methodology in health professions education research: the nominal group technique. Academic Medicine 2020; 96: 1073. DOI link
20 Koran J. Indicators per factor in confirmatory factor analysis: more is not always better. Structural Equation Modeling 2020; 27(5): 765-772. DOI link
21 Koran J. Preliminary proactive sample size determination for confirmatory factor analysis models. Measurement and Evaluation in Counselling and Development 2016; 49: 296-308. DOI link
22 Couper I, Walters L, Williams S, Campbell D, White I, Stewart R. Exploring rural doctors' early experiences of coping with the emerging COVID‐19 pandemic. The Journal of Rural Health 2022; 38(4): 923-931. DOI link, PMid:35191080
23 White I, Benson J, Elliott T, Walters L. Australian general practice registrars' experiences of training, well-being and support during the COVID-19 pandemic: a qualitative study. BMJ Open 2022; 12(6): e060307. DOI link, PMid:35725266
24 Waggoner J, Carline JD, Durning SJ. Is there a consensus on consensus methodology? Descriptions and recommendations for future consensus research. Academic Medicine 2016; 91(5): 663-668. DOI link, PMid:26796090
25 Cosgrave C. The whole-of-person retention improvement framework: a guide for addressing health workforce challenges in the rural context. International Journal of Environmental Research and Public Health 2020; 17(8): 2698. DOI link, PMid:32295246
26 Cano G, Bain-Donohue S, Moore M. Why do some medical graduates lose their intention to practise rurally? Rural and Remote Health 2021; 21(2): 5747. DOI link, PMid:34082565
27 Rattray J, Jones MC. Essential elements of questionnaire design and development. Journal of Clinical Nursing 2007; 16(2): 234-243. DOI link, PMid:17239058
28 Block ZA, Brinkmann JT, Gard SA. The utilization of consensus techniques in education and research in medical professions. Journal of Prosthetics and Orthotics 2021; 33(3): 175-183. DOI link
29 Felton M, Crowell A, Liu T. Arguing to agree: mitigating my-side bias through consensus-seeking dialogue. Written Communication 2015; 32(3): 317-331. DOI link
30 Boynton PM, Greenhalgh T. Selecting, designing, and developing your questionnaire. British Journal of Medicine 2004; 328: 1312-1315. DOI link, PMid:15166072