Original Research

A return-on-investment analysis of impacts on James Cook University medical students and rural workforce resulting from participation in extended rural placements

AUTHORS

name here
Torres Woolley1
PhD, MPHTM, Evaluation Coordinator *

name here
Tarun Sen Gupta2
FRACGP, FACRRM, Professor

name here
Ruth A Stewart3
FACRRM, DRANZCOG adv, National Rural Health Commissioner for Australia

name here
Aaron Hollins4
FACRRM, FAFPHM, Senior Lecturer

CORRESPONDENCE

*Dr Torres Woolley

AFFILIATIONS

1, 2 College of Medicine & Dentistry, James Cook University, Townsville, Qld 4811, Australia

3 College of Medicine & Dentistry, James Cook University, Victoria Parade, Thursday Island, Qld 4875, Australia

4 College of Medicine & Dentistry, James Cook University, Atherton, Qld 4883, Australia

PUBLISHED

19 October 2021 Volume 21 Issue 4

HISTORY

RECEIVED: 21 December 2020

REVISED: 11 August 2021

ACCEPTED: 12 August 2021

CITATION

Woolley T, Sen Gupta T, Stewart RA, Hollins A.  A return-on-investment analysis of impacts on James Cook University medical students and rural workforce resulting from participation in extended rural placements. Rural and Remote Health 2021; 21: 6597. https://doi.org/10.22605/RRH6597

AUTHOR CONTRIBUTIONSgo to url

This work is licensed under a Creative Commons Attribution 4.0 International Licence


abstract:

Introduction:  Although all medical students at James Cook University (JCU), Queensland, Australia, undertake rural placements throughout their course, a proportion (currently about 20 per year out of 170–190 final-year students) undertake extended rural placements in rural and remote towns – 5-month Integrated Rural Placement (IRP) or 10-month Longitudinal Integrated Clerkship (LIC) programs. This study uses a return-on-investment (ROI) approach to quantify student and rural medical workforce benefits arising from these ‘extended placements’ between 2012 and 2018.
Methods:  Seventy-two JCU MBBS graduates participated in extended rural placements between 2012 and 2018. In 2019, 46 of these graduates who had reached at least postgraduate year 2 and provided consent to be contacted for health workforce research were emailed a link to an online survey. Questions explored the key benefits to students’ development of competencies and to rural medical workforce as a direct result of student participation in the IRP/LIC activities, as well as estimations of costs to students, deadweight (how much change would have occurred without participating in an extended placement), and attribution (how much change was due to other programs or experiences). The key student and rural medical workforce benefits were each assigned a ‘financial proxy’ to allow calculation of ROI from 2013 to 2019 as a dollar value, compared with the costs to students and to the JCU medical school from implementing the IRP/LIC programs between 2012 and 2018.
Results:  Twenty-five of the 46 JCU medical graduates who undertook an extended placement responded (response rate 54%), reporting that the most common (96%) and most important benefit (56%) from their extended placement was ‘greater depth and breadth of clinical skills’. Seventy-five percent (18/24; one missing response for this question) of the respondents also reported intending to have a full-time career in rural and remote practice. The overall cost of undertaking an IRP or LIC program for students between 2012 and 2018 was calculated to be $60,264, while the cost to the JCU medical school for sending 72 students out on extended rural placements was calculated as $32,560, giving total costs of $92,824. Given the total value of benefits ($705,827) calculated for the key student benefit of increased clinical skills and confidence in the internship year from participating in an extended placement ($32,197) and for the key rural medical workforce benefit of willingness to work in a rural or remote town ($673,630), the ROI from the extended rural programs between 2013 and 2019 (after students graduated and entered the workforce) is calculated at $7.60 for every dollar spent.
Conclusion:  This study confirms that undertaking an extended placement has significant positive impacts on final-year medical students’ clinical confidence, clinical skills and communication skills into their internship year. In addition, the extended placements have longer-term impacts on the non-metropolitan health workforce by inspiring more JCU medical graduates to take up rural generalist, rural general practitioner or generalist specialist positions in rural and remote towns. This positive ROI from extended rural placements is important evidence for shifting the conversation around supporting these programs from one of cost to one of value.

Keywords:

Australia, cost–benefit, longitudinal, Longitudinal Integrated Clerkship, medical, return on investment, rural placement.

full article:

Introduction

Positive rural and remote community placements have been shown to have educational benefits for medical students that are at least equal to if not better than traditional urban placements1-5. Immediate benefits include improving clinical competencies, particularly those specific to the priority health issues of these rural and remote areas6, while longer-term benefits include increased intentions for future rural practice and for working with underserved populations7-10. There is also good evidence that if medical students are not provided with rural training opportunities, then the positive effect of having a rural background on subsequent rural practice diminishes over time6,11.

In addition, there are further benefits of having extended rural placements12-14. Longer rural placement duration has been found to be associated with later rural practice15-17 as well as with a later career in primary care18. Research has also shown extended rural or urban clinical placements have advantages over much shorter hospital-based clinical blocks with respect to improved academic results19-21, enhanced patient-centredness19,20, greater exposure to common conditions21, more meaningful learning relationships with patients and academic mentors22, and quality of student feedback23.

In 2012, the James Cook University (JCU) medical school implemented two types of extended rural placement programs for final-year students – the Integrated Rural Placement (IRP) and Longitudinal Integrated Clerkship (LIC) programs. These extended placements base students in small rural and remote towns across northern Australia for periods of 20 and 35 weeks, respectively. However, community-based training programs have extra expenses around student living and travel for both the students and the medical school, and local preceptors and administrators may need to be paid to coordinate and administer the rural programs. Quantifying the return on investment (ROI) from extended community-based medical student training would be important evidence to shift the conversation from one of cost to one of value. If the approximate overall value of extended placements to participating students and later rural medical workforce can be calculated, it may be easier to obtain community and government support for extended rural placements into the future. This study aims to quantify the financial ROI from key participant and rural workforce benefits arising from year 6 JCU medical students participating in extended rural placements from 2012 to 2018.

Methods

Study design

This study was based on accepted social ROI (SROI) methodology that was simplified to involve just a single survey of 2012–2018 JCU MBBS graduates who had experienced extended placements. The survey involved a question to quantify the costs incurred by respondents while undertaking the program as year 6 students, and a combination of closed and open-ended questions to approximate the ‘standard’ quantitative and qualitative components of an ROI evaluation for quantifying the key benefits to stakeholders. The costs incurred by the JCU medical school in delivering the programs from 2012 to 2018 were obtained from school records.

Description of the medical school’s extended placements 

The JCU medical school rural placement program underpins the JCU medical curriculum. All students undertake a total of 20 weeks of mandatory placements in rural and remote communities during the second (4 weeks), fourth (6 weeks) and sixth years (10-week ‘rural internship’) of study. Rural placements aim to expose students to rural and remote health and medical care, as well as provide experiences of living in these communities. In addition, year 6 students are able to choose a 20-week or 35-week extended placement, which includes the 10-week rural internship.

What is social return on investment methodology?

Social return on investment (SROI) methodology is distinguishable from other forms of economic valuation by the engagement of stakeholders in identifying and then valuing personal, social and community outcomes in addition to economic outcomes, the creation of a ‘theory of change’ for the program to capture the linear association between inputs (costs) and program activities leading to stakeholder outcomes/benefits (short-, intermediate- and longer-term).

A unique aspect of ROI evaluations is the assigning of a financial proxy value to each key stakeholder benefit to provide an estimation of the social and economic value generated by the program24. Financial proxies allow the ROI to be expressed as a ratio of the total monetised value of benefits (social value) divided by the total program investment. To make the estimated ratio more accurate, the total calculated social value is then ‘discounted’; that is, reduced, to account for other contributing factors. In this instance, the total value of extended placements will be reduced by taking into account: (1) how much of the key benefits would have occurred anyway if the respondents had not undertaken an extended placement (deadweight), (2) given respondents experienced an extended placement, how much of the key benefits was due to other programs or experiences (attribution), and (3) how long the key benefits last after the placement is completed (duration).

ROI evaluation of the extended rural programs

The ROI evaluation of the JCU medical school’s extended placements follows the six stages of an SROI described by Laing and Moules25, which in turn are based on the stages established in the SROI Network’s guide to SROI26. (Note: This article does not follow the traditional approach of data collection, data analysis and description of results; these are instead described under each of the six SROI stages.)

Stage 1: Establishing scope and identifying key stakeholders:  The authors, all of whom are rural workforce academics and/or rural practitioners, set the scope for the ROI analysis, and identified the stakeholder groups benefiting from the extended rural placement program. Stakeholder beneficiaries are defined as people, organisations or specific populations that experience change (positive or negative) resulting from program activities26. The scope was set at two key stakeholder groups potentially experiencing benefits from extended placements: JCU MBBS graduates who had undertaken an extended placement in their final year of medical school, and rural Australian medical workforce. In SROI evaluations, ‘true beneficiaries’ are suggested to be better placed to determine the outcomes accrued as a result of an intervention compared with ‘funder’ stakeholders27. Therefore, the stakeholder group ‘rural Australian medical workforce’ was chosen for this evaluation rather than the JCU medical school, which funded and implemented extended placements, as ‘rural Australian medical workforce’ is the intended key long-term beneficiary.

The author group also considered all potential benefits and costs to these two key stakeholder groups and summarised these into a draft survey. This draft survey was piloted with two JCU year 6 MBBS students who were currently undertaking an IRP to ensure that the full range of potential benefits for JCU graduates and rural workforce were included in the survey, and that survey questions were likely to be well understood by respondents. After validity testing, an electronic SurveyMonkey survey was sent to the 46 of the total 72 JCU MBBS graduates who had undertaken an extended placement between 2012 and 2018 (as identified through school records) and who had consented to be contacted for further studies in the final-year exit survey.

Stage 2: Mapping costs and benefits:  Stakeholder engagement via the graduate survey was used to develop a theory of change, as well as identifying and quantifying the most important outcome (key benefit) for both the graduate and rural Australian medical workforce stakeholder groups in this impact map. Mapping a theory of change is a key component of a ROI analysis and describes the relationship between inputs (money and other resources that go into running a program), activities of the program, and outcomes/benefits (short-, intermediate- and longer-term) for the stakeholders as a result of the program26. Thus, the theory of change is the ‘story’ of how extended placements make a change or difference in the participants and other stakeholders.

To map the costs to students, the survey asked JCU graduates to self-report the costs accrued to them on their extended placement. To understand the key placement activities and outcomes for students, the survey explored their intentions for a rural career and vocational specialty training pathway, benefits to their clinical competencies and confidences from their placement, how much the extended placements contributed to their clinical skills and career intentions, and any potential negative outcomes from undertaking placements. All questions are described in Appendix I, while the reported costs, activities and resulting benefits are summarised in Figure 1.

The costs to the medical school for the extended rural placements were based on the amount of money paid to students as travel subsidies to move from their home clinical school to the placement site, and for subsidies to connect to the internet via a domestic arrangement if this technology was not available at the placement accommodation. School records identified each of the 72 students’ home clinical school site in year 6 and their rural placement site, allowing calculation of the associated travel subsidy and internet subsidy, if appropriate.

The most common benefit from extended placements reported by 96% of the JCU graduates (and identified as the ‘most important’ by 56% of respondents) was ‘greater depth and breadth of clinical skills’ from the extended placements compared with the mandatory 10-week rural term and other non-rural placement experiences. An increase in clinical confidence was the second most significant outcome for graduates (reported as ‘most important’ by 38% of respondents), with 88% of respondents reporting increased clinical confidence from their placement. Most students also reported an increase in communication skills (94%) and cross-cultural communication skills with Aboriginal and Torres Strait Islander patients (78%).

From the outcome map, the most common benefit for the rural Australian workforce beneficiary group was chosen by the research team as ‘more JCU graduates choosing a full-time rural career’. Overall, 75% (18/24; one missing response for this question) of JCU medical graduate respondents reported intending to have a career in rural and remote practice (defined as 50 or above on a 0–100 scale, with 100 being ‘definitely will be a rural doctor’), with 92% (23/25) of responders also reporting they were currently pursuing a career as a generalist (rural generalist 68%, general practitioner (GP) 16%) or general specialist (8%), appropriate for rural practice. While most of the respondents were still in their early career (postgraduate years 1–3), 10 of the 25 (40%) were practising in a rural or remote town (Modified Monash Model (MMM) 4+) in 2019. The MMM is a widely accepted system in Australia for classifying metropolitan, regional, rural and remote areas into seven rurality categories according to geographical remoteness and town size.

table image Figure 1:  Theory of change for James Cook University extended rural placements describing the relationship between inputs (money and other resources required), activities and outcomes (short-, intermediate- and longer-term) for stakeholders from participating in the programs.

Stage 3: Evidencing key outcomes and giving them a value:  This stage involves uncovering collaborating objective quantitative data to confirm whether the key benefits have occurred, and then to value these benefits through a ‘monetisation’ process involving an appropriate financial proxy. In SROI methodology, financial proxies are used to estimate the net social value, in present-day dollar terms, of key benefits that do not have a market price26.

For the medical student beneficiary group, analysis of the medical school’s final-year exit surveys from 2012 to 2018 (n=738, including 46 who had undertaken an extended placement) was used to confirm that ‘greater depth and breadth of clinical skills’ had occurred in those undertaking an extended placement (Table 1). The exit surveys ask year 6 students how well the MBBS course prepared them to be a doctor with regard to ‘your clinical skills’, ‘ability to apply your knowledge’ and ‘your readiness to work in the first week of internship’. Between 2012 and 2018, year 6 students who undertook an extended placement had significantly higher mean scores than the other students in self-reported ‘clinical skills’ (mean of 4.49 vs 4.29 out of 5; p=0.042), ‘readiness to work in first week of internship’ (mean of 4.47 vs 4.09 out of 5; p=0.001) and ‘ability to apply knowledge’ (mean of 4.33 vs 4.07 out of 5; p=0.006). Statistical comparisons were undertaken using SPSS v24 for Windows (IBM; http://www.spss.com) via two-sided Student’s t-tests, with statistical tests considered significant with p<0.05.

As JCU graduate responders reported increases in both depth and breadth of clinical skills as well as clinical confidence from undertaking extended placements, the financial proxy assigned to value this outcome was based on the personal cost to students to be trained as doctors. Through the Higher Education Contribution Scheme, Australian medical students are required to pay training costs of $9792 per year at 2014 levels. Thus, JCU medical students are required to pay 6 × $9792 = $58,752 for their 6 years of training; that is, the value to JCU students of the standard 6 years training to be ‘work ready’ for internship is $58,752. However, using a ‘revealed preference’ approach27, JCU students who undertake an IRP/LIC experience report (via the exit surveys), on average, an additional 9.3% of ‘work readiness for internship’ (mean of 4.47 vs 4.09). Note that, for this key benefit, the 9.3% is also used as the percentage of deadweight in the later discounting calculation. Therefore, if students have to pay $58,752 for 6 years of training to be ‘work ready’ interns, then, using standard ROI methodology, of the 72 JCU graduates who undertook an extended placement from 2012 to 2018, 88% (based on 22 of the 25 survey responders who reported increased clinical competencies and confidence) would have each received an additional $5464 of value from their extended placement.

For the beneficiary group ‘rural medical workforce’, analysis of the JCU medical school’s graduate tracking database demonstrates that 16 of the total 72 (22.2%) JCU graduates who undertook an extended placement were working in rural or remote towns (MMM 4+) in 2019, compared with 131/1524 (8.6%) of JCU graduates who did not do an extended placement (Table 1). Thus, the extended year 6 rural placement programs are associated with an extra 10 JCU graduates ((22.2% – 8.5%) × 72 = 9.8) practising in a rural or remote town during 2019 than would have been expected without the extended rural programs.

Further evidence of the impact of extended rural placements on promoting willingness towards rural practice was also observed in the study findings. The current study found that the 25 respondents reported their extended placement contributed an average of 67% towards ‘their desire to be a rural or remote doctor’ (attribution); with seven of the 25 respondents also stating they were currently working more rurally in 2019 than they would have had they not undertaken an extended rural placement. These seven included four respondents who reported currently working (2019) in a regional city when they otherwise (if not for their extended rural placement) would have been working in a major city, plus a further one respondent now working in a rural town rather than in a regional city, and two respondents now working in a remote rather than a rural town.

These seven respondents who self-reported that they are practising more rurally than they would have without their extended placement experience were also pursuing rural generalist (5) or generalist specialist (one emergency medicine and one anaesthesia) careers – appropriate choices for a longer-term rural career. Overall, 21/25 (84%) of survey respondents reported currently pursuing a career as a rural generalist or GP, while the school’s graduate tracking database shows 57% of graduates to be currently training or have achieved fellowship as rural generalists or GPs. This increase in vocational training choice for generalist medicine after undertaking an extended placement is supported by the finding that 16/25 (64%) of respondents reported that their placement had ‘influenced the type of doctor they want to be’, with all 16 describing a change towards rural practice and/or rural generalism.

For the stakeholder group ‘rural medical workforce’, the financial proxy assigned for the key benefit ‘JCU graduates willing to work full-time in a rural or remote town’ was based on the willingness of many respondents in the current study to fill a vacant position in a rural or remote town for less than their expected salary. Overall, 50% (12/24; one missing response for this question) of respondents reported they would be willing to accept lower remuneration to secure work in a rural or remote town – a ‘stated preference’ approach27. Of the 12 reporting a willingness to accept a rural doctor position, two reported being willing to accept 0–10% less, five would accept 10–20% less, two would accept 20–30% less, two would accept 30–40% less, and one would accept 40–50% less. Combining this question with another query asking what specific salary they would expect for working in a rural town, the 24 respondents, on average, were willing to take $23,479 less remuneration to work in a rural town (using the midpoint for each percentage category; for example, for the 10–20% less category, a midpoint of 15% was used). Therefore, using standard ROI methodology, of the 72 JCU graduates who undertook an extended placement from 2012 to 2018, 75% (based on 18 of the 24 survey responders who reported a desire to work as a doctor full-time in a rural or remote town) would create $23,479 of savings to rural Australian medical workforce per year.

This financial proxy approximates the potential savings to rural workforce from more quickly filling positions in rural and remote towns because of more doctors willing to accept an available position in these areas as a result of undertaking an extended rural placement. This approximation does not necessarily mean that these graduates would accept less money; it may also cover situations where graduates take a position in a rural or remote town that historically has found it difficult to recruit permanent doctor positions even at full pay. This is a reasonable approximation given current difficulties associated with filling Australian rural doctor vacancies with graduates having appropriate procedural skills28.

Table 1:  The key identified outcomes of the Integrated Rural Placement and Longitudinal Integrated Clerkship programs with associated quantitative and qualitative evidence of outcomestable image

Stage 4: Establishing impact:  Calculating a more accurate monetary value to the key benefits must entail ‘discounting’ of the net social value of the key outcomes. Discounting relates to the proportion of the total population who experienced the outcome/benefit, how much of the key outcomes would have happened without the extended placement experience (deadweight), whether any other organisations or programs contribute to key outcomes in addition to the extended placements (attribution), and how long outcomes will last after the program has been completed (duration). In this ROI evaluation, the level of discounting for ‘proportion’, ‘deadweight’, ‘attribution’ and ‘duration’ was established through respondent self-report in the survey and external databases using the formula:

total population ×  proportion ×  $ value of financial proxy ×  % deadweight ×  attribution × duration

The financial value to the 72 students (total population) who undertook an extended placement from 2012 to 2018 is based on the self-reported key benefit ‘work ready for internship’. The survey found 88% of the 25 respondents reported both increased clinical skills and clinical confidence that improved their work readiness for up to 12 months (ie their internship year), based on survey responses where 100% of respondents gave a 6–12-month period of this benefit (duration), and, as previously stated, the deadweight is 9.3%. As there are no other programs that are involved in teaching students clinical skills and confidences in rural towns during extended placements, and as the deadweight also took into account attribution towards being work ready for internship outside of the extended placement experience, the attribution for this benefit is 100%. With an approximate value of $5464 worth of extra clinical skills and confidences to each student who reported experiencing this benefit, the value calculation to the overall population of students undertaking an extended placement is:

72 (total population) ×  88% (proportion) ×  $5464 ×  9.3% (deadweight) ×  100% (attribution) × 1 (duration in years)

= $32,197 total value for all JCU MBBS students undertaking an extended placement between 2012 and 2018.

The total financial value to ‘rural medical workforce’ between 2013 and 2019 from the 72 students (total population) who undertook an extended placement from 2012 to 2018 is based on the self-reported key benefit ‘willingness to work in a rural or remote town’. The survey found 75% (18 of 24; one missing data) of respondents reported a desire to work full-time as a doctor in a rural or remote town (proportion), and that respondents reported their extended placement contributed an average of 67% towards ‘their desire to be a rural or remote doctor’ (attribution). In addition, the school’s tracking database showed JCU graduates who experienced an extended rural placement were more likely to be working rurally in 2019 (22.2%) than those who did not (8.6%). Deadweight is calculated by removing the proportion of key outcomes expected to occur if participants had not undertaken the program; therefore, the calculation of deadweight for this benefit is: 100% – (8.6 ÷ 22.2)% = 61%. Further, the 18 graduates who reported a desire to work full-time in a rural or remote town had worked an average of 1.3 years (duration) in a rural or remote town (MMM 4–7) between 2013 and 2019. Therefore, with an approximate value of $23,479 worth of ‘willingness to work in a rural or remote community’, the value calculation from the overall population of students undertaking an extended placement from 2012 to 2018 is:

72 (total population) ×  75% (proportion) ×  $23,479 ×  61% (deadweight) ×  67% (attribution) × 1.3 (duration in years)

= $673,630 total value for ‘rural medical workforce’ between 2013 and 2019.

Stage 5: Calculating the SROI and undertaking a sensitivity analysis:  Once the total net value had been calculated for 2013 to 2019 (allowing time for students to graduate and enter the workforce) and the various discounts applied, this value was compared with the total investment from 2012 to 2018 (ie the monetised costs of running the IRP/LIC programs) to determine the ROI ratio26. In this SROI study, the net financial value of the IRP and LIC programs from 2013 to 2019 from the key benefits was $705,827 – calculated from the total value to students ($32,197) and the total value to rural medical workforce ($673,630). The total costs of the IRP/LIC program from 2012 to 2018 were calculated at $92,824, including costs to the students of $60,264, and costs to the JCU medical school for sending 72 students out on extended rural placement of $32,560. The SROI ratio, therefore, is $705,827 ÷ $92,824, equating to a ratio of 7.6:1. In other words, for every dollar invested in extended placements from 2012 to 2018, $7.60 of approximate financial value was created.

In calculating the total ROI, it was necessary to make an assumption about the reduction in salary 12 respondents reported being willing to accept for a rural doctor position. When calculating the average reduction of $23,479, only the midpoint was used for each percentage category. For example, for the 10–20% less category, a midpoint of 15% was used, although the true reduction could be anywhere between 10% and 20%. To assess how much influence this assumption may have had on the final ROI ratio of 7.6:1, a sensitivity analysis was carried out to allow the overall value to be expressed between lowest case scenario and highest case scenario for each of the 24 examples. Thus, the sensitivity analysis on the lowest percentage range for each of the 24 respondents’ reported reduction in salary gave a minimum average value of $16,667 and a maximum average value of $29,458. Thus, with an SROI ratio of 7.6:1, there is a degree of confidence that extended rural programs create between $5.50 and $9.50 of social and economic benefits for the key stakeholders for every dollar invested.

Ethics approval

Ethics approval for the study was obtained from the JCU Human Ethics Committee (#H6921).

Discussion

Traditional approaches to which decisions in health professional education are made and analysed often take only financial measures into account, or use simplistic statistical associations with a single outcome (eg future practice location). This ROI study of the value of extended rural placements, the first in health professional education, shows significant benefits to medical students in terms of clinical skills and confidence, and to rural Australian workforce in terms of increased numbers of medical graduates willing to work full-time in rural and remote areas as rural generalists, GPs or generalist specialists. These benefits have far greater value than the costs associated with running the program, and will continue to accrue benefits to rural Australian medical workforce past the 2019 evaluation period, assuming at least some of these graduates continue to work in underserved rural and remote towns – a likely outcome with 40% of the respondents in this study practising in a rural or remote town (MMM 4+) in 2019.

Furthermore, while this study shows JCU’s extended rural placements create considerable benefits to rural Australian medical workforce by producing more full-time regional, rural and remote rural generalists/GPs, other financial benefits would also accrue to rural workforce and rural community members. These other benefits would probably include a reduction in the costs of bringing in locums or international medical graduates to cover vacant positions, or the cost to rural practices and community members if a GP locum or an international medical graduate could not be employed to fill that position. Recruiting difficulties can lead to rural health services needing to employ temporary locum GPs on thousands of dollars a day – endangering the financial sustainability of practices already under fiscal strain29. A study in rural USA found each full-time equivalent rural physician vacancy carried an estimated US$990,000 annualised loss to the medical practice due to costs of employing short-term locums30.

However, much of the value in health professional education is psycho-social in nature, rather than just financial. For example, studies strongly suggest that the social value to rural Australian communities from having a permanently based, longer-term rural doctor would probably be worth considerably more than just financially based outcomes31,32. Similarly, previous research demonstrates that a strong level of clinical skills and clinical confidence has considerable positive psychological effects for intern-year doctors. The transition from medical student to intern is very stressful, with anxiety and burnout common33,34, and any negative personal experiences during this period may significantly impact on future confidence levels and professional identity35. It is likely that the extra skills and confidence that most JCU graduates received from their extended rural placement experiences might somewhat alleviate their anxiety, stress and chances of negative experiences during their intern year, which would be very much valued during internship.

In summary, this ROI study shows extended undergraduate rural placement programs result in ‘real’ value for both student participants and rural health workforce, including strong evidence of increased rural medical practice in the short and longer term. Thus, findings strongly suggest rural health workforce policy and strategy should focus on encouraging longer-term undergraduate (and probably postgraduate) rural training posts.

Limitations

There is inherent subjectivity involved with using financial proxies to represent social value, and a different research team may produce different results. ROI methodology is an approximation, as any time value is in question, it is inherently subjective. However, ROI will always tend to underestimate the total impact of intervention programs26. In particular, the current study has not factored in benefits to the community. Thus, while this evaluation counts the expenses students had while living in the community as a program cost, this is actually a financial benefit to the community. In addition, some students would also positively impact on the social fabric of the community, and, as final-year students, assist with clinical activities in hospitals and medical centres, which can also be a significant benefit to the community’s health services.

The main advantage of ROI studies is that they measure ‘true’ program impact by providing a more accurate approximation of causality through the discounting process. Discounting takes into account deadweight (how likely key outcomes are to still occur had participants not undertaken the program), attribution (other influences on participants in addition to the program’s influences) and duration of benefits. Without discounting, it would not be possible to tease out the ‘true’ influence of extended rural placements on participants’ motivations for a rural career and clinical competencies from other ‘non-programmatic’ influences that may be contributing to these outcomes. For example, the opt-in nature of the extended placement programs at JCU means students who are already motivated towards a rural career and/or are confident in their clinical skills will probably be more likely to choose an extended rural placement; this influence can be removed from the final calculation of program impact by adjusting for deadweight. The other advantages of ROI studies are that the full range of benefits arising from the program are usually uncovered, and that the financial proxy for the key benefit to each stakeholder group can approximate the overall value of the program in relation to costs – critical factors in determining whether programs should be further funded or potentially expanded.

However, it must be stated that the ROI methodology relies on approximations and assumptions from self-report to calculate the financial proxies for the key benefits. For example, the calculation for financial proxy for the student stakeholder group – $5464 each for those reporting an increase in clinical skills and confidences – is based both on a 9% increase in self-reported ‘work readiness’ in the exit survey (an approximation of increased benefit derived from a Likert scale), and the assumption that this equates to an extra $5464 of value based on students’ overall HECS fees for undergraduate medical training. Similarly, the calculation of the financial proxy for benefits to ‘rural medical workforce’ is based on graduates’ willingness to accept a lower remuneration as a direct result of their extended placement experiences, leading them to quickly fill an available position in a less desirable rural town or take lower remuneration than might be expected for such positions.

Overall, however, the key to an effective ROI is quality data. While the sample size for the survey was small at 25 respondents, the response rate was reasonable at 54%. In addition, this ROI used several approaches to improve the trustworthiness of the findings: involving multiple sources of objective data from other sources to confirm survey findings, and open-ended questions to allow mapping of a simple theory of change showing the association between key placements activities and key outcomes. However, due to budgetary constraints, this ROI evaluation, while adapted from accepted SROI methodology, only collected quantitative data from a survey without any qualitative investigations, nor were interpretations of this ROI evaluation verified with survey participants – both strongly recommended by the SROI Network26. This survey-only approach was developed as a quick and cost-efficient way of undertaking an ROI evaluation in response to a standard SROI evaluation not being feasible. Indeed, reviews of ROI in health and education24,36 have identified a significant lack of standard ROI program evaluations around health workforces due to the financial and time constraints involved. Thus, this cost-efficient, survey-based ROI approach has the advantage of usability; however, some richness and diversity of information will have been reduced by using content analysis of open-ended survey questions to approximate qualitative investigations.

In addition, the possibility exists that selection bias may be influencing the study inferences and findings, as only 25 of the total 72 graduates who had undertaken an extended rural placement in their undergraduate year 6 responded to the graduate survey, from which all inferences and findings arose. However, the authors accessed JCU medical school’s graduate tracking database to statistically compare the 25 respondents with the remaining 47 non-respondents across every variable collated in the tracking database. The analysis found the two groups were statistically similar (p>0.100) across all demographic variables (age, gender, rurality of hometown at application to medical school, ethnicity, international origin, lateral entry, accepted under a bonded medical place), undergraduate variables (scholarships awarded, Honours undertaken) and postgraduate training variables (rurality of practice location from postgraduate years 1–5, choice of specialty training college). Therefore, potential for selection bias in this study appears to be minimal.

While the findings of this survey-based ROI are encouraging, ‘traditional’ SROI evaluations involving associated qualitative investigations are further required to confirm the financial ROI of extended rural placement programs identified in this pilot study, as well as to estimate the true social value of program benefits to all student, community and health workforce stakeholders.

It should also be noted, however, that at present it is unlikely that many health professional education schools would easily be able to reproduce studies of this nature. Few researchers are experienced in the relatively new SROI methodology, and not many schools maintain a graduate tracking database containing the objective data required for SROI analyses. Alternatively, this argues for health professional education schools with government-funded rural or other special programs to develop appropriate graduate tracking databases that collate key outcome data, and to collaborate with impact evaluators experienced in SROI analyses.

Conclusion

This ROI study confirms that undertaking an extended rural placement has significant impacts on medical students’ clinical skills, clinical confidence and communication skills for their internship year, as well as having eventual longer-term impacts on rural Australian medical workforce by motivating more graduate doctors to choose a rural career.

The ROI analysis further confirms the financial value to medical students and the rural Australian health system as a direct result of an extended rural placement program – $7.60 for every dollar spent – far outweighing the costs of running the program. However, the overall value of extended placements is likely to be far greater; for example, the potential social and financial value to rural and remote community members from having more locally based doctors skilled in rural medicine. This positive ROI from extended rural placements is important evidence for shifting the conversation around supporting or expanding these programs from one of cost to one of value.

Acknowledgements

The authors acknowledge the JCU medical graduates who participated in this study.

references:

1 Sen Gupta TK, Hays RB, Kelly G, Buettner P. Are medical student results affected by allocation to different sites in a dispersed rural medical school? Rural and Remote Health 2011; 11: 1511. DOI link, PMid:21244193
2 Lenthall S, Wakerman J, Knight S. The frontline and ivory tower: a case study of service and professional-driven curriculum. Australian Journal of Rural Health 2009; 17(3): 129-133. DOI link, PMid:19469776
3 Wilson M, Cleland J. Evidence for the acceptability and academic success of an innovative remote and rural extended placement. Rural and Remote Health 2008; 8: 960. DOI link
4 McLean RG, Pallant J, Cunningham C, DeWitt DE. A multi-university evaluation of the rural clinical school experience of Australian medical students. Rural and Remote Health 2010; 10: 1492. DOI link, PMid:20818841
5 Worley P, Esterman A, Prideaux D. Cohort study of examination performance of undergraduate medical students learning in community settings. BMJ 2004; 328(7433): 207-209. DOI link, PMid:14739189
6 Strasser R, Hogenbirk JC, Lewenberg M, Story M, Kevat A. Starting rural, staying rural: how can we strengthen the pathway from rural upbringing to rural practice? Australian Journal of Rural Health 2010; 18(6): 242-248. DOI link, PMid:21114701
7 Stagg P, Greenhill JA, Worley PS. A new model to understand the career choice and practice location decisions of medical graduates. Rural and Remote Health 2009; 9: 1245. DOI link, PMid:19943714
8 Henry JA, Edwards BJ, Crotty B. Why do medical graduates choose rural careers? Rural and Remote Health 2009; 9: 1083. DOI link, PMid:19257797
9 Eley D, Young L, Baker P, Wilkinson D. Developing a rural workforce through medical education: lessons from down under. Teaching and Learning in Medicine 2008; 20(1): 53-61. DOI link, PMid:18444186
10 Larkins S, Michielsen K, Iputo J, Elsanousi S, Mammen M, Graves L, et al. Impact of selection strategies on representation of underserved populations and intention to practise: international findings. Medical Education 2015; 49(1): 60-72. DOI link, PMid:25545574
11 Hogenbirk JC, McGrail M, Strasser RP, Lacarte S, Kevat A, Lewenberg M. Urban washout: how strong is the rural-background effect? Australian Journal of Rural Health 2015; 23(3): 161-168. DOI link, PMid:25945452
12 Walters L, Greenhill J, Richards J, Ward H, Campbell N, Ash J, et al. Outcomes of longitudinal integrated clinical placements for students, clinicians and society. Medical Education 2012; 46: 1028-1041. DOI link, PMid:23078680
13 Playford DE, Nicholson A, Riley GJ, Puddey IB. Longitudinal rural clerkships: increased likelihood of more remote rural medical practice following graduation. BMC Medical Education 2015; 15: 55. DOI link, PMid:25879715
14 Hudson JN, Poncelet AN, Weston KM, Bushnell JA, Farmer EA. Longitudinal integrated clerkships. Medical Teacher 2017; 39(1): 7-13. DOI link, PMid:27832713
15 McGrail MR, O'Sullivan BG, Russell DJ. Rural training pathways: the return rate of doctors to work in the same region as their basic medical training. Human Resources for Health 2018; 16(1): 56. DOI link, PMid:30348164
16 Malhi RL, Ornstein J, Myhre D. The impact of rural rotations on urban based postgraduate learners: a literature review. Medical Teacher 2019; 41(7): 830-838. DOI link, PMid:31043111
17 Worley P, Martin A, Prideaux D, Woodman R, Worley E, Lowe M. Vocational career paths of graduate entry medical students at Flinders University: a comparison of rural, remote and tertiary tracks. Medical Journal of Australia 2008; 188(3): 177-178. DOI link, PMid:18241180
18 Halaas GW. The Rural Physician Associate Program: successful outcomes in primary care and rural practice. Rural and Remote Health 2005; 5: 453. DOI link
19 Ogur B, Hirsh D, Krupat E, Bor D. The Harvard Medical School-Cambridge integrated clerkship: an innovative model of clinical education. Academic Medicine 2007; 82(4): 397-404. DOI link, PMid:17414198
20 Ogur B, Hirsh D. Learning through longitudinal care - narratives from the Harvard Medical School-Cambridge Integrated Clerkship. Academic Medicine 2009; 84: 844-850. DOI link, PMid:19550174
21 Worley P, Strasser R, Prideaux D. Can medical students learn specialist disciplines based in rural practice: lessons from students' self-reported experience and competence. Rural and Remote Health 2004; 4: 338. DOI link
22 Worley P, Prideaux D, Strasser R, Magarey A, March R. Empirical evidence for symbiotic medical education: a comparative analysis of community and tertiary-based programmes. Medical Education 2006; 40: 109-116. DOI link, PMid:16451237
23 Mazotti L, O'Brien B, Tong L, Hauer KE. Perceptions of evaluation in longitudinal versus traditional clerkships. Medical Education 2011; 45: 464-270. DOI link, PMid:21486322
24 Banke-Thomas AO, Madaj B, Charles A, van den Boek N. Social return on investment (SROI) methodology to account for value for money of public health interventions: a systematic review. BMC Public Health 2015; 15: 582. DOI link, PMid:26099274
25 Laing CM, Moules NJ. Social return on investment: a new approach to understanding and advocating for value in healthcare. Journal of Nursing Administration 2017; 47(12): 623-628. DOI link, PMid:29135853
26 The SROI Network. A guide to social return on investment. 2012. Available: web link (Accessed 30 October 2020).
27 Louviere J, Hensher D, Swait J. Stated choice methods: analysis and applications. Cambridge, UK: Cambridge University Press, 2000. DOI link
28 Australian Institute of Health and Welfare. Rural and remote health. Web report. Canberra: AIHW, 2019. Available: web link (Accessed 30 October 2020).
29 National Rural Health Alliance. Locums and short-term contractors in the health workforce. Position paper. Canberra: NHRHA, 2012. Available: web link (Accessed 30 October 2020).
30 Schutte L. What you don't know can cost you: building a business case for recruitment and retention best practices. Journal of the Association of Staff Physician Recruiters 2012; 696. Available: web link (Accessed 16 September 2021).
31 Margolis S. Is Fly in/Fly out (FIFO) a viable interim solution to address remote workforce shortages? Rural and Remote Health 2012; 12: 2261. DOI link, PMid:23046214
32 Birks M, Mills J, Francis K, Coyle M, Davis J, Jones J. Models of health service delivery in remote or isolated areas of Queensland: a multiple case study. Australian Journal of Advanced Nursing 2010; 19(1): 36-45.
33 Sturman NJ, Tan Z, Turner J. 'A steep learning curve': junior doctor perspectives on the transition from medical student to the healthcare workplace. BMC Medical Education 2017; 17(1): 92. DOI link, PMid:28549459
34 Fitzpatrick O, Biesma R, Conroy RM, McGarvey A. Prevalence and relationship between burnout and depression in our future doctors: a cross-sectional study in a cohort of pre-clinical and clinical medical students in Ireland. BMJ Open 2019; 9(4): e023297. DOI link, PMid:31048421
35 Wald HS. Professional identity (trans)formation in medical education: reflection, relationship, resilience. Academic Medicine 2015; 90(6): 701-706. DOI link, PMid:25881651
36 Hutchinson CL, Berndt A, Gilbert-Hunt S, George S, Ratcliffe J. Valuing the impact of health and social care programmes using social return on investment analysis: how have academics advanced the methodology? A protocol for a systematic review of peer-reviewed literature. BMJ Open 2018; 8(12): e022534. DOI link, PMid:30530579

appendix I:

Survey questions listed under return-on-investment categories table image