Validating competencies for an undergraduate training program in rural medicine using the Delphi technique

full article:

Introduction

Inequality with respect to the geographic distribution of health professionals is a global issue confronting various countries, irrespective of their level of socioeconomic development. While half the world's population lives in rural or remote areas, less than 25% of doctors work in such regions¹.

The dearth of these professionals in rural areas is a multifactorial problem, related to professional isolation, lack of safety in the workplace, low salaries, irregular remuneration, lack of career advancement perspectives, difficult access to continued education, and lack of structure in the work environment and in the community^2-5. For this reason, three retention domains should be considered: professional, personal and community.

Nevertheless, some educational interventions are strongly associated with physician attraction and retention in rural and remote areas. These include rurally located medical schools, multifaceted education interventions, continuing medical education and rurally relevant curricula and rural clinical placements^6-11 .

Despite this evidence, the Flexnerian model still largely prevails in schools of medicine¹², and the majority of universities give little or no value to experiences acquired in rural learning environments. Most undergraduate clinical practice still takes place in large cities and in tertiary hospitals, sometimes with outmoded and fragmented curricula. The disparities between the competencies developed during undergraduate training and the needs of the population come, therefore, as no surprise. Hence, a reform is needed that would transform the learning experience by adopting a line of teaching based on competencies with the community in mind. This would lead to a new model of professional with a strong sense of social responsibility^13-15.

There are few data in the literature on the competencies that should be developed by an undergraduate student in a rural medicine program¹⁶. Despite the continental dimensions of Brazil and its enormous cultural diversity, only some medical schools in this country offer students the opportunity to acquire work experience focused on rural or remote areas, and to the best of the authors' knowledge there is no record of any longitudinal program in rural medicine or of an undergraduate competency-based curriculum that corresponds to the needs of rural populations^17-21.

Therefore, the objective of the present study was to elaborate and validate a framework of competencies in rural medicine to be developed as an integrated part of medical training.

Methods

This study applied a modified version of the Delphi technique to enable specialists to reach a consensus on the relevant competencies required for a longitudinal program of rural medicine. Thirty-eight doctors affiliated with the Rural Medicine Working Party Group of the Brazilian Society of Family and Community Medicine were invited to make up a panel of specialists. Data were collected in February and March 2015 at the Faculdade Pernambucana de Saúde (Pernambuco Faculty of Health) using an online questionnaire.

The selection criterion for the inclusion of the physicians in the specialist panel was that they had to be affiliated with the Family Medicine and Rural Community Group of the Brazilian Society of Family and Community Medicine, since this is the only group in Brazil that is recognized and structured for dealing with studies on this subject. The selected physicians were included in the study after agreeing to and signing an informed consent form. At the time of data collection, the Rural Medicine Working Party consisted of 40 doctors distributed throughout the various regions of the country. Fifteen physicians also worked as medical professors or preceptors, three worked as health managers and 22 worked principally in patient care. Two of them were involved with the study and for ethical reasons were not included on the panel of specialists.

For the first step in this study, a structured, five-point Likert-type questionnaire was put together with the following possible options of response: 'should not be included', 'of little importance', 'relatively important', 'very important' and 'indispensable'. This instrument, composed of questions regarding competencies in rural medicine, was elaborated following a literature review. The terms rural health, competence and medical education were used to search for articles in the Medline, CINAHL, Academic Search Alumni Edition, Education Research Complete, Education Resources Information Center (ERIC) and PUBMED databases. Studies dealing with the subject in question were selected and those defining or suggesting the competencies required for working in health care in rural areas were used to construct the first questionnaire. In addition, the online curriculum frameworks of some international institutes²² that included structured competency-based work experience in medicine or rural health and the rural populations health status were assessed, with the objective of identifying the competencies that best meet the needs of this segment of the population.

With the exclusion of competencies referring specifically to settings and situations in other countries that were not relevant to this particular setting, and to focal specialties (ie those restricted to a certain gender, age group, system or organ), the competencies found were used by the principal investigator of this study to construct the questionnaire. The competencies that according to the literature were most relevant, as well as the more general competencies, sometimes subdivided into other competencies, were classified as core competencies. The competencies contained within the core competencies, those with more specific characteristics, were classified as secondary competencies. The resulting instrument consisted of a three-page questionnaire, with space for suggesting new competencies and for comments at the end of each page. Its objective was to serve as a foundation on which to build the framework.

The first Delphi round consisted of sending the instrument to the panel of specialists to allow them to evaluate the relevance of each one of the competencies. In the second round, a new online questionnaire was sent to the members of the Rural Medicine Working Party who had agreed to participate in the study. After examining the results of the previous round, the panelists re-evaluated those competencies for which no consensus had been reached in the first round and evaluated the suggestions that had been made.

Data were collected using an online research tool, Survey Monkey. The study participants were registered in the software program by their email addresses, which were obtained from the website of the Rural Medicine Working Party or by telephone contact. In both rounds, the investigators used the program to send the instrument simultaneously to all the participants, together with a letter inviting them to participate in the study and including instructions on how to complete and return the questionnaire. The invitation letter included a protected link through which the recipient was able to gain access to the informed consent form. After agreeing to the terms of the informed consent form, the participant was then able to access the questionnaire. In both rounds, the participants were given a 15-day period in which to answer the questionnaire, with reminders being sent periodically.

The data were analyzed using spreadsheet software and Epi Info v3.5 (Centers for Disease Control and Prevention; https://wwwn.cdc.gov/epiinfo/html/prevVersion.htm). Descriptive measures such as means, 25th and 75th percentiles, and percentages of relevant (mean rate >3.0) and non-relevant scores were calculated for the responses to each item in the questionnaires. The consensus criterion established for a competency to be included in the framework was it being deemed 'very important' or 'indispensable' by a simple majority of the participants, while the criteria for excluding a competency was that a simple majority of the panel considered that it 'should not be included' or was 'of little importance'. The competencies that failed to reach a consensus were submitted to a second round of the Delphi method to allow the panelists to re-evaluate them, now in the form of dichotomized questions²³. In this way, more competencies could be evaluated in the second round for possible inclusion in the framework.

Ethics approval

The Faculdade Pernambucana de Saúde review board approved the study protocol (no. 14374714.0.0000.5569).

Results

The literature review identified 95 articles related to the topic, 15 of which listed competencies in rural medicine and were used by the principal investigator of this study to construct the first-round questionnaire. This consisted of 26 core competencies and 165 secondary competencies, which, in accordance with the multidimensional aspect of the professional competency, were later grouped into six domains: cognitive, technical, affective, relational, integrative and contextual, as proposed by Epstein and Hundert²⁴. Adhesion to the study was 52% (n=20): 80% (12/15) among medical professors and preceptors, 100% (3/3) among the health managers and 45% (9/20) among those working predominantly in patient care. In the first round of the Delphi technique, consensus was obtained for the 26 core competencies. Of these, 26.9% (n=7) were evaluated by 80% or more of the participants (n=16) as 'indispensable' competencies:

• 'to communicate effectively with the sick person, the community and the healthcare team'
• 'to show empathy for the patient and for the community'
• 'to show a high level of clinical perspicacity in performing diagnoses in a setting in which supplementary tests are rare and access to a specialist is difficult'
• 'to be able to perform necessary procedures in a setting where access to a specialist is difficult'
• 'to make home visits decisively, ethically and compassionately'
• 'to use technology to communicate and exchange information with colleagues in distant locations for the purpose of continued education and specialist orientation'
• 'to schedule patient consultation times with a view to meeting demand effectively and equitably, taking into consideration the particular difficulties and challenges of the geographical, cultural and socioeconomic contexts of rural patients' (Tables 1,2).

Of the secondary competencies, only eight failed to obtain a consensus. The majority of these were related to hospital procedures and transportation (Table 3).

Although various suggestions were made, all of them referred to items that had already been included in the questionnaire and no new competency was suggested. Nevertheless, it was proposed that rural medical training in Brazil should be provided within the specialization program in Family and Community Medicine, since medical training cannot be restricted only to the undergraduate medical course. The need for specialization to enable a doctor to work in primary healthcare was emphasized.

Fifty percent of the physicians (n=10) completed the second-round questionnaire: seven teachers (70%) and three (30%) physicians working principally in patient care. In this round, the specialists evaluated a new instrument containing dichotomized questions on those competencies that had not achieved a consensus in the first round. Of the competencies that failed to achieve a consensus in the first round, 87.5% (n=7) were excluded from the framework in the second round. There was a consensus (100%; n=10) with respect to the proposal made in the first round regarding training in rural medicine. The final framework consisted of 26 core competencies and 158 secondary competencies.

Table 1: Analysis of consensus on the relevance of the core competencies required for an undergraduate course in rural medicine according to domain: technical, cognitive, relational, Recife, Brazil, 2015.




Table 2: Analysis of consensus on the relevance of the core competencies required for an undergraduate course in rural medicine according to domain: affective, integrative, contextual, Recife, Brazil, 2015.




Table 3: Analysis of secondary competencies required for an undergraduate course in rural medicine that failed to achieve a consensus in the first round according to core competency, Recife, Brazil, 2015.

Discussion

In line with the transformation occurring in medical education worldwide, this study constructed and validated a framework of competencies for a longitudinal program of rural medicine, with a view to training physicians who will have a sense of social responsibility and a greater inclination and readiness to meet the needs of rural populations¹³.

The response rate of 52% achieved in the first round was within the expected limits for the Delphi method²⁵. Adherence was found to be greater among the doctors working predominantly in teaching and among the managers, a fact that may reflect the greater motivation of these professionals in relation to the subject matter compared to those working predominantly in patient care. A modified version of the Delphi technique was selected, with the open questions of the classic first round being substituted by a structured questionnaire constructed following a literature review. The objective of using this method was to maximize the response rate in the first round, ensuring the inclusion of important topics that could otherwise have been omitted by the specialists and increasing the likelihood of achieving a consensus in a more effective manner, as already suggested by other authors^26,27. Consequently, two rounds were sufficient for obtaining a consensus, as previously shown in a similar study²⁸. With the Delphi method, it was also possible to connect panelists distributed throughout the various regions of the country. This not only rendered the study viable but also introduced a broader perspective to the discussion regarding the particular competencies required for doctors working in different rural settings.

A study conducted in 2004 in Australia systematized the principal differences between general practitioners working in rural and urban areas through a review of the pertinent literature. It was found that rural physicians, by being isolated geographically, with few available resources and limited access to specialists, needed to be more decisive, self-reliant and independent. They worked longer hours and were often required to have skills used in other specialties such as anesthesia, surgery and emergency medicine in addition to having to deal with hospitalized patients. They needed to be highly skilled with respect to the diagnosis, treatment and clinical management of health problems, and had to know how to behave in an intercultural context²⁹, unlike other doctors working in urban environments, who were more easily able to refer their patients to specialists and had ready access to complementary tests.

The competencies deemed relevant by the specialists reflected the principal needs of rural communities. The seven most relevant competencies, evaluated as indispensable by 80% of the specialists, were related principally to communication, empathy, clinical perspicacity, performing procedures, home visits, continued education and planning.

Users' perceptions in relation to the quality of the healthcare services is strongly associated with tangible elements of the interaction between the healthcare professional and the patient, generally related to aspects of communication and expressions of empathy³⁰. Involvement with the patient through responsive attitudes to his/her concerns increases user satisfaction³¹. Other investigators have emphasized the importance of knowledge on sociohistorical and cultural aspects in improving understanding of the problems experienced by these populations and the direct or indirect effect of these issues on their health conditions³².

The relevance of clinical perspicacity and skills in performing procedures lies in meeting the needs of the individuals living in these areas, consequently resolving their health issues within or close to their communities. Some specialists suggest that centers equipped with transportation would solve these problems. However, this is controversial, with other professionals arguing that this alternative could be applied to more specific problems requiring, for example, specialist surgeries, but not for routine problems³³. For this reason, in a setting in which access to specialist services and supplementary tests is difficult, clinical perspicacity becomes even more important as well as the ability to reach diagnosis based on an accurate analysis of signs and symptoms; perform locoregional anesthesia, resection of lipomas and nevi; and to drain abscesses, etc. There is evidence that students who spend time in rural areas tend to develop better clinical skills^34,35.

The lack of healthcare professionals trained to attend child birth is a major problem in rural areas. The dearth of these professionals increases the number of infants born without adequate care and of women who have to travel to give birth in other municipalities³⁶. A study conducted in Ecuador highlighted gaps in the obstetric training of recent graduates in relation to the needs of rural populations, emphasizing the need to prepare them to attend normal childbirth, to repair perineal lacerations and to perform neonatal resuscitation³⁷, competencies that have also been included in the framework developed in the present study.

Studies indicate that rural doctors carry out more home visits, attend more weekend consultations and call-outs, and cover greater distances than their colleagues working in urban areas³⁸. For these reasons, students have to be trained to carry out home visits effectively. It is important that they know how to manage the time and resources available to allow them to meet demands equitably. The need for students to acquire knowledge on palliative care should also be emphasized, since individuals living in rural areas also report greater difficulty in accessing this type of care, and lack of knowledge in this area is a common complaint of rural doctors who feel unprepared to provide it³⁹.

The use of technology for communication and for exchanging information with colleagues in other locations for the purpose of continued education and matrix development was another competency evaluated as indispensable by the panelists. Technology has become an extremely important tool with which to manage the professional isolation experienced by doctors in rural and remote areas. Telemedicine has brought advances to rural medicine, since it permits distant specialist learning with theoretical material and video classes, discussion forums, interactive consultations and specialist orientation⁴⁰. It is a strategy that has been adopted to supervise students in rural areas⁴¹, and by government agencies to provide support to professionals working in these regions⁴².

The group considered that medical training is not an end in itself and that specialization is required for working in primary care. Therefore, rural doctors in Brazil should complete the training provided by the residency program in family and community medicine. It should be taken into consideration that the group of panelists consisted in its majority of family doctors and that, in accordance with the new national guidelines, recently graduated doctors should be qualified to work at any healthcare level⁴³.

Nevertheless, the consensus of the panelists is in agreement with the idea that primary health care is not synonymous with simple problems, but rather involves complex problems requiring additional training in order to acquire specific skills⁴⁴. Evidence indeed points to the fact that doctors training within the residency program in family and community medicine are those most likely to work with rural and remote populations⁴⁵. In some countries, the team of rural doctors includes other specialties such as surgery and medical emergencies and indeed there is an ongoing debate in the literature regarding whether rural medicine should be a separate discipline²⁹.

Evidence shows the importance of interactions between primary, secondary and tertiary care in enabling students to develop skills in hospital procedures⁴⁶. However, in the present study, the competencies evaluated as irrelevant were mostly related to hospital procedures, reflecting the training of rural doctors in Brazil, which is predominantly related to primary care.

Conclusions

Considering the various possible ways of providing training in rural medicine, it is important to adopt a training model that meets the needs of the community and of the local healthcare system, and that contemplates the full extent of rural practice⁴⁷. The debate on the terminality of the undergraduate medical course is extremely broad-reaching and topical; however, it was not one of the objectives of the present study.

The advances in the field of education in rural medicine as a subspecialty in Brazil are in their infancy, with a competency-based curriculum for a third year in the family and community medicine residency program in rural medicine having only been recently published⁴⁸. Among the competencies taken into consideration is the ability to perform surgeries in emergency situations as an advanced skill, indicating a greater interaction between the rural doctor and the hospital environment, as is the case in other countries.

This study elaborated and validated a framework of competencies for a longitudinal program of rural medicine that consists of 26 core competencies and 158 secondary competencies. This framework should contribute to the construction of competency-based curricula in rural medicine for use in medical training in Brazil.

Conflicts of interest

Two of the authors of this study are members of the Rural Medicine Working Party, but were not included in this research.

References

1. World Health Organization. Increasing access to health workers in remote and rural areas through improved retention. Geneva: World Health Organization, 2010. Available: http://www.ncbi.nlm.nih.gov/books/NBK138618/ (Accessed 19 April 2015).

2. Smith DM. Barriers facing junior doctors in rural practice. Rural and Remote Health (Internet) 2005; 5(4): 348. Available: www.rrh.org.au (Accessed 27 August 2016).

3. Hays RB, Veitch PC, Cheers B, Crossland L. Why doctors leave rural practice. Australian Journal of Rural Health 1997; 5(4): 198-203. http://dx.doi.org/10.1111/j.1440-1584.1997.tb00267.x

4. Amiresmaili M, Khosravi S, Feyzabadi VY. Factors affecting leave out of general practitioners from Rural Family Physician Program: a case of Kerman, Iran. International Journal of Preventive Medicine 2014; 5(10): 1314-1323.

5. Cameron PJ. Professional, personal and community: 3 domains of physician retention in rural communities. Canadian Journal of Rural Medicine 2012; 17(2): 47-55.

6. Dolea C, Stormont L, Braichet JM. Evaluated strategies to increase attraction and retention of health workers in remote and rural areas. Bulletin of the World Health Organization 2010; 88(5), 379-385. http://dx.doi.org/10.2471/BLT.09.070607

7. McGrail MR, Humphreys JS, Joyce CM. Nature of association between rural background and practice location: a comparison of general practitioners and specialists. BMC Health Services Research 2011; 11(1): 63. http://dx.doi.org/10.1186/1472-6963-11-63

8. Quinn KJ, Kane KY, Stevermer JJ, Webb WD, Porter JL, Williamson HA Jr, et al. Influencing residency choice and practice location through a longitudinal rural pipeline program. Academic Medicine 2011; 86(11): 1397-1406. http://dx.doi.org/10.1097/ACM.0b013e318230653f

9. Strasser R, Neusy AJ. Context counts: training health workers in and for rural and remote areas. Bulletin of the World Health Organization 2010; 88(10): 777-782. http://dx.doi.org/10.2471/BLT.09.072462

10. Walker JH, Dewitt DE, Pallant JF, Cunningham CE. Rural origin plus a rural clinical school placement is a significant predictor of medical students' intentions to practice rurally: a multi-university study. Rural and Remote Health (Internet) 2012; 12: 1908. Available: www.rrh.org.au (Accessed 27 August 2016). http://dx.doi.org/10.1186/1478-4491-12-48

11. Isaac V, Watts L, Forster L, McLachlan CS. The influence of rural clinical school experiences on medical students' levels of interest in rural careers. Human Resources for Health 2014; 12: 48.

12. Duffy TP. The Flexner report - 100 years later. The Yale Journal of Biology and Medicine 2011; 84(3): 269-276.

13. Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet 2010; 376(9756): 1923-1958. http://dx.doi.org/10.1016/S0140-6736(10)61854-5

14. Van Schalkwyk SC, Bezuidenhout J, Conradie HH, Fish T, Kok NJ, Van Heerden BH, et al. 'Going rural': driving change through a rural medical education innovation. Rural and Remote Health (Internet) 2014; 14: 2493. Available: www.rrh.org.au (Accessed 27 August, 2016).

15. Nair M, Webster P. Education for health professionals in the emerging market economies: a literature review. Medical Education 2010; 44(9): 856-863. http://dx.doi.org/10.1111/j.1365-2923.2010.03747.x

16. Zimitat C. A curriculum model for rural medicine at the undergraduate level. In: E Bell, C Zimitat, J Merrick (Eds). Rural medical education: practical strategies. New York: Nova Science Publishers, 2011; 57-66.

17. Universidade Federal do Amazonas. Informação sobre o internato rural do curso de medicina. (Internet). Available: http://proeg.ufam.edu.br/attachments/512_Informação%20nº%20058%20Informações%20sobre%20Internato%20Rural%20do%20Curso%20de%20Medicina.pdf (Accessed 27 August 2016).

18. Coelho M, Caldato F. Projeto pedagógico do curso de medicina - CESUPA. (Internet) 2012. Available: http://www.cesupa.br/Graduacao/Biologicas/docs/PROJETO_PEDAGÓGICO_MEDICINA_CESUPA.pdf (Accessed 27 August 2016).

19. Universidade Federal de Alagoas, Faculdade de Medicina. Projeto pedagógico do curso de medicina. (Internet) 2006. Available: http://www.ufal.edu.br/arquivos/prograd/cursos/campus-maceio/ppc-medicina.pdf (Accessed 27 August 2016).

20. Universidade Federal de Tocantins. Manual de Estágio Supervisionado. (Internet). Available: http://download.uft.edu.br/?d=e0ecee8c-3129-4a3f-bd09- d918ebeacbf9:manual_estagio_supervisionado_1568.pdf (Accessed 21 April 2015).

21. Pontifícia Universidade Católica de Goiás. Manual do Internato do Curso de Medicina. (Internet) 2011. Available: http://www.pucgoias.edu.br/ucg/prograd/graduacao/ArquivosUpload/50/file/manuais_2 011/Manual%20do%20m%C3%B3dulo%20Internato%20IX%202011-1.pdf (Accessed 21 April 2015).

22. WA Country Health Service. Rural and remote allied health competencies (STUDENT). (Internet) 2011. Available: http://www.wacountry.health.wa.gov.au/fileadmin/sections/allied_health/WACHS_R_AH_RuralRemoteCompetencies_Pro.pdf (Accessed 27 August 2016).

23. Fried H, Leao AT. Using Delphi technique in a consensual curriculum for periodontics. Journal of Dental Education 2007; 71(11): 1441-1446.

24. Epstein RM, Hundert EM. Defining and assessing professional competence. Journal of the American Medical Association 2002; 287(2): 226-235. http://dx.doi.org/10.1001/jama.287.2.226

25. Wright J, Giovinazzo R. Delphi: uma ferramenta de apoio ao planejamento prospectivo. Caderno de Pesquisas em Administração 2000; 01(12): 54-65.

26. Hsu CC, Sandford BA. Minimizing non-response in the Delphi process: how to respond to non-response. Practical Assessment, Research & Evaluation 2007; 12(17).

27. Custer RL, Scarcella JA, Stewart BR. The modified Delphi technique - a rotational modification. Journal of Vocational and Technical Education (Internet) 1999; 15(2). http://dx.doi.org/10.21061/jcte.v15i2.702

28. Stewart J, O'Halloran C, Harrigan P, Spencer JA, Barton JR, Singleton SJ. Identifying appropriate tasks for the preregistration year: modified Delphi technique. British Medical Journal 1999; 319(7204): 224-229. http://dx.doi.org/10.1136/bmj.319.7204.224

29. Smith J, Hays R. Is rural medicine a separate discipline? Australian Journal of Rural Health 2004; 12(2): 67-72. http://dx.doi.org/10.1111/j.1038-5282.2004.00556.x

30. Biggs J. The reflective institution: assuring and enhancing the quality of teaching and learning. Higher Education 2001; 41: 221-238. http://dx.doi.org/10.1023/A:1004181331049

31. Pollak KI, Alexander SC, Tulsky JA, Lyna P, Coffman CJ, Dolor RJ, et al. Physician empathy and listening: associations with patient satisfaction and autonomy. Journal of the American Board of Family Medicine 2011; 24(6): 665-672. http://dx.doi.org/10.3122/jabfm.2011.06.110025

32. Eiser AR, Ellis G. Viewpoint: cultural competence and the African American experience with health care: the case for specific content in cross-cultural education. Academic Medicine 2007; 82(2): 176-183. http://dx.doi.org/10.1097/ACM.0b013e31802d92ea

33. Chater AB. Supporting rural procedural practice. In AB Chater, J Rourke, ID Couper, et al. (Eds). WONCA Rural Medical Education Guidebook. (Internet) 2014. Available: http://www.globalfamilydoctor.com (Accessed 9 September 2016).

34. Schauer RW, Schieve D. Performance of medical students in a nontraditional rural clinical program, 1998-99 through 2003-04. Academic Medicine 2006; 81(7): 603-607. http://dx.doi.org/10.1097/01.ACM.0000232409.20271.7a

35. Waters B, Hughes J, Forbes K, Wilkinson D. Comparative academic performance of medical students in rural and urban clinical settings. Medical Education 2006; 40(2): 117-120. http://dx.doi.org/10.1111/j.1365-2929.2005.02363.x

36. United Nations. The Millennium Development Goals report 2014. (Internet) 2014. Available: http://www.un.org/millenniumgoals/2014%20MDG%20report/MDG%202014%20English%20web.pdf (Accessed 27 August 2016).

37. Sánchez Del Hierro G, Remmen R, Verhoeven V, Van Royen P, Hendrickx K. Are recent graduates enough prepared to perform obstetric skills in their rural and compulsory year? A study from Ecuador. BMJ Open 2014; 4(7): e005759. http://dx.doi.org/10.1136/bmjopen-2014-005759

38. Weeks WB, Wallace AE. Rural-urban differences in primary care physicians' practice patterns, characteristics, and incomes. Journal of Rural Health 2008; 24(2): 161-170. http://dx.doi.org/10.1111/j.1748-0361.2008.00153.x

39. Robinson CA, Pesut B, Bottorff JL, Mowry A, Broughton S, Fyles G. Rural palliative care: a comprehensive review. Journal of Palliative Medicine 2009; 12(3): 253-258. http://dx.doi.org/10.1089/jpm.2008.0228

40. Smith AC, Bensink M, Armfield N, Stillman J, Caffery L. Telemedicine and rural health care applications. Journal of Postgraduate Medicine 2005; 51(4): 286-293.

41. Wood JA, Miller TW, Hargrove DS. Clinical supervision in rural settings: a telehealth model. Professional Psychology: Research and Practice 2005; 36(2): 173-179. http://dx.doi.org/10.1037/0735-7028.36.2.173

42. Graziplene LR. Creating telemedicine-based medical networks for rural and frontier areas. (Internet) 2009. Available: http://observgo.uquebec.ca/observgo/fichiers/71490_Telemedicine.pdf (Accessed 15 April 2015).

43. Ministério de Saúde. Resolução CNE/CES no. 3, de 20 de junho de 2014. Institui diretrizes curriculares nacionais do curso de graduação em medicina e dá outras providências. (Internet) 2015. Available: http://portal.mec.gov.br/index.php?option=com_content&view=article&id=12812&Ite mid=866 (Accessed 8 April 2015).

44. Anderson MIP, Rodrigues RD. Formação de especialistas em Medicina de Família e Comunidade no Brasil: dilemas e perspectivas. Revista Brasileira de Medicina de Família e Comunidade 2011; 6(18): 19-20. http://dx.doi.org/10.5327/Z1809-59092011001800005

45. Katzenellenbogen J, Durey A, Haigh M, Woods J. Critical success factors for recruiting and retaining health professionals to primary health care in rural and remote locations: contemporary review of the literature. (Internet) 2013. Available: http://www.ruralhealthwest.com.au/docs/publications/rhw_uwa_cucrh-contemporary-review-of-the-literature-f-lr.pdf?sfvrsn=2 (Accessed 27 August 2016).

46. Young L, Larkins SL, Sen Gupta TK, McKenzie SH, Evans RJ, Crowe MJ, et al. Rural general practice placements alignment with the Australian Curriculum Framework for Junior Doctors. Medical Journal of Australia 2013; 199(11): 787-791. http://dx.doi.org/10.5694/mja13.10563

47. Gupta Sen TK, McKenzie A. Postgraduate pathways to rural medical practice. In AB Chater, J Rourke, ID Couper, et al. (Eds). WONCA Rural Medical Education Guidebook. (Internet) 2014. Available: http://www.globalfamilydoctor.com (Accessed 9 September 2016).

48. Sociedade Brasileira de Medicina de Família e Comunidade. Currículo Baseado em Competências para Medicina de Família e Comunidade. (Internet) 2015. Available: http://www.sbmfc.org.br/media/Curriculo Baseado em Competencias(1).pdf (Accessed 8 April 2015).