download PDF
Introduction
Globally, population decline and the concentration of people in urban centers have led to a gradual reduction in medical facilities in small towns, resulting in limited access to timely and appropriate emergency medical services (EMS)1,2. This trend has created a critical need to enhance EMS in underserved regions, where medical resources are often insufficient to meet urgent patient needs3,4.
The disparity in mortality rates among emergency patients between metropolitan and rural areas underscores the issue of rural–urban health inequities. Previous studies have consistently revealed that rural areas, often corresponding to small towns, experience higher mortality rates from major causes such as heart disease, trauma-related injuries, and chronic lower respiratory diseases compared to metropolitan cities5,6. For instance, in the US, rural patients with severe conditions such as acute myocardial infarction and heart failure exhibit higher mortality rates than their urban counterparts7. Additionally, rural patients presenting with common symptoms such as chest pain, nausea, vomiting, and abdominal pain show similarly elevated mortality rates8. This disparity was further exacerbated during the COVID-19 pandemic, as mortality rates surged across both rural and urban hospitals, disproportionately impacting rural areas because of limited medical resources and inadequate healthcare infrastructure9,10. These findings highlight an urgent need for policy interventions and innovative approaches to reduce healthcare inequities between metropolitan and rural areas, ensure equitable access to emergency care, and improve patient outcomes.
In response, South Korea has identified regions with inadequate emergency care environments and designated them as ‘EMS-vulnerable areas’11. These regions are defined as areas where more than 30% of the population cannot reach an emergency medical facility within 30 minutes or a definitive treatment center within 1 hour. To address these challenges, the Korean Government has implemented measures such as deploying emergency medical helicopters, providing personnel support, and allocating operational funds to ensure adequate patient transport systems12.
Recently, the Korean Government developed an information and communication technology-based emergency medical services (ICT-EMS) system specifically targeting North Chungcheong Province, identified as an EMS-vulnerable area13. This innovative system provides continuous support from the moment patients require medical attention until they receive definitive treatment. Its key features include real-time transmission of patient information, triage based on severity, selection of appropriate hospitals, remote medical consultations, coordinated transportation, and the secure sharing of critical patient data. This system was particularly valuable during the COVID-19 pandemic, when a shortage of isolation facilities complicated patient transfers. A preliminary study demonstrated the potential effectiveness of this system among EMS providers14. However, evaluating its effectiveness solely from the perspective of EMS providers may overlook critical aspects of its real-world applicability. Comprehensive assessments incorporating diverse stakeholder perspectives are therefore essential to thoroughly validate the system’s utility.
Therefore, this study aims to comprehensively evaluate the effectiveness and practical implications of the North Chungcheong Province Smart EMS (ICT-EMS) system from the perspective of hospital stakeholders. Specifically, through a structured survey, this study examines the system’s usability, feasibility, acceptability, appropriateness, and user satisfaction.
Methods
Study design and setting
In accordance with the national Emergency Medical Service Act 2015, emergency medical institutions in South Korea are categorized into four levels: regional emergency medical centers, local emergency medical centers, local emergency medical agencies, and emergency medical facilities. North Chungcheong Province reflects this classification and currently operates one regional emergency medical center, six local emergency medical centers, eight local emergency medical agencies, and eight additional emergency medical facilities, totaling 23 emergency institutions. Among them, 18 institutions were included in this study based on the eligibility criterion of having used the ICT-EMS system for more than 6 months since its implementation on 1 June 2023. A survey was conducted among healthcare providers at the participating medical institutions over a 3-week period, from 24 January to 13 February 2024. In-person surveys were administered at nine hospitals in the central region, while online surveys were conducted for the nine hospitals in the northern and southern regions. Online surveys were facilitated by project coordinators at each hospital, who provided instructions for completing the survey. The collected data were analyzed to evaluate the system’s impact.
Participants
A total of 260 healthcare professionals – including physicians, nurses, and emergency medical technicians (EMTs) – participated in this study. Participants were recruited through a purposive, convenience-based sampling approach, targeting individuals with direct experience of using the ICT-EMS system. As participation was voluntary, the number of respondents varied across institutions, depending on staffing levels and availability. To ensure consistent and accurate system use, all participants completed standardized training before the study, covering patient triage, hospital selection, and transportation workflows within the ICT-EMS platform.
ICT-EMS system
North Chungcheong Province is divided into 11 administrative regions, with eight designated as medically underserved in emergency medical care by the government in 2021. Of the 15 emergency medical institutions in the province, nine (60%) are concentrated in the central region, leading to significant disparities in access to emergency care. North Chungcheong Province also has the lowest number of emergency medical specialists in South Korea. As a result, the province experiences high mortality rates for patients with severe emergency conditions, primarily due to geographical barriers and insufficient medical infrastructure. Regions such as Chungju, Jecheon, Danyang, as well as the southern areas such as Boeun, Okcheon, and Yeongdong, are part of different emergency medical zones (Wonju and Daejeon, respectively), where longer travel times reduce the efficiency of EMS and the survival rates for critical patients.
To address these challenges, the government introduced the ICT-EMS system to enhance emergency medical care in North Chungcheong Province, particularly in underserved areas. Key functions of the ICT-EMS include the real-time sharing of patient information between paramedics and healthcare providers, accurate patient classification, hospital selection based on available resources, emergency treatment guidance, transportation coordination, and seamless information sharing between medical institutions. These features are designed to ensure timely and appropriate care for critical patients, even in resource-limited regions. Schematic representations of the system’s core functions are shown in Figures 1 and 2.
Figure 1: The ICT-based emergency medical service system: integrated prehospital and hospital workflow.
Figure 2: The core mobile triage module within the ICT-based emergency medical system.
Evaluation of the ICT-EMS system
Assessment tool
The ICT-EMS system was evaluated using the System Usability Scale (SUS), Feasibility of Intervention Measure (FIM), Acceptability of Intervention Measure (AIM), Appropriateness of Intervention Measure (IAM), and a satisfaction questionnaire. These tools are widely utilized in system evaluation studies14-16. The FIM, AIM, and IAM scales were developed through rigorous psychometric evaluation and have demonstrated strong content and construct validity in multiple implementation science studies13. The SUS, a widely used usability assessment tool, has also been validated across various domains, including healthcare systems15. These validated tools were selected to ensure robust measurement of our evaluation outcomes.
The SUS uses a 5-point Likert scale, with 1 representing ‘strongly disagree’ and 5 representing ‘strongly agree’. It includes 10 questions, five of which (1, 3, 5, 7, 9) are positively worded, while the other five (2, 4, 6, 8, 10) are negatively worded. Scores for each question are determined according to the number of responses for each option. The sum of all responses to the 10 questions is divided by the number of participants to calculate the mean score. For positively worded questions, 1 is subtracted from the mean, while for negatively worded questions, the mean is subtracted from 5 to compute the adjusted score. The sum of these 10 adjusted scores is then multiplied by 2.5 to generate the overall score. This score was used for comparative analyses on participant characteristics17.
The FIM, AIM, IAM, and satisfaction measures were also assessed using a 5-point scale. Each measure included four questions, except for the satisfaction questionnaire, which contained six questions. The FIM assessed the feasibility of ICT-EMS, focusing on factors such as implementability, ease of use, and practicality. The AIM evaluated acceptability by addressing aspects such as approval, appeal, and user preference. The IAM examined the appropriateness of the system in terms of compatibility and suitability for emergency care15. Reliability was determined by calculating Cronbach’s alpha coefficient, which ranged from 0.800 to 0.942, indicating acceptable reliability18. Details of the questionnaire used to evaluate the ICT-EMS system are presented in Table 1.
Table 1: Questions used in the SUS, AIM, FIM, IAM, and satisfaction surveys
| Scale | No. | Question |
Cronbach's alpha |
|---|---|---|---|
| SUS | 1 | I think that I would like to use this ICT-EMS frequently. |
0.800 |
| 2 | I think the ICT-EMS is unnecessarily complex. | ||
| 3 | I thought the ICT-EMS was easy to use. | ||
| 4 | I think that I would need the support of a technical person to be able to use this ICT-EMS. | ||
| 5 | I found the various functions in the ICT-EMS were well integrated. | ||
| 6 | I thought there was too much inconsistency in this ICT-EMS. | ||
| 7 | I believe that most medical staff or users would learn to use this ICT-EMS very quickly. | ||
| 8 | I think the ICT-EMS is very cumbersome to use. | ||
| 9 | I felt very confident using the ICT-EMS. | ||
| 10 |
I needed to learn a lot of things before I could get going with this ICT-EMS. |
||
| FIM | 1 | The ICT-EMS seems implementable. |
0.870 |
| 2 | The application of the ICT-EMS seems possible. | ||
| 3 | It seems that medical staff or users can fully participate in the ICT-EMS. | ||
| 4 | The ICT-EMS seems easy to use. | ||
| AIM | 1 | The ICT-EMS meets my approval. |
0.923 |
| 2 | The ICT-EMS is appealing to me. | ||
| 3 | I like the ICT-EMS. | ||
| 4 | I welcome the ICT-EMS. | ||
| IAM | 1 | The ICT-EMS seems fitting. |
0.942 |
| 2 | The ICT-EMS seems suitable for the Chungbuk region. | ||
| 3 | The ICT-EMS seems applicable. | ||
| 4 | The ICT-EMS seems like a good match with the Chungbuk emergency medical system. | ||
| Satisfaction | 1 | Do you believe that the ICT-EMS provides sufficient necessary information to make decisions about admitting emergency patients? |
0.931 |
| 2 | Did the ICT-EMS program assist in selecting the transfer hospital? | ||
| 3 | Do you think that using an ICT-EMS has an impact on reducing patient acceptance time? | ||
| 4 | Do your colleagues and supervisors recommend the use of ICT-EMS? | ||
| 5 | Are you overall satisfied with the emergency patient transfer system using ICT-EMS? | ||
| 6 | Would you recommend the ICT-EMS to emergency room medical staff in other areas? |
AIM, Acceptability of Intervention Measure. FIM, Feasibility of Intervention Measure. IAM, Intervention Appropriateness Measure. ICT-EMS, information and communication technology within emergency medical services. SUS, System Usability Scale.
Variables
In addition to the evaluation tools, demographic information was collected, including age, sex, education level, professional position (physician, nurse, EMT, other), total job experience at the hospital (in years), time worked in the emergency room (in years and months), region, classification of emergency medical facility, and prior participation in the ICT-EMS pilot study (yes/no). Age groups were categorized as 20–29, 30–39, and 40 years or older. Educational levels included university graduation, master degree, and doctorate. Work experience was grouped into 0–5, 5–10, and more than 10 years for hospital employment, and into 0–2, 2–4, 4–6, and more than 6 years for emergency room experience. Emergency medical institutions were classified as regional emergency medical centers (n=1), local emergency medical centers (n=6), local emergency medical agencies (n=8) or other emergency medical facilities (n=3).
Data analysis
Data analysis was conducted in four stages. First, frequency analysis was performed to describe participant characteristics. Second, Cronbach’s alpha coefficient was calculated to assess the reliability and validity of the scales used. Third, response frequencies and mean scores on the 5-point Likert scale were calculated to evaluate the ICT-EMS system. SUS scores were calculated according to Brooke's scoring system17, with scores ranging from 0 to 100. Scores above 75 were categorized as high, 50–75 as moderate, and below 50 as low19. Mean scores for FIM, AIM, IAM, and satisfaction measures were also calculated. Last, t-tests and analysis of variance (ANOVA) were conducted to determine differences in evaluation results based on participant characteristics. The significance level of 0.05 was applied. Data analysis was performed using Statistical Package for the Social Sciences v22 (IBM Corp; https://www.ibm.com/products/spss-statistics).
Ethics approval
The study was granted an exemption from the requirement for prior consent after an expedited review by the Institutional Review Board of Chungbuk National University Hospital (CBNUH 2020-11-013).
Results
Demographic characteristics of study participants
A total of 305 healthcare providers from medical institutions in the North Chungcheong Province were responsible for patient care using the ICT-EMS system. Of these, 260 participants (85.2%) completed the survey. Among the participants, 153 (58.8%) were females. The median age of the participants was 30 years (interquartile range (IQR) 27–37), with the median years of job experience and time worked in the emergency room being 5 years (IQR 2.1–10.9) and 3 years (IQR 1.5–6.9), respectively. Additionally, 184 participants (70.8%) reported no prior participation in the ICT-EMS pilot project conducted in 2021 (Table 2).
Table 2: Demographic characteristics of study participants
| Characteristic | Variable | n (%)/median (IQR) |
|---|---|---|
| Sex | Male | 107 (41.2) |
| Female |
153 (58.8) |
|
| Age (years) | 30 (27–37) | |
| 20–29 |
100 (38.5) |
|
| 30–39 |
104 (40.0) |
|
| 40–50 |
55 (21.2) |
|
| Unknown |
1 (0.4) |
|
| Education | University graduation | 231 (88.8) |
| Master degree |
21 (8.1) |
|
| Doctorate |
5 (1.9) |
|
| Unknown |
3 (1.2) |
|
| Position | Physician | 20 (7.7) |
| Nurse |
161 (61.9) |
|
| Emergency medical technician |
73 (28.1) |
|
| Other |
6 (2.3) |
|
| Job experience at the hospital (years) | 5 (2.1–10.9) | |
| 0–5 |
123 (47.3) |
|
| 5–10 |
56 (21.5) |
|
| >10 |
79 (30.4) |
|
| Unknown |
2 (0.8) |
|
| Total time worked in emergency room (years) | 3 (1.5–6.9) | |
| 0–2 |
85 (32.7) |
|
| 2–4 |
57 (21.9) |
|
| 4–6 |
37 (14.2) |
|
| >6 |
77 29.6) |
|
| Unknown |
4 (1.6) |
|
| Hospital location | City | 217 (83.5) |
| County |
43 (16.5) |
|
| Hospital classification | Regional emergency medical center | 8 (3.1) |
| Local emergency medical center |
157 (60.4) |
|
| Local emergency medical institution |
66 (25.4) |
|
| Other emergency medical facilities |
29 (11.2) |
|
| Participation in ICT-EMS pilot project | Yes | 69 (26.5) |
| No |
184 (70.8) |
|
| Unknown |
7 (2.7) |
EMT, emergency medical technician. ICT-EMS, information and communication technology within emergency medical services. IQR, interquartile range.
Usability evaluation
The usability of ICT-EMS system, as measured by the SUS, yielded an overall mean score of 52.6 (standard deviation (SD) 13.7), indicating a moderate level of usability (Table 3). Participants identified specific challenges with the system, as reflected in low converted scores for areas such as system consistency (1.9), the need for technical support (1.6), and user confidence (1.8). These findings suggest that users perceived the system as complex and occasionally difficult to operate. However, positive aspects were noted, with neutral to slightly positive converted scores ranging from 1.8 to 2.5, highlighting areas of moderate user satisfaction.
Further analysis of SUS scores by participant characteristics revealed that the usability ratings were higher among males, participants aged 40 years and older, those with doctoral degrees, physicians, participants with 10 years of hospital experience, those with less than 2 years of emergency room experience, country hospital staff at local emergency medical agencies, and non-participants of the ICT-EMS pilot project. Statistically significant differences in usability scores were observed based on age and occupation (p<0.05) (Table 4).
Table 3: SUS scores of study participants
| SUS | Score† (mean±SD) | Converted¶ score |
|---|---|---|
| 1. I think that I would like to use this ICT-EMS frequently. | 3.2±1.0 | 2.2 |
| 2. I think the ICT-EMS is unnecessarily complex. | 2.8±0.9 | 2.2 |
| 3. I thought the ICT-EMS was easy to use. | 3.4±0.9 | 2.4 |
|
4. I think that I would need the support of a technical person to be able to use this ICT-EMS. |
3.4±1.0 | 1.6 |
| 5. I found the various functions in the ICT-EMS were well integrated. | 3.0±0.9 | 2.0 |
| 6. I thought there was too much inconsistency in this ICT-EMS. | 3.1±1.0 | 1.9 |
| 7. I believe that most medical staff or users would learn to use this ICT-EMS very quickly. | 3.5±0.8 | 2.5 |
| 8. I think the ICT-EMS is very cumbersome to use. | 3.0±1.0 | 2.0 |
| 9. I felt very confident using the ICT-EMS. | 2.8±0.9 | 1.8 |
| 10. I needed to learn a lot of things before I could get going with this ICT-EMS. | 2.8±0.9 | 2.2 |
| Total converted scores x 2.5 (overall SUS score) | N/A | 52.6 |
† 1, strongly disagree; 2, somewhat disagree; 3, neutral or no opinion; 4, somewhat agree; 5, strongly agree.
¶ For items 1, 3, 5, 7, and 9, the converted score is the mean score minus 1. For items 2, 4, 6, 8, and 10, the converted score is 5 minus the mean score.
ICT-EMS, information and communication technology within emergency medical services. N/A, not applicable. SD, standard deviation. SUS, System Usability Scale.
Table 4: Overall SUS score by selected participant characteristics
| Characteristic | Variable† | n | Mean±SD | p-value |
|---|---|---|---|---|
| Sex | Male | 107 | 53.7±13.2 | 0.26 |
| Female |
153 |
51.8±14.0 | ||
| Age (years) | Less than median (<30) | 118 | 52.3±13.8 | 0.76 |
| Equal to or greater than median (≥30) |
141 |
52.9±13.7 | ||
| 20–29 |
100 |
52.6±14.5 | 0.03* | |
| 30–39 |
104 |
50.6±2.4 | ||
| 40–50 |
55 |
56.5±14.0 | ||
| Education | University graduation | 231 | 52.6±13.8 | 0.37 |
| Master degree |
21 |
50.4±14.0 | ||
| Doctorate degree |
5 |
60.0±13.0 | ||
| Occupation | Physician | 20 | 54.4±15.0 | 0.04* |
| Nurse |
161 |
52.0±12.6 | ||
| Emergency medical technician |
73 |
53.2±16.1 | ||
| Other |
6 |
54.6±5.6 | ||
| Job experience at the hospital (years) | Less than median (<5) | 123 | 52.9±13.1 | 0.65 |
| Equal to or greater than median (≥5) |
135 |
52.1±14.2 | ||
| 0–5 |
123 |
52.9±13.1 | 0.15 | |
| 5–10 |
56 |
49.5±13.3 | ||
| >10 |
79 |
54.0±14.6 | ||
| Total time worked in emergency room (years) | Less than median (<3) | 119 | 53.4±13.5 | 0.26 |
| Equal to or greater than median (≥3) |
137 |
51.5±13.9 | ||
| 0–2 |
85 |
54.6±13.4 | 0.19 | |
| 2–4 |
57 |
49.8±13.2 | ||
| 4–6 |
37 |
53.1±13.2 | ||
| >6 |
77 |
51.5±14.4 | ||
| Hospital location | City | 217 | 52.2±14.0 | 0.61 |
| County |
43 |
53.6±11.0 | ||
| Hospital classification | Regional emergency medical center | 8 | 42.8±11.4 | 0.16 |
| Local emergency medical center |
157 |
52.3±13.8 | ||
| Local emergency medical agency |
66 |
54.3±11.9 | ||
| Other emergency medical facilities |
29 |
52.8±16.7 | ||
| Participation in ICT-EMS pilot project | Yes | 69 | 51.2±13.7 | 0.25 |
| No |
184 |
53.4±13.5 |
*p<0.05, **p<0.01, ***p<0.001.
† The total number of cases for the variables within each characteristic may not sum to 260 due to missing or ‘prefer not to answer’ responses from the statistical analyses.
ICT-EMS, information and communication technology within emergency medical services. SD, standard deviation. SUS, System Usability Scale.
Feasibility, acceptability, appropriateness, and satisfaction evaluation
The evaluation of the ICT-EMS feasibility, acceptability, appropriateness, and user satisfaction is summarized in Figure 3 and Table 5. For feasibility, 10.5% of participants rated the system negatively (strongly disagree or disagree), 32.1% provided a neutral rating, and 57.4% rated it positively (agree or strongly agree). Similarly, acceptability received negative ratings from 18.9% of participants, moderate ratings from 40.6%, and positive ratings from 40.4%. Appropriateness ratings showed a similar distribution, with 17.6% negative, 39.3% moderate, and 43.1% positive feedback. Satisfaction ratings were less favorable, with 27.0% of participants rating the system negatively, 36.0% moderately, and 37.0% positively.
When examining feasibility, acceptability, appropriateness, and satisfaction scores by participant characteristics, feasibility scores were higher among males, participants aged 40 years and older, those with doctorate, physicians, participants with more than 5 years of experience, those with less than 2 years of emergency room experience, county hospital staff, those working at local emergency medical agencies, and non-participants of the ICT-EMS pilot project. A statistically significant difference in feasibility scores was observed based on hospital classification (p<0.05). Similar trends were seen for acceptability and appropriateness, with appropriateness scores showing significant differences by hospital classification (p<0.05). Satisfaction scores followed a similar trend but were notably higher among EMTs and city hospital staff (Table 6).
Table 5: FIM, AIM, IAM, and satisfaction scores of study participants
| Scale | No. | Question | Mean±SD | Overall |
|---|---|---|---|---|
| FIM | 1 |
The ICT-EMS seems implementable. |
3.5±0.9 |
3.5±0.8 |
| 2 |
The application of the ICT-EMS seems possible. |
3.4±0.9 |
||
| 3 |
It seems that medical staff or users can fully participate in the ICT-EMS. |
3.6±0.8 |
||
| 4 | The ICT-EMS seems easy to use. |
3.6±0.7 |
||
| AIM | 1 |
The ICT-EMS meets my approval. |
3.4±0.9 |
3.2±0.9 |
| 2 | The ICT-EMS is appealing to me. |
3.4±0.9 |
||
| 3 | I like the ICT-EMS. |
3.0±0.9 |
||
| 4 | I welcome the ICT-EMS. |
3.1±1.0 |
||
| IAM | 1 |
The ICT-EMS seems fitting. |
3.3±0.9 |
3.3±1.0 |
| 2 | The ICT-EMS seems suitable for the Chungbuk region. |
3.2±1.0 |
||
| 3 | The ICT-EMS seems applicable. |
3.4±0.9 |
||
| 4 | The ICT-EMS seems like a good match with the Chungbuk emergency medical system. |
3.3±0.9 |
||
| Satisfaction | 1 | Do you believe that the ICT-EMS provides sufficient necessary information to make decisions about admitting emergency patients? |
2.7±1.1 |
3.1±1.1 |
| 2 | Did the ICT-EMS program assist in selecting the transfer hospital? |
3.1±1.0 |
||
| 3 | Do you think that using an ICT-EMS has an impact on reducing patient acceptance time? |
3.2±1.0 |
||
| 4 |
Do your colleagues and supervisors recommend the use of the ICT-EMS? |
3.2±1.0 |
||
| 5 | Are you overall satisfied with the emergency patient transfer system using ICT-EMS? |
3.1±1.0 |
||
| 6 | Would you recommend the ICT-EMS to emergency room medical staff in other areas? |
3.2±1.1 |
AIM, Acceptability of Intervention Measure. FIM, Feasibility of Intervention Measure. IAM, Intervention Appropriateness Measure. ICT-EMS, information and communication technology within emergency medical services. SD, standard deviation.
Table 6: Feasibility, acceptability, appropriateness, and satisfaction evaluation by selected participant characteristics
| Characteristic | Variable† | n | FIM | p-value | AIM | p-value | IAM | p-value | Sat | p-value |
|---|---|---|---|---|---|---|---|---|---|---|
| Sex | Male | 107 | 3.6 | 0.14 | 3.3 | 0.33 | 3.3 | 0.91 | 3.1 | 0.75 |
| Female |
153 |
3.5 | 3.2 | 3.3 | 3.0 | |||||
| Age (years) | Less than median (<30) | 118 | 3.6 | 0.64 | 3.2 | 0.90 | 3.3 | 0.84 | 3.1 | 0.88 |
| Equal to or greater than median (≥30) |
141 |
3.5 | 3.3 | 3.3 | 3.1 | |||||
| 20–29 |
100 |
3.6 | 0.12 | 3.2 | 0.08 | 3.3 | 0.22 | 3.1 | 0.28 | |
| 30–39 |
104 |
3.4 | 3.1 | 3.2 | 3.0 | |||||
| 40–50 |
55 |
3.7 | 3.5 | 3.4 | 3.2 | |||||
| Education | University graduate | 231 | 3.5 | 0.43 | 3.2 | 0.09 | 3.3 | 0.41 | 3.1 | 0.54 |
| Master degree |
21 |
3.3 | 3.1 | 3.2 | 2.9 | |||||
| Doctorate |
5 |
3.7 | 4.0 | 3.8 | 3.3 | |||||
| Occupation | Physician | 20 | 3.6 | 0.37 | 3.3 | 0.94 | 3.4 | 0.52 | 3.1 | 0.54 |
| Nurse |
161 |
3.5 | 3.2 | 3.2 | 3.0 | |||||
| Emergency medical technician |
73 |
3.6 | 3.3 | 3.4 | 3.1 | |||||
| Other |
6 |
3.9 | 3.3 | 3.5 | 3.4 | |||||
| Job experience at the hospital (years) | Less than median (<5) | 123 | 3.5 | 0.89 | 3.3 | 0.64 | 3.3 | 0.78 | 3.1 | 0.70 |
| Equal to or greater than median (≥5) |
135 |
3.5 | 3.2 | 3.3 | 3.0 | |||||
| 0–5 |
123 |
3.5 | 0.95 | 3.3 | 0.49 | 3.3 | 0.76 | 3.1 | 0.35 | |
| 5–10 |
56 |
3.5 | 3.1 | 3.2 | 2.9 | |||||
| >10 |
79 |
3.5 | 3.3 | 3.3 | 3.1 | |||||
| Total time worked in emergency room (years) | Less than median (<3) | 119 | 3.6 | 0.43 | 3.3 | 0.19 | 3.4 | 0.18 | 3.1 | 0.37 |
| Equal to or greater than median (≥3) |
137 |
3.5 | 3.2 | 3.2 | 3.0 | |||||
| 0–2 |
85 |
3.6 | 0.46 | 3.4 | 0.12 | 3.4 | 0.21 | 3.2 | 0.14 | |
| 2–4 |
57 |
3.5 | 3.1 | 3.2 | 2.9 | |||||
| 4–6 |
37 |
3.6 | 3.2 | 3.3 | 3.1 | |||||
| >6 |
77 |
3.5 | 3.2 | 3.1 | 2.9 | |||||
| Hospital location | City | 217 | 3.5 | 0.10 | 3.2 | 0.77 | 3.3 | 0.73 | 3.1 | 0.95 |
| County |
43 |
3.7 | 3.3 | 3.3 | 3.0 | |||||
| Hospital classification | Regional emergency medical center | 8 | 2.8 | 0.01 | 3.1 | 0.73 | 2.5 | 0.03 | 2.4 | 0.08 |
| Local emergency medical center |
157 |
3.5 | 3.2 | 3.2 | 3.0 | |||||
| Local emergency medical agency |
66 |
3.7 | 3.2 | 3.3 | 3.1 | |||||
| Other emergency medical facilities |
29 |
3.7 | 3.4 | 3.5 | 3.3 | |||||
| Participation in ICT-EMS pilot project | Yes | 69 | 3.5 | 0.57 | 3.2 | 0.19 | 3.2 | 0.29 | 2.9 | 0.07 |
| No |
184 |
3.6 | 3.3 | 3.3 | 3.1 |
† The total number of cases for the variables within each characteristic may not sum to 260 due to missing or ‘ prefer not to answer’ responses from the statistical analyses.
AIM, Acceptability of Intervention Measure. FIM, Feasibility of Intervention Measure. IAM, Intervention Appropriateness Measure. ICT-EMS, information and communication technology within emergency medical services. Sat, satisfaction.
Figure 3: Evaluation of feasibility, acceptability, appropriateness, and satisfaction regarding the ICT-based EMS system.
Discussion
This study evaluated the usability, feasibility, acceptability, appropriateness, and satisfaction of the ICT-EMS system among hospital healthcare providers in rural areas underserved by emergency medical services. Despite a moderate overall usability score of 52.6, significant concerns persist, particularly regarding system inconsistency and the need for technical support, which undermines user confidence. These challenges indicate that while some aspects of the system are positively perceived, considerable usability issues remain. Notably, demographic factors such as age and occupation significantly influenced usability perceptions, with older participants and physicians responding more favorably. These insights underline the need for targeted improvements to enhance system usability and confidence, especially among younger and less experienced healthcare providers.
Overall feasibility, acceptability, appropriateness, and satisfaction scores were above average. However, satisfaction related to the system’s provision of adequate information for patient admission decisions was notably low, with more than 43.5% of participants expressing dissatisfaction. This underscores the critical importance of effective communication during patient handovers, a factor emphasized by both the World Health Organization and the Joint Commission for its role in preventing patient harm20. Miscommunication continues to be a significant challenge in the integration of prehospital EMS and hospital emergency department (ED), often leading to preventable patient harm20.
The ICT-EMS system aims to facilitate better communication between EMS providers and ED staff, ensuring necessary resources are prepared on patient arrival. However, improvements are crucial to ensure the system adequately supports information sharing without burdening EMS providers. Implementing new health technologies in high-stakes, urgent environments such as EMS can be perceived as an additional load, potentially hindering adoption21,22 Timely and effective communication between prehospital EMS and hospital ED providers is recognized as a critical step for time-sensitive emergency care22. However, challenges in information sharing between EMS providers and receiving ED staff persist. Reddy et al23 identified three major barriers to coordination between ED and EMS teams: ineffectiveness of current information and communication technologies, a lack of common ground, and breakdowns in information flow. The dynamic nature of prehospital patient transport often leads to incomplete or inaccurate data being provided to the receiving hospital24.
The findings of the current study indicate that regional emergency centers, often burdened with high patient volumes and intense workloads, showed lower scores across all metrics than other emergency medical agencies. This may be attributed to the intense pressure and constant demand in these settings, which probably make it more challenging for providers to integrate and adapt to new technologies. The high-stress environment may also limit the time and resources available for thorough training and familiarization with the ICT-EMS system, leading to lower perceptions of its usability and overall satisfaction.
During the pilot phase, a significantly low overall SUS score of 35.6 by EMS providers, which indicated a negative perception of the ICT-EMS system14, was reported. Although the current study shows improved scores across all metrics compared to the pilot phase, the challenges faced by EMS providers during prehospital care remain evident. EMS providers must manage patient information while simultaneously delivering care in high-pressure settings, which may explain their lower scores. This highlights the need for simplified system interfaces and features that minimize the burden on EMS providers.
The diverse participant pool from EDs, encompassing a broad spectrum of professional roles and experience levels, strengthens the generalizability of the findings and provides a comprehensive view of the ICT-EMS system’s impact. However, this study has limitations. First, the relatively short duration of ICT-EMS use among participants may have affected their adaptation and evaluation of the system. Second, while this study focused on hospital healthcare providers, EMS providers were only assessed during the pilot study phase, limiting direct comparisons. Future research should include a broader range of EMS providers to further evaluate and enhance the system’s effectiveness.
Conclusion
This study highlights the moderate usability and generally positive perceptions of the ICT-EMS system among hospital healthcare providers in an EMS-vulnerable region. Importantly, our findings demonstrate the system’s impact on facilitating communication between EMS providers and hospitals, improving hospital selection processes, and supporting clinical decision-making during emergency admissions. However, the study also identified critical limitations, particularly in providing sufficient information to support patient admission decisions.
The ICT-EMS system has had a measurable impact on enhancing coordination, reducing delays in patient transfer, and streamlining emergency workflows in underserved settings. To maximize its impact on patient outcomes, particularly for time-sensitive conditions, continuous system evaluation and targeted refinements – such as improving usability and information clarity – are imperative. The successful integration of such ICT solutions can significantly contribute to improving the efficiency and equity of EMSs in resource-limited regions
Funding
This work was supported by the Korea Ministry of Land, Infrastructure and Transport as part of the Smart City Challenge Main Project (2022–2024).
Conflicts of interest
The authors declare no conflicts of interest.
Acknowledgements
This study was supported by the Ministry of Land, Infrastructure and Transport of the Republic of Korea.