Economic evaluation and analyses of hospital-based electronic medical records (EMRs): a scoping review of international literature

In this paper, we have reported findings from the first comprehensive scoping review examining the economic impacts of EMR implementation in the hospital setting using CBAs. We wanted to first understand the types of CBAs used to evaluate EMR implementation; second, to determine how comprehensive the analyses were, and how the benefit and cost items were identified, measured and valued; and third, to determine whether the literature found that the benefits of EMR implementation justify the investment costs.

The review showed that despite a relatively rich ten-year literature, only three studies^(ID#1–#3) qualified as CBA studies, the rest focused on impact evaluation, cost/financial analysis, benchmarking (e.g., efficiency analysis) or evaluation of interventions that were EMR-powered^{(ID#13–14,#17)}. More than two-thirds of the studies originated from the United States, following the Obama Care boost, and in favor of statistical analysis approaches rather than CBA. Only three studies originated in Europe, while none was from the United Kingdom. Of the three studies conducted in Europe, one of them was a CBA, despite its narrowed scope (impacts of hospital EMR on research and development). The other two CBA studies, in China and Korea, investigated a wider scope of impacts; however, the measurement and valuation of impacts were limited to financial impacts or cost savings from reduced adverse events. Patient outcomes and experience, changes in healthcare workforce as the results of EMR implementation were either not measured or valued. These limitations were acknowledged in all studies.

While most studies pointed toward a positive impact of EMR implementation, such impact is not quantitatively comparable across studies (as summarized and described in Tables 2 and 3). This is attributable to three main reasons: heterogeneity of impact measures (both benefits and costs), different implementation conditions (despite all in the hospital setting), and a large variety of analytical methods employed to understand the economic value of EMR, thus biases associated with those methods. As EMR adoption continues to spread, the reproducibility of findings that allow comparative analyses across settings and healthcare systems is critical to know what works and what does not, how, and why.

First, there were a wide range of analysis units, measures and indicators used to report on the impacts of EMR implementation, from patient to health system levels (as summarized in Tables 2 and 3). Comparable analyses will require increasing standardization of measures of impacts at all levels. Standardization of measures is particularly important as uncertainty existed within the literature as to what mechanisms EMR acts on to create better safety outcomes for patients. Encinosa et al. (2012), gave an analogy of EMR acting in the role of a “car airbag” that mitigates damage once an adverse event has occurred, and the CPOE systems act similarly to the role of a car’s “electronic stability controls” in reducing complications^(ID#10). While there is substantial evidence that an effective EMR system allows for reductions of medical errors and waste, the causal links from EMR implementation to ultimate patient outcomes such as reduced mortality rate and shorter length of stay are less well understood in the literature. It is proposed that causal links might be better understood if studies can use direct and more precise health outcomes such as vital signs (e.g., body temperature, pulse, and respiration rates) to mediate the impact of EMR on ultimate outcomes (e.g., mortality rates). In any evaluation (especially economic evaluation), understanding the direction and magnitude of change (quantitative) can be equally important as the mechanism of change (e.g., qualitative through understanding the theory of change). The quantitative measures, when measured with standardized instruments across time and implementation units, can highlight the success/failure for lessons learnt and/or further investigations. Additionally, due to current misalignment between evaluation focuses and healthcare aims, the impacts of EMR on quality of care, patient experience, and the healthcare workforce were largely omitted (see Table 3, and further discussion below).

Second, the review identified a very large degree of heterogeneity of implementation conditions and types of EMR systems within the hospital setting. Variance in the latter is especially prominent in studies using large-scale, hospital and panel data. Due to the rapid development of EMR, later adopters were likely to use more sophisticated technologies compared with early adopters, allowing for better capture of the impacts. Variance in the former (across settings/adoption situations) was also noted in the literature, mostly related to the geographical area of adoption. For instance, Dranove et al. (2014), demonstrated that the benefits accrued to EMR were reliant on the access to and use of ICT within the local community^(ID#6). On the other hand, developing nations and low-efficiency hospitals were shown to benefit from EMR implementation, despite their relatively low level of ICT (see further discussion in theme 6). These seemingly contradicting findings suggest that the large heterogeneity in impact measurement makes it challenging to compare all the results across all the study settings.

Third, studies employed a wide range of economic and statistical methods to understand the value of EMR implementation. These methods ranged from economic evaluation methods such as CBAs^(ID#1–3), cost-effectiveness analyses^{(ID#11–12,#15)}, and efficiency and productivity analyses^{(ID#9,#18,#27)} to econometric models such as difference-in-difference^(ID#24), a system of cost equations^(ID#5), or statistical analysis such as multivariate regressions^{(ID#6,#11,#12,#13,#16,#19,#20,#21)}, interrupted time series^(ID#4,#26), generalized linear models^(ID#10), and logistic regressions^(ID#18). These models included different sets of variables and measures, many of which were specific to the datasets used or the data-collection context. All studies acknowledged methodology limitations and the potential biases (on the impact estimates of EMR implementation) resulting from the modeling and data limitations. The main sources of biases include self-selection bias of hospitals implementing EMR, self-reporting of EMR usage, and “conservative” estimates of benefits.

Self-selection bias was often examined in studies that utilized longitudinal data, in which investigators attempted to control for the self-selection endogeneity through fixed-effect models. There was a wide range of hypotheses regarding the correlates of EMR adoption in hospitals. Zhivian et al. (2012) proposed that inefficient hospitals were more likely to adopt EMR as the potential improvements were greater than in hospitals that were already more efficient^(ID#27). Other studies linked EMR adoption with teaching hospital status^(ID#18) or whether the hospital belonged to a community fund-raising scheme^(ID#20). As such, it is unclear if the endogeneity would lead to the under- or over-estimation of the EMR impacts, resulting to limited ability to interpolate EMR as a causation for the measured outcomes^(ID#9).

EMR use was commonly self-reported both by clinicians and by the organizations collectively (i.e., hospitals)^(ID#16). Self-reported savings would inevitably lead to large uncertainty around the estimation of both EMR-related benefits and costs due to self-selection bias. Clinicians who engaged in cost-saving activities could potentially overlook other benefits or costs if they were not directly or clearly related to the cost-saving targets at hand. Conversely, it was believed that hospital response rates were dependent on the successes of EMR, leading to a potential overestimation of EMR benefits.

Many studies stated that they relied on “conservative estimates” when evaluating the benefits of EMR^{(ID#1–2,#7,#8)}. Conservative estimates were applied to the loss of productivity following the initial implementation of the EMR system, in two of the CBAs, with predictions for productivity loss higher than expected^(ID#1–2). Similarly, studies also used “conservative estimates” in deciding on the counterfactual with costs associated with the counterfactual underestimated to conservatively measure the financial impact from EMR implementation^(ID#7–8).

Outside the three main questions of the scoping review, we discovered three additional important topics that warrant future research (see Fig. 3). First, the maturity of EMR implementation in the past two decades and how it would affect future evaluation of EMR. Second, the evidence of economies of scale and scope with respect to the impacts of EMR implementation. Third, the (incomplete) alignment of selected evaluation approach/framework with the quadruple aims of healthcare delivery.

Emerging themes from the literature of economic evaluation and analysis of hospital-based electronic medical records.

Maturity of EMR implementation

The literature overwhelmingly indicates that most EMR systems were still maturing^15,16,17. The full extent of the costs and benefits of EMR implementation could not be understood, measured, and valued using the current metrics or data-collection systems, both were outdated. While publication dates were within the last 11 years (since 2010), most data used for the economic evaluation came from the prior decades (late 1990s–2010).

Despite its relatively short time, a trend has emerged, that is, increasing benefits over time were observed as EMRs slowly matured. Initial studies with data sourced during the late 1990s found that the promised cost-savings from an administration perspective were not delivered through EMR implementation^(ID#6,#9). However, later evaluations that accounted for improvement in care quality, using evidence sourced from the 2000s, indicated that EMR provided quality increases at a competitive cost^{(ID#14,#25–26)}. In other words, the most recent literature found that EMR is cost-effective. Last, more recent CBAs, sourcing data from the period 2006–2012, indicated that the EMR implementation has become a solution that is financially viable, even when studies did not fully account for improved patient outcomes^(ID#1-3).

This change in viability can be attributed to five main factors. First, the capabilities of EMR technology have vastly developed throughout this timeframe. This is perhaps best reflected in the changing role of EMR. Initially, the system was developed and utilized as a system to keep track of resource utilization, costs, and outcomes^(ID#6); however, improvements in technology have enabled EMR to provide clinical support and hence provide greater utility to hospitals^(ID#22). In parallel, the system has been refined over time to improve the cost-saving utilization. ^(ID#4,#13). Second, clinicians have become more accepting of EMR implementation over time. Clinician uptake and technical ability have been marked as a key determinant in the success of EMRs. Third, the role of network effects has led to increasing benefits and decreasing costs associated with EMR^(ID#5) Fourth, there is a potential that EMR can contribute to developing improved procedures and workflow through increasing the amount of information available. Furthermore, there is the potential of EMR to be used as the building block for improving medical technologies through the emergence of predictive and prescriptive analytics driven by machine learning and artificial intelligent developments.

In addition to the natural progression of EMR maturity, developments have been made in the systems to support the EMR implementation, which together has led to increasing benefits over time. In the literature, this was often discussed as “workforce training” and “development of ICT services”. Although the perceptions of physicians towards the benefits of EMR remain widely predetermined¹⁸, larger productivity gains can be made through EMR, especially when physicians are incentivized by sharing in the benefits from the gains^(ID#18) However, EMR implementation has been consistently associated with an implementation lag where disbenefits (e.g., lower productivity) are accrued during the training phase^(ID#2,#6). The duration of the disbenefits in hospital efficiency studies has been linked with the level of general ICT adoption within a region, suggesting that the prevalence of complementary ICT skills within the community acts to both minimize system costs, and to increase benefits. As such, an increasing use of ICT systems in all operating aspects of society in the future is expected to further benefit the EMR systems.

Returns to scale and scope

Another common theme relating to returns to scale and scope is that present EMR technologies offer highly effective solutions for simple medical problems but show decreasing effectiveness for highly complex problems. In economic terms, studies found strong evidence of increasing returns to scale and scope of the EMR implementation in small and simple hospital systems but decreasing returns (approaching negative) in large and complex systems. The evidence was visible at both national health system and within the context of a hospital. For instance, great success has been observed in the implementation of the EMR in the Kumuzu Central hospital in Malawi^(ID#7). In these systems, with their relatively low stock of ICT capital and labor available, the marginal returns (e.g., improved patient processing and treatment efficiency) from the EMR adoption were significant. Similarly, a US study noted a phenomenon that EMR applications showed their highest effectiveness in the scenario of noncomplex cases^(ID#22).

The mechanism driving increasing returns to scale and scope for simple and small medical problems or systems is intuitive. For instance, if all patient information is well captured and linked in a unified EMR system (between departments within a hospital, or across hospitals within a health district/region), when a particular patient repeatedly presents at different locations, it requires less time spent on collecting past medical history and relevant personal information—examples include allergies, comorbidities and complexities. This allows faster and more effective treatment decisions while accruing much lower cost of acquiring patient information (i.e., cost of both medical staff and patient’s time and effort). However, the mechanism that drives decreasing (even negative) returns scale and scope, i.e., EMR showing minimal additional effectiveness in complex cases, is less certain. A common explanation in the literature centers around the lack of precision measures of outcomes in complex situations, such as the contribution of EMR implementation on mortality rate and length of stay^(ID#21–22). For example, medication errors that can be effectively reduced by an EMR system might contribute heavily to the mortality risk of patient of simple casemix; however, a patient with a complex casemix might have very high mortality risk, regardless of whether or not medication errors occur.

The alignment of EMR evaluation with the quadruple aims of healthcare delivery

Our review found that the current literature of economic evaluation of EMR did not sufficiently address the quadruple aim of healthcare delivery, that is, improve population health, enhance patient experience, reduce cost per patient, and improve work-life balance of the healthcare workforce¹¹.

Of the 28 studies included in the scoping review, no study explicitly addressed all aspects of the quadruple aim of health care, particularly the impacts of EMR implementation on the healthcare workforce (see Table 3). Considering the high levels of clinician burnout partially attributed to EMR often discussed in noneconomic studies, a gap in the economic literature exists: that is, quantifying the impact of EMR on the work-life balance of the healthcare workforce. Another important aim of healthcare delivery—patient experience—was addressed in only one study. Hydari et al. (2019)^(ID#24) observed qualitative benefits in “increased confidence in the health system” (by patients) after EMR adoption, no studies contemplated the patient experience beyond health outcomes or quality-adjusted life years (through the QALY estimates).

A potential explanation for this relatively narrow focus of EMR evaluation to date is the maturity of digital health across settings and countries. The economic development and project-management literatures have shown that piece-meal project implementations, even with best intentions in mind, but without proper alignment to overall goals and purposes of the sector, can result in high wastage and undesirable impacts, especially when projects compete for the same labor and capital resources to produce the same outputs^19,20.

Project implementations face constraints of time, budget (allocated funding and inputs), and set target outputs/results to achieve (number of staff trained, number of machines installed, or go live dates, etc.). If their success is defined by a narrow set of input–output metrics, rather than measures aligned with sector purposes (i.e., who will benefit, what will change systematically as a result of the project), and goals (i.e., broader sector goals of healthier population, improved patient and staff experience, and reduced cost per capita), many projects might achieve their targeted results (outputs) without contributing to achieving sector purposes and goals. Therefore, project design and implementation without proper attention to purposes and goals (and the verification indicators that allow them to measure the contribution of the project outputs to changes and impacts at a higher level) tend to fall into the situation of “crowding out”, that is, multiple projects competing for already-scarce resources (i.e., labor/workforce time, existing facility, hardware system, and so on). Such phenomenon of micro–macro paradox has been observed and studied in many economic sectors, including health²¹. That is, individual projects showing success: outputs produced as planned within the budget and timeline—but overall, there are no substantial changes toward the desired outcome—such as population health improvement, improved patient experience, etc.

It has been shown that using a systematic approach to project-cycle management that is tightly linked to sector purposes and goals and within sector-wide approach framework can result in desirable outcomes in the medium- and long term (i.e., sustainable)^{19,20,22,23,24}. While the sector-wide approach was developed specifically in relation to development practices, the concept (and its implementation tools) is translatable to other settings/practices. That is, all stakeholders within a sector (funders, lenders, providers, government agencies, and users) implement projects that collectively contribute to achieve the desirable sector goals and purposes, in order to harmonize the efforts, avoid overlapping and wastage, and more importantly magnify each other’s outputs or results. In the case of digital health, it is essential that the EMR implementation, within an organization or settings (hospital, acute care, primary or aged care), starts with the healthcare aims (sector goals) and purposes (operationalized goals).

While the review scope was limited by its focus, i.e., EMR implementation in hospital only rather than EMR/EHR for the whole healthcare system, the diversity of hospital-specific settings, research questions, and methodological analyses posed a challenge to synthesize and present all the information that might be of use for decision-makers and hospital managers. Additionally, the grouping of study types as well as impact domains is limited to our interpretation of the information presented in the original studies. Last, there might be studies qualified to be included, yet missed, in the process of review. A future update and additional in-depth examinations of specific themes identified in this review would undoubtedly progress the research frontier in EMR implementation and economic evaluation.