http://orcid.org/0000-0001-8558-5973
Figures
Abstract
Objective
To develop key performance indicators that evaluate the effectiveness of a prescription medication system.
Methods
A modified RAND/UCLA appropriateness method was used to develop key performance indicators (KPIs) for a prescription medication system. A broad list of potential KPIs was compiled. A multidisciplinary group composed of 21 experts rated the potential KPIs. A face-to-face meeting was held following the first rating exercise to discuss each potential KPI individually. The expert panel undertook a final rating of KPIs. The final set of KPIs were those indicators where at least 80 percent of experts rated the indicator highly i.e. rating of ≥ 7 on a scale from 1 to 9.
Results
292 KPIs were identified from the published literature. After removing duplicates and combining similar indicators 71 KPIs were included. The final ranking resulted in six indicators being ranked 7 or higher by 80% of the respondents and an additional seven indicators being ranked 7 or higher by ≥70 but ≤80% of respondents. The six selected indicators include four specific disease areas, measure structural and process aspects of health service delivery, and assessed three of the domains of healthcare quality: efficiency, effectiveness, and safety.
Conclusions
These indicators are recommended as a starting point to assess the current performance of prescription medication systems. Consideration should be given to developing indicators in additional disease areas as well as indicators that measure the domains of timeliness and patient–centeredness. Future work should focus on the feasibility of measuring these indicators.
Citation: Spackman E, Clement F, Allan GM, Bell CM, Bjerre LM, Blackburn DF, et al. (2019) Developing key performance indicators for prescription medication systems. PLoS ONE 14(1): e0210794. https://doi.org/10.1371/journal.pone.0210794
Editor: Yan Tang, RTI International Metals Inc, UNITED STATES
Received: August 10, 2018; Accepted: January 2, 2019; Published: January 15, 2019
Copyright: © 2019 Spackman et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are available from figshare (DOI: 10.6084/m9.figshare.7413503 and 10.6084/m9.figshare.7413506).
Funding: Funding was provided by Alberta Health and the Ontario Ministry of Health and Long-Term Care. Provincial Drug Plan Managers from Alberta Health and the Ontario Ministry of Health and Long-Term Care participated in the survey and discussions with the other experts.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Effective prescription medication systems ensure patients can access needed medications, reduce overuse of inappropriate medications, and optimize use of cost-effective medications.[1,2] Prescription medication systems include, but extend beyond publicly funded drug plans. In many jurisdictions the prescription medication system includes physicians who prescribe medications, pharmacists that dispense and sometimes prescribe medication, the public and private prescription drug plans that provide partial or full payment for prescription medications and the patients who use prescription medications.
In a recent review of Canadian provincial drug plans many differences were noted in who was covered, what medications were covered, the total medication expenditures of each province, the government share of the total expenditure, and the out-of-pocket costs borne by the patient.[3] To inform decision making about the optimal organization of a prescription medication system it is important to have metrics to evaluate the effectiveness of a jurisdiction’s prescription medication system.
Key performance indicators (KPIs) are metrics that evaluate the outcomes of an organization, or of particular activities. The assessment of performance identifies gaps between current and desired outcomes and provides an indication of progress towards closing the gaps.[4] KPIs are commonly used to measure and improve upon the performance of healthcare systems; however, no KPIs have been specifically developed to evaluate or compare prescription drug systems across jurisdictions. The Canadian Institute for Health Information (CIHI) reports a broad collection of health indicators to help provinces/territories, regional health authorities and facilities monitor the health of their populations and track how well their health systems function.[5] In England, KPIs have been used through the Quality and Outcomes Framework (QOF) to standardize and incentivize improvements in the delivery of primary medical care.[6,7] While some of the QOF indicators encourage appropriate medication use, the focus of QOF is on primary health care delivery.
In this paper, we introduce a set of KPIs for measuring the performance of a prescription medication system and describe the process we used to develop them.
Methods
We used a modified RAND/UCLA appropriateness method to develop KPIs for a prescription medication system.[8–10] This method is a formal group judgment process that combines collective judgment of experts with scientific evidence, by asking panelists to rate, discuss, and then re-rate indicators. It is used extensively to assess what constitutes appropriate and necessary care within a health care system.[11] In this modified procedure, four steps were taken: 1) Identifying published KPIs relevant to a prescription medication system; 2) Rating of KPIs by an expert panel; 3) Meeting of the expert panel to discuss and develop KPIs; and 4) A final rating of the modified list of KPIs by the expert panel. This study was approved by the ethics committee at the University of Calgary, REB16-1747.
Identifying published KPIs
We compiled a broad list of potential KPIs. The following sources were searched: US Agency for Healthcare Research and Quality (AHRQ)[12]; National Health Services Quality Outcomes Framework (QOF)[13]; World Health Organization (WHO)[14]; Healthcare Effectiveness Data and Information Set (HEDIS)[15]; the Canadian Institute of Health Information (CIHI) Indicator Library[16]; The World Bank’s World Development and Service Delivery Indicators[17]; OECD’s Health Care Quality Indicators[18]; and Indicators of Quality Prescribing in Australian General Practice (a manual developed by Australia’s National Prescribing Service)[19]. We kept our initial search broad and included verbatim every KPI of relevance to a prescription drug system from the published source. Similar indicators were combined to represent a single indicator and those that were not relevant to the Canadian system were excluded, for example ‘The estimated number of children eligible for antiretroviral therapy’ (because the prevalence of HIV is so low in children in Canada this would not have been a useful marker of appropriate medication access).
Rating of KPIs by an expert panel
A multidisciplinary group composed of 21 experts in drug evaluation, clinical therapeutics, drug policy, drug plan management, and/or performance indicator development were invited by email and consented to participate. Each expert was sent the list of identified published KPIs and a brief description of its intent and purpose. Experts were instructed to identify KPIs that satisfied three criteria: a) likely associated with patient outcomes; b) could be influenced by drug plan policies; and c) higher performance on the indicator would be considered a higher quality prescription drug system.
The experts were asked to assess the validity of each proposed indicator on two scales: 1) importance to quality care and health of the patient; and 2) sensitivity to performance of drug coverage i.e. how well the indicator is likely to change given changes in drug plan policies. Scales ranged from 1 (Not valid) to 9 (Extremely valid). Experts were also asked if they had any suggestions or comments.
Meeting of the expert panel to discuss and develop KPIs
A face-to-face meeting was held following the first rating exercise to discuss each quality indicator individually. Experts were provided with a summary of the first-round ratings and a confidential reminder of their own previous rating. The experts were divided into small groups and asked to discuss in detail a group of indicators which shared similar content areas. The discussion focused on the evidence (or lack thereof) supporting or refuting each indicator and the first-round ratings of the experts. Experts were also given the opportunity to propose alternate wording for each indicator or propose additional indicators. They were then asked to separate indicators into bins marked “Very good”, “Good”, or “Poor”. The small groups were then re-organized and the new groups reviewed the first group’s judgments. After the results of the second small groups were collated and separated into “Very good”, “Good”, or “Poor”, the entire panel met as a group. During the final session of the day, indicators that were noted to be poor were reviewed and removed, if consensus could be achieved. Comments on specific KPIs were also discussed, and instructions were provided with respect to the second round of confidential rating. During this discussion expert members noted that additional indicators relevant to step therapy should be developed. Step therapy is an approach to prescription intended to control the costs and risks by only progressing to more costly or risky therapies within specific clinical areas if necessary.
Final rating of modified list of KPIs by the expert panel
After creation of the additional KPIs as noted above, all remaining indicators were re-rated. Respondents were asked to consider whether each metric was appropriate as a performance indicator of a prescription medication system, using a scale from 0 (Not appropriate) to 9 (Completely appropriate). Consistent with the RAND methodology, the final set of indicators included those indicators where at least 80 percent of experts rated the appropriateness of the indicator highly (rating of ≥ 7).[10] Results were discussed with the experts in a series of conference calls to inform them of the results and understand their perception of the results.
Results
Identifying published KPIs
A total of 292 KPIs were identified from the published literature. After removing duplicates and combining similar indicators, a total of 108 KPIs were included. After initial discussions with the research team with the goal of excluding KPIs of very low relevance, the list of 108 indicators was reduced to 71 indicators for review by the expert panel.
Rating of KPIs by an expert panel
The first rating exercise was completed by 18 of 21 (86 percent) of the experts. The results of the first rating exercise were summarized and returned to each respondent. An example of the personalized information provided to each of the respondents for each of the 71 indicators is provided in S1 Table.
Meeting of the expert panel to discuss and develop KPIs
Of the 21 experts who were invited to the face-to-face meeting, two were unable to attend. A number of important decisions and actions arose from this meeting, including revisions to the wording of many indicators, the need to represent overall prescription drug systems rather than drug insurance activities only, and the need to develop some additional indicators for step therapy. It was also decided at the meeting to identify indicators by the Donabedian outcome framework (structure, process and outcome)[20] and the Agency for Healthcare Research and Quality (AHRQ) outcome framework (effective, safe, efficient, equitable, timely, and patient-centered)[21].
With respect to the need to focus on the entire prescription medication system, it was noted that poor performance across KPIs could be attributable to many stakeholders, including the public drug plan, private insurance plans, physicians, and pharmacists. The experts decided that it may be difficult to disentangle the effects of each of these actors on KPIs. Rather than try to develop KPIs that would solely be affected by the prescription drug plan, it was decided that a broader view would be appropriate and that indicators could represent any aspect of the prescription drug system.
The group also discussed whether the ratings of the public drug plan managers included in the expert panel should be used in the final ranking as a perception of bias could be raised given that the indicators developed may be used to assess the performance of the drug plans. After discussion, it was decided that the drug plan manager’s responses would be included because of their expertise in the drug system, but a sensitivity analysis, with and without the responses of the drug plan managers, was undertaken.
Through group discussion, 43 indicators were considered poor and were excluded from further rating; 28 of the 71 indicators were considered appropriate for further rating. After completion of the in-person meeting, the experts considered it important to include indicators of adherence to step therapy. Six additional step therapy indicators were developed with the help of three clinical experts. These indicators focused on the treatment of diabetes, chronic obstructive pulmonary disease (COPD), and rheumatoid arthritis. These diseases were selected because they are common, because of the strong evidence base regarding step therapy and the existence of national treatment guidelines.
Final rating of published and newly developed KPIs by the expert panel
The second and final rating exercise asked respondents to rate 34 KPIs. After the meeting each author was sent an electronic survey to complete. Of the 21 experts invited to undertake the final rating exercise 17 (81 percent) completed the survey. The respondents had broad expertise and were from a variety of provinces (Table 1).
Six indicators were ranked 7 or higher by 80% of the respondents (Table 2),
- The proportion of patients taking a brand name medication where a therapeutically equivalent lowest cost generic is available in class.
- The price in each province for a basket of selected medicines.
- In adults (age ≥ 30), initiating pharmacologic management of type 2 diabetes, the proportion who receive metformin as first-line therapy.
- The proportion of patients with asthma (age < 45) prescribed a long acting beta2 agonist who are NOT using an inhaled corticosteroid.
- In adults (age > 18) with rheumatoid arthritis who are receiving biologic or targeted synthetic therapy, the proportion of patients who had previously received methotrexate and at least one other traditional DMARD, either alone or in combination.
- The proportion of patients (age ≥ 18) diagnosed with chronic non-cancer pain dispensed an opioid at a dose less than the equivalent of 100mg per day of morphine.
In the sensitivity analysis, removing the drug plan manager’s responses resulted in no changes to which indicators were ranked 7 or higher by ≥80% of respondents.
An additional seven indicators were ranked 7 or higher by ≥70% but ≤80% of respondents (S2 Table).
Discussion
A panel of 21 experts considered a list of 71 KPIs of a prescription medication system, and selected six indicators as being valid KPIs. The selected indicators include four specific disease areas, measure structural and process aspects of health service delivery, and assessed three of the domains of healthcare quality: efficiency, effectiveness, and safety. None of the included indicators measured health outcomes or reflected timeliness or patient–centeredness. While none of the included indicators specifically measured equity, KPI’s 3–6 in Table 2, when measured across a variety of patient subgroups would enable an assessment of equity.
While the initial goal of this project was to develop indicators to measure the performance of prescription drug plans, discussion of the experts at the face-to-face meeting recommended it was more appropriate to develop KPIs reflecting performance of the overall prescription medication system. All agreed that it would be impossible to disentangle the effect of the many different stakeholders that contribute to the health of patients through the use of prescription medications (including publicly funded drug plans, private drug plans, physicians and pharmacists). However, experts involved felt that the selected KPIs would enable the comparison of prescription medication system performance across jurisdictions, acknowledging that performance variations could be due to a variety of factors within a system. Experts noted that understanding differences in jurisdictional performance on a particular indicator would require consideration of the jurisdictions’ demographics, the provincial health care system and the capacity of provincial drug plans. When using these KPIs, experts recommended detailed interviews with the drug plan managers to understand their current rules and regulations, and their ability to optimize appropriate use of medications within their drug plan.
The experts agreed that these indicators are not meant to judge the performance of a publicly funded drug plan, nor its managers, as each jurisdiction has evolved differently and faces different constraints (including varied budget and legal responsibilities). Moreover, as noted, many aspects of the prescription medication system will affect these metrics.
The ability to develop indicators is limited by the available evidence on optimal prescribing in disease areas. There was some concern that measuring indicators in only four disease areas would put undue attention on these diseases. This might result in prescription medication systems focusing improvements in these narrow disease areas in order to improve outcomes on these KPIs. Experts were careful to note that these should be seen as representative disease areas–and that lower or higher performance in these indicators would likely extend to other disease areas.
It was recommended that the next steps would be to develop technical definitions and measure the selected KPIs using administrative health data (where needed) across different jurisdictions to assess feasibility and establish baseline performance. An additional seven indicators (Appendix 2) were also recommended for measurement and further evaluation, based on having been ranked as a valid indicator by at least 70% of the expert group. Experts noted the importance of future meetings to consider the results of the initial evaluation, and to develop additional KPIs that reflect the performance of a prescription medication system, and possibly to remove existing KPIs as best practices change.
The prescription medication system is an important part of overall health care and plays an important role in maximizing the health of the population through cost-effective provision of medications. There are many differences in prescription medication systems across jurisdictions: eligibility criteria for beneficiaries, types of medicines covered, patient copayments, and total expenditures. KPIs were developed to evaluate the effectiveness of different prescription medication systems and inform decision makers about changes in organization.
Conclusion
A multifaceted process involving a panel of 21 experts developed and assessed KPIs of a prescription medication system. Multiple rounds of rating, conference calls and one face-to-face meeting reduced the initial list of 292 indicators to six indicators rated highly as being valid KPIs. These indicators are recommended as a starting point to assess the current performance of prescription medication systems across different jurisdictions. Consideration should be given to developing indicators in additional disease areas as well as indicators that measure the domains of timeliness and patient–centeredness. In order to understand differences–and improve performance within KPIs across jurisdictions—assessing the capacity of publicly funded medication systems will be important. Future work will focus on the feasibility of measuring these indicators.
Supporting information
S1 Table. Example response from the first rating exercise.
https://doi.org/10.1371/journal.pone.0210794.s001
(DOCX)
S2 Table. The seven key performance indicators that were highly ranked by 70 percent of respondents.
https://doi.org/10.1371/journal.pone.0210794.s002
(DOCX)
Acknowledgments
We would like to thank Irfan Dhalla, Adrian Levy, Suzanne McGurn, Leanne Jardine, Kevin Wilson and Chad Mitchell for participating in the RAND process.
References
- 1. Bigdeli M, Peters DH, Wagner AK. Medicines in Health Systems: Advancing access, affordability and appropriate use. 2014.
- 2.
Canadian Medical Association. A Prescription for Optimal Prescribing. Ottawa, ON; 2011.
- 3. Clement FM, Soril LJJ, Emery H, Campbell DJT, Manns BJ. Canadian Publicly Funded Prescription Drug Plans, Expenditures and an Overview of Patient Impacts Prepared for Alberta Health. 2016;
- 4. Stelfox HT, Niven DJ, Clement FM, Bagshaw SM, Cook DJ, McKenzie E, et al. Stakeholder Engagement to Identify Priorities for Improving the Quality and Value of Critical Care.
- 5.
Canadian Institute for Health Information. CIHI Health Indicators [Internet]. 2017.
- 6. Gillam SJ, Siriwardena AN, Steel N. Pay-for-performance in the United Kingdom: impact of the quality and outcomes framework: a systematic review. Ann Fam Med. American Academy of Family Physicians; 2012;10: 461–8. pmid:22966110
- 7. Roland M. Linking Physicians’ Pay to the Quality of Care—A Major Experiment in the United Kingdom. new Engl J Med n engl j med. 2004;35114.
- 8. Shekelle PG, Kahan JP, Bernstein SJ, Leape LL, Kamberg CJ, Park RE. The Reproducibility of a Method to Identify the Overuse and Underuse of Medical Procedures. N Engl J Med. Massachusetts Medical Society; 1998;338: 1888–1895. pmid:9637810
- 9. Shekelle PG, Chassin MR, Park RE. Assessing the predictive validity of the RAND/UCLA appropriateness method criteria for performing carotid endarterectomy. Int J Technol Assess Health Care. 1998;14: 707–27. pmid:9885461
- 10. Fitch K, Bernstein SJ, Aguilar MD, Burnand B, Lacalle JR, Lázaro P, et al. The RAND/UCLA Appropriateness Method User’s Manual Library of Congress Cataloging-in-Publication Data. 2001;
- 11. Sheldon T. Promoting health care quality: what role performance indicators? Qual Health Care. 1998;7 Suppl: S45–50.
- 12.
Agency for Healthcare Research and Quality. National Quality Measures Clearinghouse [Internet].
- 13.
NHS Digital. Quality and Outcomes Framework [Internet].
- 14.
World Health Organization. Global Health Observatory indicator views [Internet].
- 15.
National Committee for Quality Assurance. HEDIS Measures [Internet].
- 16.
Canadian Institute for Health Information. Indicator Library [Internet].
- 17.
The World Bank. Service Delivery Indicators [Internet].
- 18.
OECD. Data for Measuring Health Care Quality and Outcomes [Internet].
- 19.
National Prescribing Service (Australia). Indicators of quality prescribing in Australian general practice: a manual for users. Surry Hills, N.S.W.: National Prescribing Service; 2006.
- 20. Donabedian A. The quality of care. How can it be assessed? JAMA—J Am Med Assoc. 1988;260: 1743–8.
- 21. Agency for Healthcare Research and Quality. The Six Domains of Health Care Quality [Internet]. 2017.