http://orcid.org/0000-0002-6235-8177
Figures
Abstract
Failure to attend hospital appointments has a detrimental impact on care quality. Documented efforts to address this challenge have only modestly decreased no-show rates. Behavioral economics theory has suggested that more effective messages may lead to increased responsiveness. In complex, real-world settings, it has proven difficult to predict the optimal message composition. In this study, we aimed to systematically compare the effects of several pre-appointment message formats on no-show rates. We randomly assigned members from Clalit Health Services (CHS), the largest payer-provider healthcare organization in Israel, who had scheduled outpatient clinic appointments in 14 CHS hospitals, to one of nine groups. Each individual received a pre-appointment SMS text reminder five days before the appointment, which differed by group. No-show and advanced cancellation rates were compared between the eight alternative messages, with the previously used generic message serving as the control. There were 161,587 CHS members who received pre-appointment reminder messages who were included in this study. Five message frames significantly differed from the control group. Members who received a reminder designed to evoke emotional guilt had a no-show rates of 14.2%, compared with 21.1% in the control group (odds ratio [OR]: 0.69, 95% confidence interval [CI]: 0.67, 0.76), and an advanced cancellation rate of 26.3% compared with 17.2% in the control group (OR: 1.2, 95% CI: 1.19, 1.21). Four additional reminder formats demonstrated significantly improved impact on no-show rates, compared to the control, though not as effective as the best performing message format. Carefully selecting the narrative of pre-appointment SMS reminders can lead to a marked decrease in no-show rates. The process of a/b testing, selecting, and adopting optimal messages is a practical example of implementing the learning healthcare system paradigm, which could prevent up to one-third of the 352,000 annually unattended appointments in Israel.
Citation: Berliner Senderey A, Kornitzer T, Lawrence G, Zysman H, Hallak Y, Ariely D, et al. (2020) It’s how you say it: Systematic A/B testing of digital messaging cut hospital no-show rates. PLoS ONE 15(6): e0234817. https://doi.org/10.1371/journal.pone.0234817
Editor: Sreeram V. Ramagopalan, University of Oxford, UNITED KINGDOM
Received: March 23, 2020; Accepted: June 2, 2020; Published: June 23, 2020
Copyright: © 2020 Berliner Senderey et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data underlying the results presented in the study are available from clalit health services http://clalitresearch.org/.
Funding: The Israeli Ministry of Finance provided funds to the commercial company "Kayma Labs" to permit the development of the randomization process. No other funds were received by the authors in connection with this study. No funding bodies had any role in the study design, data collection, analysis, decision to publish, or preparation of the manuscript. The authors TK, HZ, and DA are employed by “Kayma Labs”, though this company did not fund this research paper.
Competing interests: The authors TK, HZ, and DA are employed by the commercial company “Kayma Labs”. This commercial affiliation does not alter our adherence to PLOS ONE policies on sharing data and materials.
Introduction
Unattended medical appointments are a frequent event. Hospital outpatient clinics have reported no-show rates of 19.3%-43.0% globally [1]. These events negatively impact care quality worldwide, causing major disruptions to clinical management, delays in scheduled care, and reduced patient contentedness. Health-care providers consider the no-show phenomenon as intractable and invest extensive efforts and resources to control and create workarounds such as overbooking [1–4].
Short message services (SMSs) are frequently used as pre-appointment reminders by health service providers to reduce appointment no-shows, and to provide information that relates to, and encourages, the desired behavior of canceling or keeping the appointment [5–11]. There is strong evidence that even simple SMS reminders are effective in reducing non-attendance compared to no reminders at all, though their impact is small [3, 12–14]; thus, the reminders’ performance is considered sub-optimal. Health providers who use reminders still experience substantial no-show rates of 21%-25%, resulting in decreased quality of care [3, 9, 12, 13, 15, 16]. Simple straightforward reminders implicitly assume that one key reason a patient does not attend their appointment is due to forgetfulness. Yet, there is vast evidence suggesting that other reasons for non-attendance without notification are more prominent and that a more holistic approach to this issue is needed [17–20].
Previous studies have demonstrated that the strategic narrative of the reminder may increase compliance in the healthcare domain. For example, a study that focused on human papillomavirus (HPV) infection investigated the impact of differential text messages on child HPV vaccination rates and found that persuasive text reminders, emphasizing the potential threat for the child, improved HPV vaccination rates [21].
Additionally, findings from two randomized control trials concluded that missed hospital appointments might be reduced by rephrasing appointment reminders and stating appointment costs [3]. The specific cost of the appointment manipulated guilt emotions, which in turn led to the lowest no-show rates. However, mentioning specific costs can antagonize or make patients doubt the authenticity of the reminder. We suggest that the same effect of guilt can be evoked without stating the cost, by emphasizing emotional guilt. In addition, there is no consensus as to the motivators that would optimally increase the likelihood that members will either attend their appointment or cancel it in advance.
Behavioral economics theory suggests that different motivational narratives, such as fairness to others or adherence to social norms, can dramatically increase a message’s impact as compared with a generic informative format [21–28]. Many retail and finance industries’ policies and practices reflect these theories and are designed to prompt people towards particular choices. Such policies and practices might have been relatively underused in healthcare practice.
This study aimed to assess whether using specific message formats for appointment reminders influences advanced cancellation and no-show rates. We examined whether a change in the narrative of the current SMS reminder increased members' engagement compared to the SMS reminder currently in use, and whether specific strategic narratives are more effective than others. Implementing our research results in daily clinical practice can affect members’ behavior, improve quality of care, and potentially serve as a practical example for the learning health system [29].
Materials and methods
Setting and data sources
This study was based on data from individuals with scheduled appointments to one of 596 outpatient clinics located within 14 Clalit Health Services (CHS) hospitals. CHS is the largest payer-provider healthcare organization in Israel, which provides primary, specialty, and inpatient care to over 52% of the Israeli population and has 4.4 million members. CHS’s comprehensive healthcare data warehouse combines hospital and community medical records. All Israeli citizens are covered by one of four healthcare organizations, and while it is possible to switch between the organizations, membership turnover within CHS is less than 1% annually [30], allowing for consistent longitudinal follow-up. CHS’ electronic health records (EHR) contain administrative and clinical data, socio-demographic information, diagnoses from community and hospital settings, recorded chronic diseases, clinical markers, and appointment related details. All CHS members’ information was extracted from CHS’s EHR, as of the index date (appointment date) and from current demographics.
CHS operates an appointment reminder system that automatically sends a text message five days prior to a scheduled appointment with a link to an internet-based system that allows for confirmation or cancellation the appointment in advance. Data from the CHS SMS appointment reminder system was retrieved and appended to the abovementioned data points.
Study population and design
The population eligible for this study included all CHS members who were 18 years old and older, with scheduled appointments between December 1, 2018 to March 31, 2019, at one of the 596 outpatient clinics within CHS’ 14 hospitals. All participants had a valid cell phone number in the CHS EHR and consented to receive phone-based appointment reminders. The index date was defined as the date of the scheduled appointment (Fig 1). Randomization occurred via a randomization program and participants were assigned to one out of nine messages that were issued five days in advance of the appointment. Members with multiple appointments during the study period could receive the same or differently framed messages for each appointment.
Variables definitions
Appointment reminders.
CHS members were randomly assigned to one out of nine possible message frames that reminded them of their upcoming appointment (see Table 1). Eight variations were designed based on the following principles: the ‘social norm’ versions highlighted the idea that social identity and descriptive norms potentially motivate individuals to perform certain actions [31–34]. The ‘emotional’ versions aimed to provoke an emotional reaction in order to prompt members to take action by mentioning people they care about [35] or to evoke feelings of sympathy or empathy [36]. The ‘appointment cost’ version was based on the opportunity cost effect [37]. Although members do not directly pay the healthcare organization for missed appointments, this narrative highlights the amount of money they cause the organization to lose by not showing up [38]. Both ‘professional figure’ and ‘personal’ versions relied on the messenger effect that suggests that people’s compliance to a message is affected by the figure who delivers it, for example, an actual name or authority figure rather than an automated device [39]. The ninth frame was the routinely used reminder message, in use by CHS in recent years, and was retained as the control group.
Outcomes.
The primary outcome was a no-show event, defined as a scheduled appointment that a CHS member failed to attend. The no-show rates were calculated as the number of no-show appointments out of the total number of appointments scheduled. The secondary outcome was advanced cancellation, defined as members who cancel their scheduled appointment in advance of the appointment date/time. The advanced cancellation rates were calculated as the number of cancellations out of the total number of appointment reminders sent.
Baseline measurements.
Sociodemographic variables were measured at index date and included biological sex, age (years), socioeconomic status (SES; low, medium, high; based on clinic-level data), population sector (Jewish, non-Jewish), and immigrant status (immigrated to Israel, born in Israel). Clinical characteristics included smoking status (current, former, or non-smoker, as reported in the EHR), body mass index (computed from documented weight and height measurements), and Charlson Comorbidity Index (computed from risk factors to evaluate an age-comorbidity score [40]). Comorbidity variables were evaluated as of the index date, and included cardiovascular diseases (yes/no; defined as any of the following: acute myocardial infarction unstable angina pectoris, angina pectoris, acute coronary syndrome, percutaneous transluminal coronary angioplasty, coronary artery bypass graft, ischemic heart disease, ischemic stroke), diabetes (yes/no), chronic kidney disease (yes/no; defined as the last eGFR value prior to index date less than 60 ml/min/1.73m2), celiac disease (yes/no), and inflammatory bowel disease (yes/no) (see S1 Table). We extracted these diagnoses from community and hospital records, as well as from the CHS chronic disease registry.
The appointment characteristics included past non-attendance–i.e.,‘chronic no-show’ (yes/no; defined as members who missed appointments at least two times in a row within one year prior to index date), time to an appointment (calculated as the difference in days between the date of scheduling the appointment to the date of the appointment), and clicking on the link (yes/no; defined as whether member clicked on the link in the SMS reminder message).
Statistical analysis
Socio-demographic characteristics, clinical, and appointment-related variables were calculated within the nine different SMS groups. Summaries of continuous variables are presented as means and standard deviations unless skewed, and in that case, are presented as medians and interquartile ranges. Categorical variables are presented as absolute numbers and percentages, as appropriate.
In order to assess whether any of the message frames caused a lower risk for no-shows compared to the currently existing message frame, multinomial testing was performed. Univariate and multivariate analyses using binary logistic regression models accounting for features determined via an automated generic framework to be most predictive of no-show (i.e., sociodemographic characteristics, clinical, and appointment related variables) behavior.
Message frames were considered as treatment variables, with the existing message serving as the reference group and the record of attendance as the binary outcome variable. Secondary analyses were conducted to assess the effect of the different message frames on canceling appointments in advance.
Statistical analyses were conducted using the R language (version 3.5.3, R Foundation for Statistical Computing, Vienna, Austria). All statistical tests were 2-tailed, and a 5% significance threshold was maintained.
Ethics
This study was reviewed by the IRB of the CHS organization and it was determined that this study was not a clinical trial, but rather an organizational initiative to optimize internal policy. It received an exemption for the need for individual informed consent since it was determined that the various intervention arms posed no harm to members. It was not registered as a clinical trial for these reasons. Obtaining consent would introduce a burden to the members (larger than the intervention itself); obtaining informed consent would cause serious practical problems that would undermine the trial results (particularly for the control group), and the risk of harm was low since the intervention merely consisted of small modifications to existing routine processes.
Results
During the study period, between December 2018 and March 2019, there were 218,066 scheduled appointments in CHS’s hospital outpatient clinics, of which 161,587 had a valid associated mobile telephone number with approval for receiving phone-based appointment reminders (Fig 1). Among those who received one of the nine SMS appointment reminders (Table 1), in 104,469 (64.6%) cases, reminder’s accompanying link was opened within 48 hours of receiving the message.
Socio-demographic, clinical, and appointment related characteristics by type of appointment reminder can be seen in Table 2. Approximately half of the eligible population was female (55.4%), and the average age of the population was 59.3 years. The distribution of all members’ characteristics and appointment information was similar between the nine treatment groups, and no significant differences were found (e.g., all p values > 0.05) (Table 2).
Fig 2 and Table 3 present no-show and advanced cancellation rates in the groups receiving one of the eight alternative message frames compared with the generic control. Five out of the eight alternative message reminders presented in Table 1 (‘appointment cost’, ‘emotional relatives’, ‘emotional guilt’, ‘social norm’, and ‘social identity’) had significantly lower rates of no-shows and higher rates of canceling in advance compared with the routinely used message reminder. The ‘emotional guilt’ reminder frame led to the lowest no-show and highest advanced cancellation rates. Members who received the ‘emotional guilt’ message reminder had a no-show rate of 14.2% compared with 21.1% in the control group (odds ratio [OR]: 0.69, 95% confidence interval [CI]: 0.67, 0.76), and an advanced cancellation rate of 26.3% compared with 17.2% in the control group (OR: 1.2, 95% CI: 1.19, 1.21).
Abbreviations: SMS, short message service. † Calculated as the number of people who did not attend an appointment (and did not cancel in advance) out of the total population with appointments. ‡ Calculated as the number of people who canceled an appointment in advance out of the total number who clicked on the SMS link.
Favorable results were found among members who received the ‘appointment cost’ message, with a 15.3% no-show rate (OR: 0.72, 95% CI: 0.68, 0.77) and a 27.4% advanced cancellation rate (OR: 1.25, 95% CI: 1.1, 1.27). Similar results of 15.6% (OR: 0.77, 95% CI: 0.79, 0.82) no-show rate and 23.4% (OR: 1.18, 95% CI: 1.17, 1.19) advanced cancellation rate were found among the members who received the ‘emotional relatives’ framed message.
Both ‘social norm’ and ‘social identity’ framed messages were associated with a 17.8% (OR: 0.73, 95% CI: 0.61, 0.79) and 17.7% (OR: 0.83, 95% CI: 0.76, 0.87) no-show rate and 21.8% (OR: 1.08, 95% CI: 1.07, 1.08) and 24.6% (OR: 1.19, 95% CI: 1.11, 1.24) advanced cancellation rate, respectively. The ‘standard’, ‘personal request’, and ‘professional figure’ messages did not produce significantly different results as compared with the control message (Fig 2, Table 3).
The multivariate analysis showed that the relative reduction in the risk of no-show and advanced cancellation remained quite unchanged after adjusting for socio-demographic variables, clinical and appointment-related characteristics, and past non-attendance behavior (Table 3).
Discussion
We have shown that careful design of SMS narratives based on behavioral economic principles can reduce hospital outpatient clinic no-show rates by over 30 percent. Out of nine differently framed reminders, five produced statistically significant lower no-show rates and higher advanced cancellation rates. The emotional guilt and specific cost message frames showed the greatest nominal differences in no-show rates and advanced cancellation rates compared with the control group (14.2% and 15.3% compared to 21.1% in no-show rates and 26.3% and 27.4% compared to 17.2% in advanced cancellation rates, respectively).
While many health interventions approach behavioral challenges by emphasizing the need to support and prompt the individual through reminders, these results indicate that different messages can influence no-show and cancellation rates [5, 6, 12, 41–43]. These results are aligned with behavioral economic theories. However, the current data do not offer adequate support that the varied effects have resulted from those specific psychological mechanisms. Future research is needed to explore this topic.
These results highlights the potential of introducing behavioral economics principles into multiple avenues of healthcare delivery in order to improve member adherence and reduce waste in care provision.
Our study ventures beyond the published literature by including a standard alternative message alongside the affect-based alternatives, possibly indicating that the reduction in no-show rates was in fact due to a change in context and not merely in response to a simple change in wording.
This study had several limitations. First, all reminder messages were sent five days prior to the appointment. This five day period might be considered as relatively long, as previous studies have reported shorter periods of one to three days between sending the reminder and appointment date [6]. Sending the reminder SMS five days before the appointment may have allowed the psychological effect of the remainder to decay over time, meaning that even though the members may have confirmed attendance, they may not attend the appointment. However, if at the time of the reminder, the members had forgotten about the appointment, it may have been possible that this five day period of time will enable them to rearrange schedule in order to attend.
Another possible limitation was the inability to distinguish between members who read the SMS reminders and those who did not. However, we retain our confidence in the overall conclusions since 64.6% of the receipients clickedon the link within 48 hours, indicating that the majority of people read the message. Also, the assignment to the SMS frame for each appointment was properly randomized, and there is no reason to suspect additional potential confounders. Furthermore, all members within the study period had no more than three appointments. It is possible that receiving the same message when scheduling new appointments diminished the effect over time. The multiple and varying number of appointments per patient in the sample led to correlated error variance in the study dataset, which may have produced biased error estimates. However, due to the short period of the study (December 2018 –March 2019), not many patients had repeated appointments (approximately 9.4% of the total population).
It is important to note that the expected effect of rephrasing reminder messages may be limited, as it was designed with the ‘average’ person in mind, rather than customized on an individual level. While we found that the effect of sending alternative messages was maintained after adjustment by various covariates, it is possible that interaction of individual characteristics may modify the impact of specific message frames on no-shows. Future research should focus on customizing the content per person to even further reduce no-show rates.
The major strength of our findings was that all 14 CHS hospitals, located throughout the country, were included in this study. This means that the effect of different SMS versions on no-shows was assessed among participants who came from diverse backgrounds, and thus, can be generalized to a greater scale for policy implications.
The number of unattended appointments across all outpatient clinics in CHS’ 14 hospitals is approximately 600,000 annually (18.7% of all outpatient clinic appointments). Our results indicate that replacing the current reminder message with a carefully designed message can potentially save 187,000 appointments annually. Nationally, this change can potentially result in saving approximately 350,000 unattended appointments, thereby improving the quality of care across the country.
Since May 2019, CHS changed the policy to adopt the “emotional guilt’ narrative in all outpatient clinics for all messages used in daily practice (more than 3 million appointments a year), and is monitoring the scale of the real-world impact of this change. This can serve as an example of how research knowledge gained by a learning healthcare system can be implemented into routine clinical practice and effect changes in the organization’s policy [29, 44]. It is worth noting that such a change may have additional unintended consequences other than improving visit attendance, such as a reduction in clinic or physician satisfaction ratings.
The era of digital health enables healthcare providers to systematically customize their interaction with members in order to increase the effectiveness of healthier behavior [45]. This simple example of how strategic use of traditional messaging substantially impacts members’ behaviors shows the untapped potential of smart messaging in health care. Improvement in member engagement depends on the utilization of technical add-ons, but even more so, on the nuances and characterization of the way the messaging is constructed.
Supporting information
S1 Table. Codes used for variable definitions.
https://doi.org/10.1371/journal.pone.0234817.s001
(DOCX)
Acknowledgments
We thank Oran Huberman, Meirav Visel, Tzahi Israel, and Moshe Gerlitz for contribution in the intervention conceptual framing and design; Sydney Krispin and Becca Feldman for their editorial support and reviewing the manuscript; and Galit Benbenishiti, Nachum Yosef, and Ilan Gofer for their help in the design and implementation of the randomization process.
References
- 1. Nelson A., et al., Predicting scheduled hospital attendance with artificial intelligence. npj Digital Medicine, 2019. 2(1).
- 2. Elvira C., et al., Machine-Learning-Based No Show Prediction in Outpatient Visits. International Journal of Interactive Multimedia and Artificial Intelligence, 2018. 4(7): p. 29.
- 3. Hallsworth M., et al., Stating Appointment Costs in SMS Reminders Reduces Missed Hospital Appointments: Findings from Two Randomised Controlled Trials. PLoS One, 2015. 10(9): p. e0137306. pmid:26366885
- 4. Ellis D.A., et al., Demographic and practice factors predicting repeated non-attendance in primary care: a national retrospective cohort analysis. Lancet Public Health, 2017. 2(12): p. e551–e559. pmid:29253440
- 5. de Jongh T., et al., Mobile phone messaging for facilitating self-management of long-term illnesses. Cochrane Database Syst Rev, 2012. 12: p. CD007459. pmid:23235644
- 6. Gurol‐Urganci I., et al., Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database of Systematic Reviews, 2013(12).
- 7. Rebecca Guy J.H., Wand Handan, Stott Sam,Ali Hammad and Kaldor John, How effective are SMS reminders at increasing clinic attendance-Meta-analysis and systematic review. Health Services Research, 2012. 47(2): p. 614–632. pmid:22091980
- 8. Young K.E., Franklin A.B., and Ward J.P., Infestation of northern spotted owls by hippoboscid (Diptera) flies in northwestern California. J Wildl Dis, 1993. 29(2): p. 278–83. pmid:8487378
- 9. lin CL, M.N., Boneh J, Li H, Lazebnik R, Text Message Reminders Increase Appointment Adherence in a Pediatric Clinic: A Randomized Controlled Trial. international journal of pediatrics, 2016. 2016.
- 10. Kuerbis A., van Stolk-Cooke K., and Muench F., An exploratory study of mobile messaging preferences by age: Middle-aged and older adults compared to younger adults. J Rehabil Assist Technol Eng, 2017. 4.
- 11. Kerrison R.S., et al., Text-message reminders increase uptake of routine breast screening appointments: a randomised controlled trial in a hard-to-reach population. Br J Cancer, 2015. 112(6): p. 1005–10. pmid:25668008
- 12. Haynes L.C., et al., Collection of Delinquent Fines: An Adaptive Randomized Trial to Assess the Effectiveness of Alternative Text Messages. Journal of Policy Analysis and Management, 2013. 32(4): p. 718–730.
- 13. Hallsworth M, L.J., Metcalfe R,Vlaev I, The Behavioralist As Tax Collector: Using Natural Field Experiments to Enhance Tax Compliance JPolicyAnalManag, 2013. 32: p. 718–730.
- 14. Junod Perron N., et al., Text-messaging versus telephone reminders to reduce missed appointments in an academic primary care clinic: a randomized controlled trial. BMC Health Serv Res, 2013. 13: p. 125. pmid:23557331
- 15. Dantas L.F., et al., No-shows in appointment scheduling—a systematic literature review. Health Policy, 2018. 122(4): p. 412–421. pmid:29482948
- 16. McLean S.M., et al., Appointment reminder systems are effective but not optimal: results of a systematic review and evidence synthesis employing realist principles. Patient Prefer Adherence, 2016. 10: p. 479–99. pmid:27110102
- 17. Neal R.D., et al., Reasons for and consequences of missed appointments in general practice in the UK: questionnaire survey and prospective review of medical records. BMC Fam Pract, 2005. 6: p. 47. pmid:16274481
- 18. Martin C., Perfect T., and Mantle G., Non-attendance in primary care: the views of patients and practices on its causes, impact and solutions. Family Practice, 2005. 22(6): p. 638–643. pmid:16055472
- 19. van Baar J.D., et al., Understanding reasons for asthma outpatient (non)-attendance and exploring the role of telephone and e-consulting in facilitating access to care: exploratory qualitative study. Quality and Safety in Health Care, 2006. 15(3): p. 191–195. pmid:16751469
- 20. Crosby L.E., et al., Perceived barriers to clinic appointments for adolescents with sickle cell disease. J Pediatr Hematol Oncol, 2009. 31(8): p. 571–6. pmid:19636266
- 21. Matthew S. McGlone, K.K.s., Serena A. Rodriguez, Maria E. Fernandez, Persuasive texts for prompting action: Agency assignment in HPV vaccination reminders. Vaccine, 2017. 35: p. 4295–4297.
- 22. Rice T., The behavioral economics of health and health care. Annu Rev Public Health, 2013. 34: p. 431–47. pmid:23297657
- 23. Soler R.E., et al., Nudging to Change: Using Behavioral Economics Theory to Move People and Their Health Care Partners Toward Effective Type 2 Diabetes Prevention. Diabetes Spectr, 2018. 31(4): p. 310–319. pmid:30510385
- 24. RH Thaler C.S., Nudge: improving decisions about health, wealth, and happiness. 2008.
- 25. King D., et al., Approaches based on behavioral economics could help nudge patients and providers toward lower health spending growth. Health Aff (Millwood), 2013. 32(4): p. 661–8.
- 26. Barkan R., et al., The pot calling the kettle black: distancing response to ethical dissonance. J Exp Psychol Gen, 2012. 141(4): p. 757–73. pmid:22409664
- 27. Bertoni M, C.L., Robone S, Promoting breast cancer screening take-ups with zero cost: Evidence from an Experiment on Formatting Invitation Letters in Italy. IZA discussion paper, 2019. 12193.
- 28. Ding X., et al., Designing risk prediction models for ambulatory no-shows across different specialties and clinics. J Am Med Inform Assoc, 2018. 25(8): p. 924–930. pmid:29444283
- 29.
Mcginnis JM, A.D., Olsen L, The learning healthcare system: workshop summery. National Academies press, June 2007.
- 30. Rosen B., Waitzberg R., and Merkur S., Israel: Health System Review. Health Syst Transit, 2015. 17(6): p. 1–212. pmid:27050102
- 31. Hogg M.A. and Reid S.A., Social Identity, Self-Categorization, and the Communication of Group Norms. Communication Theory, 2006. 16(1): p. 7–30.
- 32. Goldstein N.J., Cialdini R.B., and Griskevicius V., A Room with a Viewpoint: Using Social Norms to Motivate Environmental Conservation in Hotels. Journal of Consumer Research, 2008. 35(3): p. 472–482.
- 33. Griskevicius V., et al., Going along versus going alone: when fundamental motives facilitate strategic (non)conformity. J Pers Soc Psychol, 2006. 91(2): p. 281–94. pmid:16881765
- 34. Shapiro J.R. and Neuberg S.L., When do the stigmatized stigmatize? The ironic effects of being accountable to (perceived) majority group prejudice-expression norms. J Pers Soc Psychol, 2008. 95(4): p. 877–98. pmid:18808265
- 35.
Barrett, L.F., & Salovey, P., The wisdom in feeling. Guilford, New York..pdf>. 2002.
- 36. Christian R.C., & Alm J., Empathy, sympathy, and tax compliance. Journal of economic psychology, 2014. 40: p. 62–82.
- 37.
Zellermayer O., The pain of paying. unpublished dissertation, Department of Social and Decision Sciences. Carnegie Mellon University, Pittsburgh, PA., 1996.
- 38. Frederick S., et al., Opportunity Cost Neglect. Journal of Consumer Research, 2009. 36(4): p. 553–561.
- 39. Kassin S.M., Deposition Testimony and the Surrogate Witness:Evidence for a "Messenger Effect" in Persuasion. Personality and Social Psychology Bulletin, 1983. 9(2): p. 281–288.
- 40. Charlson M., et al., Validation of a combined comorbidity index. J Clin Epidemiol, 1994. 47(11): p. 1245–51. pmid:7722560
- 41. Hasvold P.E. and Wootton R., Use of telephone and SMS reminders to improve attendance at hospital appointments: a systematic review. Journal of Telemedicine and Telecare, 2011. 17(7): p. 358–364. pmid:21933898
- 42. Koshy E., Car J., and Majeed A., Effectiveness of mobile-phone short message service (SMS) reminders for ophthalmology outpatient appointments: Observational study. BMC Ophthalmology, 2008. 8(1): p. 9.
- 43. Downer S.R., et al., SMS text messaging improves outpatient attendance. Australian Health Review, 2006. 30(3): p. 389–396. pmid:16879098
- 44. Bellika A.B.a.J.G., The Learning Healthcare System Where are we now A systematic review. Journal of biomedical informatics, 2016. 64: p. 87–92. pmid:27693565
- 45. Michie S., van Stralen M.M., and West R., The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci, 2011. 6: p. 42. pmid:21513547