Correction
20 Oct 2015: Hallsworth M, Berry D, Sanders M, Sallis A, King D, et al. (2015) Correction: Stating Appointment Costs in SMS Reminders Reduces Missed Hospital Appointments: Findings from Two Randomised Controlled Trials. PLOS ONE 10(10): e0141461. https://doi.org/10.1371/journal.pone.0141461 View correction
Figures
Abstract
Background
Missed hospital appointments are a major cause of inefficiency worldwide. Healthcare providers are increasingly using Short Message Service reminders to reduce ‘Did Not Attend’ (DNA) rates. Systematic reviews show that sending such reminders is effective, but there is no evidence on whether their impact is affected by their content. Accordingly, we undertook two randomised controlled trials that tested the impact of rephrasing appointment reminders on DNA rates in the United Kingdom.
Trial Methods
Participants were outpatients with a valid mobile telephone number and an outpatient appointment between November 2013 and January 2014 (Trial One, 10,111 participants) or March and May 2014 (Trial Two, 9,848 participants). Appointments were randomly allocated to one of four reminder messages, which were issued five days in advance. Message assignment was then compared against appointment outcomes (appointment attendance, DNA, cancellation by patient).
Results
In Trial One, a message including the cost of a missed appointment to the health system produced a DNA rate of 8.4%, compared to 11.1% for the existing message (OR 0.74, 95% CI 0.61–0.89, P<0.01). Trial Two replicated this effect (DNA rate 8.2%), but also found that expressing the same concept in general terms was significantly less effective (DNA rate 9.9%, OR 1.22, 95% CI 1.00–1.48, P<0.05). Moving from the existing reminder to the more effective costs message would result in 5,800 fewer missed appointments per year in the National Health Service Trust in question, at no additional cost. The study’s main limitations are that it took place in a single location in England, and that it required accurate phone records, which were only obtained for 20% of eligible patients. We conclude that missed appointments can be reduced, for no additional cost, by introducing persuasive messages to appointment reminders. Future studies could examine the impact of varying reminder messages in other health systems.
Citation: Hallsworth M, Berry D, Sanders M, Sallis A, King D, Vlaev I, et al. (2015) Stating Appointment Costs in SMS Reminders Reduces Missed Hospital Appointments: Findings from Two Randomised Controlled Trials. PLoS ONE 10(9): e0137306. https://doi.org/10.1371/journal.pone.0137306
Editor: Vineet Gupta, University of California San Diego, UNITED STATES
Received: February 15, 2015; Accepted: August 13, 2015; Published: September 14, 2015
Copyright: © 2015 Hallsworth et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The Centre for Health Policy at Imperial College provided funds to iPlato to permit the development of the randomization procedure. No other funds were received by the authors in connection with this trial. No funding bodies had any role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work. The authors have declared that no competing interests exist.
Introduction
“Did Not Attends” (DNAs) occur when a patient unexpectedly fails to attend an appointment [1]. In 2012–13, around 5.5 million National Health Service (NHS) outpatient appointments were missed in England (9.3% of the total) [2]. Whilst it is difficult to establish the exact financial impact of DNAs, a recent estimate claimed that missed first outpatient appointments cost the NHS up to £225 million in 2012–13 [3]. DNAs lead to worse care for patients, the inefficient use of staff, and increased waiting times [4] [5]. Hospitals may also engage in complex compensating behaviours, such as overbooking, which introduce problems of their own [6].
Survey evidence indicates that the main reason for a DNA is the patient forgetting their appointment (although we recognise that other factors, such as dissatisfaction with standards of care, are also important) [7] [8] [9]. Attention has therefore focused on the effective use of reminders. Systematic reviews have found that telephone and Short Message Service (SMS) reminders significantly reduce non-attendance, although a recent Cochrane Review judged that the quality of existing evidence is only of low to moderate quality [10] [11]. There is also evidence that SMS reminders are more cost-effective than telephone reminders [12]. The use of SMS reminders is widespread in the NHS: for example, approximately five million were sent via NHS Mail per monthin 2013 and 2014 [13] [14].
There is, however, no evidence on whether or how the effectiveness of these reminders is affected by their content [10]. Other studies have shown that compliance can be increased by even small changes to the phrasing of messages [15] [16]. Therefore it is possible that changes to reminder wording could lead to savings and improved patient care for no additional cost. Given the scale of SMS usage, this is a significant opportunity for service improvement. Accordingly, we undertook two randomised controlled trials to test the impact of rephrasing appointment reminders on DNA rates amongst hospital outpatients in a London NHS Trust.
Ethics and Registration
Ethical approval was obtained from a Research Ethics Committee of the NHS National Research Ethics Service (NHS REC 13/NW/0508). Individual informed consent for participation was not obtained, as approved by the Committee. Obtaining consent would introduce burdens to the patient (larger than the intervention itself); obtaining informed consent would cause serious practical problems that would undermine the trial results (particularly for the control group); and the risk of harm was low, since the intervention merely consisted of small modifications to existing routine processes.
This trial was registered on ISRCTN after it had commenced (ISRCTN 49432571 http://www.controlled-trials.com/ISRCTN49432571/). This is because the study team received guidance in its REC approval letter that there was a period of six weeks from commencement within which to register the trial. All other SMS trials the authors are involved in are registered.
Trial One Methods
Trial design
The trial took place in an NHS Trust (Barts Health) that used an SMS provider (iPlato) to send outpatient appointment reminders. Participants were all patients with a valid mobile telephone number and an outpatient appointment in November and December 2013 and January 2014 in at least one of five specialities: rheumatology, ophthalmology, gastroenterology, neurology, and cardiology. These specialities were chosen because they were not currently the subject of any other initiatives to reduce DNAs, apart from the SMS reminders already in use.
Appointments were randomly allocated to one of four reminder messages, which were issued five days in advance. Three of these reminder messages were created by the study team in order to test the hypotheses set out below; one was the existing reminder message used by the Trust. We hypothesized that these new reminder messages would lead to DNA rates that were significantly lower than those produced by the existing message. To test this hypothesis, records of message assignment were compared against records of whether a DNA subsequently occurred. The Trust recorded a DNA as occurring if a patient did not present at an appointment and did not cancel the appointment in advance. It is important to note that none of the new reminder messages exceeded the 160 character SMS limit, and therefore costs were kept constant.
Interventions
Table 1 sets out the text of the messages. The then current reminder message was retained as the control intervention. The study team created three new reminder messages: “Easy Call”, “Social Norms”, and “Specific Costs”. The main purpose of the “Easy Call” message was to include the telephone number for cancelling appointments in the body of the SMS reminder, as opposed to referring the patient to their appointment letter. This change was based on the well-established findings that even small reductions in the effort required to perform an action can have a significant impact on behaviour [17].
The “Social Norms” message communicated the fact that the dominant social norm was to attend an appointment. There is evidence from other fields that people overestimate the extent to which others perform acts that are non-optimal (or which cause harm to others), and that correcting these perceptions can change behaviour [18] [19]. Further, the specific tactic of referring to high attendance rates has been shown to reduce DNAs in a primary care setting (although using different media) [20]. Note that we retain the telephone number in this and subsequent variations.
The “Specific Costs” message noted the cost of a missed appointment. The patient may not be aware that missing an appointment incurs a cost. Even if they are, the cost may not be salient, since it is likely to be seen as just an “opportunity cost”–i.e. the loss of an opportunity to do something more productive [21]. There is emerging evidence that making such costs salient may influence behaviour. For example, giving clinicians feedback on the cost of discretionary diagnostic tests influences subsequent demand for tests [22]. Therefore, there are good reasons to examine the effect of presenting the potential cost of a missed appointment. To identify the cost figure, we obtained the 2012–13 NHS tariff for initial and follow-up appointments in all five specialities. Across all categories, the mean cost was £161. To reflect the fact that this was a simple average, it was appropriate to represent the figure as an approximation.
Power calculations
Our decision to create four treatment groups was informed by a power analysis. On the basis of past appointment data, we calculated that the five specialities dealt with approximately 17,000 appointments per month. We also observed that SMS messages were being issued for approximately 20% of appointments in the specialities, meaning that around 3,400 appointments would fall into the sample each month. On the basis of previous message-based studies, we aimed to detect a 1.5 percentage point change in the DNA rate (representing 0.16 standard deviations) with 80% power. These calculations indicated that three months would be required to collect the 10,000 cases needed to obtain this power.
Randomisation
The randomisation was conducted as follows. The SMS provider iPlato applied a Mersenne Twister computer-generated randomisation syntax to automatically allocate appointments to reminder messages [23]. This was a simple randomisation procedure with no blocking. None of the researchers or hospital staff took part in the randomisation or were able to access allocation details until the data were provided at the end of the trial. Randomisation was applied to appointments, rather than to participants, which means that participants could receive the same or different reminder(s) over the course of the study.
Analysis
To analyse the main trial outcomes, we applied a binary logistic regression model using STATA version 13. In line with our hypothesis, the model took the message groups as treatment variables, with the existing message used by the Trust as the constant or reference group. The presence or absence of a DNA, as recorded by the Trust, was the binary outcome variable. We added in age and gender to the model, and interacted these factors with the different message treatments. We also conducted subgroup analysis by speciality, although the study was not powered for this analysis. We conducted secondary analyses in order to assess the impact of the messages on levels of appointment attendance and patient cancellation, although we did not have directional hypotheses for these outcomes. These secondary analyses consisted of two further binary logistic regressions, one with appointment attendance as the binary outcome variable, and one with patient cancellation as the binary outcome variable. Finally, we conducted a chi squared omnibus test on our three non-control messages, in order to determine if their combined effect was statistically significant.
Trial One Results
Of the 37,183 appointments occurring in the specialities during Trial One, 10,111 had mobile phone records that permitted an SMS reminder to be sent (Fig 1). Table 2 gives the results for the main trial outcomes in raw percentages, alongside key descriptive statistics. Table 3 gives the regression outputs for the primary and secondary analyses. The Specific Costs message produced a DNA rate of 8.4% (206/2445), compared to a rate of 11.1% (284/2566) for the Control message (OR 0.74, 95% CI 0.61–0.89, P<0.01). No other messages had a significant effect on DNA rates. Fig 2 shows these results as a graph. Interestingly, the Social Norms message led to more appointment cancellations (Control 8.8%, 225/2566, Social Norms 10.5%, 268/2541, OR 1.23, 95% CI 1.02–1.48, P = 0.03).
The Specific Costs message produced a DNA rate of 8.4%, compared to a rate of 11.1% for the Control message (OR 0.74, 95% CI 0.61–0.89, P<0.01). None of the other messages had a significant effect on this outcome measure.
Table 3 shows that the coefficients remain stable after controlling for age and gender, although it is worth noting that these covariates have separate effects on the outcome variables: men are more likely to attend and less likely to cancel; older patients are less likely to DNA and more likely to attend. There are no significant interactions between message treatments and age or gender, and we have not presented these results for reasons of space. Table 4 shows that the results for the DNA outcome are sustained across the five different specialities. Furthermore, including specialities as covariates in the regression does not affect the main results. Finally, the chi squared omnibus test on our three non-control messages produced a statistically significant result at the 0.05 level (p = 0.02).
Trial Two Methods
Following the results from Trial One, we amended our original trial protocol to allow us to conduct a second trial, which built on the results described above. This trial took place in March, April and May 2014. All other aspects of the trial design and methods are identical to Trial One. We carried out the same analytic procedures as in Trial One, with the exception that the Specific Costs message (i.e. the best-performing message from Trial One) was used as the control intervention.
Table 5 gives the four messages included in Trial Two. As noted, this trial retained the Specific Costs message, while introducing three new messages: “General Costs”, “Empathy” and “Recording”. The “General Costs” message was constructed to present the concept of costs in more general terms. Creating a more general message is attractive for practical reasons, since it requires less effort to adopt at scale (specific costs do not have to be calculated). However, studies have shown that tailored messages are more likely to attract attention and trigger conscious thought [24] [25]. We therefore hypothesized that the General Costs message would be less effective at preventing DNAs than the Specific Costs message.
The “Empathy” message reminded the recipient of the situation of others. There is much evidence that situationally-induced empathy (asking someone to “put yourself in others’ shoes”) leads to an increase in pro-social behaviour [26] [27]. In this case, missing an appointment often means unnecessary delay to the treatment of others, even if some form of overbooking is taking place. The final “Recording” message simply pointed out that the hospital records missed appointments. Evidence from other fields shows that increasing “visibility” of an individual’s actions through creating data records (and notifying them of that fact) increases pro-social behaviour [28] [29]. We did not have a directional hypothesis for the effect of the Empathy and Recording messages compared to the Specific Costs message.
Trial Two Results
Of the 38,989 appointments occurring in the specialities during Trial Two, 9,848 had mobile phone records that permitted an SMS reminder to be sent (Fig 3). Table 6 gives the results for the main trial outcomes in raw percentages alongside key descriptive statistics, while Table 7 gives the regression outputs. Trial Two replicates the Specific Costs result from Trial One, in that it produces a DNA rate (8.2%, 203/2461) that is similar in absolute terms to the result from the previous trial, while also being significantly lower than those resulting from the two of the three other messages tested.
As hypothesized, the General Costs message was less effective at preventing DNAs than the Specific Costs message: it produced a DNA rate of 9.9% (252/2553), compared to 8.2% for the Specific Costs message (OR 1.22, 95% CI 1.00–1.48, P = 0.046). The Empathy message was also significantly less effective than the Specific Costs message (10.7%, 261/2439, OR 1.33, 95% CI 1.10–1.62, P<0.01). The Recording message produced a DNA rate of 9.6% (230/2395), which is not significantly higher than the Specific Costs message, although it was close to significance at the 0.05 level (OR 1.18, 95% CI 0.97–1.44, P = 0.07). Fig 4 shows the main treatment effects in a graph. Table 7 shows that there are negligible changes to the results after controlling for age and gender, although it is worth noting that these covariates have separate effects on the DNA variable: men and younger patients are significantly more likely to DNA. There are no significant interactions between message treatments and age or gender, and therefore we have not presented these results. As Table 8 shows, the overall pattern of results is sustained when the data are analysed by speciality, with the exception of Neurology (the smallest group). Furthermore, including specialities as covariates in the regression does not affect the main results. Finally, a chi squared omnibus test on our three non-control messages produced a statistically significant result at the 0.05 level (p = 0.032).
The General Costs message was less effective at reducing DNAs than the Specific Costs message: it produced a DNA rate of 9.9%, compared to 8.2% for the Specific Costs message (OR 1.22, 95% CI 1.00 to 1.48, P = 0.046).
Discussion
These results show that the wording of SMS reminders significantly affects the extent to which patients miss, attend or cancel outpatient appointments. Moreover, the results show that presenting the specific tariff cost of the appointment produces a DNA rate that is approximately three percentage points lower than for other messages, a result that we replicate. The fact that there were no significant differences between the Control and Social Norms messages means we can be confident that the effect of the Specific Costs is not being driven by the “We are expecting you” message framing, or the presence of the telephone number. The changes were not anticipated to produce adverse effects for patients, not least because they retained the freedom to attend, cancel or not attend their appointment. Based on our analysis of the available outcome data, there is no reason to believe that harms had occurred. It is worth noting that the Social Norms message significantly increased the level of patient cancellations; however, we cannot interpret this as a poor outcome that increases harm, since a patient cancellation is one route to avoiding a missed appointment and attending at a future date.
The results indicate that the wording of reminders can reduce DNA rates by around a quarter in relative terms. These treatment effects should be placed in context. As already noted, in Trial Two the odds ratio between the Specific Costs and Empathy messages was 1.33 (95% CI 1.10–1.61). This compares favourably with the summary effect of 1.48 for SMS reminders per se reported by a recent meta-analysis [30] [11]. In crude terms, this comparison suggests that the content of SMS reminders may constitute a large part of their overall effect. The more general implication is that health system performance improvements do not depend on technical solutions alone.
Newly-worded messages constituted the sole element of this intervention; no other aspects of patient care were altered. It should be noted that the reminders were sent a relatively long time before the appointment (five days) compared to those in previous studies, which were generally sent between one and three days in advance [10]. Having a relatively long period between the message and appointment is likely to increase the possibility that the reminder effect decays in that period. Ideally, we would have issued the messages one to three days in advance, but we were not able to alter the existing schedule.
These results have clear policy implications. We calculate that the lower DNA rate produced by the Specific Costs message (compared to the Control message) in Trial One would result in 5,800 fewer missed appointments for Barts Health across all outpatient specialities, if sustained over the course of a year. This figure was obtained by extrapolating the total number of appointments from the six months of available data to estimate the total number of appointments for a year. The 2.6% marginal effect was then applied to this estimated total. The estimated number of appointments is based on a mobile phone coverage rate of 20%, and thus it may be conservative figure. On the same basis, full phone record coverage could result in 28,900 fewer missed appointments for the Trust annually. It also worth noting that a 2.6 percentage point change in the DNA rate across all outpatient appointments in England (assuming 20% mobile phone coverage) equates to 400,000 fewer DNAs annually. However, we wish to stress that this estimate is dependent on several factors, particularly on the extent to which the result generalizes to other specialities, and the volume of appointments in each speciality. Since these figures are extrapolations they should be treated with caution.
The wider implications of this trial are that healthcare providers with SMS technology looking to reduce DNAs should consider adopting the messages above, or testing their own wordings. More generally, there is evidence that they should ensure messages contain specific, salient information in order to maximise their effectiveness. If the 160 character limit is not exceeded, then there is no additional cost to such a change. In addition, improving mobile phone records (to extend the use of SMS messages) may be an inexpensive way of reducing DNAs.
These trials have several limitations. Scaling up the use of these messages is dependent on Trusts possessing accurate records of mobile telephone numbers. As noted above, SMS reminders were only issued to 20% of individuals with appointments in the study period. We cannot observe whether the populations with or without accurate mobile number telephone records differ systematically from each other. We cannot rule out the possibility that the population who received the SMS reminders may have been more sensitive to the intervention than those who did not (which is another reason to be cautious of the extrapolations above). The study team were also unable to access records about the proportion of messages that were actually read or received, although the randomised nature of the study means we can retain confidence in our overall conclusions.
In addition, it is uncertain how far the effect of these messages can be replicated in different regions and different health systems. The trial took place in a London Trust. London sees higher DNA rates than other parts of the UK, which means that the effect of these interventions may be smaller in those areas [2]. However, it should be noted that the best-performing message produced a DNA rate of 8.2%, which is below the 9.3% average rate for England. More broadly, it is unclear whether mentioning costs to the health system is an effective strategy in health systems with different funding structures from that of the NHS. The generalizability of such messages is a pressing area for future research.
Acknowledgments
We would like to thank Tobias Alpsten, Greg Bird, Martin Rowden, Lynn Underhill, Lynne Fenwick, Tabitha Brufal, and Barts Health and iPlato collectively. Hannah Behrendt provided excellent research support. The views expressed are those of the authors and not necessarily those of the Department of Health, Public Health England, the Cabinet Office, or the UK Government.
Author Contributions
Conceived and designed the experiments: MH DB MS AS DK IV AD. Performed the experiments: MH DB AS. Analyzed the data: MS MH. Wrote the paper: MH DB MS AS DK IV AD.
References
- 1.
Health and Social Care Information Centre. Hospital Outpatient Activity 2012–13: Reporting Definitions. 2013 Dec 12 [cited 22 September 2014]. Available: http://www.hscic.gov.uk/catalogue/PUB13005.
- 2.
NHS England. Quarterly Hospital Activity Data. 2014 Available: http://www.england.nhs.uk/statistics/statistical-work-areas/hospital-activity/quarterly-hospital-activity/qar-data/. Accessed: 22 September 2014.
- 3.
National Audit Office. NHS waiting times for elective care in England. London: National Audit Office; 2014. Available: http://www.nao.org.uk/wp-content/uploads/2014/01/NHS-waiting-times-for-elective-care-in-England.pdf.
- 4. Karter AJ, Parker MM, Moffet HH, Ahmed AT, Ferrara A, Liu JY, et al. Missed appointments and poor glycemic control: an opportunity to identify high-risk diabetic patients. Med Care 2004;42: 110–115. pmid:14734947
- 5. Murray M. Patient care: access. BMJ 2000;320: 1594. pmid:10845975
- 6. Laganga LR, Lawrence SR. Clinic overbooking to improve patient access and increase provider productivity. Decis Sci 2007;38: 251–276.
- 7. Neal RD, Hussain-Gambles M, Allgar VL, Lawlor DA, Dempsey O. Reasons for and consequences of missed appointments in general practice in the UK: questionnaire survey and prospective review of medical records. BMC Fam Pract 2005;6: 47–52. pmid:16274481
- 8. Murdock A, Rodgers C, Lindsay H, Tham TCK. Why do patients not keep their appointments? Prospective study in a gastroenterology outpatient clinic. J R Soc Med 2002;95: 284–286. pmid:12042374
- 9. Van Baar JD, Joosten H, Car J, Freeman GK, Partridge MR, Van Weel C, et al. Understanding reasons for asthma outpatient (non)-attendance and exploring the role of telephone and e-consulting in faciliating access to care: exploratory qualititative study. Qual Saf Health Care 2006;15: 191–195. pmid:16751469
- 10. Gurol-Urganci I, de Jongh T, Vodopivec-Jamsek V, Atun R, Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database Syst Rev 2013;12: CD007458. pmid:24310741
- 11. Hasvold PE, Wootton R. Use of telephone and SMS reminders to improve attendance at hospital appointments: a systematic review. J Telemed Telecare 2011;17: 358–364. pmid:21933898
- 12. Junod Perron N, Dao MD, Righini NC, Humair J-P, Broers B, Narring F, et al. Text-messaging versus telephone reminders to reduce missed appointments in an academic primary care clinic: a randomized controlled trial. BMC Health Serv Res 2013;13: 125131.
- 13.
Health and Social Care Information Centre (2013) NHSmail user survey results 2013. 2013. Available: http://systems.hscic.gov.uk/nhsmail/service/survey2013.pdf. Accessed: 22 September 2014.
- 14.
Health and Social Care Information Centre (2014) NHSmail usage statistics, March 2014. 2014. Available: http://systems.hscic.gov.uk/nhsmail/service. Accessed: 22 September 2014].
- 15. Haynes LC, Green DP, Gallagher R, John P, Torgerson DJ. Collection of Delinquent Fines: An Adaptive Randomized Trial to Assess the Effectiveness of Alternative Text Messages. J Policy Anal Manag 2013;32: 718–730.
- 16. Hallsworth M, List J, Metcalfe R, Vlaev I. The Behavioralist As Tax Collector: Using Natural Field Experiments to Enhance Tax Compliance. NBER Working Paper 2014;20007.
- 17. Hawton K, Bergen H, Simkin S, Dodd S, Pocock P, Bernal W, et al. Long term effect of reduced pack sizes of paracetamol on poisoning deaths and liver transplant activity in England and Wales: interrupted time series analyses. BMJ 2013;346: f403. Available: http://www.bmj.com/highwire/filestream/629698/field_highwire_article_pdf/0/bmj.f403.full.pdf pmid:23393081
- 18. Perkins HW. Social norms and the prevention of alcohol misuse in collegiate contexts. J Stud Alcohol Suppl 2002;14: 164–172. pmid:12022722
- 19. Wenzel M. Misperceptions of social norms about tax compliance: from theory to intervention. J Econ Psychol 2005;26: 862–883.
- 20. Martin SJ, Bassi S, Dunbar-Rees R. Commitments, norms and custard creams–a social influence approach to reducing did not attends (DNAs). J R Soc Med 2012;105: 101–104. pmid:22408079
- 21. Frederick S, Novemsky N, Wang J, Dhar R, Nowlis S. Opportunity Cost Neglect. J Consum Res 2009;36: 553–561.
- 22. Fogarty AW, Sturrock N, Premji K, Prinsloo P. Hospital clinicians’ responsiveness to assay cost feedback: a prospective blinded controlled intervention study. JAMA Intern Med 2013;173: 1654–1655. pmid:23897527
- 23. Matsumoto M, Nishimura T. Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Trans Model Comput Simul 1998;8: 3–30.
- 24. Kreuter MW, Wray RJ. Tailored and targeted health communication: strategies for enhancing information relevance. Am J Health Behav 2003;27: S227–S232. pmid:14672383
- 25.
Petty RE, Haugtvedt CP, Smith SM. Elaboration as a determinant of attitude strength: creating attitudes that are persistent, resistant, and predictive of behavior. In: Petty RE, Krosnick JA, editors. Attitude strength: Antecedents and consequences. Mahwah, NJ: Lawrence Erlbaum; 1995. Vol. 4. pp. 93–130.
- 26. Eisenberg N. Emotion, Regulation, and Moral Development. Annu Rev Psychol 2000;51: 665–697. pmid:10751984
- 27. Penner LA, Dovidio JF, Piliavin JA, Schroeder DA. Prosocial behavior: multilevel perspectives. Annu Rev Psychol 2005;56: 365–392. pmid:15709940
- 28. Iyer GS, Reckers PMJ, Sanders DL. Increasing tax compliance in Washington state: a field experiment. Natl Tax J 2010;63: 7–32.
- 29.
Slemrod J, Thoresen TO, Bø EE. Taxes on the Internet: Deterrence effects of public disclosure. 2013. CESifo Working Paper: Public Finance.
- 30. Guy R, Hocking J, Wand H, Stott S, Ali H, Kaldor J. How effective are short message service reminders at increasing clinic attendance? A meta-analysis and systematic review. Health Serv Res 2012;47: 614–632. pmid:22091980