Figures
Abstract
The Consumer Assessment of Healthcare Providers and Systems Clinician & Group Survey (CG-CAHPS) is one of the most widely studied and endorsed patient experience measures for ambulatory care. This study aimed to develop a Japanese CG-CAHPS and examine its psychometric properties. We evaluated the structural validity, criterion-related validity, internal consistency reliability, and site-level reliability of the scale. Data were analyzed for 674 outpatients aged 18 years or older in 11 internal medicine clinics. The confirmatory factor analysis supported the scale’s structural validity and the same composites (Access, Provider Communication, Care Coordination, and Office Staff) as that of the original CG-CAHPS. All site-level Pearson correlation coefficients between the Japanese CG-CAHPS composites and overall provider rating exceeded the criteria. Results of item-total correlations and Cronbach’s alpha indicated adequate internal consistency reliability. We developed the Japanese CG-CAHPS and examined its validity and reliability to measure the quality of ambulatory care based on patient experience. The results of the Japanese CG-CAHPS survey will provide useful information to providers, organizations, and policy makers for achieving a patient-centered healthcare system in Japan.
Citation: Aoki T, Taguchi K, Hama E (2021) Development and psychometric properties of the Japanese Consumer Assessment of Healthcare Providers and Systems Clinician & Group Survey (CG-CAHPS). PLoS ONE 16(4): e0250843. https://doi.org/10.1371/journal.pone.0250843
Editor: Stefan Hoefer, Medical University Innsbruck, AUSTRIA
Received: August 15, 2020; Accepted: April 15, 2021; Published: April 28, 2021
Copyright: © 2021 Aoki et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: This work was supported by JSPS KAKENHI Grant Number JP20K18849. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The EPARK provided support in the form of salaries for authors [KT and EH], but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.
Competing interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: KT and EH are an employee of EPARK. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Introduction
The quality assessment of patient-centeredness from the patient’s perspective is an important aspect of quality of health care [1]. In recent years, better patient experience has been recognized as one of the crucial goals of healthcare alongside population health and per capita cost [2]. Patient experience is integrally tied to the principles and practices of patient- and family- centered care. Embedded within patient experience is a focus on individualized care and tailoring services to meet patients’ needs and engage them as partners in their care [3]. Numerous studies have shown that better patient experience is positively associated with patient health outcomes, patient safety, and patient behaviors across a wide range of disease areas and settings [4–7].
The Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Clinician & Group Survey (CG-CAHPS) is one of the most widely studied and endorsed patient experience measures for ambulatory care [8]. This standardized scale was developed by the Agency for Healthcare Research and Quality (AHRQ) and its validity and reliability were examined [9–11]. Currently, in the United States, CG-CAHPS results are widely used as quality measures in accountability initiatives and to stimulate, guide, and monitor quality improvement efforts [12]. For example, results from the CG-CAHPS have been reported on the Center for Medicare & Medicaid Services Physician Compare website, and insurers are also increasingly including patient experience data in pay-for-performance programs.
In contrast, in Japan, voluntary activities for the assessment of patient experience have just begun in limited settings, and systematic approaches for quality assessment and improvement based on patient experience measures are still unestablished. Only a few standardized scales, which have been confirmed psychometric properties, are available to assess patient experience in Japan [13–15]. Patient experience scales, which are tailored to different care settings, are needed to help identify aspects of care that can be targeted to improve patient experience. Accordingly, the present study aimed to develop a Japanese version of the CG-CAHPS version 3.0, which asks patients about their experiences with care from an ambulatory care provider over a six-month period, and to examine its structural validity, criterion-related validity, internal consistency reliability, and site-level reliability.
Materials and methods
Design, setting, and participants
The data used in this study were collected from a multicenter cross-sectional survey in 11 internal medicine clinics from February to March 2020. The 11 participating clinics, which were cooperating facilities of the Institute of Quality and Time of Healthcare, voluntarily took part in the survey and are in urban areas in the Tokyo Metropolis and Kanagawa Prefecture, with all the clinics being privately owned and managed. In Japan, clinics are generally run by one full-time physician, nurses, and medical assistants, and they provide outpatient services and possibly home care. Outpatient services in the participating clinics were provided by physicians trained in an internal medicine–based residency program in general hospitals.
Independent surveyors distributed a self-administered questionnaire in the waiting room directly to all outpatients aged 18 years or older who visited one of the participating clinics within the survey period using a continuous sampling method. The survey period was two consecutive days. Patients who were assessed by the surveyors as unable to respond to the questionnaire due to severe physical or mental disorders were excluded. According to the sampling criteria of the original CG-CAHPS version 3.0, of the survey respondents, those who responded that they have had at least one visit with the provider in the last six months were included in our analyses.
This study was approved by the Ethics Committee of Kyoto University Graduate School of Medicine (approval number R2331). Informed consent was not obtained because this study is an analysis of existing anonymous data. Data were anonymized before access.
Measures
CG-CAHPS.
The original CG-CAHPS version 3.0 (the latest version as of February 2020) is an 18-item tool comprising four composites, a global rating, and five screening items (Q1, Q3, Q5, Q12, and Q15) [8]. The composites are Access (Q2, Q4, and Q6), Provider Communication (Q7, Q8, Q10, and Q11), Care Coordination (Q9, Q13, and Q16), and Office Staff (Q17 and Q18). The global rating is Rating of Provider (Q14).
Permission to translate the CG-CAHPS into Japanese was granted by the AHRQ. Following the translation guidelines for CAHPS® surveys provided by AHRQ [16], translation of the CG-CAHPS into Japanese was performed through the following steps. First, two bilingual translators with experience in translating survey instruments conducted forward translations from English to Japanese. Two forward translations were performed independently. The two translations were then reviewed by a translation reviewer who is a native speaker of Japanese and has experience in translating survey instruments. After reviewing the translation, the reviewer prepared a reconciled version of the translation. A committee consisting of the two translators and the reviewer then discussed and prepared the final version. Revisions were made to the reconciled version necessary for cross-cultural adaptation. The final wording of each survey item and response option was determined by consensus (S1 Appendix).
The CG-CAHPS survey uses multiple response formats: four-point scales (1 = never, 2 = sometimes, 3 = usually, and 4 = always), and a global rating scale (0 = worst to 10 = best). To make the results easier to understand, we converted all scales to scores ranging from 0 to 100 using the following formula:
Converted Score = 100 * (Respondent’s selected response value–Minimum response value on the scale) / (Maximum response value—Minimum response value)
In the Japanese version, assuming the convergence in each composite as in the original version, the score for each of the four composites was computed as the mean value for all converted scores in the scale that would fall in the range of 0–100 points, with higher scores indicating better performance.
Statistical analysis
We validated the Japanese CG-CAHPS through the following steps:
First, we carried out a confirmatory factor analysis based on Pearson correlations using the maximum likelihood estimation to evaluate the structural validity of the Japanese CG-CAHPS composites. In the factor analysis based on Pearson correlations, we hypothesized the same factor structure (four-factor solution) as that of the original CG-CAHPS. In addition, we also conducted a confirmatory factor analysis based on polychoric correlations using the diagonally weighted least squares estimation to address the concern that Pearson correlations may not be appropriate. The factor analysis based on polychoric correlations was limited to three factors: Provider Communication, Care Coordination, and Office Staff. Because many participants responded "Not applicable" to the Access items, it was difficult to estimate a polychoric correlation matrix in the four-factor structure. The appropriateness of the resulting structure was determined by examining if factor loadings were 0.40 or greater [17]. Model fitness was assessed using the comparative fit index (CFI), root mean square error of approximation (RMSEA), and standardized root mean square residual (SRMR). For CFI, a value of > 0.95 indicates goodness of fit. Previous studies suggest that models with RMSEA < 0.06 and SRMR < 0.08 are representative of models with good fit [18].
Second, we used the Japanese CG-CAHPS composite scores and the overall provider rating to examine criterion-related validity. Validity was assessed using Pearson correlation coefficients with each Japanese CG-CAHPS composite to predict the Rating of Provider (0 = Worst to 10 = Best) of the scale at the site-level. A correlation coefficient greater than 0.30 was considered meaningful [19]. Site-level correlations are a more important criterion for measurement than patient-level correlations because the former are benchmarking tools to compare one provider or facility with another. To examine site-level correlations, we used each provider’s mean score on CG-CAHPS composites and the Rating of Provider.
Internal consistency reliability was examined by item-total correlations and Cronbach’s alpha. We also examined site-level reliability for each score using a linear mixed effect model to adjust for the case-mix variables: age, education, and self-rated health. Site-level reliability was estimated by using the following formula [20]:
Reliability = (Between-site variance) / [Between-site variance + (Within-site variance)/(Sample size for site)]
For a scale to be considered sufficiently reliable, an item-total correlation of 0.30 and Cronbach’s alpha and site-level reliability value of 0.70 are recommended [21].
Finally, descriptive statistics were performed on the Japanese CG-CAHPS scores, including the mean, standard deviation, and observed range. To deal with missing data, in the confirmatory factor analysis based on Pearson correlations, we used the full information maximum likelihood estimation to enable the use of information collected from participants with missing data. In the evaluation of confirmatory factor analysis based on polychoric correlations, criterion-related validity, and reliability, we conducted complete case analyses. All statistical analyses were conducted using R version 3.6.3 (R Foundation for Statistical Computing, Vienna, Austria; www.R-project.org).
Results
Of the total 818 eligible outpatients, 787 (96.2%) responded to the survey. Of these respondents, we analyzed the data from 674 patients who responded that they have had at least one visit with the provider in the last 6 months. Table 1 shows the participants’ characteristics.
Table 2 shows the participants’ responses to each item of the Japanese CG-CAHPS. The Top Box score for each item, which is the percentage of participants who provided the most positive responses on that item, ranged from 56.1% to 73.6%. Regarding the mean Top Box score for composites, the highest score was observed for Provider Communication (70.9%), while the lowest score was for Care Coordination (59.6%). The bottom box score, which is the percentage of participants with the least positive responses on the item, ranged from 0.8% to 7.2%.
Structural validity
Fig 1 shows the path diagrams of the confirmatory factor analysis based on Pearson correlations to assess the structural validity of four-factor structure of the Japanese CG-CAHPS composites. All factor loadings of each item onto each factor were above the 0.40 criteria, ranging from 0.48 to 0.90. The correlation coefficients among factors ranged from 0.51 to 0.89. Part of the model fit indices showed good fit (CFI = 0.940, RMSEA = 0.074, and SRMR = 0.067). The confirmatory factor analysis based on polychoric correlations indicated excellent goodness of fit (CFI = 0.998, RMSEA = 0.054, and SRMR = 0.047).
Squares are observed variables (items); ellipses are latent variables (factors), values on the single-headed arrows are standardized factor loadings, values on the double-headed arrows are correlation coefficients. CG-CAHPS = Consumer Assessment of Healthcare Providers and Systems Clinician & Group Survey.
Criterion-related validity
Table 3 shows the Pearson correlation coefficients between the Japanese CG-CAHPS composites and the Rating of Provider as an overall provider rating at the site-level. All correlations exceeded the 0.30 criterion. Provider Communication (r = 0.85) had the highest correlation with the overall rating.
Reliability and descriptive statistics
Table 4 indicates the score distribution, internal consistency reliability, and site-level reliability for the Japanese CG-CAHPS. All item-total correlations were above the 0.30 criteria, ranging from 0.31 to 0.92. For Access, Provider Communication, and Office Staff, the Cronbach’s alpha and site-level reliability were above 0.70. However, for Care Coordination, the Cronbach’s alpha and site-level reliability did not exceed the 0.70 criterion.
Descriptive statistics showed that the highest scored scale was Provider Communication (mean score = 88.1), and the most poorly scored scale was Care Coordination (mean score = 78.6). The full range of possible scores was observed for all scales.
Discussion
We developed the Japanese CG-CAHPS in the form of a standardized scale for assessing the quality of ambulatory care from the patient’s perspective in Japan. In our multicenter study, the psychometric properties of the Japanese CG-CAHPS, including structural validity, criterion-related validity, internal consistency reliability, and site-level reliability were evaluated. Even in Japan, it is important to have valid and reliable measures for assessing patient experience in various settings. This scale could be used for quality improvement based on the assessment of patient experience with ambulatory care and for health services research in Japan.
The confirmatory factor analysis supported the scale’s structural validity and the same composites (Access, Provider Communication, Care Coordination, and Office Staff) as that of the original CG-CAHPS. Correlation coefficients between all Japanese CG-CAHPS composites and the overall provider rating for assessing criterion-related validity exceeded the meaningful value at the site-level. In internal consistency analyses, only Cronbach’s alpha for Care Coordination did not exceed the recommended value; however, all item-total correlations were greater than the cutoff value, which indicated acceptable internal consistency of the scales. The site-level reliability for Care Coordination was 0.68, slightly below the cutoff of 0.70, indicating that it is necessary to have more participants per provider to increase reliability to acceptable levels. The low internal consistency reliability for Care Coordination and the strongest correlation of Provider Communication with overall provider rating were consistent with the results of previous studies [10, 20].
The CG-CAHPS is one of the most widely studied patient experience scales for ambulatory care worldwide. The CG-CAHPS has been translated into many languages in order to be used in other countries so that comparisons of health service quality from the patient perspective can be made. In our study, the recovery rate for the questionnaire administered was very high, suggesting a low risk of selection bias.
However, the present study has several potential limitations. First, in this study, we evaluated the structural validity, criterion-related validity, and internal consistency reliability of the Japanese CG-CAHPS, other psychometric properties, including convergent and discriminant validity, test-retest reliability, and interpretability, have not been assessed [22]. These psychometric properties of the scale need to be evaluated in future studies. Second, our survey setting was restricted to urban areas and may not have sufficiently represented the Japanese national level. Therefore, the study results may have limited generalizability and a survey using the Japanese CG-CAHPS in other suburban and rural areas should be conducted.
Conclusions
We developed the Japanese CG-CAHPS and examined its validity and reliability to measure the quality of ambulatory care based on patient experience. The results of the Japanese CG-CAHPS survey will provide useful information to providers, organizations, and policy makers for achieving a patient-centered healthcare system in Japan.
Acknowledgments
We would like to thank Yukiko Matsuo (Ohana clinic Tokyo), So Ishii (Kudan-shita Eki-mae Coco Clinic), Tsuyoshi Sugimoto (Seint Clinic Ikebukuro Ekimae), Toshiyuki Kaneko (Tokyo skytree medical clinic) for their support during the study period.
References
- 1.
Institute of Medicine. Committee on Quality of Health Care in America. Crossing the quality chasm: A new health system for the 21st century. Washington, DC: National Academies Press; 2001.
- 2. Berwick DM, Nolan TW, Whittington J. The Triple Aim: Care, Health, And Cost. Health Aff. 2008;27(3):759–769. pmid:18474969
- 3. Wolf J, Niederhauser V, Marshburn D, LaVela SL. Defining patient experience. Patient Exp J. 2014;1:7–19.
- 4. Doyle C, Lennox L, Bell D. A systematic review of evidence on the links between patient experience and clinical safety and effectiveness. BMJ Open. 2013;3:e001570. pmid:23293244
- 5. Anhang Price R, Elliott MN, Zaslavsky AM, Hays RD, Lehrman WG, Rybowski L, et al. Examining the role of patient experience surveys in measuring health care quality. Med Care Res Rev. 2014;71:522–554. pmid:25027409
- 6. Aoki T, Miyashita J, Yamamoto Y, Ikenoue T, Kise M, Fujinuma Y, et al. Patient experience of primary care and advance care planning: A multicentre cross-sectional study in Japan. Fam Pract. 2017;34:206–212. pmid:28334740
- 7. Aoki T, Yamamoto Y, Ikenoue T, Kaneko M, Kise M, Fujinuma Y, et al. Effect of Patient Experience on Bypassing a Primary Care Gatekeeper: a Multicenter Prospective Cohort Study in Japan. J Gen Intern Med. 2018;33:722–728. pmid:29352418
- 8.
Agency for Healthcare Research and Quality. CAHPS Clinician & Group Survey. [Cited 2020 June 17]. Available from: https://www.ahrq.gov/cahps/surveys-guidance/cg/index.html
- 9.
Agency for Healthcare Research and Quality. Development of the CAHPS Clinician & Group Survey. [Cited 2020 June 17]. Available from: https://www.ahrq.gov/cahps/surveys-guidance/cg/about/Develop-CG-Surveys.html
- 10. Scholle SH, Vuong O, Ding L, et al. Development of and field test results for the CAHPS PCMH Survey. Med Care. 2012;50:S2–S10. pmid:23064272
- 11. Hays RD, Berman LJ, Kanter MH, et al. Evaluating the psychometric properties of the Cahps patient-centered medical home survey. Clin Ther. 2014;36: 689–696. pmid:24811752
- 12. Holt JM. Patient Experience in Primary Care: A Systematic Review of CG-CAHPS Surveys. J Patient Exp. 2019;6:93–102. pmid:31218253
- 13. Aoki T, Inoue M, Nakayama T. Development and validation of the Japanese version of Primary Care Assessment Tool. Fam Pract. 2015;33:112–117. pmid:26546033
- 14. Aoki T, Fukuhara S, Yamamoto Y. Development and validation of a concise scale for assessing patient experience of primary care for adults in Japan. Fam Pract. 2020;37:137–142. pmid:31325300
- 15. Kijima T, Akai K, Matsushita A, Hamano T, Onoda K, Yano S, et al. Development of the Japanese version of the general practice assessment questionnaire: Measurement of patient experience and testing of data quality. BMC Fam Pract. 2018;19:181. pmid:30486790
- 16.
Agency for Healthcare Research and Quality. Translating CAHPS Surveys. [Cited 2020 June 17]. Available from: https://www.ahrq.gov/cahps/surveys-guidance/helpful-resources/resources/cahpsGuidelines_Translation.html
- 17. Peterson RA. A Meta-Analysis of Variance Accounted for and Factor Loadings in Exploratory Factor Analysis. Marketing Letters. 2000;11:261–275.
- 18. Reeve BB, Hays RD, Bjorner JB, et al. Psychometric evaluation and calibration of health-related quality of life item banks: Plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Med Care. 2007;45:S22–S31. pmid:17443115
- 19. Cohen J. Statistical Power Analysis. Current Directions in Psychological Science. 1992;1:98–101.
- 20. Dyer N, Sorra JS, Smith SA, Cleary PD, Hays RD. Psychometric properties of the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Clinician and Group Adult Visit Survey. Med Care. 2012;50:S28–S34. pmid:23064274
- 21.
DeVellis RF. Scale Development: Theory and Applications. Newbury Park, Calif: Sage publications; 1991.
- 22. Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63(7):737–45. pmid:20494804