Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Development and psychometric properties of the Japanese Consumer Assessment of Healthcare Providers and Systems Clinician & Group Survey (CG-CAHPS)

  • Takuya Aoki ,

    Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Writing – original draft

    taoki@jikei.ac.jp

    Affiliations Division of Clinical Epidemiology, The Jikei University School of Medicine, Tokyo, Japan, Section of Clinical Epidemiology, Department of Community Medicine, Graduate School of Medicine, Kyoto University, Kyoto, Japan

  • Kuichiro Taguchi,

    Roles Conceptualization, Data curation, Investigation, Project administration, Resources

    Affiliation Institute of Quality and Time of Healthcare, Tokyo, Japan

  • Eiichi Hama

    Roles Conceptualization, Data curation, Investigation, Project administration, Resources

    Affiliation Institute of Quality and Time of Healthcare, Tokyo, Japan

Abstract

The Consumer Assessment of Healthcare Providers and Systems Clinician & Group Survey (CG-CAHPS) is one of the most widely studied and endorsed patient experience measures for ambulatory care. This study aimed to develop a Japanese CG-CAHPS and examine its psychometric properties. We evaluated the structural validity, criterion-related validity, internal consistency reliability, and site-level reliability of the scale. Data were analyzed for 674 outpatients aged 18 years or older in 11 internal medicine clinics. The confirmatory factor analysis supported the scale’s structural validity and the same composites (Access, Provider Communication, Care Coordination, and Office Staff) as that of the original CG-CAHPS. All site-level Pearson correlation coefficients between the Japanese CG-CAHPS composites and overall provider rating exceeded the criteria. Results of item-total correlations and Cronbach’s alpha indicated adequate internal consistency reliability. We developed the Japanese CG-CAHPS and examined its validity and reliability to measure the quality of ambulatory care based on patient experience. The results of the Japanese CG-CAHPS survey will provide useful information to providers, organizations, and policy makers for achieving a patient-centered healthcare system in Japan.

Introduction

The quality assessment of patient-centeredness from the patient’s perspective is an important aspect of quality of health care [1]. In recent years, better patient experience has been recognized as one of the crucial goals of healthcare alongside population health and per capita cost [2]. Patient experience is integrally tied to the principles and practices of patient- and family- centered care. Embedded within patient experience is a focus on individualized care and tailoring services to meet patients’ needs and engage them as partners in their care [3]. Numerous studies have shown that better patient experience is positively associated with patient health outcomes, patient safety, and patient behaviors across a wide range of disease areas and settings [47].

The Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Clinician & Group Survey (CG-CAHPS) is one of the most widely studied and endorsed patient experience measures for ambulatory care [8]. This standardized scale was developed by the Agency for Healthcare Research and Quality (AHRQ) and its validity and reliability were examined [911]. Currently, in the United States, CG-CAHPS results are widely used as quality measures in accountability initiatives and to stimulate, guide, and monitor quality improvement efforts [12]. For example, results from the CG-CAHPS have been reported on the Center for Medicare & Medicaid Services Physician Compare website, and insurers are also increasingly including patient experience data in pay-for-performance programs.

In contrast, in Japan, voluntary activities for the assessment of patient experience have just begun in limited settings, and systematic approaches for quality assessment and improvement based on patient experience measures are still unestablished. Only a few standardized scales, which have been confirmed psychometric properties, are available to assess patient experience in Japan [1315]. Patient experience scales, which are tailored to different care settings, are needed to help identify aspects of care that can be targeted to improve patient experience. Accordingly, the present study aimed to develop a Japanese version of the CG-CAHPS version 3.0, which asks patients about their experiences with care from an ambulatory care provider over a six-month period, and to examine its structural validity, criterion-related validity, internal consistency reliability, and site-level reliability.

Materials and methods

Design, setting, and participants

The data used in this study were collected from a multicenter cross-sectional survey in 11 internal medicine clinics from February to March 2020. The 11 participating clinics, which were cooperating facilities of the Institute of Quality and Time of Healthcare, voluntarily took part in the survey and are in urban areas in the Tokyo Metropolis and Kanagawa Prefecture, with all the clinics being privately owned and managed. In Japan, clinics are generally run by one full-time physician, nurses, and medical assistants, and they provide outpatient services and possibly home care. Outpatient services in the participating clinics were provided by physicians trained in an internal medicine–based residency program in general hospitals.

Independent surveyors distributed a self-administered questionnaire in the waiting room directly to all outpatients aged 18 years or older who visited one of the participating clinics within the survey period using a continuous sampling method. The survey period was two consecutive days. Patients who were assessed by the surveyors as unable to respond to the questionnaire due to severe physical or mental disorders were excluded. According to the sampling criteria of the original CG-CAHPS version 3.0, of the survey respondents, those who responded that they have had at least one visit with the provider in the last six months were included in our analyses.

This study was approved by the Ethics Committee of Kyoto University Graduate School of Medicine (approval number R2331). Informed consent was not obtained because this study is an analysis of existing anonymous data. Data were anonymized before access.

Measures

CG-CAHPS.

The original CG-CAHPS version 3.0 (the latest version as of February 2020) is an 18-item tool comprising four composites, a global rating, and five screening items (Q1, Q3, Q5, Q12, and Q15) [8]. The composites are Access (Q2, Q4, and Q6), Provider Communication (Q7, Q8, Q10, and Q11), Care Coordination (Q9, Q13, and Q16), and Office Staff (Q17 and Q18). The global rating is Rating of Provider (Q14).

Permission to translate the CG-CAHPS into Japanese was granted by the AHRQ. Following the translation guidelines for CAHPS® surveys provided by AHRQ [16], translation of the CG-CAHPS into Japanese was performed through the following steps. First, two bilingual translators with experience in translating survey instruments conducted forward translations from English to Japanese. Two forward translations were performed independently. The two translations were then reviewed by a translation reviewer who is a native speaker of Japanese and has experience in translating survey instruments. After reviewing the translation, the reviewer prepared a reconciled version of the translation. A committee consisting of the two translators and the reviewer then discussed and prepared the final version. Revisions were made to the reconciled version necessary for cross-cultural adaptation. The final wording of each survey item and response option was determined by consensus (S1 Appendix).

The CG-CAHPS survey uses multiple response formats: four-point scales (1 = never, 2 = sometimes, 3 = usually, and 4 = always), and a global rating scale (0 = worst to 10 = best). To make the results easier to understand, we converted all scales to scores ranging from 0 to 100 using the following formula:

Converted Score = 100 * (Respondent’s selected response value–Minimum response value on the scale) / (Maximum response value—Minimum response value)

In the Japanese version, assuming the convergence in each composite as in the original version, the score for each of the four composites was computed as the mean value for all converted scores in the scale that would fall in the range of 0–100 points, with higher scores indicating better performance.

Statistical analysis

We validated the Japanese CG-CAHPS through the following steps:

First, we carried out a confirmatory factor analysis based on Pearson correlations using the maximum likelihood estimation to evaluate the structural validity of the Japanese CG-CAHPS composites. In the factor analysis based on Pearson correlations, we hypothesized the same factor structure (four-factor solution) as that of the original CG-CAHPS. In addition, we also conducted a confirmatory factor analysis based on polychoric correlations using the diagonally weighted least squares estimation to address the concern that Pearson correlations may not be appropriate. The factor analysis based on polychoric correlations was limited to three factors: Provider Communication, Care Coordination, and Office Staff. Because many participants responded "Not applicable" to the Access items, it was difficult to estimate a polychoric correlation matrix in the four-factor structure. The appropriateness of the resulting structure was determined by examining if factor loadings were 0.40 or greater [17]. Model fitness was assessed using the comparative fit index (CFI), root mean square error of approximation (RMSEA), and standardized root mean square residual (SRMR). For CFI, a value of > 0.95 indicates goodness of fit. Previous studies suggest that models with RMSEA < 0.06 and SRMR < 0.08 are representative of models with good fit [18].

Second, we used the Japanese CG-CAHPS composite scores and the overall provider rating to examine criterion-related validity. Validity was assessed using Pearson correlation coefficients with each Japanese CG-CAHPS composite to predict the Rating of Provider (0 = Worst to 10 = Best) of the scale at the site-level. A correlation coefficient greater than 0.30 was considered meaningful [19]. Site-level correlations are a more important criterion for measurement than patient-level correlations because the former are benchmarking tools to compare one provider or facility with another. To examine site-level correlations, we used each provider’s mean score on CG-CAHPS composites and the Rating of Provider.

Internal consistency reliability was examined by item-total correlations and Cronbach’s alpha. We also examined site-level reliability for each score using a linear mixed effect model to adjust for the case-mix variables: age, education, and self-rated health. Site-level reliability was estimated by using the following formula [20]:

Reliability = (Between-site variance) / [Between-site variance + (Within-site variance)/(Sample size for site)]

For a scale to be considered sufficiently reliable, an item-total correlation of 0.30 and Cronbach’s alpha and site-level reliability value of 0.70 are recommended [21].

Finally, descriptive statistics were performed on the Japanese CG-CAHPS scores, including the mean, standard deviation, and observed range. To deal with missing data, in the confirmatory factor analysis based on Pearson correlations, we used the full information maximum likelihood estimation to enable the use of information collected from participants with missing data. In the evaluation of confirmatory factor analysis based on polychoric correlations, criterion-related validity, and reliability, we conducted complete case analyses. All statistical analyses were conducted using R version 3.6.3 (R Foundation for Statistical Computing, Vienna, Austria; www.R-project.org).

Results

Of the total 818 eligible outpatients, 787 (96.2%) responded to the survey. Of these respondents, we analyzed the data from 674 patients who responded that they have had at least one visit with the provider in the last 6 months. Table 1 shows the participants’ characteristics.

Table 2 shows the participants’ responses to each item of the Japanese CG-CAHPS. The Top Box score for each item, which is the percentage of participants who provided the most positive responses on that item, ranged from 56.1% to 73.6%. Regarding the mean Top Box score for composites, the highest score was observed for Provider Communication (70.9%), while the lowest score was for Care Coordination (59.6%). The bottom box score, which is the percentage of participants with the least positive responses on the item, ranged from 0.8% to 7.2%.

thumbnail
Table 2. Response to Japanese CG-CAHPS items (N = 674): Number (%).

https://doi.org/10.1371/journal.pone.0250843.t002

Structural validity

Fig 1 shows the path diagrams of the confirmatory factor analysis based on Pearson correlations to assess the structural validity of four-factor structure of the Japanese CG-CAHPS composites. All factor loadings of each item onto each factor were above the 0.40 criteria, ranging from 0.48 to 0.90. The correlation coefficients among factors ranged from 0.51 to 0.89. Part of the model fit indices showed good fit (CFI = 0.940, RMSEA = 0.074, and SRMR = 0.067). The confirmatory factor analysis based on polychoric correlations indicated excellent goodness of fit (CFI = 0.998, RMSEA = 0.054, and SRMR = 0.047).

thumbnail
Fig 1. Factor structure of Japanese CG-CAHPS (confirmatory factor analysis).

Squares are observed variables (items); ellipses are latent variables (factors), values on the single-headed arrows are standardized factor loadings, values on the double-headed arrows are correlation coefficients. CG-CAHPS = Consumer Assessment of Healthcare Providers and Systems Clinician & Group Survey.

https://doi.org/10.1371/journal.pone.0250843.g001

Criterion-related validity

Table 3 shows the Pearson correlation coefficients between the Japanese CG-CAHPS composites and the Rating of Provider as an overall provider rating at the site-level. All correlations exceeded the 0.30 criterion. Provider Communication (r = 0.85) had the highest correlation with the overall rating.

thumbnail
Table 3. Pearson correlation coefficients between Japanese CG-CAHPS composites and overall provider rating.

https://doi.org/10.1371/journal.pone.0250843.t003

Reliability and descriptive statistics

Table 4 indicates the score distribution, internal consistency reliability, and site-level reliability for the Japanese CG-CAHPS. All item-total correlations were above the 0.30 criteria, ranging from 0.31 to 0.92. For Access, Provider Communication, and Office Staff, the Cronbach’s alpha and site-level reliability were above 0.70. However, for Care Coordination, the Cronbach’s alpha and site-level reliability did not exceed the 0.70 criterion.

thumbnail
Table 4. Descriptive features, internal consistency reliability, and site-level reliability of Japanese CG-CAHPS (N = 674).

https://doi.org/10.1371/journal.pone.0250843.t004

Descriptive statistics showed that the highest scored scale was Provider Communication (mean score = 88.1), and the most poorly scored scale was Care Coordination (mean score = 78.6). The full range of possible scores was observed for all scales.

Discussion

We developed the Japanese CG-CAHPS in the form of a standardized scale for assessing the quality of ambulatory care from the patient’s perspective in Japan. In our multicenter study, the psychometric properties of the Japanese CG-CAHPS, including structural validity, criterion-related validity, internal consistency reliability, and site-level reliability were evaluated. Even in Japan, it is important to have valid and reliable measures for assessing patient experience in various settings. This scale could be used for quality improvement based on the assessment of patient experience with ambulatory care and for health services research in Japan.

The confirmatory factor analysis supported the scale’s structural validity and the same composites (Access, Provider Communication, Care Coordination, and Office Staff) as that of the original CG-CAHPS. Correlation coefficients between all Japanese CG-CAHPS composites and the overall provider rating for assessing criterion-related validity exceeded the meaningful value at the site-level. In internal consistency analyses, only Cronbach’s alpha for Care Coordination did not exceed the recommended value; however, all item-total correlations were greater than the cutoff value, which indicated acceptable internal consistency of the scales. The site-level reliability for Care Coordination was 0.68, slightly below the cutoff of 0.70, indicating that it is necessary to have more participants per provider to increase reliability to acceptable levels. The low internal consistency reliability for Care Coordination and the strongest correlation of Provider Communication with overall provider rating were consistent with the results of previous studies [10, 20].

The CG-CAHPS is one of the most widely studied patient experience scales for ambulatory care worldwide. The CG-CAHPS has been translated into many languages in order to be used in other countries so that comparisons of health service quality from the patient perspective can be made. In our study, the recovery rate for the questionnaire administered was very high, suggesting a low risk of selection bias.

However, the present study has several potential limitations. First, in this study, we evaluated the structural validity, criterion-related validity, and internal consistency reliability of the Japanese CG-CAHPS, other psychometric properties, including convergent and discriminant validity, test-retest reliability, and interpretability, have not been assessed [22]. These psychometric properties of the scale need to be evaluated in future studies. Second, our survey setting was restricted to urban areas and may not have sufficiently represented the Japanese national level. Therefore, the study results may have limited generalizability and a survey using the Japanese CG-CAHPS in other suburban and rural areas should be conducted.

Conclusions

We developed the Japanese CG-CAHPS and examined its validity and reliability to measure the quality of ambulatory care based on patient experience. The results of the Japanese CG-CAHPS survey will provide useful information to providers, organizations, and policy makers for achieving a patient-centered healthcare system in Japan.

Acknowledgments

We would like to thank Yukiko Matsuo (Ohana clinic Tokyo), So Ishii (Kudan-shita Eki-mae Coco Clinic), Tsuyoshi Sugimoto (Seint Clinic Ikebukuro Ekimae), Toshiyuki Kaneko (Tokyo skytree medical clinic) for their support during the study period.

References

  1. 1. Institute of Medicine. Committee on Quality of Health Care in America. Crossing the quality chasm: A new health system for the 21st century. Washington, DC: National Academies Press; 2001.
  2. 2. Berwick DM, Nolan TW, Whittington J. The Triple Aim: Care, Health, And Cost. Health Aff. 2008;27(3):759–769. pmid:18474969
  3. 3. Wolf J, Niederhauser V, Marshburn D, LaVela SL. Defining patient experience. Patient Exp J. 2014;1:7–19.
  4. 4. Doyle C, Lennox L, Bell D. A systematic review of evidence on the links between patient experience and clinical safety and effectiveness. BMJ Open. 2013;3:e001570. pmid:23293244
  5. 5. Anhang Price R, Elliott MN, Zaslavsky AM, Hays RD, Lehrman WG, Rybowski L, et al. Examining the role of patient experience surveys in measuring health care quality. Med Care Res Rev. 2014;71:522–554. pmid:25027409
  6. 6. Aoki T, Miyashita J, Yamamoto Y, Ikenoue T, Kise M, Fujinuma Y, et al. Patient experience of primary care and advance care planning: A multicentre cross-sectional study in Japan. Fam Pract. 2017;34:206–212. pmid:28334740
  7. 7. Aoki T, Yamamoto Y, Ikenoue T, Kaneko M, Kise M, Fujinuma Y, et al. Effect of Patient Experience on Bypassing a Primary Care Gatekeeper: a Multicenter Prospective Cohort Study in Japan. J Gen Intern Med. 2018;33:722–728. pmid:29352418
  8. 8. Agency for Healthcare Research and Quality. CAHPS Clinician & Group Survey. [Cited 2020 June 17]. Available from: https://www.ahrq.gov/cahps/surveys-guidance/cg/index.html
  9. 9. Agency for Healthcare Research and Quality. Development of the CAHPS Clinician & Group Survey. [Cited 2020 June 17]. Available from: https://www.ahrq.gov/cahps/surveys-guidance/cg/about/Develop-CG-Surveys.html
  10. 10. Scholle SH, Vuong O, Ding L, et al. Development of and field test results for the CAHPS PCMH Survey. Med Care. 2012;50:S2–S10. pmid:23064272
  11. 11. Hays RD, Berman LJ, Kanter MH, et al. Evaluating the psychometric properties of the Cahps patient-centered medical home survey. Clin Ther. 2014;36: 689–696. pmid:24811752
  12. 12. Holt JM. Patient Experience in Primary Care: A Systematic Review of CG-CAHPS Surveys. J Patient Exp. 2019;6:93–102. pmid:31218253
  13. 13. Aoki T, Inoue M, Nakayama T. Development and validation of the Japanese version of Primary Care Assessment Tool. Fam Pract. 2015;33:112–117. pmid:26546033
  14. 14. Aoki T, Fukuhara S, Yamamoto Y. Development and validation of a concise scale for assessing patient experience of primary care for adults in Japan. Fam Pract. 2020;37:137–142. pmid:31325300
  15. 15. Kijima T, Akai K, Matsushita A, Hamano T, Onoda K, Yano S, et al. Development of the Japanese version of the general practice assessment questionnaire: Measurement of patient experience and testing of data quality. BMC Fam Pract. 2018;19:181. pmid:30486790
  16. 16. Agency for Healthcare Research and Quality. Translating CAHPS Surveys. [Cited 2020 June 17]. Available from: https://www.ahrq.gov/cahps/surveys-guidance/helpful-resources/resources/cahpsGuidelines_Translation.html
  17. 17. Peterson RA. A Meta-Analysis of Variance Accounted for and Factor Loadings in Exploratory Factor Analysis. Marketing Letters. 2000;11:261–275.
  18. 18. Reeve BB, Hays RD, Bjorner JB, et al. Psychometric evaluation and calibration of health-related quality of life item banks: Plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Med Care. 2007;45:S22–S31. pmid:17443115
  19. 19. Cohen J. Statistical Power Analysis. Current Directions in Psychological Science. 1992;1:98–101.
  20. 20. Dyer N, Sorra JS, Smith SA, Cleary PD, Hays RD. Psychometric properties of the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Clinician and Group Adult Visit Survey. Med Care. 2012;50:S28–S34. pmid:23064274
  21. 21. DeVellis RF. Scale Development: Theory and Applications. Newbury Park, Calif: Sage publications; 1991.
  22. 22. Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63(7):737–45. pmid:20494804