http://orcid.org/0000-0002-2866-1519
Figures
Abstract
Objective
To adapt the consumer version of the Primary Care Assessment Tool (PCAT) for Vietnam and determine its internal consistency and validity.
Participants
Total of 3289 people who used health care services at health facility at least once over the past two years.
Results
The Vietnamese adult expanded consumer version of the PCAT (VN PCAT-AE) is an instrument for evaluation of primary care in Vietnam with 70 items comprising six scales representing four core primary care domains, and three additional scales representing three derivative domains. Sixteen other items from the original tool were not included in the final instrument, due to problems with missing values, floor or ceiling effects, and item-total correlations. All the retained scales have a Cronbach’s alpha above 0.70 except for the subscale of Family Centeredness.
Conclusions
The VN PCAT-AE demonstrates adequate internal consistency and validity to be used as an effective tool for measuring the quality of primary care in Vietnam from the consumer perspective. Additional work in the future to optimize valid measurement in all domains consistent with the original version of the tool may be helpful as the primary care system in Vietnam further develops.
Citation: Hoa NT, Tam NM, Peersman W, Derese A, Markuns JF (2018) Development and validation of the Vietnamese primary care assessment tool. PLoS ONE 13(1): e0191181. https://doi.org/10.1371/journal.pone.0191181
Editor: Christophe Leroyer, Universite de Bretagne Occidentale, FRANCE
Received: April 8, 2017; Accepted: December 31, 2017; Published: January 11, 2018
This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This work was supported by the Atlantic Philanthropies [14613, 21627] and the VLIR Inter-University Cooperation Programme VLIR-IUC with Hue University [ZIUC2014AP026, ZIUC2015AP026, ZIUC2016AP026]. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Quality primary care is an essential component of strong health care systems with good health outcomes [1]. In 1978 at Alma Ata, the World Health Organization (WHO) promoted “primary care” as essential for all health systems. Research from industrialized countries has shown that stronger primary care systems are associated with lower costs and better population health outcomes [1–5]. Studies in the United States and in low- and middle-income countries have also suggested that greater primary care availability is correlated with improved health and a decrease in utilization of high cost health services [6–8]. In 2008, the World Health Organization reiterated their call for all countries to strengthen primary care systems and use primary care as a model to provide care that is equitable and efficient [9, 10].
Primary care in Vietnam is mainly provided by a network of more than 11,000 commune health care centers that provide basic and essential health services to people in every commune. A commune health center (CHC) is usually staffed with a general doctor and some ancillary staff such as a midwife, nurse, assistant doctor of traditional medicine or pharmacist. This network is supplemented by additional outpatient “polyclinics” (staffed by multiple primary care and subspecialist physicians) and district hospitals. People with public health insurance may seek health care services at their registered primary health facility, normally their local commune health center, and can then be referred to a higher level if needed such as district, provincial or central hospitals. Although those with public health insurance generally have free or low-cost access to primary care services through the CHCs, many people believe the quality to be poor and so bypass their CHC at the grassroots level and instead choose to self-pay for services directly at private clinics or hospitals. This pattern of care-seeking behavior has led to serious overcrowding in most upper level referral hospitals, despite potential compromises in quality due to extensive waiting times and short consultations under extreme time pressure. As a result, Vietnam has begun a variety of interventions since 2013 to improve the primary care system [11, 12]. Correspondingly, there is now a great need for valid tools to measure the quality of primary care and assist in evaluating these interventions and their effectiveness.
There are a variety of tools for measuring elements of primary care, however, the Primary Care Assessment Tool (PCAT) developed by Barbara Starfield at the Johns Hopkins Primary Care Policy Center focuses on the core principles of primary care and is one of the few tools designed to assess both structural and process features of primary care [13, 14]. The PCAT family of instruments includes four surveys: the adult consumer-client survey (PCAT-AE), the child consumer-client survey (PCAT-CE), a provider survey and a facility survey. The PCAT-AE is designed to collect information from consumers regarding their experience using health care resources, and it may be used to survey target populations [14].
The PCAT gauges the organizational resources and processes of grassroots health care by evaluating four essential features or core domains of primary care: first contact care (access), longitudinality (continuity), comprehensiveness and coordination. Three other derivative domains are also included in the PCAT: family-centered care, community-oriented care and culturally competent care [15]. Each domain is represented by one or two small scales. Six scales represent the four core domains of primary care: first contact, longitudinal care, coordination of services (coordination domain), comprehensive services available and comprehensive services provided (comprehensiveness domain). Three additional scales represent the three ancillary domains of family centeredness, community orientation and cultural competence. Thus, the original PCAT-AE consists of nine scales representing seven domains [14].
The PCAT-AE has been used and validated in multiple countries and is perhaps one of the most widely studied and applied tools for measuring quality of primary care across the globe [16–19]. Given the proven utility of the tool worldwide, we presumed it to be a useful tool to gauge the quality of primary care as an emerging component of the healthcare system in Vietnam. Although the PCAT-AE has been validated in a variety of countries, specificities of local health systems and patients’ cultural understanding of key concepts may make some elements of the tool less useful or valid. In this study, we developed the Vietnamese Primary Care Assessment Tool based on the consumer-client version of the adult expanded PCAT (VN PCAT-AE) and examined its internal consistency and validity.
Method
Translation and adaptation of the PCAT for Vietnam
A toolkit developed by the Johns Hopkins Primary Care Policy Center for use of the PCAT in international settings contains a set of recommended steps for valid linguistic and cultural translation of the tool (available upon request from the Center). In our initial adaptation of the tool for Vietnam, all of the recommended translation steps were successfully performed at least once as part of the translation process as shown in the Fig 1. Details of the process used are as follows:
- Step 1: Forward translation performed by a bilingual physician and a PhD student whose native tongue was Vietnamese, with experience translating documents from Vietnamese to English, and who was also familiar with use of the PCAT. Translation prioritized preserving the intent over the literal meaning of the items.
- Step 2: Qualitative review of the translated survey completed by a group of doctors and researchers from Hanoi Medical School in a focus group discussion; every translated item was reviewed to ensure its clarity, use of common language and conceptual adequacy.
- Step 3: Backward translation completed by a woman whose native language is American English and has lived in the US long enough to know the language and routines of daily life but was not already familiar with the specific wording of the original PCAT terms.
- Step 4: Doctors and health experts in Vietnam and translators jointly reviewed the forward and backward translations to assess items that were not effectively translated and those which were confusing or generated concerns. A few modifications were made and a consensus translation was produced that was determined appropriate for use in Vietnam.
- Step 5: Lay panel review occurred by two different panels of non-subjects (consumers and physicians) to review the translation, identify troublesome items, and propose alternatives.
- Step 6: Pilot testing was implemented using a final translated version. The translated version was administered to 104 representative patients who were native Vietnamese speakers and representative in terms of age, gender, and socioeconomic status. Basic descriptive analyses were conducted to ensure adequate distribution of responses. Respondents were debriefed to identify any wording or comprehension problems.
Based on challenges experienced in early efforts to utilize the tool, some important steps were repeated to improve and ensure the high quality of the questionnaire including another qualitative review to re-address the cultural relevance of each item. The research team produced a list of problematic items and proposed solutions, with subsequent backward translation. An expert panel, including family medicine leaders from all medical universities in Vietnam with the specialty of family medicine, reviewed the suitability of each item as well as the words used in the questionnaire, resulting in an updated translation of the questionnaire.
An additional pilot study was then conducted with 30 people living in two communes, and some words and cultural references in specific items were identified for further revision. A final revision was done by the research team after review of all the items and obtaining additional advice from international experts with experience in PCAT validation. The final translated version of the questionnaire for this study was then produced.
The most contentious issue throughout the process was what term to use in place of “primary care provider (PCP)” as this is a completely unknown term in the Vietnamese context. Efforts to address this also impacted the decision to repeat some translation and validation steps. Ultimately through the lay and expert review processes, the term “general doctor” was chosen to most closely represent this concept. Additional substantive changes were to replace or reword items that are not typically present in Vietnam with those that were more contextually relevant. For instance, descriptions of the types of facilities in the affiliation section were changed to use more appropriate terminology relevant to Vietnam. Similarly, some clinical services in the comprehensiveness domain were replaced to ensure items were sufficiently relevant to the Vietnamese context, similar to changes in PCAT versions from other countries[20]. Different country versions of the PCAT often have varying numbers of items to assess this domain, and so two items in the Comprehensiveness (services available) domain (G21 and G22) were completely eliminated and not replaced in the final expert review round due to consensus on the extreme scarcity of the services. Table 1 shows the original and translated items for the items that were most substantially modified.
Remaining as consistent as possible with the original tool, the translated questionnaire contained 9 scales with 84 questions representing the primary care domains using a 4-point Likert scale response format (1 = definitely not; 2 = probably not; 3 = probably; and 4 = definitely). An additional “don’t know/don’t remember” option was provided for each item. The questionnaire also included demographic questions such as age, gender, and occupation as well as health condition and degree of affiliation with a usual source of care.
Three questions were refined to identify an individual’s usual source of care as a particular person or place and the strength of that affiliation: (1) “Is there a doctor or place that you usually go if you are sick or need advice about your health?” (2) “Is there a doctor or place that knows you best as a person?” and (3) “Is there a doctor or place that is most responsible for your health care?” A person was considered to have a usual source of care if he or she answered affirmatively to any one of these three questions, and no usual source of care if they provided a negative answer to all three questions. An algorithm based on the responses to these three questions was then used to categorize the strength of affiliation with a primary care source. If all three physicians/places were the same, this was considered evidence of a very strong affiliation. If the response to the first question was the same as for either of the other two questions, then that site was used although the affiliation was categorized as less strong. If the response to the first question was different from the other two responses but the other two responses were the same, then the site where both were the same was used as their primary care source and categorized as a weak affiliation. If all three responses were different, then the site identified in the first question was used and categorized as a very weak affiliation. All subsequent questions asked were intended to refer to this specific person or place. For those with no identifiable source of primary care, subsequent questions were asked about the last place that was visited.
Data collection
To evaluate the feasibility, internal consistency and validity of the Vietnamese Primary Care Assessment Tool (VN PCAT-AE), a quantitative cross-sectional study was conducted. A multistage and purposive sampling approach was used to select the study sites. Three provinces were chosen purposively to capture the diverse characteristics of central Vietnam: Khanh Hoa, Thua Thien Hue and Quang Tri. To obtain a sample representing the diversity of the country, we purposively selected two to four districts from each province, depending on the number of commune health centers with working physicians. In addition, within these constraints, we chose at least one lowland district, one mountainous district and one urban district when possible. Specifically, in Thua Thien Hue, the survey was done with 24 communes in four districts (six communes per district); in Quang Tri, 14 communes in three districts (one district with six communes and two other districts with four communes); and in Khanh Hoa, two districts with a total of 18 communes were selected, for an overall total of 56 communes.
From each commune, 30 households were selected. Half (15) of the households were selected from a list of patients recently treated at the local CHC. The other households were selected from a commune household list. Another 15 from each list were placed on reserve lists for later use in the case of refusals or non-respondents. On the patient list from the CHC, we started with the household of the first person on the examination list of the CHC (i.e. the most recent patient), and then selected every 10th patient who followed (patients 11, 21, 31…) until the intended sample size was reached. Using a similar technique, we selected every tenth household from a separate list of households in the commune.
Each selected household was visited and the head of household surveyed, as well as one other willing adult (≥18 years old) if available during this home visit. Data collection was conducted from January through August of 2014 and questionnaires were administered through in-person interviews. Only participants who had utilized health care services at a health facility at least once over the past two years were surveyed.
Before the interview, participants received a full explanation of the study’s content and purpose and signed a consent form if they agreed to participate. Refusals were rare and so a response rate was not specifically tracked, but surveyors estimated the refusal and non-response rates at less than 5%. If a household refused or could not be reached after three attempts, then another household was chosen at random from the reserve list. Participants were compensated for their time with small gifts of appreciation (worth $2.50 USD) upon completion of the interview.
This study obtained ethical approval from the Scientific Committee of Hue University of Medicine and Pharmacy on 18th March 2014 and IRB review from Boston University (H-31432).
Data analysis
All collected questionnaires were cleaned and scanned into a computer for storage and convenient review in the future, followed by entry into EpiData by a group of six students working in pairs. Double data entry was used to check for errors in data entry. Data analysis was performed using SPSS software version 23.0.
Subsequent full validation involved several steps. First, individual items were evaluated on several criteria. Items with a high percentage (≥20%) of item non-response or “don’t know/don’t remember” responses, or items with a large floor or ceiling effect (>80% of respondents chose the lowest or highest answering category) were removed. Next, the item-total correlation for the remaining items in each scale was calculated (item-total correlation before review). Items were removed if the item-total correlation was below 0.30 or if Cronbach’s coefficient alpha for that scale improved substantially when the item was removed. Finally, item-discriminant validity was tested: for each item, the item-total correlation (item-total correlation after review) with the hypothesized scale should be substantially higher than the correlation with the other scales. In the second phase, Cronbach’s coefficient alpha was used to examine how well all items measured the same construct (internal consistency). A value of 0.70 is very often seen as a minimum[21].
The recoding progress and calculation for the sum mean score of domains and subdomains of primary care strictly complied with the guideline PCAT manual issued by John Hopkins University in 1998. For calculating the sum mean scores of domains and subdomains, a mean value was assigned to “not sure/don’t remember” answers as well as to missing values.
Results
Characteristics of study population
Table 2 describes the characteristics of the 3289 participants with valid questionnaires. For the extent of affiliation with a place or doctor, results suggested that most participants have a strong (35%) or very strong (29%) affiliation with their doctor, while approximately a third report a weak (29%) or very weak (6.1%) affiliation.
Evaluation of the individual items
Evaluation of the individual items shows that fourteen items were problematic (Table 3). Because of a high percentage of “don’t know/don’t remember” or missing answers (≥20%), two items were removed from the domain of First contact—accessibility (C8 and C10), in addition to three items from the domain Comprehensiveness (services available) (G16, G17, G18) and one item (J12) from the domain of Community orientation.
Next, items with a large floor or ceiling effect (>80%) were identified, including one item from the domain of First contact—accessibility (C3) and two items from the domain of Ongoing care (D2 and D3). Item-total correlations for the remaining items in each scale were then used to identify those whose item-total correlation was below 0.30 including two items from the domain of First contact—accessibility (C11, C12) and two items from the domain of Ongoing care (D14 and D15). Finally, Cronbach’s alpha was assessed (see Table 4) and improved substantially (from 0.65 to 0.71) for the first contact-access domain when item C9 was removed. For all items, the item-total correlation with the hypothesized scale was higher than the correlation with the other scales (see S1 Table).
Internal consistency of the different scales
Based on these parameters, 70 items of the VN PCAT-AE were determined to be appropriate for use in this population, to represent four core domains with six scales and three derivative domains with three scales (Table 4). Except for the scale of Family Centeredness, all of the retained scales have a Cronbach’s alpha above 0.70.
Evaluation within subpopulations
The robustness of the results was explored in different subpopulations such as rural and urban populations, provinces, populations from the CHC consumer’s list and from the community household list. The obtained results are highly stable, however there were a few items that showed a poorer fit in some subpopulations: item C2 and item G2 in Quang Tri province, item G1 in Khanh Hoa province and item G23 in the urban population.
Discussion
Strictly applying standardized guidelines for translation and adaptation followed by a routine psychometric validation method, we confirmed the Vietnamese PCAT (VN PCAT-AE) to be a valid and reliable instrument for the Vietnamese context, making this the first proven tool developed in Vietnam for comprehensive evaluation of primary care.
The VN PCAT-AE successfully measures all of the important domains of primary care with six scales representing four core primary care domains: first contact accessibility and utilization (first contact domain), ongoing care, coordination care, comprehensiveness-services available and comprehensiveness-services provided (comprehensiveness domain). It also successfully measures another three derivative domains of family centeredness, community orientation and cultural competence.
The VN PCAT-AE retains most major characteristics of the original PCAT version with 70 valid items. It is quite similar to PCAT versions in Argentina and South Africa with a few items determined not to be appropriate in these settings and with the addition of questions more relevant to their contexts [20, 22]. In other countries, some researchers have shortened the questionnaires by rearranging items into different scales or the addition or subtraction of scales [16, 17, 19]. We however sought to maintain the integrity of the original tool to the utmost degree possible.
It is also important to note, however, that the total absence or gross inadequacy of services in a specific domain in a certain country or setting is likely to result in psychometric qualities that threaten the validity of the tool in that domain. In Vietnam, despite of a series of great strides and improvements over the last 20 years, the primary care system is still in an early phase of development and many improvements have not yet been widely and systematically implemented throughout the entire country. In particular, a substantial floor effect may be found as a vast majority of patients in this study reported the absence of a variety of services. For instance, many questions related to appointments were removed from the access domain because of the absence of appointment systems in Vietnam, and thus resulted in removing half of the questions from this domain. While the access domain in the VN PCAT-AE remains an overall valid measure of validity, with the removal of so many items related to appointments, it may no longer maintain the same level of integrity in this domain compared with the original tool.
With primary care services in Vietnam improving, however, it is possible that some questions removed from the tool may become more valid in the future as the primary care system becomes more sophisticated and thus future researchers may want to consider reintegrating some of these questions in the tool and reassessing their validity. Recent positive changes in policy and planning by the Ministry of Health and other government entities for family medicine development and strengthening of the primary care network are anticipated to lead to significant system improvements in the future.
This study has several potential limitations. Firstly, the sample was not recruited randomly in an effort to purposively capture the diverse characteristics of the population in the Central region. Secondly, it was a home survey in which the head of household and one additional adult member were surveyed at time of the visit without a systemic method in place for choosing the additional adult member if more than one might be available, and therefore potentially introducing another source of bias.
In spite of these limitations, the Vietnamese PCAT version VN PCAT-AE demonstrates adequate validity and reliability to be used as an effective tool for comprehensively measuring the quality of primary care in Vietnam from the consumer perspective.
Supporting information
S1 Table. Item correlation with domain scores after review (item convergent validity and item discriminant validity).
https://doi.org/10.1371/journal.pone.0191181.s001
(DOCX)
Acknowledgments
We would like to acknowledge Elizabeth Henry, Ly Lai and Kristin Shaw from Boston University, the many staff in the Family Medicine department at Hue University of Medicine & Pharmacy, and the many family medicine leaders across Vietnam who reviewed and made comments on this tool for their support throughout this work.
References
- 1. Macinko J, Starfield B, Shi L. The contribution of primary care systems to health outcomes within Organization for Economic Cooperation and Development (OECD) countries, 1970–1998. Health services research. 2003;38(3):831–65. pmid:12822915
- 2. Starfield B, Shi L. Policy relevant determinants of health: an international perspective. Health Policy. 2002;60(3):201–18. pmid:11965331
- 3. Niti M, Ng TP. Avoidable hospitalisation rates in Singapore, 1991–1998: assessing trends and inequities of quality in primary care. J Epidemiol Community Health. 2003;57(1):17–22. pmid:12490643
- 4. Starfield B. Primary care and health. A cross-national comparison. JAMA. 1991;266(16):2268–71. pmid:1920727
- 5. Starfield B, Shi L, Macinko J. Contribution of primary care to health systems and health. Milbank quarterly. 2005;83(3):457–502. pmid:16202000
- 6. Chang CH, Stukel TA, Flood AB, Goodman DC. Primary care physician workforce and Medicare beneficiaries' health outcomes. JAMA. 2011;305(20):2096–104. pmid:21610242
- 7. Kravet SJ, Shore AD, Miller R, Green GB, Kolodner K, Wright SM. Health care utilization and the proportion of primary care physicians. Am J Med. 2008;121(2):142–8. pmid:18261503
- 8. Macinko J, Starfield B, Erinosho T. The impact of primary healthcare on population health in low- and middle-income countries. J Ambul Care Manage. 2009;32(2):150–71. pmid:19305227
- 9.
World Health Organization. The World Health Report 2008: Primary health care _Now more than ever. 2008.
- 10.
World Health Organization Commission on Social Determinants of Change. Closing the gap in a generation: health equity through action on the social determinants of health. 2008.
- 11.
Prime Minister. Decision No.92/QĐ-TTg January 9, 2013, Approval of the project for reducing overcrowding at hospitals in period 2013–2020 (in Vietnamese). Quyết định Số 92/QĐ-TTg: Phê duyệt đề án giảm quá tải bệnh viện giai đoạn 2013–2020.
- 12.
Ministry of Health. Decision No.935/QĐ-BYT March 22, 2013, Approval of the project for the building and development of the family medicine clinic model in the period 2013–2020 (in Vietnamese). Quyết định Số 935/QĐ-BYT: Phê duyệt đề án xây dựng và phát triển mô hình phòng khám bác sỹ gia đình giai đoạn 2013–2020.
- 13. Malouin RA, Starfield B, Sepulveda MJ. Evaluating the tools used to assess the medical home. Managed Care (Langhorne, Pa). 2009;18(6):44–8.
- 14. Shi L, Starfield B, Xu J. Validating the adult primary care assessment tool. Journal of Family Practice. 2001;50(2):161–75.
- 15. Shi L, Pinto DM, Guanais FC. Measurement of Primary Care: Report on the Johns Hopkins Primary Care Assessment Tool. Inter-American Development Bank; 2013.
- 16. Rocha KB, Rodriguez-Sanz M, Pasarin MI, Berra S, Gotsens M, Borrell C. Assessment of primary care in health surveys: a population perspective. Eur J Public Health. 2012;22(1):14–9. pmid:21470974
- 17. Aoki T, Inoue M, Nakayama T. Development and validation of the Japanese version of Primary Care Assessment Tool. Family practice. 2016;33(1):112–7. pmid:26546033
- 18. Wang W, Shi L, Yin A, Lai Y, Maitland E, Nicholas S. Development and validation of the Tibetan primary care assessment tool. BioMed research international. 2014;2014:308739. pmid:24967349
- 19. Yang H, Shi L, Lebrun LA, Zhou X, Liu J, Wang H. Development of the Chinese primary care assessment tool: data quality and measurement properties. International journal for quality in health care: journal of the International Society for Quality in Health Care / ISQua. 2013;25(1):92–105.
- 20. Berra S, Hauser L, Audisio Y, Mantaras J, Nicora V, de Oliveira MM, et al. [Validity and reliability of the Argentine version of the PCAT-AE for the evaluation of primary health care]. Rev Panam Salud Publica. 2013;33(1):30–9. pmid:23440155
- 21. Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. Journal of clinical epidemiology. 2007;60(1):34–42. pmid:17161752
- 22. Bresick G, Sayed A-R, le Grange C, Bhagwan S, Manga N. Adaptation and cross-cultural validation of the United States Primary Care Assessment Tool (expanded version) for use in South Africa. African journal of primary health care & family medicine. 2015;7(1):1–11.