Introduction

Autism spectrum disorder (ASD) refers to a group of neurodevelopmental disorders characterized by impairments in social communication and repetitive behaviors and restricted interests. Like many other conditions described in the Diagnostic and Statistical Manual of Mental Disorders (DSM), a diagnosis of ASD is based on the observation of behavioral features [1] and the specific cause of ASD is not known in many, if not most, cases. [2] A major challenge for ASD surveillance systems—and large studies of ASD in general—is reliable ascertainment of ASD.

Although rigorous ASD diagnostic instruments exist, clinicians use a variety of tools and approaches in everyday practice. [3] It is often infeasible for large-scale or population-based studies to classify ASD using the “gold-standard” practices used in clinical settings. Instead, many epidemiological studies rely—sometimes exclusively—upon existing “administrative” designations for ASD classification: International Classification of Diseases, 9^th Revision (ICD-9) billing codes, special education categories, or eligibility for disability benefits and services specific to autism (such as Medicaid). [4–9] In the United States, there is considerable variability in the utilization of these classifications and, because their primary intended purposes are to ensure appropriate service provision to individuals rather than to classify disabilities, these systems do not uniformly identify all individuals that meet ASD criteria in the population. [10–12]

To address the limitations of relying solely on existing codes or classifications for population-based tracking, the Centers for Disease Control and Prevention (CDC) developed a population-based ASD surveillance protocol that uses information from multiple health and education sources, and does not rely entirely upon existing ASD diagnoses or classifications. The Autism and Developmental Disabilities Monitoring (ADDM) Network uses a detailed process in which each site collects developmental evaluations from clinics and schools in their community. ADDM staff abstract verbatim descriptions from the evaluations, and experienced ADDM clinicians review children’s composite information to determine whether the descriptions of symptoms are consistent with ASD diagnostic criteria described in the DSM. [13] Approximately 20% of children meeting ADDM ASD criteria do not have a previously documented ASD diagnosis or classification. [14]

Given this labor-intensive review process, the timeliness and scope of the surveillance system are challenged by the continually-increasing volume of information that must be manually reviewed. While the overall reported prevalence of ASD has increased 120% between 2000 and 2010, there has been an even more dramatic increase in the annual number of evaluations that ADDM Network clinicians must review. For example, clinicians from the Georgia ADDM Network site reviewed 1,152 evaluations in 2000 and 9,811 in 2010—an increase of 750%.

To potentially improve the efficiency and timeliness of ASD surveillance, we developed and evaluated a machine learning-based algorithm that predicts whether a child will meet ASD surveillance criteria, using the words and phrases contained in a child’s evaluations.

Methods

Results

The final algorithm (using 90 terms) was trained on the 2008 Georgia ADDM data, and the terms are shown in Table A in S1 File. “Autism” was the single most important term, followed by phrases that include the word “autism” and references to symptoms such as eye contact or social interaction.

We tested the model using 2010 Georgia ADDM surveillance data, which took one second to classify (after about three minutes to process the evaluations). The distributions of classification scores are shown in Fig 1, and Table 1 describes the summary performance measures. When applied to 2010 data, the algorithm agreed with the clinician-assigned case status for 86.5% of the 1,450 children (kappa = 0.73). Sensitivity, specificity, PPV and NPV for 2010 ranged from 83.7% to 89.4%. The area under the ROC curve was 0.932. We observed similar overall performance for the 2008 data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Histograms of prediction scores (x-axis) compared to clinician-assigned surveillance case definition (blue: autism spectrum disorder (ASD), red: non-ASD).

Horizontal bar represents classification score threshold. Upper panel: classifications for 2008 (training) data. Bottom panel: 2010 (test) data, discordant classifications are highlighted in a lighter shade of blue or red.

https://doi.org/10.1371/journal.pone.0168224.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Comparison between clinician-assigned surveillance autism spectrum disorder (ASD) case status and predictions from random forest algorithm.

https://doi.org/10.1371/journal.pone.0168224.t001

Estimating ASD “prevalence” via algorithm

We compared previously-published ASD prevalence estimates [14] for the 2010 Georgia ADDM Network site to an estimate determined entirely by the algorithm, as shown in Table 2. Overall, the algorithm predicted that 708 of the 1450 children in the surveillance system would meet ASD case status, leading to an estimated prevalence of 14.6 per 1,000 children, approximately 6% lower than the previously-published estimate of 15.5 per 1,000. Similarly, most subgroup prevalence estimates for boys and racial/ethnic groups were all between 5–8% lower than-published estimates. For girls, the algorithm’s estimate was 11% lower than the previously-published estimate.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Autism spectrum disorder (ASD) prevalence per 1,000 children (with 95% confidence interval) for 2010 Georgia Autism and Developmental Disabilities Monitoring Network site: comparison between published and algorithm-derived estimates.

https://doi.org/10.1371/journal.pone.0168224.t002

Characteristics of children with concordant and discordant classifications

The algorithm classification was discordant with the clinician-determined classification for 13.4% (195 of 1,450) of the children in the 2010 surveillance year (highlighted in lower panel of Fig 1). The characteristics of the concordant and discordant groups are described in Table 3. The “true-positive” (algorithm and clinician classified ASD, n = 633) differed from the “true-negative” (algorithm and clinician classified as non-ASD, n = 621) in several ways: “true-positives” had more evaluations (median of 7 versus 4), were more likely to have had a specific ASD or autism diagnosis or special education classification documented in their records (92.3% versus 4.7%), had a younger median age at first evaluation (40 versus 53 months), and were more likely to have evaluations from both school and non-school sources (65.9% versus 35.3%). The “false-positive” (algorithm: ASD, clinician: non-ASD, n = 75) group was similar to the “true-positive” group in terms of median number of evaluations (7), age at first evaluation (41 months), and proportion of children with both health and school sources (56.0%). In comparison, the “false-negative” (algorithm: non-ASD, clinician: ASD, n = 121) group was more similar to the “true-negative” group in terms of median number of evaluations (3), age at first evaluation (53months), and proportion of children with both health and school sources (38.8%). Previous ASD diagnoses or classifications were fairly common in both discordant groups (36.0% of false positives, and 44.6% of false-negatives).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Characteristics of children, by algorithm-clinician concordance on autism spectrum disorder (ASD) case status.

https://doi.org/10.1371/journal.pone.0168224.t003

ADDM clinicians were less likely to request secondary reviews (to resolve uncertainty) for children in the true-positives group (11.4%) than the true-negatives (31.7%). However, children in the two discordant groups had the highest proportions with secondary reviews (44.6% false negatives; 64.0% false positives). Figs D and E in S1 File illustrate how the algorithms scores compare to secondary reviews and clinician certainty scores.

Discussion

These results demonstrate that a machine learning algorithm can discriminate between children that do and do not meet ASD surveillance criteria, among children with developmental concerns. Currently, the ADDM Network employs highly-trained clinicians to manually review each child’s developmental evaluations (often multiple evaluations per child), requiring an average of 45 to 60 minutes per child. Therefore, if the system must review an increasing number of records, it will require proportional increases in the resources needed to complete this task. In contrast, an automated approach requires relatively fixed resources for nearly any amount of information, and offers the potential to improve the efficiency and timeliness of the surveillance system.

Using only the words and phrases contained in a child’s records, the algorithm correctly predicted the clinician-assigned ASD case definition for 86.5% (kappa = 0.73) of the children captured by the surveillance system. This is slightly lower than the clinician inter-rater agreement observed for the overall 2010 ADDM Network (90.7%, kappa = 0.80). [14] Because the algorithm is trained on the clinician-assigned ratings, it is unlikely that agreement between the algorithm and a clinician would ever exceed inter-rater clinician agreement. On the other hand, the algorithm will have perfect intra-rater reliability, as it will always make the same classification for a given set of evaluations. An essential question is: what level of performance—if any—would be considered “acceptable” in order to trust the algorithm’s predictions? Of note, the algorithm-clinician agreement was similar to the inter-rater agreement reported by two other groups doing similar ASD classification on the basis of health records (one reported a kappa of 0.73 [21], and the other 88% agreement [22]).

The algorithm was more likely to misclassify children with certain characteristics. In particular, it was less sensitive to classifying ASD among children with fewer evaluations and those that were older when first evaluated. We also observed that the algorithm was more likely to misclassify children that underwent a secondary review by the ADDM clinicians (compared to those that did not undergo secondary review), suggesting these might be more difficult for the clinicians, as well. It might be possible to address some shortcomings by allowing the algorithm to consider the source or number of evaluations, or the age of the child at each evaluation. Alternately, the current model might serve as a useful “filter” to select the records that need manual review. As shown in Fig 1, the predictive values at the extreme ends of the range are quite high, with more misclassification in the middle. These scores could be used to identify records that need clinician reviews (e.g., a score of 0.50) versus those that are “safe bets” (e.g., scores over 0.80 or below 0.20). If, in the future, the surveillance system is able to electronically receive the contents of medical and educational evaluations, this type of “filter” could be immensely useful.

A previous study used an analogous approach using early intervention (birth to three years) records to predict which children would later be diagnosed with ASD. [23] The best-performing model from that study reported 91.4% precision (PPV) and 58.2% recall (sensitivity); the lower sensitivity is possibly due to a highly imbalanced ratio of ASD to non-ASD children. While this study had somewhat different goals from ours, the two studies suggest that text-based machine learning techniques may one day be useful in a variety of public health applications concerning ASD.

Other recent studies utilizing electronic health information have focused on using medical billing (ICD) codes to detect individuals with ASD. [24,25] These approaches are likely well-suited for case-control studies, where PPV might be more important than sensitivity, but will not detect individuals with ASD that do not have ICD codes. Because the algorithm we developed does not consider ICD-9 codes (as special education records do not assign them), the two approaches could be used to jointly classify ASD when both ICD-9 codes and evaluation text are available. In the future it may be possible to train classification algorithms on ADDM data and distribute them to help identify individuals with ASD from electronic records.

Although these results are promising, additional work is needed to evaluate the utility of this approach for ongoing ASD surveillance. For instance, performance characteristics—such as NPV or specificity—could be different in other populations. We trained the algorithm on a single year of data from one ADDM site and tested it on the following year’s data from the same site; we would need to evaluate whether similar performance could be achieved across ADDM sites or in other populations. We would also need to monitor performance so that it does not drift or degrade over time. In particular, the relatively recent changes to the ASD diagnosis in the DSM-5 could affect the terms used to describe ASD symptoms. Likewise, the surveillance case definition for the ADDM Network may change to reflect the DSM-5 criteria. For these reasons—and others—it is likely that any long-running system would require some level of continued manual review to assess the performance and quality of the system. Nevertheless, even a partially automated approach—in which a clinician might confirm or augment the algorithm’s predictions—could result in a substantial reduction in required resources.

The ADDM clinicians currently code a variety of behavioral symptoms and produce much more information than a dichotomous case classification; it remains to be seen whether these methods could reliably classify specific symptoms in addition to the overall ASD classification. We plan to pursue much more granular and classification algorithms for specific symptoms or for different populations were our current performance was weakest (such as girls, children only seen after age 6, or children without an intellectual disability). It would also be useful to estimate how well the algorithm (and the ADDM methods in general) compare to other ASD classifications, such as in-person assessments. Quantifying this textual information in a reproducible way will provide novel opportunities to better understand how children are evaluated for ASD in typical community settings.

This study is based on a large, population-based surveillance system that has routinely performed ASD surveillance in metropolitan Atlanta for more than a decade. The ASD surveillance case definition uses a well-established protocol for ascertaining ASD from record review, including extensive documentation, training materials, and inter-rater reliability for this procedure. As a by-product of conducting surveillance, the ADDM Network generates information that is useful for training text-based ASD classification algorithms. With relatively small modifications, it could efficiently produce a large volume of very specific examples that could be used to identify particular symptoms or behaviors. Ultimately, the approach piloted in this study could be trained on a much larger sample representing a diversity of community providers and behavioral evaluations.

Conclusion

Public health surveillance systems are constantly challenged to become faster, better, or to provide the same information for lower cost. [26] We observed that an automated approach could predict—with high agreement—whether a child would meet ASD surveillance criteria. While there are many logistical issues to consider, these results hint at the potential for using machine learning approaches to identify ASD from unstructured text data.

Supporting Information

Acknowledgments

Disclaimer: The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.