Evaluating Computerised Assessment of Motor Imitation (CAMI) for identifying autism-specific difficulties not observed for attention-deficit hyperactivity disorder or neurotypical development

Participants were 183 children aged 7–13 years old: 35 with ADHD (without ASD), 63 with ASD and co-occurring ADHD (ASD + ADHD), 21 with ASD without ADHD (ASD only) and 65 neurotypical children (see Table 1 for detailed sample characteristics). One additional child from the ASD-only group was tested but excluded from analysis because their CAMI score was more than three times above the interquartile range of the ASD sample. ADHD subtypes within the ADHD group were 12 inattentive, 22 combined and one not otherwise specified. ADHD subtypes within the ASD + ADHD group were 25 inattentive, six hyperactive/impulsive, 31 combined and one not otherwise specified.

Testing took place as part of a 2-day visit at the Center for Neurodevelopmental and Imaging Research, Kennedy Krieger Institute, Baltimore, Maryland, USA. All procedures complied with the ethical standards of the national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008. All procedures involving human participants were approved by the Johns Hopkins University School of Medicine Institutional Review Board (approval number IRB00269589). Parents of all participants provided written informed consent, and the children provided verbal assent before data collection. Parents were reimbursed $50 for participation.

ASD diagnosis was confirmed with the clinician-administered Autism Diagnostic Observation Schedule-Second Edition, Module 3 (ADOS-2);²⁰ and ADHD diagnosis was confirmed with the Kiddie Schedule for Affective Disorders and Schizophrenia²¹ and Conners,²² according to DSM-5 criteria. All participants had a full-scale IQ of ≥70, as measured by the Wechsler Intelligence Scale for Children, Fifth Edition (WISC-V),²³ to ensure they could follow the instructions.

Children completed an imitation task, from which their CAMI scores were obtained. The children's parents filled out the Social Responsiveness Scale-Second Edition (SRS-2)²⁴ and the Conners-3 or Conners-4²² rating scale, which were used as measures of the children's autism and ADHD traits, respectively. In addition, children completed the Physical and Neurological Examination for Soft Signs (PANESS)²⁵ as a measure of their motor ability, and the children in the ASD group were administered the ADOS-2.²⁰

All analyses were conducted in RStudio for Windows version 2023.06.0+21 (RStudio Team, PBC, Boston, MA; see http://www.rstudio.com/↗), using lme4²⁹ and tidyr³⁰ packages. To examine whether children with ASD would show poorer imitation than children with ADHD and neurotypical children, we used a linear regression model, with the predictor variable being the four diagnostic groups (ADHD versus ASD + ADHD versus ASD only versus neurotypical), the outcome variable being the children's imitation performance as measured by CAMI score, and WISC-V General Ability Index and participant gender as the covariate, given the observed diagnostic group differences in the latter two variables. Bonferroni-corrected pairwise t-test was conducted to examine which diagnostic groups differed from each other, and Cohen's d-statistic was calculated to indicate effect sizes of diagnostic group differences.

To examine whether CAMI performance is associated with core ASD traits rather than with ADHD traits or motor ability, we used a linear regression model within the ASD group (ASD + ADHD and ASD-only combined), with the predictor variables being dimensional metrics of autism traits (SRS-2 and ADOS-2 total scores), ADHD traits (Conners inattention and hyperactivity/impulsivity scores) and motor ability (PANESS scores), and the outcome variable being CAMI scores. To explore whether the associations between the dimensional metrics and CAMI held similarly in other populations, linear regression models were conducted within the ADHD and neurotypical groups as well. Because of the high amount of missing data for the SRS-2 and PANESS from the ADHD group, and the ADOS-2 not being administered to the ADHD and neurotypical groups (see Table 1), these dimensional metrics were excluded from the predictors of the respective models. Multicollinearity between predictors was satisfactory for all models (generalised variance inflation factor (GVIF) range: 1.01–1.59). In addition, different dimensions of autism traits were explored for children with ASD, with the predictor variable being ADOS-2 subscale scores (i.e. social affect and restricted and repetitive behaviours), the outcome variable being CAMI scores and the ASD subgroups (i.e. ASD + ADHD versus ASD only) being the moderator variable. For all models, adjusted R²-values are provided to indicate effect sizes by accounting for the number of predictors in regressions models.

To examine the diagnostic classification ability of CAMI scores, we trained SVM classifiers by using three-fold cross-validation (i.e. two folds for training, one fold for test). The folds were balanced on age, gender and IQ. SVM with linear kernel was used as a classifier, with the misclassification cost being inversely proportional to the number of samples per class to address the class imbalance. To improve interpretability of the results, we used a very simple model consisting of a linear SVM with only two learnable parameters, which means that the classification performance using our sample size versus an infinite amount of samples would be negligible, i.e. <0.2%.³¹ In addition, we computed the ROC of the children's CAMI score across trials to confirm that the classification ability of CAMI is not specific to machine learning. The ROC curve is generated by plotting the true positive rate versus false positive rate for 500 classification thresholds uniformly distributed in the [0 1] range. The area under the curve (AUC) quantifies the diagnostic classification ability of the scores, with larger values indicating better classification.

Linear regression model examining diagnostic group differences in imitation scores was significant (F(5,167) = 14.32, P < 0.0001, adjusted R² = 0.28), with diagnosis (F(3,167) = 18.01, P < 0.0001; see Fig. 1), WISC-V General Ability Index scores (F(1,167) = 5.74, P = 0.02) and participant gender (F(1,167) = 11.83, P = 0.001) significantly predicting CAMI scores. Of note, linear regressions with the WISC-V General Ability Index scores and participant gender as predictors of CAMI scores within each diagnostic group revealed no significant associations for WISC-V General Ability Index scores in either group (all P > 0.05), but a significant gender effect within the neurotypical group only (F(1,57) = 8.24, P = 0.006), such that neurotypical girls (mean 0.62, s.d. = 0.21) had higher CAMI scores than neurotypical boys (mean 0.46, s.d. = 0.21).

Bonferroni-corrected pairwise comparisons examining the diagnostic group differences revealed that children with ADHD performed similarly to neurotypical children (P = 0.13, d = 0.44). In contrast, children in both the ASD + ADHD (P < 0.0001, d = 1.20) and ASD-only (P < 0.0001, d = 1.13) groups had significantly lower imitation scores than neurotypical children. Children in the ASD + ADHD group had significantly lower imitation scores than children with ADHD (P = 0.007, d = 0.71), whereas children in the ASD-only group performed similarly to children with ADHD (P = 0.12, d = 0.64). These results largely support our hypothesis in that imitation difficulties as measured by CAMI are observed specifically in ASD, and not in ADHD, as compared with neurotypical populations. However, contrary to our predictions, the results did not show a significant difference between ADHD and ASD-only groups, which, given the similarity in the distributions of the two ASD groups, likely stems from the low sample size in the ASD-only group (see Discussion for more detailed discussion of this point).

Within the ASD group, the model assessing how dimensional metrics of ASD, ADHD and motor features predict CAMI was significant (F(5,29) = 2.99, P = 0.03, adjusted R² = 0.23). Among the predictors, only ADOS-2 total scores significantly predicted CAMI (F(1,29) = 5.90, P = 0.02), with poorer imitation being observed as autism traits increased. Additional linear models examining each ADOS-2 subscale further showed that higher scores in both the social affect (F(3,74) = 3.30, P = 0.03, adjusted R² = 0.08) and the restricted and repetitive behaviours (F(3,74) = 5.40, P = 0.002, adjusted R² = 0.15) subscales significantly predicted lower CAMI scores. These findings support our secondary hypothesis in that imitation performance in ASD is related with core autism traits, not ADHD traits or motor ability. Examination of how these associations manifest within the ADHD and neurotypical groups revealed that the dimensional metrics model was not significant within ADHD (F(2,27) = 0.41, P = 0.67, adjusted R² = −0.04), but it was significant within the neurotypical group (F(4,37) = 8.08, P < 0.0001, adjusted R² = 0.41). For neurotypical children, Conners inattention (F1,37) = 10.43, P = 0.003) and PANESS (F(1,37) = 21.75, P < 0.0001) scores significantly predicted CAMI scores, with higher attention and motor ability being associated with better CAMI performance. Associations of CAMI with SRS-2, PANESS and Conners metrics for the ADHD, ASD and neurotypical groups are shown in Fig. 2(a)–2(d), and associations with ADOS-2 subscale and total scores for the ASD subgroups are shown in Fig. 2(e). Altogether, these findings support our secondary hypothesis by showing that in ASD, imitation is associated specifically with autism traits, not ADHD traits or motor ability, whereas in neurotypical children, imitation is associated with attentional and motor abilities.

The ROC analysis revealed that the true positive rate, as indicated by the AUC, was 80% for the CAMI's ability to distinguish children with ASD from neurotypical children and 70% for its ability to distinguish children with ASD from children with ADHD, whereas the AUC for distinguishing children with ADHD from neurotypical children was 60% (Fig. 3(a)). Chi-squared analysis with Yates correction for each of these comparisons revealed that although children with ASD were correctly classified significantly more than chance as compared with both neurotypical children (χ²(1) = 15.15, P < 0.0001) and children with ADHD (χ²(1) = 6.41, P = 0.01), the classification of children with ADHD from neurotypical children was not significantly different from chance levels (χ²(1) =0.25, P= 0.62). These results align with SVM analyses, which revealed that CAMI scores alone could distinguish children with ASD from neurotypical children with 71.6% accuracy, children with ASD from children with ADHD with 66.4% accuracy, and children with ADHD from neurotypical children with 59.2% accuracy (Fig. 3(b)). Cumulatively, these findings evidence CAMI's specificity as an autism marker – CAMI can distinguish children with ASD from neurotypical children and children with ADHD with a success rate significantly above chance. In contrast, children with ADHD and neurotypical children show similar motor profiles in terms of their CAMI performance.

Given the heterogeneity of ASD and ADHD presentation, sample sizes of n = 20 for the ASD-only group and n = 35 for the ADHD group are relatively small. This sample size limitation may have accounted for our finding of statistically non-significant differences between the ADHD and ASD-only groups, calling for replication of these findings with larger sample sizes. In addition, missing data on several metrics in the ADHD group did not allow examination of autism traits and motor ability metrics within ADHD. It remains to be understood whether imitation performance in ADHD is associated with the same factors as neurotypical children. Additionally, given that imitation skills develop in early childhood,⁸ future studies should adapt CAMI for use with younger children. Doing so will increase the utility of CAMI as a diagnostic and intervention tool, given that the average age of ASD diagnosis is 10 years old³⁷ and earlier diagnosis enables more effective interventions. Further, our ASD sample primarily included children of relatively high intellectual ability (i.e. an IQ score of >70). Future iterations of the task should aim to simplify instructions so they are accessible to those with intellectual disability, which would more wholly represent the wide intellectual range within the ASD population.

In conclusion, by using multiple levels of analyses – diagnostic group comparisons, dimensional associations, classification analyses using ROC and SVM – this study provides evidence that, as compared with neurotypical children, motor imitation difficulties, measured using the highly reliable and scalable CAMI method, are specific to ASD, and not shared with the frequently co-occurring neurodevelopmental condition ADHD. Imitation skills appear to have divergent pathways in neurotypical children and children with ADHD and ASD: although imitation is associated with attention and motor skills in neurotypical children, for children with ASD, imitation is related to core autism traits. Finally, ROC and SVM analyses showed strong agreement and further confirm that imitation ability, as measured by CAMI alone, distinguishes children with ASD from both neurotypical children (80% true positive rate) and children with ADHD (70% true positive rate) with high accuracy, whereas the CAMI performances of children with ADHD and neurotypical children are nearly indistinguishable (60% true positive rate). These findings provide the basis for establishing the CAMI as a biomarker specific to ASD.

In line with the funder's requirements, anonymised data will be made available at the National Database for Autism Research upon publication (http://ndar.nih.gov/↗). Research materials and analysis scripts will be made available on the Open Science Framework page of the corresponding author, B.T.