A multicentre study conducted by the World Health Organization in 1973 demonstrated that the majority of individuals diagnosed with schizophrenia exhibited a marked lack of insight,Reference Carpenter, Strauss and Bartko1 which was subsequently replicated.Reference Amador, Flaum, Andreasen, Strauss, Yale and Clark2 Impaired insight has thus become a core feature of psychosis from early phases and is associated with premorbid personality traits.Reference Ayesa-Arriola, Rodríguez-Sánchez, Morelli, Pelayo-Terán, Pérez-Iglesias and Mata3 The multidimensional models of insight – illness awareness, symptom relabelling, treatment compliance and awareness of the social consequences, proposed in the early 1990sReference Amador, Strauss, Yale and Gorman7,Reference David8 – led to three decades of extensive research in the area.Reference David9 Thanks to this work, insight in psychosis has been consistently linked to positive outcomes – greater insight and better outcomes, namely reduced psychotic symptom severity,Reference Subotnik, Ventura, Hellemann, Zito, Agee and Nuechterlein10 reduced use of coercive treatmentsReference David9 and better psychosocial functioning.Reference Canal-Rivero, Ayesa-Arriola, Ruiz-Veguilla, Ortiz-García de la Foz, Labad and Crespo-Facorro11,Reference Lysaker, Pattison, Leonhardt, Phelps and Vohs12 More controversially, insight gain may result in depression,Reference Subotnik, Ventura, Hellemann, Zito, Agee and Nuechterlein10,Reference Belvederi Murri, Respino, Innamorati, Cervetti, Calcagno and Pompili13 increased risk of suicidal behaviourReference Ayesa-Arriola, Alcaraz, Hernández, Pérez-Iglesias, López Moríñigo and Duta14–Reference Lopez-Morinigo, Di Forti, Ajnakina, Wiffen, Morgan and Doody17 and poorer self-perceived quality of life.Reference Davis, Lysaker, Salyers and Minor18 Although so-called insight paradoxReference Lysaker, Roe and Yanos19 remains far from clear,Reference Lopez-Morinigo and David20 the aforementioned strong association of insight with positive outcomes results in insight assessment becoming a matter of major clinical relevance.Reference Mervis, Vohs and Lysaker21
Insight measurement can be challenging in both routine clinical practice and research, especially given its conceptual complexity. The first insight assessments relied on patients’ narrative accounts of their mental health issues, which could not be used for research purposes. To address this issue, investigators designed interviews that classified patients as having ‘good’ or ‘poor’ insight: for instance, the Present State Examination.Reference Wing, Cooper, Sartorius, Wing, Cooper and Sartorius22 Subsequently, semi-structured scored interviews were validated to measure insight unidimensional, such as the Insight and Treatment Attitudes Questionnaire (ITAQ).Reference McEvoy, Apperson, Appelbaum, Ortlip, Brecosky and Hammill23 Finally, in order to capture the aforementioned multidimensionality of insight,Reference Amador, Strauss, Yale and Gorman7,Reference David8 two scales were validated, namely the Scale to Assess Unawareness of Mental Disorder (SUMD)Reference Amador, Strauss, Yale, Flaum, Endicott and Gorman24 and its more commonly used shortened version,Reference Amador, Flaum, Andreasen, Strauss, Yale and Clark2 and the Schedule for Assessment of Insight (SAI),Reference David8 which also had an expanded version (SAI-E).Reference Kemp and David25 Further insight assessments include self-reports such as the Birchwood Insight Scale,Reference Birchwood, Smith, Drury, Healy, Macmillan and Slade26 the Markova and Berrios Insight Scale (MBIS)Reference Marková and Berrios27 and the specific insight item of two large scales of general psychopathology, the Positive and Negative Symptoms Scale (PANSS)Reference Kay, Fiszbein and Opler28 and the Manual for the Assessment and Documentation of Psychopathology (AMDP system).Reference Guy and Ban29 However, these insight assessments require considerable time input and must be rated by trained staff.Reference Østergaard, Opler and Correll30 Moreover, some degree of cooperation is required that may introduce a selection bias into research studies, namely the exclusion of potentially eligible participants with poor insight, especially in randomised controlled trials.Reference Lopez-Morinigo, Martínez, Barrigón, Escobedo-Aedo, Ruiz-Ruano and Sánchez-Alonso31
These limitations of previous approaches to insight measurement are more pronounced in real-world epidemiological studies, given the scant incorporation of psychometric tests into daily clinical practice.Reference Østergaard, Opler and Correll30 Alternatively, insight in psychosis (hereafter referred to as simply insight) could be assessed in time-constrained clinical settings with proxy measures, which can be defined as ‘indirect measures of the outcome of interest (i.e. insight), with which there is strong correlation’. Proxy measures are commonly used when the outcome variable cannot be directly observed/assessed,Reference Hrisos, Eccles, Francis, Dickinson, Kaner and Beyer32 and have had their utility proven in mental health science research. A variety of proxy measures have been developed and validated;Reference Sanz-Gomez, Alacreu-Crespo, Guija and Giner33,Reference Kirkpatrick, Buchanan, Breier and Carpenter34 those derived from either sociodemographic variables or unstructured electronic health records (EHR) data have proved particularly useful in practice-oriented research studies.Reference López-Díaz, Ayesa-Arriola, Garrido-Torres, Otíz-García de la Foz, Suárez-Pinilla and Ramírez-Bonilla35,Reference López-Díaz, Palermo-Zeballos, Gutierrez-Rojas, Alameda, Gotor-Sánchez-Luengo and Garrido-Torres36 Nevertheless, the widespread implementation of proxy measures in routine clinical care and EHR-based research remains a pending subject.
Aims
The objective of this study was to assess the validity and reliability of a SUMD-based proxy measure for insight assessment in psychosis using EHR. Specifically, we examined the extent to which a three-level clinician rating of insight, derived from routine EHR notes, aligns with SUMD scores in subjects with psychosis. If validated, this approach could provide a valuable tool for both routine clinical practice and real-world EHR-based research.
Method
Study population and inclusion/exclusion criteria
Data for this study were retrieved from the 3-year follow-up, first-episode psychosis (FEP) Programa de Atención a Fases Iniciales de Psicosis (PAFIP) cohort, a publicly funded early intervention service (EIS) for people with psychosis, at Marqués de Valdecilla University Hospital (Santander, Spain). A detailed description of PAFIP is available elsewhere.Reference Lopez-Diaz, Ayesa-Arriola, Ortiz-Garcia de la Foz, Suarez-Pinilla, Ramirez-Bonilla and Vazquez-Bourgon37,Reference Pelayo-Teran, Perez-Iglesias, Ramirez-Bonilla, Gonzalez-Blanch, Martinez-Garcia and Pardo-Garcia38 Briefly, referrals to PAFIP came from local/regional in- and out-patient mental health services, emergency departments and primary care trusts in Cantabria (on the northern coast of Spain, with a catchment area population estimated at approximately 535 000). Potentially eligible candidates were screened against the following inclusion criteria: (a) age between 16 and 60 years, both inclusive; (b) living in the catchment area; (c) experiencing a FEP as outlined below; (d) meeting DSM-IV criteria for a non-affective psychotic disorder, which were confirmed by the Structured Clinical Interview for the DSM; and (e) no previous treatment with antipsychotic medication or, if previously treated, a total treatment duration of less than 6 weeks. Drug dependence (excluding nicotine), having an IQ <70 according to the Wechsler Adult Intelligent Scale-III vocabulary subtest and a history of neurological disease or head injury were exclusion criteria. The project was approved by the local research ethics committee (CEIm Cantabria; internal code of approval, no. 2014.245) in accordance with international standards for research ethics. Participants provided written informed consent as approved by the local research ethics committee. The characteristics of the PAFIP cohort were reported to be generalisable to the target population.Reference Pelayo-Teran, Perez-Iglesias, Ramirez-Bonilla, Gonzalez-Blanch, Martinez-Garcia and Pardo-Garcia38 This study was reported in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement guidelines.Reference von Elm, Altman, Egger, Pocock, Gotzsche and Vandenbroucke39
Selected demographic, clinical and psychometric variables
Data on demographic variables, premorbid characteristics and clinical status were collected from face-to-face interviews with patients, key informants and medical records at the time of PAFIP inception. Sociodemographic variables included age, gender, ethnicity, marital or cohabitation status, employment status, educational level, urbanicity status (classified as living in a municipality with more than 10 000 inhabitants), socioeconomic status and living arrangements at the onset of psychosis. Clinical characteristics included family history of psychosis, history of substance use (defined as harmful or hazardous use of alcohol, cannabis or other illicit drugs), premorbid adjustment, premorbid IQ, age at onset of psychosis, duration of untreated psychosis (DUP), disease duration (from the onset of psychosis until clinical records were examined for SUMD proxy scoring), level of psychosocial functioning and neurocognitive performance and type of psychotic disorder (i.e. DSM-IV coding).
Socioeconomic status was assessed using the Hollingshead and Redlich scale. Premorbid psychosocial adjustment was measured using the Cannon–Spoor Premorbid Adjustment Scale (PAS), which, for the sake of interpretation, was dichotomised into good/poor premorbid adjustment through the median. The Wechsler Adult Intelligence Scale (third edition, WAIS-III) Vocabulary Subtest estimated premorbid IQ. Social functioning was assessed with the Disability Assessment Schedule (DAS). The level of global cognitive functioning (GCF) provided an overall measure of neurocognitive performance.
PAFIP full assessments were carried out at baseline, 6 weeks, 12 months and 36 months, in accordance with the PAFIP protocol. In addition, follow-up clinical appointments were arranged on a monthly basis over the follow-up. Psychopathological symptoms were assessed using the expanded version of the Brief Psychiatric Rating Scale (BPRS), the Scale for the Assessment of Positive Symptoms (SAPS) and the Scale for the Assessment of Negative Symptoms (SANS). Symptomatic remission was defined according to the criteria set out by Andreasen et al.Reference Andreasen, Carpenter, Kane, Lasser, Marder and Weinberger40
Insight assessment
Lack of insight was measured with the shortened version of SUMD. Specifically, the three general items of the shortened version of SUMD – (a) awareness of having a mental disorder, (b) awareness of the need for treatment and (c) awareness of the social consequences of the disorder – were considered. Scores for each item ranged from 1 to 5, with higher scores indicating poorer insight, which were summed to create total SUMD scores ranging from 3 to 15. Those with a SUMD total score of 6 or below had ‘good insight’, while a total score of 7 or above indicated ‘poor insight’.Reference Dumas, Baumstarck, Michel, Lancon, Auquier and Boyer41 SUMD has been widely used in both clinics and research, and was demonstrated to have good reliability and validity.Reference Dumas, Baumstarck, Michel, Lancon, Auquier and Boyer41,Reference Michel, Baumstarck, Auquier, Amador, Dumas and Fernandez42
Proxy measures for insight assessment from SUMD
The proxy version of SUMD is shown in Fig. 1, according to which, total insight scores were classified unidimensional on a 3-point Likert scale as either good (score of 3), partial (score of 9) or poor insight (score of 15). To rate the SUMD proxy, clinicians carefully read the patient’s EHR and considered their clinical status at that point (acutely psychotic or clinically stable). Where no insight assessment was documented (which is relatively common in routine clinical practice due to time constraints during visits), raters considered patient psychopathological/behavioural information, as well as clinical setting (in-/out-patient) and their clinical experience and understanding of psychotic disorders, to provide scores on the SUMD proxies. Some representative examples of the scoring method are shown in Fig. 2.
Fig. 1 Scale to Assess Unawareness of Mental Disorder (SUMD) proxy-based assessment: examiner’s guide.
Fig. 2 Examples of clinical notes and Scale to Assess Unawareness of Mental Disorder (SUMD) proxy scoring method.
Standardised procedure for validation of the SUMD proxy measure
For study purposes, the proxy version of SUMD was validated using EHR and psychometric assessments from the 1- and 3-year follow-up appointments of the PAFIP programme. Raters were trained in the use of the proxy tool, and followed a specific guide designed by the first author (L.-D.A.). Raters’ experience (less than or over 5 years) was recorded and tested as a potential moderator of proxy scoring. Cases were selected if: (a) EHR were available at the time of the 1- and 3-year follow-up PAFIP psychometric assessments; (b) the clinician recording the patient’s mental state (i.e. EHR) was not the evaluator who conducted the psychometric assessment; (c) the SUMD score was unavailable in the EHR; and (d) the third clinician evaluating the EHR to rate the proxy measure of insight was blinded to the SUMD scores. If an EHR lacked sufficient information pertaining to the proxy insight assessment (e.g. ‘no changes from previous visit’), the closest valid EHR (within a 1-month time frame) to the SUMD assessment was used instead. In order to evaluate the psychometric validity of the proxy measures, the degree of concordance between SUMD and proxy scores from EHR was examined using a variety of statistical methods detailed below.
Data analysis
Descriptive statistics were used to examine the demographic and clinical characteristics of the sample, which are reported as either percentages, mean or median and standard deviation or interquartile range (IQR), as appropriate. Test statistics of the proxy measure were specifically assessed using four types of psychometric validity: criterion, content, discriminant and external validity, and interrater reliability.
In order to test the criterion validity (i.e. the extent to which the proxy measure was correlated with SUMD scores), both the concurrent and predictive validity were examined. First, unadjusted univariate correlation analyses between proxy SUMD and SUMD scores were performed and reported as Pearson’s or Spearman’s coefficient, depending on whether the variable followed a normal distribution, respectively, according to the Kolmogorov–Smirnov normality test. Second, partial correlations were run between proxy SUMD and SUMD scores while controlling for potential confounders, such as type of psychotic disorder (first-episode schizophrenia versus all other psychotic disorder diagnoses), clinical status (remission versus non-remission status according to Andreasen’s criteria) and the rater’s previous experience in scoring proxies (less than or more than 5 years). Finally, the extent to which the proxy SUMD predicted those cases with good/poor insight was determined by a binary logistic regression analysis (using the above cut-off points). Odds ratios and the corresponding 95% confidence intervals were estimated. Nagelkerke’s R 2 was used to measure the goodness of fit of the resulting model, and the area under the receiver operating characteristic (ROC) curve (AUC) evaluated the classifier’s performance of the proxy.
For content validity (i.e. the extent to which the proxy assessed and represented all facets of the abbreviated SUMD construct), the correlation coefficients between proxy SUMD scores and those for the three main insight domains of the SUMD were examined. Discriminant validity (i.e. the extent to which the proxy measure was not related to other clinical variables or different constructs) was determined by comparing proxy SUMD median scores across patients’ age, gender, educational level, living status and illness duration. External validity (i.e. the extent to which the results of the proxy tool can be generalised or transferred to other psychometric tests) was assessed by bivariate correlations between proxy SUMD scores and BPRS, SAPS, SANS and DAS scores.
The interrater reliability of the proxy method was evaluated using the intraclass correlation coefficient (ICC), a two-way random-effects model and absolute agreement, which was calculated from three clinical notes randomly selected and rated by six researchers from all participating centres.
The handling of missing data was conducted in accordance with the pairwise deletion method. The level of significance was set at P < 0.05. Post hoc power analyses were performed using G-Power software (version 3.1.9.4, Franz Faul, University of Kiel, Germany), assuming a medium effect size (P = 0.3) and α = 0.05. The remaining statistical analyses were conducted with IBM SPSS Statistics software, version 29 for MacOS.
Results
Sample characteristics
A total of N = 134 clinical notes from n = 106 patients from the PAFIP programme were included. The majority of the clinical notes (57.5%, n = 77) were from the PAFIP 1-year follow-up appointment, 46.3% (n = 62) were from first-episode schizophrenia cases and 59% (n = 79) were from patients in remission. Seventy-six (56.7%) EHR notes were assessed by experienced clinicians (i.e. with >5 years of clinical experience) for proxy insight scoring. Sociodemographic and clinical features of the cases analysed are shown in Table 1.
Table 1 Sociodemographic, clinical and psychometric characteristics of the cases analysed
DAS, Disability Assessment Schedule; IQR, interquartile range.
a. A total of 134 clinical notes from 106 patients were included in the analyses.
b. The median score (2.05) on the Premorbid Adjustment Scale (PAS) was used as the cut-off point for determination of good premorbid adjustment.
The psychometric characteristics of the sample are summarised in Table 1. The median SUMD total score was 3 (IQR 3–7) and the median SUMD proxy score was 3 (IQR 3–9). In terms of good insight categorisation, 71.6% (n = 96) of patients assessed with SUMD were classified as having good insight compared with 58.2% (n = 78) who met the proxy criteria. The post hoc power analysis (1 – b error probability) yielded a value of 0.95, indicating that the sample size had sufficient statistical power to detect significant associations between SUMD and its proxy version.
Criterion validity
SUMD score significantly correlated with its proxy (r = 0.61, 95% CI 0.48–0.71, P < 0.001). After controlling for confounders, the partial rank correlation coefficient result was similar (r = 0.58, 95% CI 0.45–0.68, P < 0.001), thus indicating good concurrent validity of the proxy tool. Regression analysis showed that the proxy SUMD predicted good insight status (odds ratio 20.95, 95% CI 7.32–59.91, P < 0.001), with the model accounting for 42% of the variance on insight status. In total, 79.1% of cases were correctly classified in terms of good/poor insight. The AUC was 0.76 (95% CI 0.67–0.85, P < 0.001), indicating that the proxy SUMD had accurate predictive validity.
Content and discriminant and external validity
Correlation coefficients between the SUMD proxy and each of the three SUMD domains (awareness of mental disorder, awareness of need for treatment and awareness of the social consequences) ranged from r = 0.55 to r = 0.60 (all of which were significant at P < 0.001), indicating good content validity. No significant differences in age, gender, educational level, living arrangement or disease duration were observed, demonstrating excellent discriminant validity of the proxy version. SUMD proxy scores showed low to medium significant correlations with BPRS (r = 0.33, 95% CI 0.17–0.48, P < 0.001), SAPS (r = 0.49, 95% CI 0.34–0.61, P < 0.001), SANS (r = 0.17, 95% CI 0–0.34, P < 0.047) and DAS scores (r = 0.21, 95% CI 0.03–0.37, P < 0.018), indicating optimal external validity of the proxy tool.
Interrater reliability
The ICC of the SUMD proxy was 0.88 (95% CI 0.59–1.00, P < 0.001), which indicated good interrater reliability. The psychometric validity results are detailed in Table 2.
Table 2 Summary of SUMD scores and psychometric properties of its proxy version
AUC, area under the curve; OR, odds ratio; IQR, interquartile range; NS, not significant; RPB, point biserial correlation; SUMD, Scale to Assess Unawareness of Mental Disorder.
*P < 0.05.
Discussion
Main findings
This study aimed to assess the validity and reliability of a SUMD-based proxy measure for insight assessment in psychosis using EHR. To this end, EHR and psychometric assessments from a 3-year follow-up FEP cohort of patients under an EIS (PAFIP programme) from Santander (Spain) were used. In order to evaluate the validity of this proxy tool, the concordance between SUMD and its proxy version scores was examined while controlling for the effects of other explanatory variables, including the type of psychotic disorder (first-episode schizophrenia versus all other diagnoses), in-/out-patient status and raters’ experience. Of relevance, the SUMD-based proxy measure was demonstrated to have robust construct and content validity with SUMD, along with excellent discriminant validity against other unrelated constructs and optimal external validity in relation to other validated instruments for assessment of symptom severity and disability. To the best of our knowledge, this is the first study to validate an EHR-based proxy tool for the assessment of insight in patients with psychotic disorders under real-world conditions.
Overall performance of this insight proxy measure, including four types of validity – criterion, content, discriminant and external validity, and interrater reliability – was found to be good to excellent. In terms of construct validity, the psychometric properties of the SUMD proxy fulfilled quality standards for the validation of proxy tools.Reference Hrisos, Eccles, Francis, Dickinson, Kaner and Beyer32 When compared with other validated clinician-rated proxy measures in the field of psychosis, the correlation coefficient for the proxy SUMD (r = 0.61) was slightly higher than in a previous validation study of a proxy measure of PAS (r = 0.57).Reference López-Díaz, Ayesa-Arriola, Garrido-Torres, Otíz-García de la Foz, Suárez-Pinilla and Ramírez-Bonilla35 Of note, this proxy version of SUMD correctly classified more than 3 of 4 individuals in terms of good/poor insight status (79.1%), in line with external validation of the PANSS-6, Young Mania Rating Scale 6 (YMRS-6) and Montgomery–Asberg Depression Rating Scale 6 (MADRS-6) proxies, which ranged from 76.7 to 83.3%.Reference López-Díaz, Palermo-Zeballos, Gutierrez-Rojas, Alameda, Gotor-Sánchez-Luengo and Garrido-Torres36 The AUC for the proxy SUMD (0.76) demonstrated slightly enhanced discriminative power compared with a proxy measure of the PANSS remission criteria based on Clinical Global Impressions Scale (CGI) scores (0.73).Reference Masand, OGorman and Mandel43 In terms of interrater reliability, the SUMD proxy method ICC (0.88) was higher than that in a previous validation study of proxy versions of PANSS-6, YMRS-6 and MADRS-6.Reference López-Díaz, Palermo-Zeballos, Gutierrez-Rojas, Alameda, Gotor-Sánchez-Luengo and Garrido-Torres36 Finally, outside the framework of clinician-rated proxy tools but within the realm of third-party proxy tools, it should be noted that the EHR-based proxy SUMD had a stronger correlation (r = 0.61) with SUMD than the validated caregivers’ version of this scale (r i = 0.48).Reference Brent, Giuliano, Zimmet, Keshavan and Seidman44 Above and beyond significant differences between clinician- and caregiver-rated scales,Reference Grover, Chakrabarti, Ghormode, Dutt, Kate and Kulhara45 this finding suggests that the proxy SUMD version may provide a more accurate estimate of patient insight from review of EHR than the daily impressions of their close relatives or key informants. Taken together, these comparisons appear to indicate that the proxy SUMD may become a valid and reliable instrument for measurement of clinical insight in psychotic disorders, in both daily practice and EHR-based research.
A novel contribution to insight assessment in psychosis
In 1934, Sir Aubrey Lewis characterised insight as ‘a correct attitude to a morbid change in oneself’, emphasising that it could be inferred only from the patient’s behaviour, particularly their verbal expressions.Reference Lewis46 Based on the multidimensional models of insight proposed in the 1990s,Reference Amador, Strauss, Yale and Gorman7,Reference David8 scales measuring multiple insight dimensions, such as SUMD and the SAI-E, were validated.Reference Amador, Strauss, Yale, Flaum, Endicott and Gorman24,Reference Kemp and David25 In spite of the conceptual complexity involved, these validation studies demonstrated that insight can be measured, especially given the good level of agreement between these scales.Reference Mervis, Vohs and Lysaker21,Reference Sanz, Constable, Lopez-Ibor, Kemp and David47 In today’s mental health crisis, however, clinicians could be tempted to avoid time-related issues, hence assessing insight in an unidimensional manner as present/absent, which cannot generate data for research purposes and may prevent them from monitoring potential insight changes over time. By building on previous validation studies of EHR-based proxy measures in psychosis,Reference López-Díaz, Ayesa-Arriola, Garrido-Torres, Otíz-García de la Foz, Suárez-Pinilla and Ramírez-Bonilla35,Reference López-Díaz, Palermo-Zeballos, Gutierrez-Rojas, Alameda, Gotor-Sánchez-Luengo and Garrido-Torres36 the present study developed and validated a SUMD-based proxy tool for the assessment of insight in psychosis, which provides an overall measure of patients’ insight level from EHR. This clinical insight information could be considered a good predictor of patient outcomes, and may provide valuable information for early intervention and treatment planning.
Furthermore, as alluded to by LewisReference Lewis46, insight cannot be measured entirely objectively like a biological measure which relies, to some degree, on patient accounts. However, the vast majority of patients with psychotic disorders deny having a mental illnessReference Carpenter, Strauss and Bartko1,Reference Amador, Flaum, Andreasen, Strauss, Yale and Clark2 from first presentation,Reference Ayesa-Arriola, Moríñigo, David, Pérez-Iglesias, Rodríguez-Sánchez and Crespo-Facorro48 suggesting that insight and psychopathology are semi-independent domains.Reference Cuesta, Peralta and Zarzuela49 Nonetheless, no technology, including artificial intelligence, enables clinicians to read the thoughts of patients, thus leaving insight assessment based solely on traditional face-to-face approaches such as interviews and questionnaires, which may increase the likelihood of a patient adapting their answers to satisfy the clinician or researcher. Responses may be indicative of conflicting motivational dynamics: individuals may deny an illness they recognise as a means of preserving self-image or, alternatively, express acceptance of an illness they do not genuinely endorse, potentially to satisfy perceived expectations from clinicians or to increase the likelihood of hospital discharge. Some degree of expertise from the interviewer is therefore needed. While self-report measures of insight may reduce any censorship, they may provide an overly simplistic measure of insight. Also, previous results for the agreement between researcher-rated and self-report measures are mixed.Reference Capdevielle, Norton, Aouizerate, Berna, Chereau and DAmato50 Of relevance, the present SUMD-based proxy measure for assessment of insight in psychosis may have overcome these issues. In particular, it is worth noting that no cooperation from patients – this validation study was based on EHR – or significant rater experience were required, while the proxy SUMD has been shown to correlate strongly with the three insight dimensions from SUMD, namely awareness of having a mental illness, awareness of the need for treatment and awareness of the social consequences of the disorder.
Methodological limitations
Three potential limitations should be considered when interpreting the study findings. First, this proxy instrument provided an unidimensional, three-point Likert score to measure such a complex phenomenological construct as insight in psychosis, which may therefore fail to capture its multidimensionality. It is important to note, however, that this unidimensional proxy measure of insight has been shown to have a robust correlation with the three principal insight domains of SUMD. Second, this study relied on previously collected data from the PAFIP programme, and only the three main SUMD items were considered, which may have affected content validity. More specifically, potential correlations of insight levels with severity of specific symptoms, such as hallucinatory experiences, delusions, disorganised thoughts, blunted affect, anhedonia or lack of sociability, could not be assessed. This is a common limitation of studies using SUMD, given the great heterogeneity in its administration.Reference Dumas, Baumstarck, Michel, Lancon, Auquier and Boyer41 Third, the proxy tool was validated using EHR from a FEP cohort receiving care under an EIS, which tend to be more resourced than non-EIS mental health settings,Reference OConnell, O’Connor, McGrath, Vagge, Mockler and Jennings51 including higher-quality documentation, which may have limited the generalisability of our results. Finally, inclusion of an external validation cohort was not feasible, which should be acknowledged as an important limitation of the present study.
Clinical implications and future directions for research
The validation of this proxy version for SUMD has clinical implications. Since the original SUMD has been primarily employed in clinical trials and epidemiological studies, and to a lesser extent in clinical practice, this proxy version may aid in generating insight data from real-world patients, thus making a major contribution to practice-oriented research. This proxy measure may also be useful in standardising clinical data from case report-based systematic reviews, thereby facilitating research into less common psychotic disorders such as shared delusional disorder or Huntington’s disease schizophrenia-like psychosis. Also, this EHR-based proxy SUMD may have implications for management – for instance, as an outcome measure when auditing and monitoring mental health services in the psychosis pathway. The observed correlations between the proxy and other validated psychometric instruments, such as BPRS, SAPS and SANS, replicated the well-known relationship between lack of insight and psychotic symptom severity – poorer insight and worse psychosis.Reference Mintz, Dobson and Romney52 However, future studies should investigate whether this proxy for insight can also predict other outcomes such as psychotic relapse,Reference Berge, Mane, Salgado, Cortizo, Garnier and Gomez53 suicidal behaviourReference Ayesa-Arriola, Terán, Moríñigo, Rivero, Setién-Suero and Al-Halabi15 and/or quality of life.Reference Davis, Lysaker, Salyers and Minor18 Whether the SUMD proxy can be administered by other mental health professionals, such as nurses or social workers, remains to be demonstrated. In addition, it would be interesting to replicate these findings in less resourced mental health services – for instance, in low- and middle-income countries. Finally, this proxy may be automated in EHR-based research applying natural language-processing techniques,Reference Jackson, Patel, Jayatilleke, Kolliakou, Ball and Gorrell54 which requires further research.
In conclusion, while the assessment of clinical insight is paramount in the management of patients with psychotic disorders, previous validated psychometric scales remain underutilised in daily practice, representing an unmet clinical need. Alternatively, the proposed SUMD-based proxy tool for assessment of insight using EHR notes may offer a feasible, valid and reliable alternative to standard psychometric procedures for measuring insight in real-world settings. This proxy SUMD may be a particularly valuable tool in patient-oriented research, thus bridging the gap between research and routine clinical care in mental health.
Data availability
The SUMD proxy measure, which was developed and used in this study, is available to download from the supplementary material. Any additional statistical data that support the findings of this article are available from the corresponding author upon reasonable request.
Acknowledgements
The authors thank all members of the PAFIP research team. We thank all participants and their relatives who made this research work possible. We also thank Deborah Fuldauer for her proofreading services.
Author contributions
All authors have contributed to, and approved, the final version of the manuscript. L.-D.A.: devised the study concept and design, collected data, coordinated data collection, managed statistical analyses, interpreted the results, wrote the first draft of the manuscript and edited the final version. J.-D.L.-M.: design, interpretation of data and drafting of the article. V.O.-G.d.l.F.: coordinated data collection and managed statistical analyses. H.M-.M., M.D.O.-J., J.D.H.-V., G.G.-T. and C.G.-B.: collected and interpreted data and revised the article. B.C.-F. and M.J.C.: interpreted results and edited the final version. R.A.-A.: conception and design, interpreted data, revised the article and edited the final version.
Funding
The PAFIP cohort-based projects were carried out at University Hospital Marqués de Valdecilla and Research Institute Valdecilla, and were supported by the Carlos III Health Institute (nos PI14/00639, PI17/00221 and PI20/00066). This work was supported by the Government of Navarra (grant no. 20/21) and by the Carlos III Health Institute (nos ICI21_00089, ICI19_00026 and FEDER Funds) from the Spanish Ministry of Economy and Competitivity (no. 19/1698).
Declaration of interest
J.-D.L.-M. is a member of the BJPsych Open editorial board. He had no involvement in the peer review or decision-making process for this manuscript. The other authors declare that they have no conflicts of interest.
Ethical standards
The PAFIP programme was publicly funded, approved (internal code of approval, no. 2014.245) by the local institutional review board (Ethics Committee for Research involving Medicines and Medical Devices of Cantabria, CEIm Cantabria) and registered at ClinicalTrials.gov (clinical trial nos NCT0235832 and NCT02534363). All participants gave written informed consent.
Transparency declaration
The lead author certifies that this manuscript is an honest, accurate and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as originally planned have been explained above.
Open access
eLetters
No eLetters have been published for this article.