Mental healthcare has unique features

It is important to delineate the distinct features of mental healthcare before considering how a conceptual model might support the development and governance of artificial intelligence systems that can be used safely in the clinical domain. It is also of note that the majority of conversations about the applications of artificial intelligence in mental health services have, to date, focused on clinical ‘expert models’ that replicate the (psychological) therapeutic task via chatbot interfaces. This paper starts from the position that ‘artificial intelligence therapy’ is not likely to be the main use case for generative artificial intelligence in complex mental health provision. Rather, we believe that the use case will centre on expert models that undertake “cognitive space” work such as case formulation, and therefore augment clinician cognition, to improve productivity and increase the time available for direct clinical contact. This is particularly relevant in secondary and tertiary care services, where information load and clinical administration burden are high, and where patient safety is a function of good global awareness of historical and current clinical factors. We also envisage that areas of the world with significant shortages of trained mental health clinicians would benefit from access to a generative artificial intelligence cognitive space that mirrors that of a highly trained clinician, particularly in the context of specialism-specific demand.

Patient safety in mental healthcare is often a function of ‘weak signal’ detection in unstructured data-sets

Serious adverse events related to mental health, such as suicide and homicide, are rare in the general population. While the incidence of these is higher in specific clinical mental health populations, general purpose language models may not be equipped to identify these correctly in unstructured data-sets. The clinical record in mental health is dominated by unstructured data. ‘Weak signals’ in these data, such as a single sentence in a clinical note that reports a particular risk indicator, or patterns in collective clinical data across a single site, may in hindsight be seen as drivers of significant harm. ^{Reference Wilson6} Serious incident reviews in the field are replete with such examples. ⁷ Arguably, AI-assisted automation of clinical history review is likely to improve signal detection. However, even given the known variabilities of humans in their ability to digest and detect weak signals in large data-sets, the accountability of the agent (the clinician) would still apply, post hoc (for example, in a serious incident investigation), even if a weak signal was missed by either an AI or a human agent. When applying artificial intelligence to mental health clinical review, a transparent governance framework is critical to ensure sufficient clinician ‘reach’ into the processes of signal definition, signal detection and the activation of a response to both weak and strong signals in the data, in order to maintain patient safety. ^{Reference Lee, Goldberg and Kohane8}

Conversely, overemphasising weak signals due to lack of clinical experience can lead to iatrogenic harm through overtreatment and stigmatisation – for example, the misinterpretation of normative yet subjectively severe psychological distress and precipitous use of diagnosis and treatment. For example, this can occur in grief or adjustment disorder states, which may be misconstrued as severe depressive episodes. Any governance framework for the use of autonomous clinical agents will also need to be able to protect against this risk.

Large language models make it possible, depending on the interface used, for clinicians to question in vivo the information presented to them. This would permit trained clinicians to ask the right questions and properly examine weak signals. Furthermore, the domain knowledge that practising clinicians have would allow them to question the absence of signals where they might otherwise be expected. Taking this a step further, the clinician can explore the relationships between weak signals and related biopsychosocial factors, as established in the literature, with relative ease.

Fig. 1 An outline proposal: the Clinical Reach into the Cognitive Space (CRITiCS) framework for generative artificial intelligence patient safety in mental healthcare. LLM, large language model.

Mental healthcare delivery is driven by idiographic case formulation

There are some components of mental healthcare that yield to diagnostically based heuristics (for example, protocols for antipsychotic use in schizophrenia and other psychotic illnesses). However, in the majority of cases the overall treatment heuristic incorporates some degree of idiographic case formulation that drives ongoing treatment and risk management. Case formulation may either be theory naive – for example, generic ‘5P’ models as outlined by Winiarski; ^{Reference Winiarski9} single-theory dependent – for example, cognitive behavioural formulation models; ^{Reference Bieling and Kuyken10} or procedurally theoretically integrated – for example, process-based therapy. ^{Reference Hofmann and Hayes11}

The National Institute for Health and Care Excellence in the UK provides guidance on the broad theoretical constraints around which treatment for specific conditions should be provided. However, given that in mental healthcare, multiple morbidity is the norm, clinicians often undertake case formulation from a theory-diverse perspective in order to construct an individual map of the presenting difficulty, alongside the patient wherever possible. This supports treatment selection, clinical staging and outcome evaluation. Formulation is a highly idiographic task that depends heavily on clinician available knowledge, training level and both clinician and patient preference. There is limited predictive power to idiographic formulation, particularly in complex cases, such that a hindsight review can definitively conclude that the formulation was ‘wrong’ in a manner analogous to a treatment or assessment omission in the management of a physical disease. This has implications in the use of case formulations as ‘fine-tuning’ material for both clinicians and generative artificial agents. Further, as treatment histories elongate, multiple formulations typically exist in the clinical record.

Additionally, the passage of time may invalidate formulations, simply because they are missing more recent data points. Case formulations are substantially approximate representations of the clinical situation. Suitably skilled mental health clinicians have the ability to distil a sometimes vague sense of meaning into more concrete signals via learned clinical reasoning (for example, by noticing particular signs, inflections or case history details). This skill is hard to operationalise within the terms of a formal cognitive space in the context of mental healthcare, perhaps even more so than in physical healthcare. Finally, the nature of human mental health is that it is profoundly influenced by stochastic variables external to the person (such as social and economic circumstances and life events that may impact on the genesis and pathway of illness and, in many instances, define it). However, there have been recent advances in the literature supporting improved definition of this cognitive space. ^{Reference Wilcox, Schroeder and Drefs12,Reference Dissanayake, Colicchio and Cimino13}