I. Introduction
Foundation models – models trained on broad data that can be adapted to a wide range of downstream tasksFootnote 1 – can pose significant risks.Footnote 2 For example, they can be misused to create non-consensual deepfake pornography or child sexual abuse material.Footnote 3 More capable foundation models could enhance cybercriminals’ ability to identify software vulnerabilitiesFootnote 4 or terrorists’ ability to acquire and deploy biological weapons.Footnote 5 Even well-intentioned users might cause accidental harm in high-risk contexts if they lack sufficient information about the models they are using.
To mitigate these risks, regulators are starting to impose requirements on developers of foundation models. For example, the EU AI ActFootnote 6 already sets out obligations for providers of general-purpose AI models.Footnote 7 Among other things, providers will need to make information and documentation about their models available to certain users.Footnote 8 Providers of general-purpose AI models with systemic risk will also be required to, for example, perform model evaluations and ensure an adequate level of cybersecurity protection.Footnote 9 Some of these obligations are further concretised in a Code of Practice.Footnote 10 Similarly, an upcoming bill in the UK is expected to introduce obligations for developers of “the most powerful AI models,”Footnote 11 alongside more targeted measures to combat AI-generated child sexual abuse material.Footnote 12 Concentrating regulatory efforts on “upstream” developers of foundation models is a sensible starting point because they are generally well-resourced and can take effective measures to mitigate risks.Footnote 13
Much less attention has been paid to the regulation of downstream developersFootnote 14 – actors who fine-tune or otherwise modify foundational models. This could leave a major gap in regulatory frameworks because downstream developers could make modifications that undermine safety measures taken upstream.Footnote 15 For example, an upstream developer might train their model to refuse requests for advice on how to acquire chemical weapons, but a downstream developer could potentially override that training. Policymakers could choose to accept this potential loophole, relying on existing rules to try to address downstream modification risks. Alternatively, they could introduce rules designed to close it. They could either impose additional requirements on upstream developers or regulate downstream developers directly.Footnote 16 However, each approach poses its own set of challenges. Notably, direct regulation of downstream developers could significantly expand the scope of the regime.
This limited focus on downstream developers is also reflected in the literature on foundation model regulation. Although there has been some work on the AI value chain,Footnote 17 existing research on regulation has mainly focused on upstream developers.Footnote 18 However, there is a growing body of technical work illustrating how downstream developers can increase the risks posed by foundation models, either through modifications that improve their capabilitiesFootnote 19 or compromise their safety features.Footnote 20 This has already led some scholars to suggest that additional measures may be needed to address the risks from downstream modifications.Footnote 21 However, research into how policymakers might do this, and the trade-offs associated with different policy approaches, has been neglected. This article seeks to address this gap by answering the following two research questions:
-
• What regulatory challenges do downstream developers pose?
-
• How might policymakers address risks posed by downstream developers?
The scope of this article is restricted in three ways. First, it focusses on the EU and UK, given their regulatory efforts to date and potential for international influence. Second, it assumes that policymakers will introduce regulation on upstream foundation model developers in an attempt to mitigate the risks posed by these models. Third, it considers whether regulation should also apply to downstream developers who modify these models (as opposed to those who use a model as-is for a specific purpose, which would be out of scope).
The article proceeds as follows. Section II sets out why downstream developers’ ability to increase risk poses distinct challenges to policymakers. Section III sets out three broad approaches that policymakers could take in relation to downstream developers: (i) introduce regulation directly on downstream developers (“regulate downstream”), (ii) introduce regulation on upstream developers to mitigate downstream modification risks (“regulate upstream”), and (iii) do not impose any new regulations in relation to downstream modification risks, but clarify existing rules or issue voluntary guidance. Section IV sets out policy recommendations. Section V concludes with suggestions for further research.
II. What regulatory challenges do downstream developers pose?
In this section, we explore the extent to which downstream developers pose regulatory challenges. Specifically, we discuss the key role that downstream developers play in the value chain (Section II.1), the ways downstream developers can increase risk (Section II.2), and why this presents distinct challenges for policymakers (Section II.3).
1. Downstream developers play a key role in the value chain
The AI value chain is complex and involves many actors. Below, we set out the value chain and the role of downstream developers within it. Table 1 provides an overview of the different actors. Figure 1 illustrates how these actors relate to each other.
Table 1. Overview of actors along the AI value chainFootnote 22
Figure 1. Overview of the AI value chain.
Overview of the AI value chain. The AI value chain involves multiple actors which interact in various ways, as illustrated in Figure 1. Upstream foundation model developers obtain compute, cloud storage, and data from a range of sources to develop their models. Downstream developers then access these models – either through a model hub or hosting provider, or directly via upstream developers (e.g., OpenAI’s API). End users, in turn, can access the models directly from the upstream developer, or indirectly through downstream developers or distribution platforms. Notably, one actor can fulfil multiple roles – for example, a foundation model developer may also serve as a distribution platform (e.g., OpenAI offers GPT models and operates the GPT Store).Footnote 23 Although regulation could target a variety of actors,Footnote 24 this article focusses specifically on downstream developers.
Defining "downstream developer". There is no generally accepted definition of the term “downstream developer.” For the purposes of this article, we define downstream developer as any actor that fine-tunes or makes other modifications to a foundation model that was developed by another actor. “Fine-tuning” refers to further training after pre-training – typically using a much smaller dataset – often to elicit specific capabilities from the model.Footnote 25 “Other modifications” refers to other techniques to alter the model’s internals (i.e., the model’s weights) or externals (e.g., through the use of scaffolding). We provide some examples of downstream modifications below.Footnote 26 Our definition is intentionally broad to capture the full range of actors in this segment of the value chain. We discuss how this definition could be narrowed for use in a regulatory context below.Footnote 27
Number of downstream developers. Downstream developers represent a large and growing segment of the AI value chain. The number of downstream developers appears to be in the thousands, with the Hugging Face Hub alone reportedly hosting over 900,000 models, many of which seem to be modified versions of pre-existing models.Footnote 28 One key reason why this segment is so large is the relatively low barrier to entry: downstream developers require far less capital than upstream developers. Their number seems to be growing rapidly. For example, Microsoft’s Azure OpenAI service, which permits downstream modifications to models, grew from 1,000+ organisations in early 2023Footnote 29 to 18,000+ by late 2023.Footnote 30 This upward trajectory is likely driven by continued development of user-friendly tools, growing demand (and funding) for bespoke models, and an ever-widening range of applications. As foundation models become more capable, and our understanding of them deepens, we should expect this trend to continue.
Composition of downstream developers. Downstream developers are varied in nature. They include individuals, academic researchers, start-ups, small and medium-sized enterprises, and multinational corporations. Their expertise varies by domain. While some specialise in finance, healthcare, or cybersecurity, others develop more general-purpose AI tools. This diverse community also includes malicious actors who may deliberately seek to increase risk. Because their motivations, resources, and objectives differ widely, downstream developers play several roles in shaping the AI ecosystem.
Role of downstream developers. Downstream developers play at least three important roles in the AI value chain. First, they can adapt foundation models for specific domains, tasks and applications.Footnote 31 Upstream developers are unlikely to do this themselves because of limited capacity and a lack of sector-specific knowledge. Second, downstream developers may improve a foundation model more generally – for example, by combining post-training modification techniques in different ways.Footnote 32 These techniques may become even more important for AI progress if current scaling laws in pre-training start to show diminishing returns, as some have suggested.Footnote 33 Finally, they can contribute to improvements in AI safety by identifying safety issues and developing techniques to address them. For example, a cyber security firm that adopts and modifies a foundation model might make modifications to improve its cyber defence capabilities.
2. Downstream developers can increase risks
Downstream developers can fine-tune or otherwise modify foundation models in various ways. These modifications could amplify existing risks or create new ones. Below, we discuss how different types of modifications might increase risk, the circumstances under which this is more likely, and examples of real-world harm.
Removing or evading safety features. Downstream developers can increase risk by removing or evading a foundation model’s safety mechanisms. One way to do this is through fine-tuning. Research has shown that current models’ safety fine-tuning can be undone at a relatively low cost, while largely preserving the model’s capabilities.Footnote 34 More concerning, these changes can be made covertly or even unintentionally. For example, one study demonstrated how it was possible to covertly fine-tune OpenAI’s GPT-4 via an API so that it would act on harmful instructions.Footnote 35 Another study found that fine-tuning with benign and commonly used datasets on both open and API-accessible models can inadvertently degrade their safety alignment.Footnote 36 Beyond fine-tuning, other types of downstream modifications can also subvert safety features. For example, prompt engineering has been used to override safety controls in models developed by OpenAI, Anthropic, and Google DeepMind.Footnote 37 Improving the robustness of safeguards is an active research topic amongst upstream developers.Footnote 38
Improving model capabilities. Downstream developers can also increase risk by improving a foundation model’s capabilities, either in a specific domain or more generally. Davidson et al identify a range of techniques that can significantly improve a model’s performance without expensive re-training (see Table 2). To measure these enhancements, they introduce the concept of “compute equivalent gains,” which represents how much additional training compute would be needed to achieve similar improvements. Most of the post-training enhancements they studied improved benchmark performance by more than a 5x increase in training compute, with many exceeding 20x. Notably, these improvements were relatively inexpensive. For example, fine-tuning costs were typically less than 1 per cent of the model’s original training cost.
Table 2. Examples of downstream modificationsFootnote 39
Modifying open models. Post-training modifications can potentially increase risk in all types of foundation models, regardless of how they are released.Footnote 45 Model releases can range from fully closed to fully open.Footnote 46 Downstream developers can typically make more significant modifications to fully open models because they have access to the model’s weights. In addition, foundation model developers are less likely to have oversight or control over downstream modifications of an open model. This may have implications for the policy approach.Footnote 47 For example, a downstream developer who has access to a model’s weights may be better equipped to identify and mitigate risks from any modifications they have made, relative to a downstream developer that has fine-tuned a model via an API.
Modifying highly capable models. Policymakers are increasingly categorising foundation models by their level of capability. The underlying assumption is that model capabilities are a proxy for risk.Footnote 48 For example, the EU AI Act imposes more extensive obligations on general-purpose AI models if they have “high-impact capabilities",Footnote 49 while the UK plans to focus regulation on the “most powerful” models.Footnote 50 This is a sensible approach, as it targets obligations on developers of models that pose the most significant risks. The same principle should arguably also apply to downstream modifications, because modifying a highly capable model may be more likely to create unacceptable risks. At the same time, regulators should recognise that, in theory, even a less advanced model, if significantly enhanced by a downstream developer, may pose unacceptable risks.
Examples of real-world harm. Verifiable examples of downstream developers causing concrete harm are surprisingly scarce, potentially due to the difficulty of proving a direct causal link with a modification. Media coverage also tends to focus on the misuse of AI systems, rather than the underlying model and whether it was modified. However, the potential for real-world harm is there. For example, downstream modifications to generative image models such as Stable Diffusion have facilitated the creation of more realistic deepfakes.Footnote 51 We have also seen media coverage of deepfakes being used in the creation of non-consensual pornographic images,Footnote 52 as well as to spread disinformation.Footnote 53 Although we do not know whether these deepfakes were created using modified models, it seems plausible that this was the case in at least some instances. Disclaimers on liability for downstream modifications also suggest that concerns about downstream modifications exist amongst upstream developers.Footnote 54 As the number of downstream developers continues to grow, we should expect to see more harms like these materialise.
Downstream developers are able to modify foundation models in ways that increase risk, and it seems likely this will increase in future. As industry self-regulation may be insufficient,Footnote 55 policy interventions to address the risks from downstream modifications should be explored.
3. Downstream developers present distinct challenges to policymakers
Despite the potential need to address the risks posed by downstream developers, doing so poses a number of challenges. Below, we set out these challenges.
Downstream developers can undermine regulatory efforts upstream. Downstream developers could modify foundation models in ways that undermine emerging regulatory efforts upstream. For example, a downstream developer might make a modification that evades safeguards that an upstream developer has put in place to meet their regulatory obligations.Footnote 56
Downstream developers may have limited visibility and control. Upstream and downstream developers have control over different aspects of a model, resulting in “distributed responsibility” along the AI value chain.Footnote 57 Since downstream developers may only have partial knowledge of how the underlying model was developed and limited access to it, it may be difficult for them to identify risks (e.g., if they unknowingly degrade a model’s safeguards) and mitigate them (e.g., if they only have API access). Inversely, it is unlikely that upstream developers will be able to predict or address risks stemming from all potential downstream modifications to their model.
Downstream developers are numerous and diverse in nature. Given downstream developers represent such a large and diverse segment,Footnote 58 a one-size-fits-all policy approach is unlikely to be effective. Therefore, policymakers may need to combine several policy tools to effectively address downstream modification risks.
Downstream developers have considerable economic value. The role of downstream developers (as discussed in Section II.1) means that they are important for innovation and may play a role in achieving governments’ economic ambitions. Therefore, any intervention (either upstream or downstream) would need to be carefully designed to ensure that it is not overly burdensome or restrictive.
Fortunately, policymakers have a range of options for addressing the risks posed by downstream developers. We turn to these next.
III. How might policymakers address risks posed by downstream developers?
There are three broad approaches that policymakers could take with regards to downstream developers. First, they could impose regulations directly on downstream developers (“regulate downstream”) (Section III.1). Second, they could require upstream developers to implement measures that mitigate downstream modification risks (“regulate upstream”) (Section III.2). Finally, they could use alternative policy tools, such as clarifying how tort law applies or issuing voluntary guidance (Section III.3). The following section describes each approach in more detail and discusses their respective advantages and disadvantages, with an overview provided in Table 3. It is important to note that these approaches are not mutually exclusive and that they may be best used in combination. For broader context, Appendix A offers examples of regulatory mechanisms used in other high-stakes industries with complex value chains.
Table 3. Approaches policymakers could take with regards to risks from downstream modifications
1. Introduce regulation directly on downstream developers
One option for addressing the risks from downstream modifications is to introduce obligations directly on downstream developers. Note that the EU AI Act already acknowledges that general-purpose AI models can be modified into new models.Footnote 59 This implies that downstream developers could be subject to some obligations. However, more clarity is needed about the circumstances under which this would be the case. The EU AI Office has indicated that they expect very few downstream developers to be in scope.Footnote 60
How could the scope of downstream regulation be narrowed? Ideally, downstream developers should only be subject to regulation if two conditions are met. First, they have made – or are sufficiently likely to have made – modifications that either increase existing risks to an unacceptable level or introduce new, unacceptable risks. Second, imposing obligations on the downstream developer (in addition to the upstream developer) would help to meaningfully reduce these risks. To ensure that only a subset of downstream developers are in scope, policymakers could apply criteria.Footnote 62 These criteria may be specific (e.g., “a downstream developer is in scope only if they have used more than X amount of compute to modify the model”) or general (e.g., “a downstream developer is in scope only if they have reason to believe they have made modifications that are likely to increase risk to an unacceptable level”). Table 4 presents examples of specific criteria across five dimensions. While general criteria may better align with the regulation’s overall objectives and would likely require fewer updates as post-training modification techniques evolve, they can also create greater uncertainty regarding who is subject to regulation. We propose one possible approach which would combine a set of specific and general criteria in Section IV.
Table 4. Examples of specific criteria that could be used to target a subset of downstream developers
What obligations could apply? Obligations for downstream developers could range from high-level principles to more specific rules and requirements.Footnote 63 Principles for downstream developers could mirror those for upstream developers, but their implementation may differ. For instance, if a principle states that developers must assess and mitigate unacceptable risks arising from their models, one way an upstream developer might fulfil this is through extensive testing and evaluation of their model, including the production of safety cases. In contrast, a downstream developer might be able to meet the same principle by ensuring that the modified model remains within the bounds of the safety case produced by the upstream developer. More specific requirements could include obligations to avoid certain “risk-producing” activities, such as fine-tuning a model on data related to chemical, biological, radiological and nuclear (CBRN) materials. They could also include obligations to undertake certain “risk-reducing” activities, such as those outlined in Table 5. Any obligations should presumably be proportionate to the modifications made to the model. For example, downstream developers might be required to augment technical documentation provided by the upstream developer to reflect the changes they have made, rather than creating the documentation from scratch. Policymakers would need to carefully consider this when designing regulation. For example, they may need to introduce upstream regulation to ensure that downstream developers have appropriate access to relevant documentation. The type of obligations imposed should also be guided by the government’s specific policy aims. In particular, obligations to address potentially catastrophic risks from malicious use will look different to obligations to prevent accidental harm.Footnote 64
Table 5. Types of obligations that could apply to downstream developers
Advantages. Placing obligations directly on downstream developers offers several advantages. First, this approach empowers regulators to directly intervene if a downstream developer has made modifications that increase risk, rather than relying on the upstream developer to take action. This would help to strengthen a regime that could otherwise be undermined by downstream modifications. Second, it would provide clarity to well-meaning downstream developers about what they should do to identify and mitigate risks that could result from their modifications, which may be important in light of existing regulatory and civil liability obligations. Third, certain “light-touch” obligations, such as registration or documentation, could be used to enhance regulators’ understanding of downstream developers and the risks they pose, helping to inform decisions about whether more intensive regulations are needed. It may also increase consumer trust and uptake of these models.
Disadvantages. Regulating downstream developers raises two significant challenges. First, it broadens the regulatory regime to include a larger and more diverse set of actors. Such expansion could stifle innovation and increase market concentration, because regulatory burdens often have a disproportionate impact on smaller players, such as start-ups. Indeed, about half of all new tech “unicorns” in 2024 were AI-related start-ups.Footnote 65 Confronted with high compliance costs, downstream developers might decide not to enter the market, or they may be forced to leave it. Alternatively, they could decide not to comply with regulation if they lack the resources to do so, or if they believe the risk of enforcement to be low. This reluctance or inability to comply would create enforcement challenges – challenges that would be amplified by the need for greater resources to monitor and enforce against a larger number of actors.Footnote 66 As discussed, one strategy to manage this challenge is to focus obligations on a subset of downstream developers, though this would make it very difficult to capture all risks. Expanding the regulatory scope to include more entities also raises concerns about the potential for the quality of safety practices that require expertise to decline (e.g., producing safety cases).Footnote 67 Second, there is uncertainty about the extent to which downstream developers can identify and mitigate risks.Footnote 68 Downstream developers may unintentionally disable safety features, or lack sufficient access to the model to implement robust safety practices. Any obligations must address, or at least account for, these limitations. Collectively, these factors might deter policymakers and industry stakeholders, making any legislation difficult to pass.
2. Introduce regulation on upstream developers to mitigate downstream modification risks
Another option would be to impose obligations on upstream developers to mitigate the risks from downstream modifications. We suggest some examples in Table 6. Importantly, the EU AI Act already requires providers of general-purpose AI models to take some steps to mitigate downstream harms, such as adopting proportionate transparency measures (e.g., sharing information about the model with downstream actors), and cooperating with downstream actors to enable their compliance with relevant obligations.Footnote 69
Table 6. Types of obligations that could apply to upstream developers to reduce the risks from downstream modifications
Advantages. Introducing obligations on upstream developers offers three main benefits. First, obligations can help to indirectly mitigate risks from downstream modifications, for example, by ensuring that safeguards remain intact (e.g., through monitoring and modification restrictions), preventing bad actors from modifying models (e.g., by conducting KYC screenings), reducing the likelihood that modifications will increase risk (e.g., by including “buffers” in safety cases), and ensuring downstream developers know what modifications are likely to increase risk (e.g., by specifying them in licences and agreements).Footnote 74 Second, it builds on work already underway by several upstream developers. For example, many upstream developers already claim to monitor downstream activity (e.g., to detect violations of their terms),Footnote 75 restrict modifications (e.g., with fine-tuning rate limits),Footnote 76 include buffers in their capability assessments,Footnote 77 experiment with post-training modifications to investigate whether they might increase risk (e.g., agent evaluations to examine how models perform with agent scaffolding),Footnote 78 and include information on what modifications are permitted in their terms and licence agreements (e.g., by forbidding any attempts to circumvent safety features).Footnote 79 Mandating these practices would ensure that all upstream actors are taking precautions. They may also be easier for downstream developers to adhere to if they were standardised. Third, this approach would avoid expanding the scope of the regime, which is undesirable for the reasons set out above.Footnote 80
Disadvantages. There are, however, three significant downsides to this approach. First, it may leave the regulator unable to directly intervene if a downstream developer had modified a model which increased risk to an unacceptable level. Without such intervention, downstream developers could make harmful modifications without being subject to regulatory enforcement. This situation also introduces the potential for regulatory arbitrage. For instance, an upstream developer could establish a separate subsidiary to fine-tune a model on prohibited datasets if only the primary business were restricted from training on those types of data.Footnote 81 Second, this approach could stifle innovation and entrench upstream power. For example, if upstream developers are incentivised to prohibit downstream modifications outright, this may result in an unfavourable “misuse-use trade-off".Footnote 82 It may also encourage widespread surveillance, which would raise important concerns about data privacy, particularly if extensive monitoring, such as storing chat logs for prolonged periods, becomes necessary for ensuring compliance. Third, there is uncertainty over its effectiveness. Many of the potential upstream obligations identified would be impractical for open models.Footnote 83 Even in the case of closed models, upstream developers are unlikely to be able to manage all potentially risky downstream modifications given the difficulty predicting and analysing all possible modifications. Furthermore, downstream developers may still find ways to circumvent any controls (e.g., through covert fine-tuning).Footnote 84
3. Do not impose any new regulations in relation to downstream modification risks
A third option would be to refrain from imposing any new regulations on downstream developers or on upstream developers for actions taken downstream. Instead, policymakers could clarify how tort law applies to downstream developers or issue voluntary guidance to help manage downstream modification risks.
Tort law. Policymakers could clarify or alter the application of tort with respect to downstream developers. In principle, tort law imposes a general duty on everybody to take reasonable care to avoid causing harm to persons or property. Previous work has discussed how US tort law might fill a gap in the governance of upstream developers before new regulation is enacted, and serve as a valuable auxiliary mechanism once regulation is in place.Footnote 85 It is likely that this will also be the case for UK and EU jurisdictions (at least to some extent). This rationale also applies to downstream developers, particularly because the risks they introduce may be more readily foreseeable (e.g., if the modifications are domain-specific). Importantly, any contract between the upstream and downstream developer can impact which of them bears the risk of liability. For example, the terms of service could include an indemnity clause in either party’s favour. There are also ways policymakers might adapt tort law in this context (e.g., by creating an evidentiary presumption of responsibility). Policymakers could even go so far as to create new tort rules specifically dedicated to the AI value chain.Footnote 86 However, a detailed analysis of tort law and such reforms is beyond the scope of this article.
Voluntary guidance. Another option is to introduce voluntary guidance for downstream developers, upstream developers, or both, regarding downstream modification risks.Footnote 87 Guidance would not need to be legally binding (similar to the Frontier AI Safety Commitments).Footnote 88 Guidance could draw on existing frameworks, such as procurement guidance developed by the US Office of Management and Budget (OMB).Footnote 89 Guidance could serve as a foundation for future regulation or prove sufficient without new regulatory obligations. In addition, government-issued guidance might affect how courts assess whether a developer exercised reasonable care, thus interacting with tort law.
Advantages. These non-regulatory approaches offer several potential advantages. Both avoid expanding the regulatory regime beyond upstream developers, which may be desirable for the reasons discussed above.Footnote 90 There are several further benefits in relation to voluntary guidance. First, it may appeal to policymakers because it avoids the need to pass new legislation.Footnote 91 Relatedly, thoughtfully designed guidelines may be attractive to downstream developers if, for example, they enable them to press for risk-relevant information from upstream developers. Additionally, voluntary guidance can serve as a trial mechanism, allowing policymakers to test potential requirements before mandating them, a strategy that could help reduce the risk of regulatory flight. Tort law, on the other hand, has the advantage of being able to address currently unknown risks without necessarily requiring policymakers to specify rules or guidance in advance. Furthermore, because it only imposes liability after harm has occurred, it only targets risks that are realised (versus those that are speculative). This feature reduces the risk of overregulation and may also reduce compliance costs for downstream developers.
Disadvantages. Despite certain advantages, these alternative approaches may not fully address the risks from downstream modifications. Encouraging widespread adoption of voluntary guidance could be particularly challenging given the sheer number and rapid growth of downstream developers, many of whom have limited resources. Tort law also has several limitations, including the asymmetry of information between plaintiffs and developers regarding the use of foundation models and how they operate. This imbalance could make it difficult for plaintiffs to prove that a developer (either upstream or downstream) failed to take reasonable care and that this failure caused harm. Moreover, tort doctrine tends to develop slowly, potentially undermining its ability to incentivise developers to prioritise risk mitigation. Consequently, tort law may require significant adaptation to effectively manage the risks arising from downstream modifications, and even then, may fall short given the complexity of the AI value chain.
IV. Policy recommendations
In this section, we make high-level policy recommendations. In short, we argue that policymakers should develop voluntary guidance to help mitigate downstream modification risks (Section IV.1), introduce regulation on upstream developers to mitigate downstream modification risks (Section IV.2), and monitor the downstream developer ecosystem to inform whether downstream regulation is warranted (Section IV.3).
1. Develop voluntary guidance to help mitigate downstream modification risks
Given the potential for downstream modifications to increase risk, policymakers should, at the very least, develop voluntary guidance for upstream and downstream developers to help mitigate risks from downstream modifications. This guidance could help establish best practice and form the basis of future regulation.
Voluntary guidance for downstream developers. Voluntary guidance for downstream developers could include the following recommendations: (i) Only modify foundation models whose developers have established responsible use policies that facilitate effective risk assessment and mitigation, and adhere to these policies. (ii) Only modify foundation models from developers that enable incident reporting, and promptly report any incidents arising from modified versions of the model. (iii) Only modify foundation models from developers that provide technical documentation, and update it to reflect any modifications. (iv) Whenever possible, refrain from modifying foundation models in ways that are likely to increase risk. (v) If modifications are suspected to have increased risk, conduct an assessment, potentially in coordination with the upstream developer, and only proceed with deployment if these risks can be reduced to an acceptable level.
Voluntary guidance for upstream developers. Voluntary guidance for upstream developers could include the following recommendations: (i) Provide downstream developers with relevant information so they can better understand whether a modification is likely to increase risk. (ii) Establish responsible use policies that provide assurance to downstream developers that their modifications are safe (e.g., by clearly setting out any prohibited modifications). (iii) Wherever possible, notify downstream developers if their modifications appear likely to have increased risk. (iv) Invest in research aimed at enhancing the robustness of safeguards to prevent risk-increasing modifications. (v) Encourage downstream developers to report any incidents that arise from modified versions of the model. (vi) Provide clear versioning practices to improve traceability for downstream developers and other stakeholders (e.g., regulators). (vii) Provide or recommend evaluation benchmarks and other methodologies for testing the modified model. (viii) Collaborate with policymakers to share insights on which modifications have the greatest potential to increase risks.
2. Introduce regulation on upstream developers to mitigate downstream modification risks
If and when policymakers introduce regulation requiring upstream developers to mitigate risks posed by their models, they should include requirements to address risks from potential downstream modifications. In particular, policymakers should clarify that upstream developers must take into account how downstream developers might modify their models when conducting risk assessments. Policymakers could impose a duty on upstream developers to determine which modifications are likely to increase risks to unacceptable levels and to implement safeguards where feasible and appropriate. While responsibility to determine which modifications pose such risks would rest with the upstream developer, relevant authorities could provide guidance on identifying and evaluating them. Additionally, policymakers could specify practices to fulfil this duty, such as thoroughly ensuring the model’s capacity for harmful capabilities is sufficiently explored, monitoring the effectiveness of existing safeguards, and setting safe operating parameters.Footnote 92 Some flexibility in these measures will be necessary, because the measures available to upstream developers vary depending on factors like the method of release. Given the rapid pace of research and innovation, there is also an inherent uncertainty about how effective these measures will prove in practice, so policymakers should monitor their impact.
3. Monitor the downstream developer ecosystem to inform whether downstream regulation is needed
While obligations on upstream developers may significantly help to mitigate the risks from downstream modifications, we expect that a subset of downstream developers may eventually require certain obligations as well. Before introducing such requirements, we recommend that policymakers closely monitor instances of harm involving foundation models and investigate whether those harms resulted from modified versions. If sufficient harms arise from modified models, despite voluntary guidance and upstream obligations, this would strengthen the case for downstream regulation. Nonetheless, it is paramount that any obligations are targeted and proportionate to avoid unacceptable impacts on the downstream developer ecosystem.
One possible approach is illustrated in Fig. 2. The goal of this approach is to impose obligations on downstream developers only when their modifications could have resulted in an unacceptable level of risk. This would be challenging to specify in regulation, but we think the suggested combination of criteria could strike a balance between capturing modifications that sufficiently increase risk, without being overly inclusive.
Figure 2. One possible approach for identifying downstream developers whose modifications might warrant regulatory intervention.
Is the original model in scope of the regime? Downstream developers would only have the potential to be in scope of obligations if they modify a model that is already regulated. This reflects the principle that obligations on downstream developers aim to reinforce the regulatory framework, rather than expand it to models that would not otherwise be subject to oversight.
Have safeguards been intentionally or knowingly altered in a significant way? Any downstream developer that has significantly altered the safeguards of a model in scope of the regime – either intentionally or knowingly – would be subject to obligations.Footnote 93 This captures the scenario where an upstream developer has implemented protections against known risks, and a downstream actor has actively circumvented or disabled them – an action typically prohibited by upstream developers’ terms of use but one that regulators should be able to address directly.
Does the downstream developer have other reasons to believe that the modification could have resulted in unacceptable risks? Any downstream developer that has made modifications that could have increased risk to unacceptable levels would also be subject to obligations. This criterion acts as a “backstop” for capturing risk-increasing modifications that do not involve removing safeguards. Regulators could identify conditions indicating when risk may be unacceptable, such as: (i) the upstream developer explicitly warning that the modification could result in an unacceptable level of risk; (ii) the modified model being significantly more capable than the original on tasks used in high-risk domains; and (iii) the modification having used a substantial amount of compute.Footnote 94
Under this approach, two primary obligations could apply to downstream developers in scope. First, they would be required to register with the relevant authority, providing information about the model they have modified, the modifications they made, and what they intend to do with the model. Second, they would need to demonstrate either that (i) their modifications do not increase risk to unacceptable levels (e.g., by showing that the modifications do not materially change the original’s safety case), or (ii) they have implemented measures to effectively mitigate the risk (e.g., through additional safeguards or confining the model’s use to a restricted environment).
Although additional research is needed to be confident this is the right approach, we think it balances the trade-offs outlined in this article. There are at least three reasons for this. First, it ensures that upstream developers retain accountability for downstream modifications to their models. Second, it begins with a relatively narrow scope, reducing the risk of stifling innovation or overburdening enforcement, but leaves room for gradual expansion if needed.Footnote 95 Third, by combining specific and general criteria, it targets downstream developers that are more likely to increase risk to unacceptable levels and means it would be relatively difficult to game.
Nonetheless, we recognise this approach has limitations. For example, incorporating a compute threshold may pose practical challenges, as downstream developers may not always be able to accurately assess their compute usage. One alternative could be to consider the amount of data used for fine-tuning. In addition, using general criteria (e.g., whether a modification may have resulted in unacceptable risk) places responsibility on downstream developers to make this judgment. However, we view this flexibility as necessary, given the wide range of possible modifications and the rapidly evolving nature of these techniques.
V. Conclusion
The article has made three main contributions. First, it has highlighted the importance of downstream developers as a potentially overlooked regulatory target. Second, it has evaluated policy options for mitigating downstream modification risks. Third, it has proposed a regulatory approach for mitigating downstream modification risks that avoids unnecessary burdens on downstream developers.
At the same time, several important questions remain. Further work could: (i) Provide a comprehensive mapping of the downstream developer ecosystem. (ii) Investigate how likely it is that different types of downstream modifications will increase risk – intentionally or otherwise. (iii) Analyse the potential economic impacts of regulatory interventions aimed at mitigating the risk from downstream modifications. (iv) Examine whether other actors along the value chain warrant regulatory intervention (e.g., model hubs and hosting providers). (v) Explore how regulation at different points along the entire value chain can be designed in the most streamlined way.
Ultimately, policymakers need to take a holistic approach when designing AI regulations. Many current efforts focus on regulating upstream developers, but downstream developers are often overlooked. If this gap remains, the regulatory framework could be vulnerable to loopholes and fail to manage risks appropriately. Although this is a complex issue, policymakers have several options. By combining these measures, they can effectively mitigate risks without stifling downstream innovation.
Acknowledgments
We are grateful for valuable comments and feedback from Alan Chan, Alexander Erben, Ben Bucknall, Ben Garfinkel, Christoph Winter, Connor Dunlop, Cristian Trout, Irene Solaiman, Jaime Sevilla, John Lidiard, Lauritz Morlock, Leia Wang, Lennart Heim, Leonie Koessler, Marie Buhl, Maximilian Negele, Michael Chen, Peter Wills, Sam Manning, Sayash Kapoor, Stephen Clare and Vinay Hiremath (in alphabetical order). All remaining errors are our own.
Competing interests
The authors declare none.
Appendix A. Case studies from other industries
Regulators in other high-stakes industries with complex value chains also need to manage risks that arise downstream. Below, we provide some examples of regulatory mechanisms used to mitigate downstream risks in other industries, namely pharmaceuticals (Appendix A.1), aviation (Appendix A.2), chemicals and food (Appendix A.3), and consider how they might be applied in the context of downstream AI regulation.
A.1 Pharmaceuticals: regulating generic drugs and drug alterations
The pharmaceutical value chain spans initial drug discovery and development, through manufacturing, distribution, and marketing. Downstream manufacturers (e.g., generic drug companies) produce pharmaceuticals based on original formulas, and even the original manufacturer of a drug may make post-approval changes (e.g., new formulations, manufacturing tweaks). These modifications carry risks: a generic that is not equivalent to the original could be less effective or unsafe, and a process change might alter the purity or potency of a drug. Regulators therefore treat any deviation from the approved medicine as potentially risky until proven otherwise.
Approval process for generic drugs. In the US, UK, and EU, generic drugs must go through a regulatory approval process before they can be sold. The generic manufacturer must demonstrate “equivalence” to the brand-name drug on multiple fronts.Footnote 96 For example, the US FDA requires evidence that a generic has the same active ingredient, dosage form and strength.Footnote 97
Change control for approved drugs. Regulators also control any post-approval changes to medicines. In the EU and UK, changes to a licensed drug (called “variations”) are classified by risk: minor changes (like slight manufacturing tweaks or editorial label updates) can be implemented with notification, while major changes that could affect a drug’s safety, quality or efficacy require prior regulatory approval.Footnote 98 A significant change in formulation triggers a review and re-approval by authorities before the modified drug can be marketed.
Key takeaways. Before deploying a modified foundation model, the downstream developer could be required to demonstrate that it meets safety benchmarks like the original (or to an agreed standard) – this would be analogous to showing a generic has the same therapeutic effect as the brand drug. Like pharmaceuticals, a change control process could also be established for foundation models. For example, if foundation models needed regulatory approval before release, then major downstream modifications may warrant a re-approval process to confirm they do not increase risk.
A.2 Aviation: certification of aircraft modifications
The aviation industry’s supply chain involves aircraft manufacturers, engine makers, airlines, and maintenance and repair organisations. Airplanes often serve for decades, during which time they undergo upgrades and fixes – new software in flight control systems, replacement of parts with improved versions, or retrofitting of new technology into older airframes. Any downstream modification can carry safety-critical risk.
Type certificates and supplemental type certificates. In the US and EU, no aircraft or major component can be used commercially without a type certificate proving it meets safety regulations. When an actor (e.g., the manufacturer or a third-party company) wants to modify an already-certified aircraft, they must obtain a Supplemental Type Certificate (STC).Footnote 99 For example, if an aviation company develops a winglet retrofit to improve fuel efficiency, an STC is required to certify that change. The STC forces the modifier to test and document the effects of the change on the aircraft’s performance and safety and confirms that the proposed modification is airworthy and compatible with the original design. Notably, the STC does not just approve the new part, but how it affects the original aircraft design, to ensure the whole system remains safe.
Tiered approach. Aviation regulators differentiate between minor changes (which do not appreciably affect weight, balance, structural strength, reliability or operational characteristics) and major changes. If a modification is extensive (e.g., a fundamental redesign of a flight control software), authorities may require a completely new type certificate, essentially treating it as a new aircraft design.
Key takeaways. This example underscores the importance of evaluating the impact of a modification in context. Just as an STC ensures a new aircraft component does not destabilise the plane, an evaluation of a modified model could ensure that it is still safe for use. Another lesson is the tiered approach: minor model updates might be allowed with minimal or no oversight, whereas major updates could trigger a full re-evaluation.
A.3 Chemicals and food: upstream controls to prevent downstream misuse and contamination
In chemical and food supply chains, products often pass through many actors before reaching end users. Upstream producers (e.g., a chemical manufacturer or a food ingredient supplier) may not have any direct contact with end users, but improper downstream use can cause serious harms (e.g., industrial chemicals diverted to make explosives or a contaminated ingredient causing food-borne illness in food products). Because of the potential downstream risks, there are duties on upstream suppliers to help mitigate them.
Buyer verification. Under the EU Explosives Precursors Regulation,Footnote 100 certain chemicals that can be used to manufacture bombs (e.g., ammonium nitrate) are tightly controlled. Suppliers must verify that any purchaser has a legitimate purpose for the explosive, and must obtain documentation (e.g., a licence) before sale. They are also obliged to report suspicious transactions or unusually large purchases.
Usage instructions. In the US food additive industry: any approved additive has strict limits for safe use, so its manufacturer must include “adequate directions for use” on the label to ensure downstream food producers stay within those safety limits.Footnote 101 This means if an actor sells an additive, they are obliged to tell the downstream manufacturer how to use it safely (e.g., concentration should not exceed X%, or store under X conditions).
Traceability. The EU General Food LawFootnote 102 establishes a “one step back – one step forward” traceability mandate: businesses must be able to identify who they got a food product from and to whom they supplied it.Footnote 103 If a contaminant or allergen issue is found in an ingredient, regulators can trace it upstream to the source and downstream to all final products, and the upstream supplier often must initiate a recall.
Key takeaways. The chemical and food sectors highlight the importance of knowing one’s downstream users and engaging in practices that promote safe downstream use. Just as chemical suppliers verify customers and restrict sales of dual-use substances, foundation model providers might need to implement vetting or licensing for those who want to modify their models in significant ways and report any suspicious activity. Furthermore, foundation model developers may need to set out safe operating parameters for downstream developers, akin to usage directions on food additives. AI supply chains could also borrow traceability concepts from the food industry: for example, regulators might require downstream developers to keep records of the original model and how they modified it. This would enable the regulator to trace back to the underlying model if a major incident were to occur.
Appendix B. Examples of specific criteria that could be used to target a subset of downstream developers
Open access
