Several criminal offences can originate from or culminate with the creation of content. Sexual abuse can be perpetrated by producing intimate material without the subject’s consent, while incitement to criminal activity can begin with a simple conversation. When the task of generating content is entrusted to artificial agents, it becomes necessary to delve into the associated risks posed by this technology. Generative AI changes criminal affordances because it simplifies access to harmful or dangerous content, amplifies the range of recipients, creates new kinds of harmful content, and can exploit cognitive vulnerabilities to manipulate user behaviour. Given this evolving landscape, the question that arises is whether criminal law should be involved in the policies aimed at fighting and preventing Generative AI-related harms. The bulk of criminal law scholarship to date would not criminalise AI harms on the theory that AI lacks moral agency. However, when a serious harm occurs, responsibility needs to be distributed considering the guilt of the agents involved, and, if it is lacking, it needs to fall back because of their innocence. Legal systems need to start exploring whether and how guilt can be preserved when the actus reus is completely or partially delegated to Generative AI.