Despite growing consensus that cognitive differences between Neanderthals and modern humans were not as significant as once assumed, visual art remains disproportionately associated with Homo sapiens. This paper explores why that is. Rather than appealing to underlying cognition, we argue that early art was pushed and pulled by social dynamics and demographic contexts on a larger scale and eventually formed through cultural transmission. Drawing on the archaeological record, we focus on pigment use, ornamentation, engravings and painting. We suggest that these practices initially remained ad hoc while still functioning as tools for social signalling and intergroup communication in the small-scale communities of Neanderthals and early modern humans. We argue that visual art moved beyond ad hoc practices under conditions of increased population density and interaction – first via the crystallization of art as a tradition and second, as cumulative cultural art. Lack of cognitive ability is not a well-supported explanation for the empirical low frequency of Neanderthal art. Instead, we propose effects of differences in social connectivity and population densities as an alternative. By reframing art as a flexible, context-dependent behaviour, we challenge essentialist models and advocate for a pluralistic view of cognitive and cultural expression across different hominin groups.