Living in groups is more stressful and less straightforward than we usually assume. In order to live in large groups, mammals need to find solutions that allow these stresses to be defused. I show that, in primates, this has involved successively adding increasingly costly structural, behavioural, and cognitive solutions that are dependent on the evolution of large brains. Primate social evolution consists of a series of glass ceilings on group size that restrict the habitats that species could occupy.