Between 1847 and 1876, the textile factory Todos os Santos operated in Bahia. During these almost three decades, it was the largest textile factory in Brazil and came to employ more than four hundred workers. Until recently, many aspects of the factory’s labour force were hidden. There was a hegemonic narrative that all of these workers were free and waged individuals and that their living and working conditions were extremely progressive for the period. Meanwhile, there was a silence about the employment of enslaved people in the institution as well as a lack of in-depth analysis concerning the legally free workers. This article analyses labour at the Todos os Santos factory. On the one hand, it provides evidence on why the myth about the exclusive use of free and waged workers in the factory was formulated and the interests behind this narrative. On the other, through analysis of data from newspapers, philanthropic institutions, and legal and government documents, it reveals the profiles of the supposedly different classes of free and enslaved workers employed at Todos os Santos—men, women, and children of different colours—showing how complex, and often how similar, their living and working conditions were.