This paper addresses the conceptual gap between the expected benefits of federalism in managing ethnicity-based conflict and its actual governance outcomes in the African context. One of the main reasons for this gap is the conflation of federalism with decentralization. In response, we develop and configure polycentric federalism as a praxis-oriented framework with three institutional parameters, administrative devolution, peaceful competition among governance units, and individual choice of alternating governance structures. Through this framework, we analyse federal institutional design in Nigeria and Ethiopia to illustrate why federalism fails to effectively manage ethnicity-based conflict in African states. Despite the varying approaches to federalism in the two cases, institutional design falls far short of achieving the parameters of polycentric federalism, a necessary condition for effectively managing diversity through federalism. Beyond the policy implications, our analysis contributes to institutional economics by illustrating how federal institutional design affects identity-based group dynamics in conflict-ridden multiethnic polities.