Many aspects of brain development depend on experience. Because the major macro-morphological events of brain development occur over the first 2–3 years of postnatal life, this time period can be considered both a period of opportunity as well as a period of vulnerability. In this paper we describe how experience with stress early in life can have a negative impact on certain aspects of brain development, and specifically, those neural circuits that underlie memory. We also describe the effects of traumatic events on the development of the neural basis of memory. In support of our argument, we review the literature on brain, stress, and memory in the context of development. Based on this review, we suggest that the developing brain is particularly vulnerable to the harmful physiological effects of stress, which in turn has the potential to lead to impairments in memory. Unfortunately, there are few empirical data that directly address this hypothesis. In this context we offer a number of suggestions for future research.