What factors limit where a microbial species grows and thrives? Population growth rate is a function of two demographic parameters – rates of birth and death. Analyzing changes in birth and death rates in response to environmental stress can help to identify physiological constraints that limit microbial niches; however measuring them in microbial populations is challenging. In order to overcome this obstacle, we developed a novel cellular mark-release-recapture technique called "TrackScar" that enables single-cell measurement of age-structured birth and death rates in budding yeast. We used this method to study the demographic and physiological factors that limit population growth during heat stress in a genetically diverse panel of Saccharomyces cerevisiae strains. We find that the average population growth rate during stress is often a poor predictor of the behavior of individual cells because of heterogeneity in fecundity. Age-structured mortality is one cause of this heterogeneity and is also highly variable across strains. Some genetic backgrounds senesce prematurely during heat stress, while others show the opposite pattern, with elevated rates of early life mortality. Age structuring of birth and death rates points to potential genotype-by-environment effects on processes that regulate asymmetric cell division; consistent with this we find that a mitochondrial inheritance defect explains the early life mortality phenotype of one of the strains we studied. This study demonstrates that characterizing the behavior of individual cells is critical to understanding the physiology of microbes and highlights how the interplay of cellular physiology, genetic variation, and environment influences where microbial populations survive and flourish.