Immune cell populations represent a first line of defense against a variety of pathogens and immune insults. The composition of these cell populations within a given organ system determines the type and fit of potential responses. Here we utilize a set of approximately 100 F1 crosses between Collaborative Cross recombinant inbred lines to assess both variation in as well as genetic control of immune cell composition and activity levels in the lungs of unperturbed animals. We identified 30 cell populations within the lungs of these animals, and were able to identify 9 significant and 12 suggestive QTL regulating individual cell populations. We conditioned on these QTL loci, and were able to identify additional loci driving the composition of the immune cell population. Furthermore, we identified a population of immune cells who are only present in a subset of strains. Multiple genetic loci contribute to the presence of this population, and C57BL/6J alleles drive the absence of this population, suggesting that this commonly used mouse strain does not have this population of cells. In order to validate the underlying relationships between various cell populations, as well as those genetic loci we have identified, we conducted an analysis of immune cell populations from 100 Diversity Outbred animals. Our results highlight both the strong and complex genetic control of basal immune compositions, as well as the importance of studying immune cell populations across a range of genetically diverse models.