Gene-environment interaction (G×E) refers to the phenomenon that the same mutation has different phenotypic effects in different environments, and is believed to be common in all organisms. Previous studies in humans and model organisms identified a number of genes or quantitative trait loci (QTLs) exhibiting G×E. However, little is known about the general properties of G×E and its underlying QTLs. Here we use the genotype data of 1005 segregants from a cross between two Saccharomyces cerevisiae strains and the growth rates of these segregants in 47 different environments to identify QTLs underlying growth rate variation in each environment (gQTLs) and QTLs that have different effects in each pair of environments (g×eQTLs). We found that 31% of gQTLs are g×eQTLs and on average 78% of g×eQTLs belong to gQTLs, supporting the current practice of identifying genes/QTLs showing different effects in different environments from genes/QTLs with effects in individual environments. Interestingly, most g×eQTLs identified from gQTLs have concordant rather than antagonistic effects in different environments, but as the effect size of a mutation in one environment increases, the probability that it is antagonistic in another environment becomes greater. Antagonistic g×eQTLs are overrepresented in environments rarely encountered by yeast in nature. We found gQTLs to be enriched in nonsynonymous sites and nonsense sites, while g×eQTLs are enriched in introns. Gene Ontology (GO) analysis identified >10 times the number of enriched GO terms for g×eQTLs than gQTLs. Simulations based on the yeast data showed that ignoring G×E causes substantial missing heritability. Together, our findings reveal the genomic architecture of interactions between natural polymorphisms and environments in yeast growth and demonstrate the importance of considering G×E in explaining fitness variation and missing heritability.