Understanding the genetic architecture of complex traits is a fundamental aim of many branches of genetics. Genome wide association studies (GWAS) have been successful at identifying some loci affecting complex traits. However, those loci account for just a very small proportion of the total genetic variation, the widely known phenomenon called “missing heritability”. As a result, prediction of phenotypes based on the loci uncovered by GWAS has had low accuracy. Methods that regress phenotypes on hundreds of thousands of markers concurrently (whole genome regression, WGR,) may be able to capture a consistent amount of the genetic variation of complex traits and, thereby, increase predictive ability. However, most WGR methods assume strict additivity and traits that are potentially affected by non-additive (epistatic) interactions have not shown any gain in predictive ability. Here, we investigated the effect of sample size and genetic architecture on the accuracy of genomic prediction of complex traits. We used G-BLUP methodology and the unique resource of the Drosophila Genetic Reference Panel (DGRP), a collection of 205 fully sequenced inbred lines that have been phenotyped for many quantitative traits, as well as simulated data. The results show that the accuracy of prediction increases as the sample size increases, conditional on the genetic architecture of the trait examined being taken into account in the statistical model used. In particular, strict additive models in presence of an epistatic component being part of the genetic architecture may fail completely, even if we have knowledge about the true variants affecting that trait and no matter the size of the sample. However, when an epistatic model is fitted, the accuracy of prediction rises, even with small sample size. In summary, this study shows the importance of accounting for their genetic architecture to increase the accuracy of genomic prediction of complex traits.