This article draws upon the experiences of four recent efforts to synthesize the findings of quantitative studies in science education research. After establishing the need for research syntheses in advancing generalizable knowledge and causal effects research, the authors identify a set themes that emerged in the process of conducting these syntheses. The results show that many impact study designs were not conducive to quantitative synthesis, reporting practices of studies were often insufficient for inclusion in research synthesis, and very few replications were performed. These findings have the potential to limit the advancement of knowledge and discovery in science education research.