Primary literature offers rich opportunities to teach students how to “think like a scientist.” Calls for increased incorporation of original scientific literature into science education have been issued by a variety of educational organizations (e.g., “Vision and Change”, AAAS, 2011). We describe a Master’s-level course that offers a structured analysis of four recent papers from diverse fields of biology: one flawed paper, one exemplary paper, and a pair of conflicting papers. Students who took this course self-reported a significant increase in a variety of skills associated with critical engagement with primary literature (e.g., being able to critically analyze a paper’s data, independently draw conclusions, propose a follow-up experiment). However, objective measures of the same students’ skills detected only an improved ability to design experiments. We will also present the first analysis, to our knowledge, of what these recent biology undergraduates perceive as the most challenging aspects of engaging with the primary literature. We analyzed 69 pairs of pre- and post-course free responses to the question: “What aspects of reading and analyzing primary literature do you find most challenging?” Thematic analysis of these data was conducted by three raters who were blind to both the identity of the students and the pre-/post- status of the response. Before instruction, the challenges that students reported centered around unfamiliar experimental techniques, background, and jargon-rich scientific language. After instruction, drawing independent conclusions and evaluating authors’ conclusions became the most frequently identified challenge. In addition, after taking this course, the frequency of challenges aligned with the Higher Order Cognitive Skills (Analysis, Synthesis, Evaluation) increased significantly. Together, these changes are consistent with a more competent, critical approach to reading scientific papers. We discuss the implications of our findings to instructional programs that utilize scientific primary literature.