Study: Perceptions of Graduate Student Performance Can be Flawed

A new study examined how well graduate students and their mentors in STEM fields do at assessing the students’ skills.

And in most cases, they don’t do better than chance when compared to the students’ actual performance.

The results were published earlier this fall in the American Educational Research Journal. The paper’s lead author is David Feldon, who directs the STE2M Center at Utah State University. He is also an associate professor in the Instructional Technology and Learning Sciences Department within the Emma Eccles Jones College of Education and Human Services at Utah State University.

“…Relatively few studies examine the assessment of graduate students’ skill development,” the study says. “The studies that do exist typically … address either the faculty mentors’ perceptions or the student mentees’ perceptions, but not both.”

Feldon said the study was unusual because it compared human impressions to actual performance — and found that those judgments were often wrong.

The research team interviewed 69 matched mentor/mentee pairs in the fall and 49 matched pairs in the spring of one academic year. The students also submitted written research proposals in early fall, then revised and resubmitted in the spring. Those proposals were assessed using a rubric for evaluating scientific research skills.

Here are some findings:

During the fall, 71 evaluative comments by mentors of their mentees’ research skills and knowledge were relevant to rubric criteria. Of these, none predicted performance on the rubric at levels significantly greater than chance… Comments regarding mentees’ data analysis skills predicted rubric-assessed performance at a rate significantly worse than chance.
… Like the mentors, mentees’ self-assessments were mostly inconsistent with their rubric-assessed performance.
The students were able to more accurately predict their skills at performing a literature review, possibly because it was a task they did more often.
The study highlights questions about effective teaching and mentorship.

“It’s hard to tease apart which specific tasks as part of a mentorship affect performance, since, as the study highlighted, perceptions of whether or not a student is learning required skills are flawed,” Feldon told eCampus News in an interview earlier this month. “Much more performance-based data would be needed to develop meaningful guidelines, and that’s where future research comes in.”

Michelle A Maher and Briana Timmerman from the University of South Carolina and Melissa Hurst from the University of Virginia co-authored the study.

Related links:

Contact: David Feldon, david.feldon@usu.edu

Writer: JoLynne Lyon, 435-797-1463