We have been talking about peer review for a while now. I use several assignments that have a peer review step. It is good practice for students to learn how to review the work of others and provide constructive feedback. Delivery of honest and useful feedback is a skills. I have used software to automatically assign documents for peer review and also shared documents by email or GoogleDocs. I heard about Kritik through a colleague taking a Quality Matters course with me. Eli Samarasinghe and colleagues from Kritik spoke about “How Peer Evaluation Significantly Reduces Grade Disputes” at the 2021 Lilly Conference online. They started by polling the audience. Participants thought teaching assistants usually are tougher graders. They discussed barriers to reducing grade disputes including time, uniformity, and coordinating effective activities as well as the factors that complicate grading and result in grade disputes. Data from 140,000 peer evaluations from Kritik revealed that “<4 % disputed grades” resulting in “60% reduction in grade disputes.” Samarasinghe described a calibration element the software has and how it closes the feedback loop by allowing students to respond back to the evaluators. The session included faculty talking about their experience with Kritik. One faculty member mentioned that they were able to implement peer feedback at a large scale with hundreds of students thanks to the software. They also explained how they made each “phase” of the peer review process different grade weights. Another faculty member has been using Kritik for three years to teach organic chemistry to 700 students. It was interesting to hear that students learned different ways of representing molecular and why they were correct or incorrect. Both faculty members spoke about the importance of clear questions and calibration. One also talked about how they have specific messaging about what a re-grade means: the peer evaluations will not be considered and your assignment will be evaluated by the instructor. The presenters polled the audience about the percentage of grade disputes they encounter in the courses they teach, and this ranged but focused about less than ten percent. Both faculty members mentioned that they would keep Kritik in their face-to-face courses. Since students are paying for Kritik, one faculty member emphasized that they wanted to maximize the use of the software. The co-founder is doing studies on the effectiveness of Kritik as a faculty member at Indiana. The last couple of minutes were devoted to a demonstration of Kritik. Students upload a document, evaluate the work of others, and provide feedback on the evaluation. Closing the feedback loop with that third step and clear messaging about the expectations are evidently critical components that are somewhat streamlined by Kritik. I am interested to read and learn about the studies that were mentioned and the details of implementation of this software. I will keep Kritik in mind an d for now try to use the current system we use…
