Tonight I started watching some of the JMBE Live sessions I missed. This session was from May 17, 2024 and Greg Crowther and Thomas Knight shared their paper. Stanley Maloy, JMBE Editor-in-Chief, facilitated the session. Knight started by defining higher and lower-order cognitive skills (HOCS and LOCS). Knight defined lower-order cognitive skills from Bloom’s levels 1 and 2. And higher-order cognitive skills levels 3 and above. Knight cited Momsen et al. 2010 that identified course design and the levels of Bloom’s that were actually tested. A more recent publication presented similar findings. Knight noted that building higher-order multiple choice questions (MCQs) is a challenge. NSF has funded frameworks such as PHYS-MAPS and BIO-MAPS and others to share HOCS assessment questions. Knight and team are building HOCS assessment questions aligned with HOCS active learning items using novel contexts. Crowther then shared the methods the team used for this study. The experimental subjects were Chatbots as Models of Students! Advantages of using chatbots as models of students were shared by attendees: no IRB, cheaper, bigger sample size! The team tested several different chats including ChatGPT and others. The research team had a challenge: how to create questions that bots have not been drained on! Crowther shared hypothetical examples of ion gradients. Crowther invited Sir Elton John to sing “I’m Still Grading.” After grading the chatbots, the scores were worse on hypothetical questions. The newer versions of ChatGPT did better on the hypothetical questions. Crowther polled the audience and the chatbots turned out to do worse on redefinition questions. Knight asked: do humans also do worse on redefinition questions? The research team concluded that hypothetical context questions seem to be more challenging. They want to test similar ideas in human students. Knight recommended using their Test Question Templates (TQT). The TQT framework promotes “transparent, learning-aligned assessment of novel scenarios within carefully chosen Lesson Learning Objectives.” The research team is developing a guide for developing TQTs and a website to help people develop TQTs. Knight noted during the question and answer session that even though they were using multiple-choice questions, the chatbots responded with the answer and an explanation. Knight and Crowther responded to Maloy’s question about the hurdle of overcoming prior knowledge in redefinition questions (even when asked to forget prior knowledge). Knight explained that the challenging questions (HOCS) may be a way to share the goals. Maloy spoke about thinking being hard… and fun!
