Exam maintenance workshops are the life blood of any exam. I’ve worked with many different SME programs from organizations that are employing just a few dozen SMEs to thousands of SMEs. After calls with so many organizations to discuss their programs, I’ve compiled a few simple tips for how you can improve your item writing workshops and in time improve your entire SME program and produce better outcomes.
We live in a big data world; why should your exam maintenance be any different. There are so many data points that you can track in your item writing workshops that you can use to inform your future workshops. This may sound like a basic concept, but there are seemingly basic data points that a lot of SME programs aren’t tracking. Which of your SMEs excel at writing at multiple choice questions, and which ones excel at writing long form questions? Does a particular SME tend to write questions that are more elaborate? How many of their items are passing review, and how are their items performing on the Exam? All of these are examples of data points your exam team can use to group your SMEs for success in your next workshop. By recording this data and using it to inform your next workshop, you can have more efficient and productive meetings!
At Lineup, we talk to a lot of organizations about their SME programs; it’s our job! If there’s one consistent thing we hear from credentialing organizations is that they are evaluating their SMEs but on an informal basis. They get together after a workshop to discuss how the workshop went and the particulars of individual performance. Maybe they store the details of that conversation somewhere, maybe they don’t, but storing that information in a transparent place where your team can find and learn from it is an important step in improving your workshops. It’s opportunity for learning. Is a particular SME standing out that should be fast-tracked for a leadership role? Is there someone that is struggling a bit that should be paired with a more experienced item writer? Maybe a SME has been with your program too long, and it’s time to transition their efforts into a new capacity. All this critical evaluation data should be collected and stored in a single repository, instantly giving your team access to valuable insights to make informed decision for your next successful workshop.
I’m sure you’ve got a tight group of item writers that you can absolutely count on to get quality items written in a timely manner. But your exam team should constantly be trying to expand that pool no matter how small your item writing needs are. Not only does this mantra lend itself to the diversity topic discussed above, but we believe that SMEs have a useful shelf life for item writing and eventually need move on to helping your organization in other ways. SMEs that have been in your item writing program for too long will tend to dominate meetings and items will have a bias toward their viewpoint of the profession or material at hand. Recruit new SMEs so that your program can work new viewpoints into your workshops and keep any one person from dominating the material.
For the Lineup team, diversity is an issue that is close to our hearts. We strive for diversity, and our goal to build more diverse teams is a founding principle that lead us to build Lineup. It’s not just a shiny object discussion that will fade with time. You should take it seriously, not just because it’s the right thing to do but also because it will produce better outcomes! There is consensus among the scientific community that creating more diverse teams will lead that team to avoid bias, process facts more clearly, and avoid groupthink. In addition, diverse teams have been scientifically shown to be more innovative. Your exam team should strive for all types of diversity. Push to include newcomers to your field in item writers. We’ve even come across organizations that are including interns in their item writing workshops! Avoid cultural bias by including SMEs of different age groups and regions of the country. Incorporating diversity into your item writing workshop will not only lead to better outcomes in your workshops, but a more valid and defensible exam!
An item writing workshop is not only an important event in properly maintaining a defensible exam, it’s an extension of your entire SME program. Your team should be constantly striving to improve the engagement and experience of your SMEs to work towards creating a better exam. While the tips listed in this blog aren’t the only way to improve your workshops, I think they are a fantastic step in the right direction for any SME program!