# Comparative Judgment (Workshop) > A ready-to-run PD workshop on assessing complex work through pairwise comparative judgment. Run-of-show, handout, follow-up. *A ready-to-run professional learning workshop from K12 Academics. Source page: https://pd.k12academics.com/comparative-judgment-workshop.html* For complex qualities like writing or reasoning, people are far better at judging which of two pieces is better than at scoring each against a rubric. Comparative judgment uses this, repeatedly comparing pairs of student work, to produce reliable assessments of things rubrics struggle to capture. People judge which of two pieces is better far more reliably than they score each against a rubric; comparative judgment uses repeated pairwise comparisons to assess complex qualities, like writing or reasoning, that rubrics struggle to capture. ## Why it works Scoring complex work (essays, reasoning, open problems, portfolios) against rubrics is often unreliable, different scorers disagree, and some qualities resist being broken into rubric criteria. Comparative judgment takes a different approach rooted in a robust finding: humans are much more reliable at comparative decisions (which of these two is better?) than absolute ones (what score is this?). By having judges make many pairwise comparisons of student work, the results can be combined statistically into a reliable rank order and scores, capturing holistic quality that rubrics miss. For teachers, comparative judgment can be a powerful way to assess complex work reliably, moderate judgments across a team, and build shared understanding of quality, though it is a different model than rubric scoring and suits some purposes better than others. **The research:** Research on comparative judgment in assessment. ## The run-of-show ### Staff meeting (15 minutes) A fast introduction to plant one routine. **Goal:** everyone leaves with one thing to try. | Start | Length | Segment | What happens | Facilitator note | | --- | --- | --- | --- | --- | | 0:00 | 3 min | Hook | Do you and colleagues score the same essay the same? Name the gap. | You are surfacing rubric unreliability. | | 0:03 | 5 min | Mini-teach | Judge comparisons, not absolute scores. One slide. | | | 0:08 | 5 min | Plan one | Name complex work to compare instead of score. Turn and talk. | | | 0:13 | 2 min | Commit | Each person names a use to consider. | | ### PLC / team meeting (45 minutes) The standard version. **Goal:** every teacher leaves with something designed for a real upcoming lesson. | Start | Length | Segment | What happens | Facilitator note | | --- | --- | --- | --- | --- | | 0:00 | 5 min | Hook | Scoring each vs comparing pairs. Which is more reliable? | Let them see the difference. | | 0:05 | 8 min | Mini-teach | Comparative judgment and how it works. | | | 0:13 | 12 min | See it in action | Try comparing a few pairs of work. | Discuss quality. | | 0:25 | 12 min | Design | Plan assessing complex work by comparing. | Protect the full time. | | 0:37 | 5 min | Share | Compare judgments in pairs; build shared standards. | Keep it in pairs. | | 0:42 | 3 min | Commit + follow-up | Lock it in and set the check-in date. | | ### Half-day institute (~3 hours) A deep session for a PD day. **Goal:** built, practiced, and a rollout planned. | Start | Length | Segment | What happens | Facilitator note | | --- | --- | --- | --- | --- | | Block 1 | 45 min | Experience + why | Rubric disagreement, then the research. | | | | 15 min | Break | | | | Block 2 | 60 min | Try it | Teams compare work and build shared standards. | Discuss quality. | | Block 3 | 45 min | Practice | Plan a comparative-judgment assessment. | Reliable holistic assessment is the aim. | | Close | 15 min | Plan the rollout | Try comparative judgment; set the check-in. | | ## The core activity: assess complex work by comparing Teachers leave understanding how to assess complex work by comparison. - Pick complex work that rubrics struggle to score reliably (writing, reasoning). - Have judges compare pairs (which is better?) across many comparisons. - Combine the comparisons into a rank order and shared understanding of quality. - Use it to assess reliably and build shared standards. ## Facilitator notes - Use for complex, holistic work, and compare pairs rather than score each. - Combine comparisons into a rank. - Build shared quality understanding. ## Adapt it - **Elementary and secondary:** compare work, build shared standards. - **Team use:** moderation, shared understanding. - **All settings:** compare pairs for complex work. ## Participant handout One page for every teacher. Print it, or save it as a PDF. ### Comparative Judgment: compare, don't just score **Reliable holistic assessment:** compare pairs of work, combine into a rank, and build shared understanding of quality. - The complex work that rubrics struggle to score: __________ - How judges compare pairs (which is better?): __________ - How comparisons combine into a rank and shared understanding: __________ - When comparative judgment fits better than rubrics: __________ - A use to try: __________ ## Make it stick Compare, don't just score: - Assess complex work by comparing. - Build shared standards. - Bring a comparative-judgment use to the next PLC. ## Common questions **What is comparative judgment?** Assessing by repeatedly comparing pairs of student work (which is better?) rather than scoring each against a rubric. Many comparisons are combined statistically into a reliable rank and scores. It leverages that people judge comparisons more reliably than absolute scores. **Why use it instead of rubrics?** For complex, holistic qualities (writing, reasoning) that rubrics struggle to capture and where rubric scoring is unreliable, comparative judgment can produce more reliable results and shared understanding of quality. **When is it worth it?** For assessing complex work reliably, moderating across a team, or building shared quality standards. It is a different model than rubric scoring and suits some purposes (holistic quality) better than others. ## Go deeper - [The Learning Scientists](https://www.learningscientists.org) ## Build it into a bigger day - [Rubric Design](https://pd.k12academics.com/rubric-design-workshop.html) - [Rubric Calibration & Moderation](https://pd.k12academics.com/rubric-calibration-moderation-workshop.html) - [Analyzing Student Work](https://pd.k12academics.com/analyzing-student-work-workshop.html) --- Full workshop page: https://pd.k12academics.com/comparative-judgment-workshop.html All workshops: https://pd.k12academics.com/