Vol 01. Kappa and the procurement checklist

Good morning.

This week we published the first piece of the Reliability Primer, on the Kappa statistic and why percent agreement flatters every vendor. If you are on a procurement committee this quarter, print these six questions.

1. Report Cohen Kappa or Krippendorff Alpha, not raw agreement.
2. Quadratic weighted Kappa for any ordinal rubric.
3. Publish the confusion matrix.
4. Give a 95 percent confidence interval.
5. Break the number down by rubric row.
6. Report per subgroup.

Read the full piece at bestaigrading.com/blog/kappa-statistic-ai-grading.

Next Sunday, we run part two, on sample size.