Pabak-OS Calculator

compute
clear
scale categories:
3 4 5 6 7 8 9


near misses:

miss by 2:

Output

N
hits
near miss
miss by two
Po
Pc
Pabak-OS
Z-score
Z P val
SE
CI 85%
CI 90%
CI 95%

PABAK-OS (Prevalance and Bias Adjusted Kappa-Ordinal Scale)


PABAK-OS is a method for calculating inter-rater reliability between two raters using an ordinal rating scale of 3,4,5,6 or 7 categories. The -OS stands for “ordinal scale”. Simple PABAK is an established index of rater agreement which controls for chance agreement (chance calculated solely from the number of categories in a scale). PABAK stands for “prevalence and bias adjusted Kappa”, and aims to avoid the peculiar, unintuitive results sometimes obtained from Cohen’s Kappa. The main limitation of simple PABAK is its suitability for nominal scales only–it is not sensitive to order. An alternative to PABAK-OS presently is Kappa-LW (linear weighted Kappa), which Cohen developed to extend Kappa to ordinal scales. Kappa-LW is a good technique, but given imbalanced agreement matrices, it can yield peculiar results, as Kappa can with ordinal scales. A better technique is the little-known and unfortunately complex “Kripendorff’s Alpha.” PABAK-OS is a simple and flexible index, permitting you to select your own “partial credit weights” for “near misses” and for “misses by 2 categories”. For more detailed explanations, references and field tests regarding PABAK-OS, we suggest you read the soon-to-be-published manuscript Parker, Vannest, & Davis (in press) PABAK-OS. Email us for a copy at kvannest (at) tamu.edu
  1. summarize your agreements in an agreement matrix, where each tally is the joint occurrence of ratings by two raters.
  2. on the PABAK-OS page, select # of scale categories.
  3. choose the partial credit weights you wish to give a “near miss” and a “miss by two” (if you wish to do so). Exact agreement receives a default weight of 1.
  4. enter your agreement matrix data into the matrix provided. The exact agreement diagonal is colored brown, “near misses” colored tan, and “misses by two” colored off-white or cream.
  5. click “compute” button to obtain a PABAK-OS summary and supplemental data. The maximum PABAK-OS is always 1. PABAK-OS can be defined as: “chance-adjusted percent of agreement on an ordinal scale, including partial credit for near misses”