Hostname: page-component-5b777bbd6c-f9nfp Total loading time: 0 Render date: 2025-06-19T12:12:21.447Z Has data issue: false hasContentIssue false
  • English
  • Français

Measuring Nominal Scale Agreement Between a Judge and a Known Standard

Published online by Cambridge University Press:  01 January 2025

D. D. Wackerly ,
J. T. McClave  and
P. V. Rao
D. D. Wackerly*
Affiliation:
University of Florida
J. T. McClave
Affiliation:
University of Florida
P. V. Rao
Affiliation:
University of Florida
*
Requests for reprints should be sent to D. D. Wackerly, Department of Statistics, Nuclear Sciences Center, University of Florida, Gainesville, Florida 32611.
Get access Rights & Permissions [Opens in a new window]

Abstract

Two designs for comparing a judge's ratings with a known standard are presented and compared. Design A pertains to the situation where the judge is asked to categorize each of N subjects into one of r (known) classes with no knowledge of the actual number in each class. Design B is employed when the judge is given the actual number in each class and is asked to categorize the individuals subject to these constraints. The probability distribution of the total number of correct choices is developed in each case. A power comparison of the two procedures is undertaken.

Keywords

categorical datanominal dataFisher’s tea-tasting experimentFleiss’ Kappamultinomial distributionpower comparisonscomputer algorithm
Type
Original Paper
Information
Psychometrika , Volume 43 , Issue 2 , June 1978 , pp. 213 - 223
Copyright
Copyright © 1978 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

The authors would like to thank G. E. Blume for her permission to use the data and description which appear in the introduction. Computer funding was provided by the Northeast Regional Data Center at the University of Florida. Our gratitude is also extended to the referees who made helpful suggestions regarding exposition and relevant references.

References

Agresti, A., & Wackerly, D. D. Some exact conditional tests of independence for R × C cross-classification tables. Psychometrika, 1977, 42, 111–125.CrossRefGoogle Scholar
Blume, G. E. A comparative study of dreams and related fantasies. Unpublished doctoral dissertation, University of Florida, 1977.Google Scholar
Boulton, D. M. Remark on algorithm 434. Communications of the ACM, 1974, 17, 326–326.CrossRefGoogle Scholar
Fisher, R. A. The Design of experiments, 9th ed., New York: Hafner Press, 1971.Google Scholar
Fleiss, Joseph L. Statistical methods for rates and proportions, 1973, New York: John Wiley & Sons, Inc..Google Scholar
Freeman, G. H. & Halton, J. H. Note on an exact treatment of contingency, goodness of fit and other problems of significance. Biometrika, 1951, 38, 141–149.CrossRefGoogle Scholar
Gridgemen, N. T. The lady tasting tea and allied topics. Journal of the American Statistical Association, 1959, 54, 776–783.CrossRefGoogle Scholar
Morton, R. On the efficiency of Fisher's tea-tasting designs. Journal of the Royal Statistical Society, Series B, 1975, 37, 49–53.CrossRefGoogle Scholar
Radlow, R. & Alf, E. Jr. An alternate multinomial assessment of the accuracy of the x 2 test of goodness of fit. Journal of the American Statistical Association, 1975, 70, 811–813.Google Scholar
Tocher, K. D. Extentions of the Neyman-Pearson theory to tests of discontinuous variates. Biometrika, 1950, 37, 130–144.CrossRefGoogle Scholar