The Fusion Model Skills Diagnosis System

doi:10.1017/CBO9780511611186.010

10 - The Fusion Model Skills Diagnosis System

Published online by Cambridge University Press: 23 November 2009

Robert A. Henson and

Edited by

Jacqueline Leighton and

Mark Gierl

Show author details

Louis A. Roussos: Affiliation:
Senior Psychometrician, Measured Progress
Louis V. DiBello: Affiliation:
Research Professor of Psychology, University of Illinois at Chicago
William Stout: Affiliation:
Professor of Statistics, University of Illinois at Urbana-Champaign
Sarah M. Hartz: Affiliation:
Resident, Department of Psychiatry, University of Iowa
Robert A. Henson: Affiliation:
Assistant Professor of Education Research and Methodology, University of North Carolina at Greensboro
Jonathan L. Templin: Affiliation:
Assistant Professor of Psychology, University of Kansas
Jacqueline Leighton: Affiliation:
University of Alberta
Mark Gierl: Affiliation:
University of Alberta

Book contents

Get access

Summary

INTRODUCTION

There is a long history of calls for combining cognitive science and psychometrics (Cronbach, 1975; Snow & Lohman, 1989). The U.S. standards movement, begun more than 20 years ago (McKnight et al., 1987; National Council of Teachers of Mathematics, 1989), sought to articulate public standards for learning that would define and promote successful performance by all students; establish a common base for curriculum development and instructional practice; and provide a foundation for measuring progress for students, teachers and programs. The standards movement provided the first widespread call for assessment systems that directly support learning. For success, such systems must satisfy a number of conditions having to do with cognitive-science–based design, psychometrics, and implementation. This chapter focuses on the psychometric aspects of one particular system that builds on a carefully designed test and a user-selected set of relevant skills measured by that test to assess student mastery of each of the chosen skills. This type of test-based skills level assessment is called skills diagnosis. The system that the chapter describes in detail is called the Fusion Model system.

This chapter focuses on the statistical and psychometric aspects of the Fusion Model system, with skills diagnosis researchers and practitioners in mind who may be interested in working with this system. We view the statistical and psychometric aspects as situated within a comprehensive framework for diagnostic assessment test design and implementation.

Information

Type: Chapter
Information: Cognitive Diagnostic Assessment for Education
Theory and Applications
, pp. 275 - 318

DOI: https://doi.org/10.1017/CBO9780511611186.010 [Opens in a new window]

Publisher: Cambridge University Press

Print publication year: 2007

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Book purchase

Temporarily unavailable

References

Birenbaum, M., Tatsuoka, C., & Yamada, Y. (2004). Diagnostic assessment in TIMMS-R: Between countries and within country comparisons of eighth graders' mathematics performance. Studies in Educational Evaluation, 30, 151–173.Google Scholar

Carlin, B., Xia, H., Devine, O., Tollbert, P., & Mulholland, J. (1999). Spatio-temporal hierarchical models for analyzing Atlanta pediatric asthma ER visit rates. In Gatsonis, C., Kass, R., Carlin, B., Carriquiry, A., Gelman, A., Verdinelli, I., & West, M. (Eds.), Case studies in Bayesian statistics (Vol. IV, pp. 303–320). New York: Springer-Verlag.CrossRef Google Scholar

The Commission on Instructionally Supportive Assessment. (2001, October). Building tests to support instruction and accountability: A guide for policymakers. Retrieved November 29, 2006, from http://www.testaccountability.org/

Cowles, M.K., & Carlin, B.P. (1996). Markov chain Monte Carlo convergence diagnostics: A comparative review. Journal of the American Statistical Association, 91, 883–904.CrossRef Google Scholar

Cronbach, L.J. (1975). Beyond the two disciplines of scientific psychology. American Psychologist, 30, 116–127.CrossRef Google Scholar

Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353.CrossRef Google Scholar

DiBello, L.V., Stout, W.F., & Hartz, S.M. (2000, June). Model parameterization and identifiability issues. Paper presented at the international meeting of the Psychometric Society, University of British Columbia, Vancouver.Google Scholar

DiBello, L.V., Stout, W.F., & Roussos, L.A. (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In Nichols, P.D., Chipman, S.F., & Brennan, R.L. (Eds.), Cognitively diagnostic assessment (pp. 361–389). Mahwah, NJ: Erlbaum.Google Scholar

DiBello, L.V., Stout, W.F., Roussos, L.A., Henson, R.A., Templin, J.L., & Fife, J. (2006). Skill profiles for groups of students at a given NAEP scale level: Development and demonstration. Unpublished project report, Educational Testing Service, Princeton, NJ.Google Scholar

Dorans, N.J. (1989). Two new approaches to assessing differential item functioning: Standardization and the Mantel-Haenszel method. Applied Measurement in Education, 2, 217–233.CrossRef Google Scholar

Embretson, S. (1984). A general latent trait model for response processes. Psychometrika, 49, 175–186.CrossRef Google Scholar

Gelman, A., Carlin, J.B., Stern, H.S., & Rubin, D.R. (1995). Bayesian data analysis. London: Chapman & Hall.Google Scholar

Gelman, A., & Meng, X.-L. (1996). Model checking and model improvement. In Gilks, W.R., Richardson, S., & Spiegelhalter, D.J. (Eds.), Markov chain Monte Carlo in practice. Boca Raton, FL: Chapman & Hall/CRC.Google Scholar

Hambleton, R.K., & Swaminathan, H. (1985). Item response theory. New York: Kluwer-Nijhoff.CrossRef Google Scholar

Hartz, S.M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Unpublished doctoral dissertation, University of Illinois, Champaign.Google Scholar

Hartz, S.M., & Roussos, L.A. (in press). The fusion model for skills diagnosis: Blending theory with practice (ETS Research Report). Princeton, NJ: Educational Testing Service.Google Scholar

Henson, R.A., He, X., & Roussos, L.A. (in press). Cognitive diagnosis attribute level discrimination indices. Applied Psychological Measurement.Google Scholar

Henson, R.A., Roussos, L.A., & Templin, J.L. (2004). Cognitive diagnostic “fit” indices. Unpublished ETS Project Report, Princeton, NJ.Google Scholar

Henson, R.A., Roussos, L.A., & Templin, J.L. (2005). Fusion model “fit” indices. Unpublished ETS Project Report, Princeton, NJ.Google Scholar

Henson, R.A., Stout, W.F., & Templin, J. (2005). Using cognitive model-based sum-scores to do skills diagnosis. Unpublished ETS External Team Project Report, Educational Testing Service, Princeton, NJ.Google Scholar

Henson, R.A., Templin, J.L., & Porch, F. (2004). Description of the underlying algorithm of the improved Arpeggio. Unpublished ETS External Team Project Report, Educational Testing Service, Princeton, NJ.Google Scholar

Henson, R.A., & Templin, J.L. (in press). Using model-based subscoring to do skills diagnosis. Journal of Educational Measurement.Google Scholar

Jang, E.E. (2005). A validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG TOEFL. Unpublished doctoral dissertation, University of Illinois, Champaign.Google Scholar

Jang, E.E. (2006, April). Pedagogical implications of cognitive skills diagnostic assessment for teaching and learning. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.Google Scholar

Jiang, H. (1996). Applications of computational statistics in cognitive diagnosis and IRT modeling. Unpublished doctoral dissertation, University of Illinois, Champaign.Google Scholar

Junker, B.W. (1999, November 30). Some statistical models and computational methods that may be useful for cognitively-relevant assessment. Prepared for the Committee on the Foundations of Assessment, National Research Council. Retrieved November 29, 2006, from http://www.stat.cmu.edu/∼brian/nrc/cfa/.Google Scholar

Junker, B. (2000). Some topics in nonparametric and parametric IRT, with some thoughts about the future. Unpublished manuscript, Carnegie Mellon University, Pittsburgh.Google Scholar

Landis, J.R., & Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.CrossRef Google Scholar PubMed

Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187–212.CrossRef Google Scholar

McKnight, C., Crosswhite, F., Dossey, J., Kifer, E., Swafford, J., Travers, K., & Cooney, T. (1987). The underachieving curriculum: Assessing U.S. school mathematics from an international perspective. Champaign, IL: Stripes.Google Scholar

National Council of Teachers of Mathematics. (1989). Assessment standards for school mathematics. Reston, VA: Author.

Patz, R.J., & Junker, B. (1999a). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24, 146–178.CrossRef Google Scholar

Patz, R.J., & Junker, B. (1999b). Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses. Journal of Educational and Behavioral Statistics, 24, 342–366.CrossRef Google Scholar

Pellegrino, J.W., & Chudowsky, N. (2003). The foundations of assessment. Measurement: Interdisciplinary Research and Perspectives, 1(2), 103–148.Google Scholar

Popham, W.J., Keller, T., Moulding, B., Pellegrino, J.W., & Sandifer, P. (2005). Instructionally supportive accountability tests in science: A viable assessment option?Measurement: Interdisciplinary Research and Perspectives, 3(3), 121–179.Google Scholar

Rasch, G. (1961). On general laws and the meaning of measurement in psychology. In Neyman, J. (Ed.), Proceedings of the fourth Berkeley symposium on mathematical statistics and probability (Vol. 4). Berkeley: University of California Press.Google Scholar

DiBello, L.V., Roussos, L.A., & Stout, W. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In Rao, C.R. & Sinharay, S. (Eds.), Handbook of statistics: Volume 26, psychometrics (pp. 979–1030). Amsterdam: Elsevier.Google Scholar

Roussos, L.A., DiBello, L.V., Henson, R.A., Jang, E.E., & Templin, J.L. (in press). Skills diagnosis for education and psychology with IRT-based parametric latent class models. In Embretson, S. & Roberts, J. (Eds.), New directions in psychological measurement with model-based approaches. Washington, DC: American Psychological Association.Google Scholar

Samejima, F. (1994, April). Cognitive diagnosis using latent trait models. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans.Google Scholar

Samejima, F. (1995). A cognitive diagnosis method using latent trait models: Competency space approach and its relationship with DiBello and Stout's unified cognitive-psychometric diagnosis model. In Nichols, P.D., Chipman, S.F., & Brennan, R.L. (Eds.), Cognitively diagnostic assessment (pp. 391–410). Mahwah, NJ: Erlbaum.Google Scholar

Shavelson, R.J., Black, P.J., William, D., & Coffey, J. (in press). On linking formative and summative functions in the design of large-scale assessment systems. Educational Evaluation and Policy Analysis.Google Scholar

Shepard, L.A. (2000). The role of classroom assessment in teaching and learning (CSE Technical Report 517). Los Angeles: National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles.Google Scholar

Sinharay, S. (2004). Experiences with Markov chain Monte Carlo convergence assessment in two psycho-metric examples. Journal of Educational and Behavioral Statistics, 29, 461–488.CrossRef Google Scholar

Sinharay, S. (2005). Assessing of unidimensional IRT models using a Bayesian approach. Journal of Educational Measurement, 42, 375–394.CrossRef Google Scholar

Sinharay, S. (2006). Model diagnostics for Bayesian networks. Journal of Educational and Behavioral Statistics, 31, 1–33.CrossRef Google Scholar

Sinharay, S., & Johnson, M.S. (2003). Simulation studies applying posterior model checking for assessment of the common item response theory models. (ETS Research Report No. RR-03–28). Princeton, NJ: Educational Testing Service.Google Scholar

Snow, R.E., & Lohman, D.F. (1989). Implications of cognitive psychology for educational measurement. In Linn, R.L. (Ed.), Educational measurement (3rd ed., pp. 263–331). New York: Macmillan.Google Scholar

Tatsuoka, K.K. (1990). Toward an integration of item-response theory and cognitive error diagnoses. In Frederiksen, N., Glaser, R.L., Lesgold, A.M., & Shafto, M.G. (Eds.), Diagnostic monitoring of skill and knowledge acquisition. (pp 453–488). Hillsdale, NJ: Erlbaum.Google Scholar

Tatsuoka, K.K., Corter, J., & Guerrero, A. (2003). Manual of attribute-coding for general mathematics in TIMSS studies. New York: Columbia University, Teachers College.Google Scholar

Tatsuoka, K.K., Corter, J., & Tatsuoka, C. (2004). Patterns of diagnosed mathematical content and process skills in TIMMS-R across a sample of 20 countries. American Educational Research Journal, 41, 901–926.CrossRef Google Scholar

Xin, T. (2004). Latent trait and latent class models for multi-rater, multi-attribute agreement (UMI No. 3129052). Dissertation Abstracts International, 65(04), 1337A.Google Scholar

Xin, T., Xu, Z., & Tatsuoka, K.K. (2004). Linkage between teacher quality, student achievement, and cognitive skills: A rule-space model. Studies in Educational Evaluation, 30, 205–223.CrossRef Google Scholar

Accessibility standard: Unknown

Why this information is here

This section outlines the accessibility features of this content - including support for screen readers, full keyboard navigation and high-contrast display options. This may not be relevant for you.

Accessibility Information

Accessibility compliance for the PDF of this book is currently unknown and may be updated in the future.