Hostname: page-component-745bb68f8f-l4dxg Total loading time: 0 Render date: 2025-02-04T10:34:12.999Z Has data issue: false hasContentIssue false
  • English
  • Français

Assessing Growth in a Diagnostic Classification Model Framework

Published online by Cambridge University Press:  01 January 2025

Matthew J. Madison  and
Laine P. Bradshaw
Matthew J. Madison*
Affiliation:
Clemson University
Laine P. Bradshaw
Affiliation:
University of Georgia
*
Correspondence should be made to Matthew J. Madison, Department of Education and Human Development, Clemson University, 226 Holtzendorff Hall, Clemson, SC 29634, USA. Email: mjmadis@clemson.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

A common assessment research design is the single-group pre-test/post-test design in which examinees are administered an assessment before instruction and then another assessment after instruction. In this type of study, the primary objective is to measure growth in examinees, individually and collectively. In an item response theory (IRT) framework, longitudinal IRT models can be used to assess growth in examinee ability over time. In a diagnostic classification model (DCM) framework, assessing growth translates to measuring changes in attribute mastery status over time, thereby providing a categorical, criterion-referenced interpretation of growth. This study introduces the Transition Diagnostic Classification Model (TDCM), which combines latent transition analysis with the log-linear cognitive diagnosis model to provide methodology for analyzing growth in a general DCM framework. Simulation study results indicate that the proposed model is flexible, provides accurate and reliable classifications, and is quite robust to violations to measurement invariance over time. The TDCM is used to analyze pre-test/post-test data from a diagnostic mathematics assessment.

Keywords

diagnostic classification modelcognitive diagnosis modellatent transition analysisitem parameter driftmeasurement invariancegrowthpre-test/post-test design
Type
Original Paper
Information
Psychometrika , Volume 83 , Issue 4 , December 2018 , pp. 963 - 990
Copyright
Copyright © 2018 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Andersen, E. B.(1985).Estimating latent correlations between repeated testings.Psychometrika,50,3–16.Google Scholar
Baum, L. E., &Petrie, T.(1966).Statistical inference for probabilistic functions of finite state Markov chains.The Annals of Mathematical Statistics,37(6),1554–1563.CrossRefGoogle Scholar
Bock, R. D.,Muraki, E., &Pfeiffenberger, W.(1988).Item pool maintenance in the presence of item parameter drift.Journal of Educational Measurement,25,275–285.CrossRefGoogle Scholar
Bottge, B. A.,Heinrichs, M.,Chan, S., &Serlin, R.(2001).Anchoring adolescents’ understanding of math concepts in rich problem solving environments.Remedial and Special Education,22,299–314.CrossRefGoogle Scholar
Bottge, B. A.,Heinrichs, M.,Chan, S. Y.,Mehta, Z. D., &Watson, E.(2003).Effects of video-based and applied problems on the procedural Math skills of average- and low achieving adolescents.Journal of Special Education Technology,18(2),5–22.CrossRefGoogle Scholar
Bottge, B. A.,Ma, X.,Gassaway, L.,Toland, M.,Butler, M., &Cho, S. J.(2014).Effects of blended instructional models on math performance.Exceptional Children,80,237–255.CrossRefGoogle Scholar
Bottge, B. A.,Toland, M.,Gassaway, L.,Butler, M.,Choo, S.,Griffen, A. K., &Ma, X.(2015).Impact of enhanced anchored instruction in inclusive math classrooms.Exceptional Children,81,(2),158–175.CrossRefGoogle Scholar
Bradshaw, L.Rupp, A. A., &Leighton, J. P.(2016).Diagnostic classification models.The handbook of cognition and assessment: Frameworks, methodologies, and applications,West SussexWiley-Blackwell 297–327.CrossRefGoogle Scholar
Bradshaw, L., &Madison, M. J.(2016).Invariance properties for general diagnostic classification models.International Journal of Testing,16,(2),99–118.CrossRefGoogle Scholar
Bradshaw, L., & Templin, J. (2014). The little model that couldn’t: How the DINA model misclassifies students and hides important effects. Paper presented at the annual meeting of the Northeastern Educational Research Association in Trumbull, CT.Google Scholar
Collins, L. M.,Lanza, S. T.(2010).Latent class and latent transition analysis: With applications in the social, behavioral, and health sciences,New YorkWileyGoogle Scholar
Collins, L. M.,Wugalter, S. E.(1992).Latent class models for stage-sequential dynamic latent variables.Multivariate Behavioral Research,27,131–157.CrossRefGoogle Scholar
Corbett, A. T.,Anderson, J. R.(1995).Knowledge tracing: Modeling the acquisition of procedural knowledge.User Modeling and User-adapted Interaction,4,253–278.CrossRefGoogle Scholar
de la Torre, J.(2011).The generalized DINA model framework.Psychometrika,76,(2),179–199.CrossRefGoogle Scholar
Embretson, S. E.(1991).A multidimensional latent trait model for measuring learning and change.Psychometrika,56,495–515.CrossRefGoogle Scholar
Fischer, G. H.de Gruijter, D. NM.,van der Kamp, L. JT.(1976).Some probabilistic models for measuring change.Advances in psychological and educational measurement,New YorkWiley 97–110.Google Scholar
Fischer, G. H.(1989).An IRT-based model for dichotomous longitudinal data.Psychometrika,54,599–624.CrossRefGoogle Scholar
George, A. C.,Robitzsch, A.(2014).Multiple group cognitive diagnosis models, with an emphasis on differential item functioning.Psychological Test and Assessment Modeling,56,(4),405–432.Google Scholar
Goldstein, H.(1983).Measuring changes in educational attainment over time: Problems and possibilities.Journal of Educational Measurement,33,315–332.Google Scholar
Han, K. T., & Guo, F. (2011). Potential impact of item parameter drift due to practice and curriculum change on item calibration in computerized adaptive testing. GMAC Research Reports, RR-11-02.Google Scholar
Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign.Google Scholar
Haertel, E. H.(1989).Using restricted latent class models to map the skill structure of achievement items.Journal of Educational Measurement,26,333–352.CrossRefGoogle Scholar
Henson, R.,Douglas, J.(2005).Test construction for cognitive diagnostics.Applied Psychological Measurement,29,262–277.CrossRefGoogle Scholar
Henson, R.,Templin, J.,Willse, J.(2009).Defining a family of cognitive diagnosis models using log linear models with latent variables.Psychometrika,74,191–210.CrossRefGoogle Scholar
Hochberg, Y.(1988).A sharper Bonferroni procedure for multiple tests of significance.Biometrika,75,(4),800–802.CrossRefGoogle Scholar
Holland, P. W.,Wainer, H.(1993).Differential item functioning,Hillsdale, NJErlbaumGoogle Scholar
Huff, K.,Goodman, D. P.Leighton, J. P.,Gierl, M. J.(2007).The demand for cognitive diagnostic assessment.Cognitive diagnostic assessment for education: theory and applications,LondonCambridge University Press 19–60.CrossRefGoogle Scholar
Jang, E. E.(2005).A validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG TOEFL (unpublished doctoral dissertation),Champaign, ILUniversity of IllinoisGoogle Scholar
Junker, B. W.,Sijtsma, K.(2001).Cognitive assessment models with few assumptions, and connections with nonparametric item response theory.Applied Psychological Measurement,25,258–272.CrossRefGoogle Scholar
Jurich, D. P.,Bradshaw, L. P.(2014).An illustration of diagnostic classification modeling in student learning outcomes assessment.International Journal of Testing,14,49–72.CrossRefGoogle Scholar
Kaya, Y.,Leite, W.(2017).Assessing change in latent skills across time with longitudinal cognitive diagnosis modeling: An evaluation of model performance.Educational and Psychological Measurement,77,(3),369–388.CrossRefGoogle ScholarPubMed
Kunina-Habenicht, O.,Rupp, A. A.,Wilhem, O.(2012).The impact of model misspecification on parameter estimation and item-fit assessment in log-linear diagnostic classification models.Journal of Educational Measurement,49,59–81.CrossRefGoogle Scholar
Langeheine, R.,Pannekoek, J.,van de Pol, F.(1996).Bootstrapping goodness-of-fit measures in categorical data analysis.Sociological Methods and Research,24,492–516.CrossRefGoogle Scholar
Lanza, S. T.,Collins, L. M.(2008).A new SAS procedure for latent transition analysis: Transitions in dating and sexual behavior.Developmental Psychology,42,(2),446–456.CrossRefGoogle Scholar
Lanza, S. T.,Patrick, M. E.,Maggs, J. L.(2010).Latent transition analysis: Benefits of a latent variable approach to modeling transitions in substance use.Journal of Drug Issues,40,93–120.CrossRefGoogle ScholarPubMed
Lao, H., & Templin, J. (2016). Estimation of diagnostic classification models without constraints: Issues with class label switching. Paper presented at the annual meeting of the National Council on measurement in education in Washington, DC.Google Scholar
Lee, W.,Cho, S. J.(2017).The consequences of ignoring item parameter drift in longitudinal item response models.Applied Measurement in Education,30,(2),129–146.CrossRefGoogle Scholar
Li, F.,Cohen, A.,Bottge, B.,Templin, J.(2015).A latent transition analysis model for assessing change in cognitive skills.Educational and Psychological Measurement,76,(2),181–204.CrossRefGoogle ScholarPubMed
Maydeu-Olivares, A.Reise, S. P.,Revicki, D. A.(2015).Evaluating the fit of IRT models.Handbook of item response theory modeling: Applications to typical performance assessment,New York, NYTaylor & Francis (Routledge) 111–127.Google Scholar
Meredith, W.,Millsap, R. E.(1992).On the misuse of manifest variables in the detection of measurement invariance.Psychometrika,57,(2),289–311.CrossRefGoogle Scholar
Muthén, L. K., & Muthén, B. O. (1998–2012). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén.Google Scholar
Raju, N. S.(1988).The area between two item characteristic curves.Psychometrika,53,(4),495–502.CrossRefGoogle Scholar
Roberts, T. J.,Ward, S. E.(2011).Using latent transition analysis in nursing research to explore change over time.Nursing Research,60,(1),73–79.CrossRefGoogle ScholarPubMed
Rupp, A. A.,Templin, J.,Henson, R.(2010).Diagnostic measurement: Theory, methods, and applications,New YorkGuilfordGoogle Scholar
Sinharay, S.,Haberman, S. J.(2014).How often is the misfit of item response theory models practically significant?.Educational Measurement: Issues and Practice,33,(1),23–35.CrossRefGoogle Scholar
Templin, J.,Bradshaw, L.(2013).Measuring the reliability of diagnostic classification model examinee estimates.Journal of Classification,30,251–275.CrossRefGoogle Scholar
Templin, J.,Henson, R.(2006).Measurement of psychological disorders using cognitive diagnosis models.Psychological Methods,11,287–305.CrossRefGoogle ScholarPubMed
Templin, J.,Hoffman, L.(2013).Obtaining diagnostic classification model estimates using Mplus.Educational Measurement: Issues and Practice,32,(3),37–50.CrossRefGoogle Scholar
von Davier, M. (2005). A general diagnostic model applied to language testing data (Research Report No. RR-05–16). Princeton, NJ: Educational Testing Service.Google Scholar
Wang, S.,Yang, Y.,Culpepper, S., &Douglas, J.(2018).Tracking skill acquisition with cognitive diagnosis models: A higher-order hidden Markov model with covariates.Journal of Educational and Behavioral Statistics,47,57–87.CrossRefGoogle Scholar
Wells, C. S.,Subkoviak, M. J., &Serlin, R. C.(2002).The effect of item parameter drift on examinee ability estimates.Applied Psychological Measurement,26,77–87.CrossRefGoogle Scholar