Hostname: page-component-f554764f5-fr72s Total loading time: 0 Render date: 2025-04-23T05:49:10.682Z Has data issue: false hasContentIssue false
  • English
  • Français

Reliability of a Longitudinal Sequence of Scale Ratings

Published online by Cambridge University Press:  01 January 2025

Annouschka Laenen ,
Ariel Alonso ,
Geert Molenberghs  and
Tony Vangeneugden
Annouschka Laenen*
Affiliation:
Hasselt University
Ariel Alonso
Affiliation:
Hasselt University
Geert Molenberghs
Affiliation:
Hasselt University
Tony Vangeneugden
Affiliation:
Tibotec, Johnson & Johnson
*
Requests for reprints should be sent to Annouschka Laenen, Hasselt University, Hasselt, Belgium. E-mail: annouschka.laenen@uhasselt.be
Get access Rights & Permissions [Opens in a new window]

Abstract

Reliability captures the influence of error on a measurement and, in the classical setting, is defined as one minus the ratio of the error variance to the total variance. Laenen, Alonso, and Molenberghs (Psychometrika 73:443–448, 2007) proposed an axiomatic definition of reliability and introduced the RT coefficient, a measure of reliability extending the classical approach to a more general longitudinal scenario. The RT coefficient can be interpreted as the average reliability over different time points and can also be calculated for each time point separately. In this paper, we introduce a new and complementary measure, the so-called RΛ, which implies a new way of thinking about reliability. In a longitudinal context, each measurement brings additional knowledge and leads to more reliable information. The RΛ captures this intuitive idea and expresses the reliability of the entire longitudinal sequence, in contrast to an average or occasion-specific measure. We study the measure’s properties using both theoretical arguments and simulations, establish its connections with previous proposals, and elucidate its performance in a real case study.

Keywords

reliabilitylinear mixed modellongitudinal datapsychiatryrating scale
Type
Theory and Methods
Information
Psychometrika , Volume 74 , Issue 1 , March 2009 , pp. 49 - 64
Copyright
Copyright © 2008 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

The authors are grateful to J&J PRD for kind permission to use their data. We gratefully acknowledge support from Belgian IUAP/PAI network “Statistical Techniques and Modeling for Complex Substantive Questions with Complex Data.”

References

Alonso, A., Geys, H., Molenberghs, G., Vangeneugden, T. (2002). Investigating the criterion validity of psychiatric symptom scales using surrogate marker validation methodology. Journal of Biopharmaceutical Statistics, 12, 161–179.CrossRefGoogle ScholarPubMed
Alonso, A., Geys, H., Molenberghs, G., Kenward, M.G. (2004). Validation of surrogate markers in multiple randomized clinical trials with repeated measurements: canonical correlation approach. Biometrics, 60, 845–853.CrossRefGoogle ScholarPubMed
Bost, J.E. (1995). The effect of correlated errors on generalizability and dependability coefficients. Applied Psychological Measurement, 19(2), 191–203.CrossRefGoogle Scholar
Brown, W. (1910). Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3, 296–322.Google Scholar
Cole, D.A., Martin, N.C., Steiger, J.H. (2005). Empirical and conceptual problems with longitudinal trait-state models: introducing a trait-state-occasion model. Psychological Methods, 10(1), 3–20.CrossRefGoogle ScholarPubMed
Cronbach, L.J., Gleser, G.C., Nanda, H., Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles, New York: Wiley.Google Scholar
Diggle, P.J., Liang, K.-Y., Zeger, S.L. (1994). Analysis of longitudinal data, Oxford: Clarendon Press.Google Scholar
Heise, D.R. (1969). Separating reliability and stability in test-retest correlation. American Sociological Review, 34, 93–101.CrossRefGoogle Scholar
Hertzog, C., Nesselroade, J.R. (1987). Beyond autoregressive models: some implications of the trait-state distinction for the structural modeling of developmental change. Child Development, 58, 93–109.CrossRefGoogle ScholarPubMed
Jagodzinski, W., Kühnel, S.M. (1987). Estimation of reliability and stability in single-indicator multiple-wave models. Sociological Methods and Research, 15, 219–258.CrossRefGoogle Scholar
Johnson, R.A., Wichern, D.W. (1998). Applied multivariate statistical analysis, (4th ed.). Englewood Cliffs: Prentice-Hall.Google Scholar
Kenny, D.A., Zautra, A. (1995). The trait-state-error model for multiwave data. Journal of Consulting and Clinical Psychology, 63(1), 52–59.CrossRefGoogle ScholarPubMed
Laenen, A., Alonso, A., Molenberghs, G. (2007). A measure for the reliability of a rating scale based on longitudinal clinical trial data. Psychometrika, 73, 443–448.CrossRefGoogle Scholar
Laenen, A., Alonso, A., Molenberghs, G., Vangeneugden, T. (2009). A family of parameters to investigate the reliability of a psychiatric symptom scale. Journal of the Royal Statistical Society, Series A, 172, 1–17.Google Scholar
Liang, K.-Y., Zeger, S.L. (1986). Longitudinal data analysis using generalized linear models. Biometrika, 73, 13–22.CrossRefGoogle Scholar
Lord, F.M., Novick, M.R. (1968). Statistical theories of mental test scores, Reading: Addison-Wesley.Google Scholar
Molenberghs, G., Kenward, M.G. (2007). Missing data in clinical studies, Chichester: Wiley.CrossRefGoogle Scholar
Peuskens, J.the Risperidone Study Group (1995). Risperidone in the treatment of chronic schizophrenic patients: a multinational, multicentre, double-blind, parallel-group study versus haloperidol. British Journal of Psychiatry, 166, 712–726.CrossRefGoogle ScholarPubMed
Raykov, T. (2000). A method for examining stability in reliability. Multivariate Behavioral Research, 35(3), 289–305.CrossRefGoogle ScholarPubMed
Royston, P., Atman, D.G. (1994). Regression using fractional polynomials of continuous covariates: parametric modelling. Applied Statistics, 43(3), 429–467.CrossRefGoogle Scholar
Rubin, D.B. (1976). Inference and missing data. Biometrika, 63, 581–592.CrossRefGoogle Scholar
Searle, S.R. (1982). Matrix algebra useful for statistics, New York: Wiley.Google Scholar
Smith, P.L., Luecht, R.M. (1992). Correlated effects in generalizability studies. Applied Psychological Measurement, 16(3), 229–235.CrossRefGoogle Scholar
Spearman, C. (1910). Correlation calculate from faulty data. British Journal of Psychology, 3, 271–295.Google Scholar
Tisak, J., Tisak, M.S. (1996). Longitudinal models of reliability and validity: a latent curve approach. Applied Psychological Measurement, 20, 275–288.CrossRefGoogle Scholar
Vangeneugden, T., Laenen, A., Geys, H., Renard, D., Molenberghs, G. (2004). Applying linear mixed models to estimate reliability in clinical trial data with repeated measurements. Controlled Clinical Trials, 25, 13–30.CrossRefGoogle ScholarPubMed
Verbeke, G., Molenberghs, G. (2000). Linear mixed models for longitudinal data, New York: Springer.Google Scholar
Verbyla, A.P., Cullis, B.R., Kenward, M.G., Welham, S.J. (1999). The analysis of designed experiments and longitudinal data by using smoothing splines. Applied Statistics, 48, 269–311.Google Scholar
Werts, C.E., Linn, C.E., Jøreskog, K.G. (1977). A simplex model for analyzing academic growth. Educational and Psychological Measurement, 37(3), 745–756.CrossRefGoogle Scholar
Werts, C.E., Breland, H.M., Grandy, J., Rock, D.R. (1980). Using longitudinal data to estimate reliability in the presence of correlated measurement errors. Educational and Psychological Measurement, 40, 19–29.CrossRefGoogle Scholar
Wiley, D.E., Wiley, J.A. (1970). The estimation of measurement error in panel data. American Sociological Review, 35, 112–117.CrossRefGoogle Scholar