SCALES OF MEASUREMENT

In Chapter 1, we indicated that variables can take on different values. Here, we will deal with the process of associating values with variables and how to summarize the sets of values we obtain from that process. This will prove to be important in guiding us toward situations in which it is appropriate to use the ANOVA technique.

THE PROCESS OF MEASUREMENT

Measurement represents a set of rules informing us of how values are assigned to objects or events. A scale of measurement describes a specific set of rules. Psychology became intimately familiar with scales of measurement when S. S. Stevens, in 1946, wrote a response to a committee of the British Association for the Advancement of Science. That committee was dealing with the possibility of measuring sensory events, and the members could not agree on whether or not it was possible to perform such measurements. Because they had focused on the work that Stevens had done to measure loudness of sound, he felt that a response to the committee from him was appropriate. But instead of arguing with the committee, Stevens adopted a different strategy, presenting instead a theory of measurement. He elaborated on this theory in 1951 when, as the editor of the Handbook of Experimental Psychology, he gave prominent treatment to this topic in his lead chapter.

Stevens identified four scales in his theory: nominal, ordinal, interval, and ratio scales in that order.

A three-way within-subjects design is one in which the levels of three within-subjects variables are combined factorially. Given that it is a within-subjects design, we still require that the participants contribute a data point to each of the separate cells (each of the unique combinations of the levels of the independent variables) of the design. The logistics of running a three-way study can be challenging but, partly because the subjects are their own controls and partly because each within-subject effect of interest is associated with its own error term (thus partitioning the total within-subjects error into separate error components), such a design has a considerable amount of statistical power. The key is to find appropriate applications of this design that are relatively immune to carry-over contamination effects.

A NUMERICAL EXAMPLE OF A THREE-WAY WITHIN-SUBJECTS DESIGN

The hypothetical data set that will serve as our example uses three time-independent variables as shown in Figure 12.1. We are engaged in market research studying the product look of powdered laundry detergent. One independent variable is the color of the powder: it is colored white, sea green, or baby blue. The second independent variable is whether or not the powder contains granules (flecks of bright shiny additions that in reality contribute nothing to the cleaning power of the detergent). The third independent variable is the color of the box: It is colored either aqua or orange.

WHAT IS ANALYSIS OF VARIANCE?

Analysis of variance (ANOVA) is a statistical technique used to evaluate the size of the difference between sets of scores. For example, a group of researchers might wish to learn if the room color in which college students are asked to respond to questions assessing their mood can affect their reported mood. Students are randomly assigned to complete a mood inventory in one of two rooms. Random assignment, one of the hallmarks of experimental design, is used in an attempt to assure that there is no bias in who is placed into which group by making it equally likely that any one person could have been assigned to either group. One of the rooms is painted a soft shade of blue that was expected to exert a calming effect on the students; the other room is painted a bright red that was presumed to be more agitating than calming. Higher numerical scores on the mood inventory indicate a more relaxed mood. At the end of the study, we score the mood inventory for all participants.

The research question in this example is whether mood as indexed by the score on the mood inventory was affected by room color. To answer this question, we would want to compare the mood scores of the two groups. If the mood scores obtained in the blue room were higher overall than those obtained in the red room, we might be inclined to believe that room color influenced mood.

HISTORICAL OVERVIEW

The SPSS Web site provides a comprehensive history of the company and the software they have produced. Here is a synopsis of that information. SPSS, an acronym for Statistical Package for the Social Sciences, is a set of statistical analysis procedures housed together within a large application. The software was developed in 1968 by two doctoral students at Stanford (Norman H. Nie and Dale H. Brent) and one recent Stanford graduate (Hadlai Hull). Nie and Hull brought the software to the University of Chicago and published the first manual in 1970. They incorporated the company in Illinois in 1975.

SPSS began to be used on mainframe computers, the only kind of computer available in many organizations such as universities. A user would sit down at a computer terminal and type SPSS syntax (the programming language) onto computer cards via a card-reader machine, and in the early 1980s, onto a blank screen. The syntax would look like word strings but conformed to rules that enabled the software to perform the statistical analysis that was specified by the syntax.

SPSS was first marketed for personal computers in 1984. The revolutionary difference in that marketplace was the use of a graphical user interface (abbreviated GUI but thought of by most people today as using a mouse to point and click) to make selections from dialog screens. These selections were translated “behind the scenes” to SPSS syntax, but the syntax was not directly presented on the screen to the human user.

EXPERIMENTAL AND STATISTICAL CONTROL

The designs we have discussed to this point represent techniques of experimental control in order to reduce error variance in a research study (Kirk, 1995). It is also possible to reduce error variance by employing statistical control using analysis of covariance (ANCOVA). Statistical control is used when we know how participants stand (we know their score) on an additional variable that was not part of or could not readily be incorporated into the experimental design. This additional variable is brought into the data analysis by treating it as a covariate. Although covariates can be categorical or quantitative, we will limit our discussion and data analysis procedures to quantitatively measured covariates.

The covariate represents a potential source of variance that has not been experimentally controlled but could covary with the dependent variable and could therefore confound our interpretation of the results. By identifying that variable as a covariate and collecting measures of it on the study participants, it is possible to “remove” or “neutralize” its effect on the dependent variable (by statistical procedures described in this chapter) prior to analyzing the effects of the independent variable on the dependent variable.

A SIMPLE ILLUSTRATION OF COVARIANCE

Consider a test of math skills that we might administer to school children in a particular grade. We wish to compare girls and boys in their math problem solving skills.

THE CONCEPT OF WITHIN-SUBJECTS VARIANCE

In between-subjects designs, participants contribute a single value of the dependent variable, and different participants are represented under each level of the independent variable. The situation is different for within-subjects designs. Here, we register a value of the dependent variable in each and every condition of the study for every one of the participants or cases in the study. That is, each participant is measured under each level of the independent variable.

Within-subjects designs are still univariate studies in that there is only one dependent variable in the design. As is true for between-subjects designs, the dependent variable is operationalized in a particular manner. What differentiates between-subjects and within-subjects designs is the number of such measurements in the data file that are associated with each participant: In between-subjects designs, researchers record just one data point (measurement) for each participant; in within-subjects designs, researchers record as many measurements for each participant as there are conditions in the study.

Within-subjects designs can theoretically contain any number of independent variables combined in a factorial manner. Practically, because all participants must contribute data under each level of each independent variable, the logistics of the data collection procedures can become quite challenging with several independent variables in the design. Also, there is the possibility that subjects could be affected or changed by exposure to one level of an independent variable, a situation known as carry-over effects, such that it may not be appropriate to expose them to more than that single level; the likelihood of eliminating all of the possible carry-over effects diminishes with each additional independent variable included in the design (we will discuss carry-over effects more in Section 10.4).

TO THE INSTRUCTOR

The present text is an exploration of univariate methodology where the effects of one or more independent variables are assessed on a single dependent variable. Such univariate designs are ubiquitous in the social, behavioral, and biological science literature. We have chosen, in this book, to focus our efforts on analysis of variance (ANOVA). Issues concerning multivariate methodology, including multiple regression analysis, are not covered in the present text as a result of space limitations, but they are addressed in a companion text (see Meyers, Gamst, & Guarino, 2006).

This book owes both a conceptual and computational debt to early ANOVA pioneers, beginning with the seminal work of Fisher (1925, 1935), who focused on solving agricultural problems with experimental methods. Fisher's early work was adapted to other fields, including the social and behavioral sciences, and in doing so moved from childhood to early adolescence with the work of Baxter (1940, 1941), Crutchfield (1938), Garrett and Zubin (1943), Lindquist (1940), Snedecor (1934), and Yates (1937). By the 1950s, ANOVA procedures were well established within most social and behavioral sciences (e.g., Cochran & Cox, 1957; Lindquist, 1953; Scheffé, 1959).

Beginning in the early 1960s, ANOVA procedures were further delineated and popularized by Winer (1962, 1971) and Winer, Brown, and Michels (1991). These works, while sometimes challenging to read, were considered the “gold standard” by many ANOVA practitioners.

PARTITIONING OF THE VARIANCE

The term analysis of variance is very descriptive of the process we use in the statistical treatment of the data. In a general sense, to analyze something is to examine the individual elements of a whole. In ANOVA, we start with a totality and break it apart – partition it – into portions that we then examine separately.

The totality that we break apart in ANOVA, as you might suspect from the name of the procedure, is the variance of the scores on the dependent variable. As you recall from Chapter 2, variance is equal to the sum of squares – the sum of the squared deviations from the mean – divided by the degrees of freedom. In an ANOVA design, it is the variance of the total set of scores that is being partitioned. The various ANOVA designs are different precisely because they allow this total variance to be partitioned in different ways.

ANOVA is a statistical procedure that allows us to partition (divide) the total variance measured in the study into its sources or component parts. This total measured variance is the variance of the scores that we have obtained when participants were measured on the dependent variable. Therefore, whenever we talk about the variance associated with a given partition or whenever we talk about the total variance, we are always referring to the variance of the dependent variable.

COMBINING TWO WITHIN-SUBJECTS FACTORS

A two-way within-subjects design is one in which the levels of two within-subjects variables are combined factorially. Recall from Chapter 11 that within-subjects independent variables can be either time related or not. There are three possible combinations of types of within-subjects variables that can be represented in a two-way design: (a) both can be time independent, (b) one can be time independent and the other time based, and (c) both can be time based. This latter combination is not often found in the research literature; the other two are more common, and we provide illustrations of them here.

• Both variables time independent: Students can be asked to indicate if short bursts of music are either rock or classical (music variable) under either silent or noisy conditions (noise variable). In such a situation, participants would be exposed randomly to the various combinations of the independent variables. The dependent variable could be speed of recognition of the music.

• […]

Introduction

Traditional, frequentist-based, statistical methods have provided researchers with a lens through which they could fruitfully examine the world. These well-known methods for both continuous (e.g., regression) and discrete outcome data (e.g., contingency table procedures) have proved their worth innumerable times. Yet, all methods have limits of applicability and these are no exceptions to that rule. The growth of analogous robust procedures have helped to provide the unwary analyst with protection against unexpected events – usually longer-tailed-than-Gaussian error distributions – but despite the proven robustness of these methods, careful thought must accompany any analysis. If one blindly utilizes statistical methods even those as well established as the Mantel–Haenszel (MH) test considered here, one does so with some risk. The computationally intensive procedures that we have used in fitting the Bayesian testlet models are continually becoming “cheaper and cheaper” and have made a Bayesian approach to many problems a practical lens through which phenomena can be viewed. Such methods provide further protection against the unknown. Remember Donald Rubin's comment, quoted in Chapter 7, that

In this chapter, we provide a vivid illustration of this in our study of differential item functioning (DIF).