ASSESSMENT AND INTERVENTION
ADVANCES IN LEARNING AND BEHAVIORAL DISABILITIES Series Editors: Thomas E. Scruggs and Margo A. Mastropieri Recent Volumes: Volume 12:
Edited by Thomas E. Scruggs and Margo A. Mastropieri
Volume 13:
Edited by Thomas E. Scruggs and Margo A. Mastropieri
Volume 14:
Educational Interventions – Edited by Thomas E. Scruggs and Margo A. Mastropieri
Volume 15:
Technological Applications – Edited by Thomas E. Scruggs and Margo A. Mastropieri
Volume 16:
Identification and Assessment – Edited by Thomas E. Scruggs and Margo A. Mastropieri
Volume 17:
Research in Secondary Schools – Edited by Thomas E. Scruggs and Margo A. Mastropieri
Volume 18:
Cognition and Learning in Diverse Settings – Edited by Thomas E. Scruggs and Margo A. Mastropieri
Volume 19:
Applications of Research Methodology – Edited by Thomas E. Scruggs and Margo A. Mastropieri
Volume 20:
International Perspectives – Edited by Thomas E. Scruggs and Margo A. Mastropieri
Volume 21:
Personnel Preparation – Edited by Thomas E. Scruggs and Margo A. Mastropieri
Volume 22:
Policy and Practice – Edited by Thomas E. Scruggs and Margo A. Mastropieri
Volume 23:
Literacy and Learning – Edited by Thomas E. Scruggs and Margo A. Mastropieri
ADVANCES IN LEARNING AND BEHAVIORAL DISABILITIES VOLUME 24
ASSESSMENT AND INTERVENTION EDITED BY
THOMAS E. SCRUGGS George Mason University, Fairfax, VA, USA
MARGO A. MASTROPIERI George Mason University, Fairfax, VA, USA
United Kingdom – North America – Japan India – Malaysia – China
Emerald Group Publishing Limited Howard House, Wagon Lane, Bingley BD16 1WA, UK First edition 2011 Copyright r 2011 Emerald Group Publishing Limited Reprints and permission service Contact:
[email protected] No part of this book may be reproduced, stored in a retrieval system, transmitted in any form or by any means electronic, mechanical, photocopying, recording or otherwise without either the prior written permission of the publisher or a licence permitting restricted copying issued in the UK by The Copyright Licensing Agency and in the USA by The Copyright Clearance Center. No responsibility is accepted for the accuracy of information contained in the text, illustrations or advertisements. The opinions expressed in these chapters are not necessarily those of the Editor or the publisher. British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN: 978-0-85724-829-9 ISSN: 0735-004X (Series)
Emerald Group Publishing Limited, Howard House, Environmental Management System has been certified by ISOQAR to ISO 14001:2004 standards Awarded in recognition of Emerald’s production department’s adherence to quality systems and processes when preparing scholarly journals for print
CONTENTS LIST OF CONTRIBUTORS
vii
PREDICTIVE VALIDITY OF DYNAMIC TESTING AND WORKING MEMORY AS IT RELATES TO READING GROWTH IN CHILDREN WITH READING DISABILITIES H. Lee Swanson and Michael Orosco
1
APPLICATIONS OF CURRICULUM-BASED MEASURES IN MAKING DECISIONS WITH MULTIPLE REFERENCE POINTS Gerald Tindal and Joseph F. T. Nese
31
EVOLVING STANDARDS OF DIAGNOSTIC ACCURACY IN PREDICTING AND AVOIDING ACADEMIC FAILURE Amanda M. VanDerHeyden
59
HOW TO USE PSYCHOLOGICAL TESTS FOR FUNCTIONAL DIAGNOSIS: THE CASE OF ASSESSMENT OF LEARNING DISABILITIES Giulia Balboni and Roberto Cubelli
79
A COMPARISON OF OBSERVATIONAL TECHNIQUES FOR ASSESSING STUDENTS’ SOCIAL BEHAVIOR Jugnu Agrawal, Dannette Allen-Bronaugh and Margo A. Mastropieri CURRICULUM-BASED MEASUREMENT FOR BEGINNING WRITERS: RECENT DEVELOPMENTS AND FUTURE DIRECTIONS Kristen L. McMaster, Kristen D. Ritchey and Erica Lembke v
93
111
vi
CONTENTS
ISSUES IN ASSESSMENT FOR INTERVENTION IN IMPLEMENTATION OF RESPONSIVENESS TO INTERVENTION MODELS Rollanda E. O’Connor and Victoria Sanchez COMORBIDITY BETWEEN ATTENTION DEFICIT HYPERACTIVITY DISORDER AND READING DISABILITIES: IMPLICATIONS FOR ASSESSMENT AND TREATMENT Ana Miranda, Marı´a Jesu´s Presentacio´n, Rebeca Siegenthaler, Carla Colomer and Vicente Pinto ASSESSMENT AND INTERVENTION IN SELFDETERMINATION Michael L. Wehmeyer SUCCESS AND FAILURE WITH TIER-2 SRSD FOR TIMED-WRITING TESTS AMONG SECONDTHROUGH FIFTH-GRADE STUDENTS WITH WRITING AND BEHAVIORAL DIFFICULTIES: IMPLICATIONS FOR EVIDENCE-BASED PRACTICE Karin Sandmel, Kristen D. Wilson, Karen R. Harris, Kathleen Lynne Lane, Steve Graham, Wendy P. Oakes, Sharlene A. Kiuhara and Trish D. Steinbrecher
149
171
213
251
DEVELOPING WRITING FLUENCY FOR ADOLESCENTS WITH DISABILITIES Linda H. Mason and Richard M. Kubina
295
ENGAGING YOUTH WITH DISABILITIES IN SCHOOL: BUILDING AND SUSTAINING POSITIVE RELATIONSHIPS Elizabeth Talbott and Lisa S. Cushing
321
IMPROVING INTERVENTION EFFECTIVENESS WITH UNIVERSITY–PUBLIC SCHOOL COHORT PARTNERSHIPS Margo A. Mastropieri, Thomas E. Scruggs, Nicole Conners, Mary Kealy, Nancy Morrison, Tina Diamond and Terry Werner
341
LIST OF CONTRIBUTORS Jugnu Agrawal
College of Education and Human Development, George Mason University, Fairfax, VA, USA
Dannette Allen-Bronaugh
College of Education and Human Development, George Mason University, Fairfax, VA, USA
Giulia Balboni
Faculty of Psychology, University of Valle d’Aosta, Aosta, Italy
Carla Colomer
Department of Developmental and Educational Psychology, University of Valencia, Valencia, Spain
Nicole Conners
Fairfax County Public Schools, Fairfax, VA, USA
Roberto Cubelli
Department of Cognitive Sciences and Education, Center for Mind/Brain Sciences, University of Trento, Trento, Italy
Lisa S. Cushing
Department of Special Education, University of Illinois, Chicago, IL, USA
Tina Diamond
College of Education and Human Development, George Mason University, Fairfax, VA, USA
Steve Graham
Department of Special Education, Vanderbilt University, Nashville, TN, USA
Karen R. Harris
Department of Special Education, Vanderbilt University, Nashville, TN, USA
Mary Kealy
Loudoun County Public Schools, Loudoun, VA, USA vii
viii
LIST OF CONTRIBUTORS
Sharlene A. Kiuhara
Department of Special Education, Vanderbilt University, Nashville, TN, USA
Richard M. Kubina
Department of Educational and School Psychology and Special Education, the Pennsylvania State University, University Park, PA, USA
Kathleen Lynne Lane
Department of Special Education, Vanderbilt University, Nashville, TN, USA
Erica Lembke
University of Missouri, Columbia, MO, USA
Linda H. Mason
Department of Educational and School Psychology and Special Education, the Pennsylvania State University, University Park, PA, USA
Margo A. Mastropieri
College of Education and Human Development, George Mason University, Fairfax, VA, USA
Kristen L. McMaster
University of Minnesota, Minneapolis, MN, USA
Ana Miranda
Department of Developmental and Educational Psychology, University of Valencia, Spain
Nancy Morrison
Fairfax County Public Schools, Fairfax, VA, USA
Joseph F. T. Nese
Department of Special Education, University of Oregon, Eugene, OR, USA
Rollanda E. O’Connor
Graduate School of Education, University of California at Riverside, CA, USA
Wendy P. Oakes
Department of Special Education, Vanderbilt University, Nashville, TN, USA
Michael Orosco
Graduate School of Education, University of California at Riverside, Riverside, CA, USA
ix
List of Contributors
Vicente Pinto
Department of Developmental, Educational and Social Psychology, and Methodology, Jaume I University, Castello´n, Spain
Marı´a Jesu´s Presentacio´n
Department of Developmental, Educational and Social Psychology, and Methodology, Jaume I University, Castello´n, Spain
Kristen D. Ritchey
School of Education, University of Delaware, Newark, DE, USA
Victoria Sanchez
Graduate School of Education, University of California at Riverside, Riverside, CA, USA
Karin Sandmel
School of Education, Johns Hopkins University, Maryland, Baltimore, MD, USA
Thomas E. Scruggs
College of Education and Human Development, George Mason University, Virginia, Fairfax, VA, USA
Rebeca Siegenthaler
Department of Developmental, Educational and Social Psychology, and Methodology, Jaume I University, Castello´n, Spain
Trish D. Steinbrecher
Department of Special Education, Vanderbilt University, Nashville, TN, USA
H. Lee Swanson
Graduate School of Education, University of California, Riverside, CA, USA
Elizabeth Talbott
Department of Special Education, University of IL, Chicago, USA
Gerald Tindal
Department of Special Education, University of Oregon, Eugene, OR, USA
Amanda M. VanDerHeyden
Education Research & Consulting, Fairhope, AL, USA
x
LIST OF CONTRIBUTORS
Michael L. Wehmeyer
Department of Special Education, University of Kansas, Lawrence, KS, USA
Terry Werner
College of Education and Human Development, George Mason University, Fairfax, VA, USA
Kristen D. Wilson
Department of Special Education, Vanderbilt University, Nashville, TN, USA
PREDICTIVE VALIDITY OF DYNAMIC TESTING AND WORKING MEMORY AS IT RELATES TO READING GROWTH IN CHILDREN WITH READING DISABILITIES H. Lee Swanson and Michael Orosco ABSTRACT The purpose of this chapter is to review our findings related to the question ‘‘Do outcomes related to dynamic assessment on a cognitive measure predict reading growth?’’ Our discussion related to the predictive validity of such procedures focused on outcomes related to a battery of memory and reading measures administered over a three-year period to 78 children (11.6 years) with and without reading disabilities (RD). Working memory (WM) tasks were presented under initial, gain, and maintenance testing conditions. The preliminary results suggested that maintenance testing conditions were significant moderators of comprehension and vocabulary growth, whereas probe scores and gain testing
Assessment and Intervention Advances in Learning and Behavioral Disabilities, Volume 24, 1–30 Copyright r 2011 by Emerald Group Publishing Limited All rights of reproduction in any form reserved ISSN: 0735-004X/doi:10.1108/S0735-004X(2011)0000024003
1
2
H. LEE SWANSON AND MICHAEL OROSCO
conditions were significant moderators of nonword fluency growth. Overall, the results suggested that the dynamic assessment of WM added significant variance in predicting later reading performance.
Children with normal intelligence but who suffer from reading disabilities (RD) experience considerable difficulty on working memory (WM) tasks (e.g., see De Beni, Palladino, Pazzaglia, & Cornoldi, 1998; de Jong, 1998; Gathercole, Alloway, Willis, & Adams, 2006; Siegel & Ryan, 1989; Swanson & Siegel, 2001; Willcutt, Pennington, Olson, Chhabildas, & Hulslander, 2005). Although WM is integrally related to a number of academic behaviors (e.g., Gathercole & Pickering, 2000a, 2000b; Pickering, 2006; Swanson, 2008; Swanson, Jerman, & Zheng, 2008), relatively few studies have been undertaken to systematically explore whether WM can be improved upon and whether such improvements predict later academic performance. The purpose of this chapter is to review our findings on whether procedures that attempt to improve WM performance in children with RD predict reading growth. The procedures we use to assess improvements in WM performance come under the rubric of dynamic assessment (DA). Dynamic assessment is one of clearest links between assessment and instruction (e.g., Caffrey, Fuchs, & Fuchs, 2008; Sternberg & Grigorenko, 2002; Swanson & Lussier, 2001) because it incorporates a child’s response to instructional feedback. Although DA is a term used to characterize a number of approaches, its distinctive feature is to determine the learners’ potential for change when given assistance (e.g., Embretson, 1992; see Grigorenko & Sternberg, 1998 for a review), and to provide a prospective measure of performance change independent of assistance (e.g., Brown & Ferrara, 1999). Unlike traditional testing procedures, score changes due to DA methods are not viewed as threatening task validity. In fact, some authors argue that construct validity will increase (e.g., Carlson & Wiedl, 1979; however see Caffrey et al., 2008; Swanson & Lussier, 2001, for a critical review). For example, DA procedures have been found to contribute unique variance to concurrent criterion measures of achievement even when the influence of traditional assessment procedures was partialed out in the analysis (e.g., Swanson, 1999). Although the literature is clear that impairments in WM in children with RD are related to reading performance, whether DA procedures add additional information to the prediction of growth in reading in children with RD beyond traditional testing procedures has not been tested. Perhaps
Predictive Validity of Dynamic Testing
3
the closest study on this issue is a cross-sectional study by Swanson (2003) which determined whether improvements in WM differentiated children with and without RD. The study compared RD and skilled readers (N ¼ 226) across four (7, 10, 13, 20) age groups on WM tasks under various WM conditions: initial (no probes or cues), gain (cues that bring performance to an asymptotic level), and maintenance conditions (asymptotic conditions without cues). The general findings were that skilled readers performed better than children with RD in all processing conditions and that concurrent reading comprehension performance was best predicted by the maintenance testing than the other testing conditions. Further, the magnitude of the difference (effect size) between RD and skilled readers increased on gain and maintenance testing conditions when compared with the initial conditions, suggesting that performance differences between ability groups were enhanced by using DA procedures. However, there was no follow-up testing of these children, raising the questions as to whether the measures were predictive of later reading performance. The work we describe in this chapter outlines our attempts to improve WM performance, via adapting procedures used in DA, and then predict later reading performance. The question of interest was whether DA procedures provide additional information (contributed unique variance) in predicting children’s reading performance relative to traditional testing procedures. To answer this question we assessed WM performance under three conditions across three years. These conditions included: (1) presentation of WM tasks without cues to assess initial performance (initial condition), (2) presentation of graduated cues to help participants access forgotten information from the initial condition and to continue the use of cues until span scores could no longer be improved upon (referred to as the gain or asymptotic testing condition), and (3) presentation of the highest span level achieved for the gain condition after a brief interlude, but without the support of cues (referred to as the maintenance condition). The administration of WM tasks under initial testing conditions were assumed to tap individual differences in idiosyncratic processing, as well as individual differences in accessing newly presented items in storage. To assess individual differences in item accessibility, probes (probes and cues are used interchangeably) were presented to help children reinstate the memory trace and/or retrieve forgotten items. This condition, referred to as the gain testing condition, allowed participants to use as many cues as necessary to access previously forgotten information. The probes were ordered in terms of explicitness, with the general prompts given first and more explicit prompts later. To assess constraints in activating previously retrieved information, the
4
H. LEE SWANSON AND MICHAEL OROSCO
testing procedures reinstated the highest level achieved successfully under gain conditions, but without cues. This condition, referred to as the maintenance condition, presented the same items retrieved in the gain conditions. The presentation of items in the maintenance testing condition was calibrated to the individual’s highest span level and allowed for the assessment of individual processing differences beyond the simple learning of items. Prior to reviewing the outcomes, we briefly describe our sample, procedures, and analysis. In this study, subgroups of children with RD were tested across three testing waves approximately one year apart. We predicted that DA procedures would differentiate WM performance among subgroups of children with RD. For example, a previous study by Swanson and Howard (2005) found that children with RD-only and children with general achievement problems could be separated in terms of performance using DA procedures. They showed that skilled readers and children considered poor readers (children with low IQ and reading scores) were more likely to change and maintain their WM score from initial testing conditions than children with RD-only or children with a combination of math/reading disabilities. Overall, the results suggested that WM deficits were less changeable for children with RD when compared to low IQ readers and skilled readers. To this end, our most recent study included 78 children selected from a school in Southern California. The mean age of the sample in the first testing wave was 11.62 years (SD ¼ 2.28) and included 27 girls and 51 boys. Sixty-six children were Anglo-American, eight AfricanAmerican, and four were of Hispanic heritage. All children were from middle-upper to upper class homes. The criterion for the average reading group was a score above the 45th percentile in reading word recognition, as well as an IQ score above 90. Classification of subgroups of children with RD was based upon standard scores from the word-reading and arithmetic subtests of the Wide Range Achievement Test 3 (WRAT3, Wilkinson, 1993). Operational criteria for children defined as the RD-only group were based upon: (a) IQ scores (in this case scores on the Raven Colored Progressive Matrices Test, Raven, 1976) above 90, (b) reading recognition scores at or below the 25th percentile, and (c) math calculation scores above the 25th percentile. These cut-off score criteria for children with RD-only matched the operational definition of RD outlined by Siegel and Ryan (1989). Children with average fluid intelligence (scores on the Raven Progressive Colored Matrices Test) and verbal IQs but whose performance on the WRAT3 in both word
Predictive Validity of Dynamic Testing
5
recognition and calculation were below the 25th percentile were classified as RD children with comorbid deficits (RD þ MD). Children whose fluid intelligence (Raven Progressive Colored Matrices Test) were in the average range but tested low (o25th percentile) in both arithmetic and word recognition and obtained low verbal IQs (o85, subtests on the WISCIII that included information and vocabulary), were defined as low verbal IQ readers. All children were monolingual in English. For comparison purposes, participants were divided into four ability groups: children with RD-only (low word recognition/average arithmetic, n ¼ 13), children with both reading and calculation deficits (comorbid group, n ¼ 15), low verbal IQ readers (low verbal IQ, low word recognition, and low arithmetic, n ¼ 27), and skilled readers (high word recognition, high arithmetic, n ¼ 23). No significant differences emerged among subgroups related to gender, ethnicity, or chronological age. Because ability groups were categorized on measures of word recognition and math skills, it was important to follow growth on measures related to other forms of reading achievement independent of the classification measures. Three criterion measures were selected: reading comprehension, nonword reading fluency, and receptive vocabulary. Because real sight words were used in the classification of reading groups, we selected pseudoword reading speed because of its strong association with phonological skills. Further, because expressive vocabulary was used to establish verbal IQ, we selected the receptive language test (PPVT-III) as a criterion measure. Reading comprehension was assessed by the Passage Comprehension subtest from the Woodcock Reading Mastery Test Revised (WRMT-R; Woodcock, 1998). Nonword reading efficiency was assessed by the Phonemic Decoding Efficiency subtest from the Test of Word Reading Efficiency (TOWRE; Wagner & Torgesen, 1999). Receptive vocabulary was assessed by The Peabody Picture Vocabulary Test-Third Edition (PPVT-III; Dunn, Dunn, Dunn, & Dunn 1997).
WORKING MEMORY MEASURES AND TESTING CONDITIONS Four WM subtests from a battery of 11 of the S-Cognitive Processing Test were selected because they represent verbal and visual-spatial processing (Swanson, 1995a, 1995b).
6
H. LEE SWANSON AND MICHAEL OROSCO
Verbal Working Memory Rhyming The purpose of this task was to assess the child’s recall of acoustically similar words. The child listened to sets of words that rhyme. Each successive word in the set was presented every 2 seconds. There were 9 word sets that ranged from 2 to 14 monosyllabic words. The dependent measure was the number of sets recalled. Before recalling the words, the child was asked whether a particular word was included in the set. For example, the child was presented the words ‘‘lip-slip-clip’’ and then asked if ‘‘ship or lip’’ was presented in the word set. They were then asked to recall the previously presented words (lip-slip-clip) in order. The dependent measure was the number of sets recalled correctly (range of 0–9). If the child omitted, inserted, or incorrectly ordered the words, a series of probe responses were presented. Probe responses continued until the child could no longer provide the correct response. For example, consider the sample item ‘‘car-star-bar-far’’ (Item #3) and the process question ‘‘Which word did I say – jar or star?’’ Consider the Probe Sequence: 1. The last word in the sequence was ‘‘far,’’ now can you tell me all the words in order? 2. The first word in the sequence was ‘‘car,’’ now can you tell me all the words in order? 3. The middle words in the sequence are ‘‘star’’ and ‘‘bar,’’ now can you tell me all the words in order? 4. All the words in order are ‘‘car-star-bar-far,’’ now can you tell me all the words? For each set of items not recalled in the correct order or for items left out or substituted, the experimenter provided a series of hints based on the error that was closest to Probe 1. That is, probes went from the least obvious hint (Probe 1) to the next explicit hint that facilitated recall of the answer. Once the appropriate hint had been identified, based on the location of the error, probes were presented in order until the correct sequence was given. For example, suppose the child for Item #3 responded car-bar-far. The child obviously left out a word in the middle so the experimenter would provide a hint related to the middle words (Probe 3 in this case). If Probe 3 did not provide the correct response, the experimenter would then move to Probe 4. In contrast, if a child responded initially by saying only car, the sequence began with Probe 1 and proceeded through all probes until the correct response was given. If a correct response did not occur after probing, the
Predictive Validity of Dynamic Testing
7
task was discontinued and the next task was administered. If a correct response did occur, the next set of items of increased difficulty was presented. The format of the probe procedures above was used for the other subtests. Digit/Sentence Task The purpose of this task was to assess the participant’s ability to remember numerical information embedded in a short sentence. The administration of items and probes followed the same format as the rhyming task. Prior to stimulus presentation, the participant was shown a figure (see Swanson, 1993, Fig. 1) depicting four strategies for recalling numerical information. These strategies were pictorial representations of rehearsal, chunking, associating, and elaborating of information. The general instructions for introducing the strategies were as follows: I’m going to read you some sentences that have information I want you to remember. All the sentences have to do with remembering an address, but I would like you to pay attention to all the information in the sentence because I will ask you a question about the sentence. After I present this information, and before you recall it, I will ask you to choose a strategy (for children under ten – the phrase, ‘‘A way of remembering the information’’ was used) that you think will best help you remember.
The experimenter then showed four pictures each depicting a person thinking about using one of the four strategies (see Swanson, 1993). As the experimenter explained each strategy, they pointed to the picture that matched the description. The experimenter stated that Some of the ways that may help you remember are: (1) saying the numbers over to yourself. For example, if I say ‘‘2-4-6-3 Bader Street,’’ you would say to yourself ‘‘24-6-3’’ over and over again, or (2) you might say some numbers together in pairs. For example, if I say the numbers ‘‘2-4-6-3 Bader Street,’’ you would say ‘‘24 and 63,’’ or (3) you may just want to remember that the numbers go with a particular street and location. For example, if I say ‘‘2-4-6-3 Bader Street’’ you would remember that 2-4-6-3 and Bader Street goes together, or (4) you might think of other things that go with the numbers. For example, if I say ‘‘2-4-6-3’’ you might think 2-4-6-3 I have to go climb a tree.
These four pictorial representations of strategies generally reflect rehearsal, chunking, associating, and elaborating of information, respectively. After all strategies had been explained, participants were then presented item sets that included numbers in a sentence context. They were then told that they must recall the numbers in the sentence in order shortly after they select from (point to) a pictorial array representing the strategy that best approximates how he or she will attempt to remember the information.
8
H. LEE SWANSON AND MICHAEL OROSCO
No further information about the strategies shown in the picture was provided to the participant. Participants were allowed 10 seconds in which to make a decision. The range of recall difficulty was 3 digits to 14 digits, and the dependent measure was the highest number of sets correctly recalled (range of difficulty 0–9). Thus, the sequence of the steps for administration after the introduction was as follows: (1) The participant was orally read a sentence (the numbers in the sentence were presented at the rate of approximately 1 every 2 seconds), (2) the participant was asked a process question which required them to give the name of the street referred to in the target sentence, (3) the participant was asked to select one of the four strategies that were represented pictorially that were most like the one they would use to remember the order of the street numbers, (4) the participant was asked to recall the numbers of the address in the order in which they were originally presented, and (5) if an error in recall occurred, the probe questions were implemented. Probing procedures followed the same format as the rhyming task: hints were provided sequentially based on the type of error, and ranged from least obvious hint (Probe 1) to the next explicit hint that facilitated recall of the answer. If probing did not elicit a correct response, the task was discontinued and the next task was administered. If a correct response did occur, the next set of items of increased difficulty was presented.
Visual-Spatial Working Memory Visual Matrix The purpose of this task was to assess the participant’s ability to remember visual sequences within a matrix. The participant was presented a series of dots in a matrix and was allowed 5 seconds to study the matrix. The matrix was removed and the participant was asked a discrimination question, i.e., ‘‘Are there any dots in the first column?’’ To ensure the understanding of columns, the experimenter pointed to the first column on a blank matrix (a grid with no dots). After answering the discrimination question, the participant was asked to draw the dots in the correct boxes on the blank matrix. The task difficulty ranged from a matrix of 4 squares and 2 dots to a matrix of 45 squares and 12 dots. The dependent measure was the number of matrices recalled correctly (range of 0–11). If an error occurred, probe questions were started.
Predictive Validity of Dynamic Testing
9
Probe procedures followed the same format as the verbal subtests, except that the matrices were represented as columns that reflect recency, primacy, and middle positions. For example, when probing for errors that occurred in the recency position, the experimenter drew dots on a blank matrix in the appropriate last columns and said, ‘‘Now can you show me where the rest of the dots go?’’ During probing, the matrices response form in which initial scores were established was turned over. For each hint provided to the participant, a new matrix was presented, the experimenter provided the hints (demonstrates by filling in the appropriate dots) on this matrix and then the participant was asked to fill in the correct remaining dots. If some hints do not have dots in which to demonstrate, the participant was told so. The instructions related to probing were as follows: ‘‘You missed placing some dots in the right boxes. I think you can do it correctly if I provide you with some hints.’’ Probe 1. On the blank matrix, the experimenter correctly draws in the last column(s) of dots. Then the experimenter says, ‘‘Now can you draw where the rest of the dots go?’’ Probe 2. On a blank matrix, the experimenter draws in dots for the first column. Then the experimenter says, ‘‘Now can you draw where the rest of the dots go?’’ Probe 3. On a blank matrix, the experimenter draws in the dots for the middle (between the first and last column) and says, ‘‘Now can you draw where the rest of the dots go?’’ Probe 4. The stimulus card is shown for 2 seconds. The model matrix is removed and the participant is asked to fill in all the dots on a blank matrix.
Mapping and Directions The purpose of this task was to determine whether the participants could remember a sequence of directions on a map that is void of labels (see Swanson, 1995a for detailed instructions). The experimenter presented the participant with a street map with lines connected to a number of dots that illustrated the direction a car would go to get out of the city (see Swanson, 1993). The dots represented stoplights, and the lines represented the direction the car should go. The participant was given 5 seconds to study the map. After the map was removed, the participant was then asked a process question and to point to the strategy (picture) they thought they would use to remember the street directions. Finally, they were asked to draw on a blank map the street directions (lines) and stop lights (dots). The process question was ‘‘Were there any dots in the first street (column)?’’ Using the same pictorial format as the Digit/Sentence Task, strategies were pictorial representations of elemental, global, sectional, or backward processing of patterns. The range of difficulty included dots that ranged
10
H. LEE SWANSON AND MICHAEL OROSCO
in number from 4 to 19. The dependent measure was the number of maps drawn correctly (range of 0–9). If an error was made, a series of probes was presented following the same format as the visual-matrix task procedures. Probing continued until the participant could no longer provide the correct response. The task was subsequently discontinued and the next task was administered.
Validity and Reliability of WM Measures The psychometric qualities of the measures are reviewed in Swanson (1995a, 1995b, 1996). Cronbach alpha measures on the subtest vary from .78 to .96. The reliabilities for the WM measures have been sustained when translated into Spanish (e.g., Swanson, Saez, Gerber, & Leafstedt, 2004) and Japanese (e.g., Hwang et al., 2006). Correlations with each WM subtest vary on intelligence, achievement, and Listening Span (Daneman & Carpenter, 1980) measures from .40 to .80 (e.g., see Swanson, 1992, Tables 4 and 7, Swanson, 1996, see Table 4).
Procedure Individual testing of children was performed by five doctoral students over the course of three years. All items for the WM measures for the initial testing condition were administered until (a) a process question was missed or (b) an error in retrieval occurred. If an error in retrieval occurred (a participant omitted, inserted, or incorrectly ordered the numbers, dots, words, related to the appropriate task), probes were administered. The only stipulation for instituting the probing condition was that the process question be answered correctly. Probes were administered based on the type of error made (i.e., whether the error was related to recency, primacy, or middle items), and probing procedures continued until all targeted items could not be recalled correctly. After all four subtests from the S-CPT were administered under initial and gain testing conditions, participants were re-administered the items for the highest successful set (highest set of items established under gain conditions) for each task. In order to compare ability groups across verbal and visual-spatial WM measures, mean composite z-scores for verbal (rhyming þ auditory sequence)
Predictive Validity of Dynamic Testing
11
and visual-spatial (matrix þ mapping/direction) WM performance from the initial testing conditions were computed. Z-scores for the gain and maintenance testing conditions were based on the means and SDs from the initial condition. The composite z-scores for wave (year) 1, 2, and 3 as a function of ability group and testing condition are shown in the appendix. The z-scores for wave 2 and 3 were based on the mean and standard deviations of span scores in the initial testing condition at wave 1. As a preliminary analysis of the findings we calculated effect sizes for the gain [(mean gainmean initial)/SDgain] and maintenance testing conditions [(mean gainmean initial)/SDgain] relative to the initial testing conditions. The standard deviations from the DA conditions (in this case gain or maintenance scores) were selected because pretest scores (in this case initial testing scores) are generally inflated (see Swanson & Lussier, 2001, for discussion).1 Further adjustments were made in the calculations because of the high correlations among tasks with repeated measures (see Rosenthal, 1994, p. 241, for formula). Table 1 shows the magnitude of the effect sizes (ES) as a function of gain or maintenance testing conditions. Also shown are the ES corrected for the correlation between the initial and DA conditions. Also shown is the ratio of inflation in standard deviations by dividing the standard deviation for the DA measures (gain or maintenance) by the standard deviation for initial testing measure. A previous synthesis of DA measures (Swanson & Lussier, 2001) suggested that ratios greater than 1.25 reflect practice effects rather than responsiveness to DA techniques. That is, initial testing conditions rather than DA served as the learning event. Using Cohen’s (1988) criterion for large (.80) and moderate (.50) effect sizes, Table 1 shows that all ability groups yielded high effect sizes for the gain testing condition and moderate effect sizes for the maintenance testing conditions. Using 1.25 ratio as a critical value, practice effects from the initial test condition were most notable for the RD þ MD group (4 of 6 values were greater than 1.25). Another important finding was related to the reduction in performance from the gain to maintenance testing condition. The largest average drop in performance from the gain to the maintenance testing condition (ES for gain condition minus ES for maintenance condition) occurred for the RD-only group (.86 ES difference), whereas the remaining groups yielded a reduction in ES that were in the same range [RD þ MD (.54 ES difference), low verbal IQ reader (.56 difference), and skilled readers (.48 difference). Thus, the RD-only group showed the largest drop in performance (3/4 of SD) from the gain to the maintenance testing condition when compared to the other groups.
12
Table 1.
RD-only Wave 1 Wave 2 Wave 3 RD þ MD Wave 1 Wave 2 Wave 3 Low VIQ Wave 1 Wave 2 Wave 3 Skilled Wave 1 Wave 2 Wave 3
H. LEE SWANSON AND MICHAEL OROSCO
Effect Sizes (ES), Retesting Sensitivity, and Corrected Effect Sizes by Reading Group. GainES
MaintenanceES
Gain Ratioa
Maintenance Ratioa
Corrected Gain ES
Corrected Maintenance ES
1.64
1.01
1.02
0.90
1.03
0.63
1.88
0.48
0.86
1.35
1.19
0.3
2.69
0.64
0.92
1.43
1.7
0.4
1.2
0.45
1.67
1.46
0.76
0.28
1.41
0.55
1.98
2.10
0.89
0.35
2.10
1.18
0.92
0.94
1.33
0.74
1.21
0.51
1.32
1.09
0.77
0.32
1.76
0.73
1.33
1.23
1.12
0.46
1.69
0.72
1.12
1.24
1.07
0.45
1.71
0.69
0.89
0.96
1.08
0.43
1.27
0.7
1.74
1.08
0.81
0.44
1.67
1.02
0.99
1.11
1.06
0.64
a
Ratio of inflation in standard deviations by dividing the standard deviation for the DA measures (gain or maintenance) by the standard deviation for initial testing measure.
Summary A descriptive analysis of ES between gain and maintenance conditions suggested that the RD-only group was less likely to maintain performance when compared to the other groups.
Predictive Validity of Dynamic Testing
13
Repeated Measures Prior to determining whether the various testing conditions of WM contributed to predictions of growth on measures of reading comprehension, nonword fluency, and receptive language, we used mixed modeling with repeated measures across testing waves to determine if significant differences in growth emerged between ability groups as a function of testing conditions. The mean z-scores for each reading group as a function of domain (verbal vs. visual) across the testing waves are shown in the appendix. Because span scores were greater in the verbal than visual-spatial domain under DA conditions at wave 1, only verbal WM span scores were considered in the subsequent analysis. In addition, because initial testing may have served as a learning event, initial testing scores were partialed in the analysis. Further, because chronological age at initial testing in wave 1 may interact with the amount of change across testing waves, chronological age at wave 1 was a covariate in the analysis. The results indicated that skilled readers outperformed the RD subgroups across all testing waves and DA conditions. The DA conditions did not differentiate between subgroups of less skilled readers. Because the RD subgroups could not be differentiated across the testing waves, we combined the performance RD subgroups in the subsequent analyses.
Correlations An important question to address was whether DA procedures in wave 1 were related to achievement measures in wave 3. As shown in Table 2, the majority of verbal WM measures in wave 1 were significantly related to the criterion measures in wave 3. However, it is unclear from the correlation analysis which of the testing conditions in wave 1 uniquely moderated growth on the criterion measures. To address this issue, latent growth modeling was conducted to determine which of the testing conditions in wave 1 moderated growth on the criterion measures. Verbal WM measures were used since the majority of significant correlations shown in Table 2 were related to the verbal WM measures.
Growth Modeling To examine growth in reading, we used a multilevel framework (Singer, 2002; Singer & Willet, 2003). We applied individual growth modeling to the
14
H. LEE SWANSON AND MICHAEL OROSCO
Table 2. Correlations with Working Memory Measures as a Function of Testing Conditions (Initial, Gain, Maintenance) and Criterion Measures for the Total Sample. Wave 3 Measures Passage comprehension Composite wave 1 Verbal WM measures Initial Gain Main Probes
0.51 0.42 0.50 0.13
Visual-spatial WM measures Initial 0.29 Gain 0.23 Maintenance 0.27 Probes 0.06
Nonword fluency
0.27 0.28 0.34 0.02 0.11 0.06 0.05 0.06
Receptive language
0.56 0.51 0.37 0.20 0.31 0.29 0.38 0.17
po.05, po.01, po.001.
study of intra-individual change in reading as function of WM testing conditions over the three-year period via the PROC MIXED program (SAS, 2003). This hierarchical linear modeling (HLM, Bryk & Raudenbush, 2002) allowed us to determine both the average rate of change and individual variability in change over time. Age was the variable that represented the passage of time in our growth model. To interpret the results, we centered age at 12.48 (sample mean chronological age at wave 2), so that intercepts reflected the expected performance at that age. Our growth model yielded parameter estimates that defined both the overall trajectory of the sample (fixed effects) and deviations in the overall trajectories (random effects). This unconditional linear random regression model was expressed as: yij ¼ b0 þ b1 ðageij Þ þ U 0j þ U 1j ðageij Þ þ Rij where yij is the dependent variable (e.g., passage comprehension) measured at time i in child j, ageij is child j’s age at time i, b0 is the average intercept at 12.48. b1 is the linear slope. U0j is the random intercept for child j in the sample as a whole. U1j is the random age slope for child j. Rij is the residual for child j at time i. The between-child variance components, t20 ¼ VarðU 0j Þ; t201 ¼ VarðU 01j Þ and t21 ¼ VarðU 1j Þ reflected individual differences
Predictive Validity of Dynamic Testing
15
in level of performance for the sample as a whole, and the rate of change between children, respectively. We estimated the association between the outcome (e.g., reading comprehension) and repeated measures of age across the three-year time periods. Both unconditional and conditional models are reported. For the unconditional model, reported are the fixed effects for the intercept value for wave 2 at age 12.48 and the average rate of change across individuals. For the random effects, the estimates are the variance around the sample intercept and the slope related to change overtime. Significant random effects indicated that children differed in intercepts and/or rate of change (slopes). For the conditional model, we tested whether entering WM span scores from the DA conditions into the model explained any statistically significant associations related to fixed and random effects. When one or more predictors are introduced in the conditional model, there are reductions in the magnitude of the various components. These reductions are analogous to effect sizes (Snijders & Bosker, 1999). To evaluate the compatibility of the data with our conditional model, we tested the significance of the model change. This was done by comparing the differences between the deviance values (i.e., the likelihood value for the correspondence between model and data) from the unconditional and conditional growth models. These are chi-square values, and the number of parameters added for the conditional model serves as degrees of freedom. In general, models with lower deviance fit better than models with higher deviance values. For the present study, a significantly lower deviance score for the conditional model indicated that the conditional model showed a better fit to the data than the unconditional model. Prior to the analysis, we converted raw scores for the criterion measures (passage comprehension, nonword fluency, receptive language) to z-scores. Wave 2 and 3 measures were z-scored based upon the means and standard deviations of wave 1. The results were centered on wave 2. The values were centered at wave 2 because group differences were firmly established on a number of measures at wave 1. Age in months at each testing wave was used to measure change. Tables 3–5 show the estimates related to growth and the moderating effects of DA conditions for reading comprehension, fluency, and vocabulary, respectively. For the conditional models, Model 2 included the categorical variable for skilled versus children with RD (combined performance of RD-only, RD þ MD, low verbal IQ poor readers). Model 3 added the testing conditions for the verbal WM measures. Also of interest
16
H. LEE SWANSON AND MICHAEL OROSCO
Hierarchical Linear Modeling for Growth in Passage Comprehension.
Table 3. Model
Passage Comprehension 1
2
3
Random effects Intercepts Slope Residual Deviance
Variance .28 .10 .17 Fit-index 338.0
SE .16 .02 .02
Estimate .15
SE .05
Variance .34 .06 .16 Fit-index 296.1
SE .17 .02 .02
Variance .26 .04 .14
SE .04 .01 .02 Fit-index 255.9
SE .09 .11 .004 .0001
Estimate .30 .23 .01 .001
SE .14 .14 .004 .001
Fixed effects Intercept RD Slope RD Moderators Initial Gain Main. Probe Interaction Gain Main.
0.01
0.001
Estimate .69 .75 .014 .0002
.29 .11 .31 .17 .32 .13
.11 .16 .14 .12 .14 .15
po.05, po.01.
Note: Interactions, ability group dynamic testing condition; main., maintenance testing condition; gain, gain testing condition; initial, initial testing condition; probe, number of probes used to establish gain condition; RD, less skilled readers (combined subgroups of children with RD).
was whether DA testing conditions interacted with ability group performance. Passage Comprehension (WRMT) Table 3 shows the outcomes for one unconditional and two conditional models. To interpret the table, first consider the unconditional model for the fixed effects. The fixed effects model provided an estimate of the intercept (average z-score score at wave 2) for the total sample. As shown, the average
17
Predictive Validity of Dynamic Testing
Table 4.
Hierarchical Linear Modeling for Growth in Nonword Fluency.
Model
TOWRE: Nonword Fluency 1
2
3
Random effects Intercept Slope Residual Deviance
Variance 1.07 0.27 0.32 Fit-index 447.9
SE 0.29 0.11 0.04
Variance 0.88 0.22 0.24 Fit-index 383.5
SE 0.22 0.08 0.03
Variance .73 .07 .20
Estimate 0.20
SE 0.07
Estimate 1.06 1.20 0.01 0.002
SE 0.12 0.15 0.004 0.005
Estimate .57 .83 .007 .0002
SE .32 .02 .03 Fit-index 365.1
Fixed effects Intercept RD Slope RD Moderators Initial Gain Main. Probe Interaction Gain Main.
.011
.002
SE .20 .19 .005 .002
.29 .57 .14 .35
.16 .22 .19 .18
.34 .18
.20 .21
po.05, po.01. Note: Interactions, ability group dynamic testing condition; main., maintenance testing condition; gain, gain testing condition; initial, initial testing condition; probe, number of probes used to establish gain condition; RD, less skilled readers (combined subgroups of children with RD).
passage comprehension z-score (mean z-score of test administered at time 2) for all children was .15. The slope (.01) indicated that the average child in the total sample improved .01 units for each testing wave. Also presented in Table 3 for the unconditional model were the random effects portions of the model for the total sample. The random effects included the variance of conditions related to the intercept (.28) and slope (.10). The residual error was .17. The random effects reveal that 70% [.28 þ .10)/(.28 þ .10 þ .17)] of the between-subject variance in passage comprehension was related to random effects of child differences at the intercept and slope.
18
H. LEE SWANSON AND MICHAEL OROSCO
Table 5.
Hierarchical Linear Modeling for Growth in Receptive Language.
Model
Receptive Language 1
2
3
Random effects Intercept Slope Residual
Deviance
Variance 0.44 0.08 0.13 Fit-index
SE 0.15 0.02 0.02
320.7
Variance 0.34 0.06 0.13 Fit-index
SE 0.12 0.01 0.02
Variance .33 .04 .11
301.5
SE .11 .01 .01 Fit-index 265.1
Fixed effects Intercept RD Slope RD Moderators Initial Gain Main. Probe Interactions Gain Main.
Estimate 0.17 0.028
SE 0.05 0.002
Estimate 0.55 0.53 0.02 0.001
SE 0.10 0.11 0.003 0.004
Estimate .30 .22 .007 .003
SE .16 .15 .004 .001
.29 .07 .37 .08
.12 .18 .15 .14
.10 .48
.15 .16
p o .05, p o .01.
Note: Interactions, ability group dynamic testing condition; main., maintenance testing condition; gain, gain testing condition; initial, initial testing condition; probe, number of probes used to establish gain condition; RD, less skilled readers (combined subgroups of children with RD).
The next column (Model 2) in Table 3 shows the conditional model that included the entry of the classification variable (skilled vs. children with RD). The model indicated that the average passage comprehension z-score for the children in wave 2 without reading problems was .69. Thus, the skilled reader at wave 2 had gained approximately 3/4 of a standard deviation in passage comprehension performance. Model 2 also showed that without entering moderator variables related to testing conditions, passage
Predictive Validity of Dynamic Testing
19
reading comprehension performance was .75 z-score points lower (estimate was .75) for children with RD when compared to skilled readers. The growth rate for skilled readers was approximately .02 (.014) units of improvement per testing session. Although the estimate for children with RD showed a lower growth rate (.0002), the estimate was not significantly different from skilled readers. A comparison was made between the unconditional and conditional models on the random intercept (.28 vs. .34) and slope (.10 vs. .06). The addition of the classification measure accounted for approximately 40% of the explainable variance in growth [(.10–.06)/.10)]. However, the variance increased between children on the intercept values, suggesting the model was not a good fit to the data. Model 3 added probe scores and span scores from the WM testing conditions. Also included were potential interactions between reading ability group and the DA condition. The results indicated that WM performance related to initial and maintenance testing for the verbal tasks and the ability group gain testing condition moderated passage comprehension. To interpret these findings it is important to note that predictor variables were centered. That is, the estimate for passage comprehension was .30 when predictors were set to zero. The estimate for the maintenance testing condition was .31. Taken together, the results indicated that children who differ by 1 point on WM for the maintenance testing condition differ by .31 points on passage comprehension. The negative estimates for the interaction indicated that skilled readers were more likely to benefit from the gain conditions than children with RD. A deviance test was computed to determine the fit of Model 3. A deviance test indicated that the Model 3 provided a better fit to the data than Model 2 for passage comprehension, w2 (df ¼ 6) ¼ 40.9 (296.1–255.9), po.001. When compared to Model 2, the measurement of WM across various testing conditions in Model 3 accounted for 24% of the explainable variance in intercepts [(.34–.26)/.34] and 33% of the explainable variance between children in growth [(.06–.04)/.06].
Nonword Fluency (TOWRE) Table 4 shows the results for predicting reading fluency for nonwords. The total sample average z-score (mean z-score of test administered at time 2) for all children was .20. The slope (.01) indicated that children improved .01 units for each testing wave. The random effects for the unconditional model were significant for the intercept (1.07), slope (.27), and the residual error (.32.) The random effects reveal that 81% [1.07 þ .27)/(1.07 þ .27 þ .32)] of
20
H. LEE SWANSON AND MICHAEL OROSCO
the between-subject variance in nonword fluency was related to child differences at the intercept and slope. Model 2 indicated that the nonword fluency z-scores for skilled reading children at wave 2 was 1.06. Children with RD were 1.20 z-score points lower than skilled readers. However, no differences were found between the two groups in growth. The addition of the classification measure in Model 2 explained about 19% of the variance in growth [(.27–.22)/.27)]. Model 3 added probe scores and span scores for the verbal WM testing conditions. As shown for the random effect, the intercepts and slopes decreased when compared to Model 2. The results indicated that span scores from the gain condition and probe scores significantly moderated nonword fluency. The results indicated children who differed by 1.0 with respect to the scores on the gain testing condition for verbal WM tasks differed in intercept values by .57 in the nonword fluency. The deviance test indicated that Model 3 showed a good fit to the data when compared to Model 2, w2 (df ¼ 6) ¼ 18.41 (383.50–365.1), po.001. When compared to Model 2, the measurement of WM across various testing conditions in Model 3 accounted for 17% of the explainable variance in intercepts [(.88–.73)/.88] and 68% of the explainable variance between children in growth [(.22–.07)/.22].
Vocabulary As shown in Table 5, the fixed effects in Model 3 indicated that the slopes were higher for children with RD than skilled readers. The important findings were that the initial and maintenance conditions moderated receptive vocabulary. As shown, the fixed effects for Model 3 eliminated group differences at wave 2 for the intercept estimates. Performance on the maintenance testing condition was a significant moderator of receptive vocabulary as was the ability group maintenance testing condition interaction. The results suggested that children skilled in reading were better able to sustain WM performance at their highest level (based on gain testing conditions) without cues than children with RD. The deviance test indicated that the Model 3 showed a good fit to the data when compared to Model 2, w2 (df ¼ 6) ¼ 36.40 (301.50–265.1), po.001. When compared to Model 2, the measurement of WM across various testing conditions in Model 3 accounted for only 3% of the explainable variance in intercepts [(.34–.33)/.24], but 33% of the explainable variance between children in growth [(.06–.04)/.06].
Predictive Validity of Dynamic Testing
21
Summary The growth modeling showed that various DA conditions moderated outcomes on the criterion measures. Maintenance testing conditions were important moderators in the growth model of passage comprehension and receptive vocabulary, whereas gain testing conditions and probing moderated nonword fluency. The conditional model that included measures of WM accounted for approximately 33% of the explainable growth variance in passage comprehension and receptive vocabulary and 68% of growth in nonword fluency. Ability group interactions emerged for gain testing conditions for passage comprehension, and maintenance testing conditions for vocabulary (receptive language). These interactions indicated that skilled readers were more likely to benefit from those DA conditions than children with RD.
SUMMARY OF FINDINGS AND IMPLICATIONS We will now consider the two questions that motivated this preliminary work. 1. Does dynamic assessment contribute to later predictions of achievement in children with RD? The results of the HLM analysis support the hypothesis that DA adds important variance in predicting later reading achievement. These results showed that various DA conditions (e.g., conditions related to gain, probing, and/or maintenance) significantly moderated passage comprehension, nonword reading fluency, and receptive language. Interestingly, although intercept values favored skilled readers on all criterion measures, growth on the reading (passage comprehension, nonword fluency) and receptive language measures was statistically comparable between skilled and children with RD. These findings complement some of our earlier work. Our earliest study (Swanson, 1992, Experiment 2) assessed whether dynamic testing of WM predicted concurrent reading recognition performance when initial testing conditions were partialed out in the analysis. In this study, children (mean age 10 years) were administered WM tasks under initial, gain, and maintenance conditions. Effect sizes (ES) for gain and maintenance performance relative to initial conditions (ES .90 for gain scores and .47 for maintenance scores) were in the same range as the present study. The
22
H. LEE SWANSON AND MICHAEL OROSCO
regression analysis found that DA conditions contributed a significant variance (approximately 5%) to concurrent performance on measures of word recognition (see Sternberg & Grigorenko, 2002, for a critical analysis of this finding). This earlier study was broaden to a larger sample ranging in ages 5–85 (Swanson, 1999) and that again found that DA procedures contributed approximately 5% of the variance reading and math performance. So what does this small amount of variance buy us? To answer that question we need to consider the logic of the testing conditions (also see Swanson, 2003, for a more detailed discussion). The initial testing condition reflects the baseline for each participant’s self-initiated processes to access information. The gain testing condition enhances the access to stored items by tailoring cues to help participants reinstate memory traces or to retrieve forgotten items from the initial (or baseline) conditions. Our previous studies (Swanson, 1992, 1999; Swanson, Ashbaker, & Lee, 1996) and the current study have shown that the gain conditions improve performance by as much as 1 standard deviation. We assume this occurs because the systematic cuing procedures emphasize sequential processing strategies and thereby reduces the number of competing strategies employed. We also assume that if the locus of WM problems in children with RD is in the retrieval phase, one would expect a reduction in residual difference between children with and without RD when compared to the initial testing condition. As shown in this study and our earlier studies we have been unable to reduce the residual differences between reading groups in the gain condition. Thus, one could argue that children with RD suffer WM deficits related to retrieval efficiency. One reservation in arguing that WM problems in children with RD are related to problems in retrieval efficiency (i.e., improved access to items previously forgotten), however, is that the manipulations between the initial (noncued) and gain (cued) condition are limited to individual differences as to what was actually attended to and stored by the child. Thus, the maintenance condition allowed for us to examine whether WM deficits reflect capacity constraints in accessing what has been previously stored. For this condition, the same WM tasks that matched each participant’s highest WM span level (gain score) are again administered, but without cues. Thus, each participant is presented items calibrated to their asymptotic level of WM performance. Calibrating this condition allows us to capture processing differences between groups beyond the simple learning of items. The inability to access items that have been previously learned (stored and retrieved) and have been calibrated to the individual span level we assume reflects fundamental constraints in WM capacity.
Predictive Validity of Dynamic Testing
23
Based on the above logic, we consider the results that are related to the growth curve analysis. The conditional models showed that maintenance testing conditions moderated passage comprehension and receptive language, whereas gain testing conditions moderated nonword fluency. Consistent with the above logic, we assume that individual differences as a function of maintenance testing conditions reflects constraints in WM capacity. These constraints were not directly tied to simple retrieval strategies (e.g., feedback) or previous learning (e.g., item familiarity). In contrast, individual differences in gain conditions reflect limitations in the activation of new information. Thus, DA informs us to some degree that constraints in accessing previously remember information may play a role in some areas of reading (comprehension) and the efficiency in accessing new information in other areas of reading (nonword fluency). However, these results are qualified by the ability group DA interactions. The interaction with reading group gain testing condition when predicting passage comprehension suggests that a failure to activate new information in storage is an important determinant of reading skill. Further it appears from the reading group maintenance interaction that when predicting receptive vocabulary the ability to sustain performance after environmental (probes) support is removed maybe constrained in children with RD. Clearly, these conclusions are speculative on our part but the results do suggest that different processes may be activated as a function of DA when predicting performance on the criterion measures. 2. Do children with RD differ from children with more generalized achievement deficits as a function of dynamic assessment conditions? As expected, skilled readers outperformed children with RD on the WM measures. This occurred even though children were statistically matched on measures of fluid intelligence. Interestingly, no significant differences were found on measures of WM between children with RD who exhibited low verbal IQ from children with RD-only and RD þ MD. This finding is in contrast to other studies (e.g., van der Sluis, van der Leij, & de Jong, 2005) which have suggested that RD-only children exhibit no problems across domains that draw upon executive processing (such as updating as required by WM tasks, see Miyake, Friedman, Emerson, Witzki, & Howerter, 2000), whereas children who show reading plus arithmetic difficulties have problems in executive processing (domain general processing). We did not find support for the notion that more generalized WM deficits occur only in the comorbid group when compared to children with RD only. Further, because more generalized deficits in achievement (problems in both reading
24
H. LEE SWANSON AND MICHAEL OROSCO
and math) may be associated with verbal IQ, we also explored whether group differences in WM measures were merely an artifact of verbal ability. No significant differences were found between poor readers with low verbal IQ and children with RD-only on measures of WM. Thus, we did not find support for the notion that WM deficits in children with RD are merely an artifact of verbal IQ or comorbidity. Although the results indicated that ability group differences were larger on verbal WM tasks than visual-spatial WM tasks, performance among subgroups of children with RD was in the same low range. We now consider in more detail the findings related to the weak differentiation of subgroups as a function of DA and the domain general findings related to WM tasks.
Subgroups and Dynamic Assessment The present findings of this study partially support the previous work of Swanson and Howard (2005), which showed that that WM performance and DA procedures enhanced predictions of reading. The results of Swanson and Howard showed that poor readers and children with RDonly or MD þ RD generally perform in the same low range on verbal WM tasks. These results also coincide with others’ showing that poor readers and children with RD are difficult to separate on cognitive measures (e.g., Siegel, 1992; see Hoskyn & Swanson, 2000, for a review). The Swanson and Howard study showed children with RD and/or RD þ MD were more likely than the other subgroups to return to their initial score performance after presentation of probes (feedback) had been stopped. In contrast, the present study found that effect size scores related to gain and maintenance testing for all ability groups were in the same range. However, the present results did suggest that a drop in performance from the gain to maintenance conditions was more likely to occur for the RD-only group and the RD þ MD group was more likely influenced by initial testing conditions than the actual probing. No doubt, the differences between the previous study and the present study may be due to sampling. The earlier study focused on younger children of middle to low SES background. In contrast, the present sample was older, came from higher SES homes (in terms of parent education), more schooling, and therefore both cognitive and reading performance may have been more stable across the testing sessions. It is important to note that our inability to separate children with RDonly from other children with reading problems is not unique to this study.
Predictive Validity of Dynamic Testing
25
For example, a meta-analysis comparing children with MD-only, RD-only, and RD þ MD on measures independent of the classification measures placed their performance on cognitive measures in the same range (Swanson & Jerman, 2006). For example, Swanson and Jerman found that the magnitude of ES across cognitive measures were miniscule between children with RD-only and MD-only (.10) and between MDonly and RDMD (M ES ¼ .26). More important, the means ES between RD-only and MD-only groups were .06 for STM of words, .03 for STM digits, .07 verbal WM, and .30 for visual-spatial WM. In general, these results suggested that MD and RD children share common memory deficits. Unraveling how the common memory deficits manifest themselves in different forms on achievement measures, of course, needs to be explored.
Limitations There are, of course, several limitations to our preliminary work. Four are apparent. First, we focused on children who came from average to high SES homes with highly educated parents. We assumed that previous studies that have tested children with RD had failed to control for SES (or focused on low SES participants) and therefore differences in reading in previous studies could be attributed to low SES and/or instructional factors. Regardless, our findings can only be generalized to a select group of children. Second, children with undiagnosed reading problems may yield different results. The children with RD in this study had the diagnosis of learning disabilities and the information from their cumulative file indicated a persistent reading problem in their elementary school years. This study sample was also unique in that all children had fluid intelligence scores (Raven test performance) in the normal range. If fluid intelligence scores were in the same low range as their reading scores, the results may have been different. Third, attrition in the sample occurred at wave 3. Although we did not find significant differences between the retained and nonretained samples across the various measures, the study was limited in terms of sample size when correlating wave 1 and wave 3 testing. Finally, these results need to be established by researchers independent of our research team. Although our DA measures are normed referenced (Swanson, 1995a), the majority of studies independent of our research team using the measures have been isolated to dissertations.
26
H. LEE SWANSON AND MICHAEL OROSCO
Summary In summary, DA procedures moderated later performance on measures of passage comprehension, nonword fluency, and receptive language. The results indicated that children with RD and skilled readers both benefited from DA procedures, but DA procedures did not statistically differentiate between the subgroups of children with RD. A descriptive analysis suggested that greater reductions occur between gain and maintenance testing conditions for the RD only group and the effect sizes related to DA conditions RD þ MD group are more likely related initial testing sensitivity than to procedures that test the limits. Clearly, additional research on the properties of DA and cognitive measures is needed, especially studies linking the malleability of WM to long-term instructional gains.
NOTE 1. Under most training circumstances with no ceiling or floor effects, pretest and posttest variable standard deviations are expected to be similar (see Carlson & Schmidt, 1999, p. 853; for a review). Carlson and Schmidt (1999) have argued that if exposure to the pretest acted as a learning event, then this would have caused posttest training scores to be higher, but responses to be more variable. Posttest dependent variable standard deviations would be systematically larger than pretest standard deviations, and as a result posttest standard deviations would result in smaller effect sizes than those calculated using pretest standard deviations. On the other hand, if changes in performance are a function of treatment (in this case feedback to the examinee) rather than merely retesting, then posttest means would be higher than pretest means, but posttest standard deviations would be comparable to pretest standard deviations (Carlson & Schmidt, 1999). In our case, we assumed that initial scores served as a pretest measure and gain and maintenance reflected posttesting.
ACKNOWLEDGMENTS The authors thank Marshal Raskin and Eleanor Higgins for their administrative assistance in testing children over the three- to five-year period. The authors also thank the various testers (Crystal Howard, Leilani Saez, Olga Jerman, Diane Luxenberg, and Diane Dowds) who as doctoral students played an important role in monitoring children’s progress. The technical aspects of this preliminary work are detailed for reading comprehension (Swanson, in press) and fluency (Swanson, 2010) elsewhere.
Predictive Validity of Dynamic Testing
27
REFERENCES Brown, A. L., & Ferrara, R. A. (1999). Diagnosing zones of proximal development. In: P. Lloyd (Ed.), L.Vygotsky: Critical assessments: The zones of proximal development (Vol. III, pp. 225–256). New York: Routledge. Bryk, A., & Raudenbush, S. W. (2002). Hierarchical linear models (ed.). Newbury Park, CA: Sage. Caffrey, E., Fuchs, D., & Fuchs, L. S. (2008). The predictive validity of dynamic assessment: A review. The Journal of Special Education, 41, 254–270. Carlson, J. S., & Wiedl, K. H. (1979). Toward a differential testing approach: Testing-the-limits employing the Raven matrices. Intelligence, 3, 323–344. Carlson, K. D., & Schmidt, F. L. (1999). Impact of experimental design on effect size: Findings from the research literature on training. Journal of Applied Psychology, 84, 851–862. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum. Daneman, M., & Carpenter, P. A. (1980). Individual differences in working memory and reading. Journal of Verbal Learning and Verbal Behavior, 19, 450–466. De Beni, R., Palladino, P., Pazzaglia, F., & Cornoldi, C. (1998). Increases in intrusion errors and working memory deficit of poor comprehenders. Quarterly Journal of Experimental Psychology: Human Experimental Psychology, 51, 305–320. De Jong, P. (1998). Working memory deficits of reading disabled children. Journal of Experimental Child Psychology, 70, 75–95. Dunn, L., Dunn, L. M., Dunn, L. M., & Dunn, D. M. (1997). The peabody picture vocabulary test (3rd ed.). Circle Pines, MN: American Guidance Services, Inc. Embretson, S. E. (1992). Measuring and validating cognitive modifiability as an ability: A study in the spatial domain. Journal of Educational Measurement, 29, 25–50. Gathercole, S. E., Alloway, T. P., Willis, C., & Adams, A. (2006). Working memory in children with reading disabilities. Journal of Experimental Child Psychology, 93, 265–281. Gathercole, S. E., & Pickering, S. J. (2000a). Assessment of working memory in six- and sevenyear-old children. Journal of Educational Psychology, 92, 377–390. Gathercole, S. E., & Pickering, S. J. (2000b). Working memory deficits in children with low achievements in the national curriculum at 7 years of age. British Journal of Education Psychology, 70, 177–194. Grigorenko, E. L., & Sternberg, R. J. (1998). Dynamic testing. Psychological Bulletin, 124, 75–111. Hoskyn, M., & Swanson, H. L. (2000). Cognitive processing of low achievers and children with reading disabilities: A selective meta-analytic review of the published literature. School Psychology Review, 29, 102–119. Hwang, Y., Hosokawa, T., Swanson, H. L., Ishizaka, I., Kifune, N., Ohira, D., et al. (2006). A Japanese short form of the Swanson-Cognitive Processing Test to measure working memory: Reliability, validity, and differences in scores between primary school children of the United States and Japan. Psychological Reports, 99, 27–38. Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., & Howerter, A. (2000). The unity and diversity of executive functions and their contributions to complex frontal lobe tasks: A latent variable analysis. Cognitive Psychology, 41, 49–100. Pickering, S. J. (2006). Working memory in dyslexia. In: T. P. Alloway & S. E. Gathercole (Eds), Working memory and neurodevelopmental disorders (pp. 7–40). New York: Psychology Press.
28
H. LEE SWANSON AND MICHAEL OROSCO
Raven, J. C. (1976). Colored progressive matrices. London, England: H. K. Lewis & Co. Ltd. Rosenthal, R. (1994). Parametric measures of effect size. In: H. Cooper & L. V. Hedges (Eds), The handbook of research synthesis (pp. 232–260). New York: Russell Sage Foundation. SAS Institute Inc. (2003). SAS/STAT user’s guide. Cary, NC: SAS Institute. Siegel, L. S. (1992). An evaluation of the discrepancy definition if dyslexia. Journal of Learning Disabilities, 25, 618–629. Siegel, L. S., & Ryan, E. B. (1989). The development of working memory in normally achieving and subtypes of learning disabled. Child Development, 60, 973–980. Singer, J. D. (2002). Fitting individual growth models using SAS PROC MIXED. In: Moskowitz & S. Hersberger (Eds), Modeling intraindividual variability with repeated measures data: Methods and applications (pp. 135–170). Mahwah, NJ: Erlbaum. Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis. New York: Oxford University Press. Snijders, T. A., & Bosker, R. J. (1999). Multilevel analysis. Thousand Oaks, CA: Sage. Sternberg, R. J., & Grigorenko, E. L. (2002). Dynamic testing: The nature and measurement of learning potential. New York: Cambridge University Press. Swanson, H. L. (1992). Generality and modifiability of working memory among skilled and less skilled readers. Journal of Educational Psychology, 84, 473–488. Swanson, H. L. (1993). Working memory in learning disability subgroups. Journal of Experimental Child Psychology, 56, 87–114. Swanson, H. L. (1995a). S-cognitive processing test. Austin, TX: PRO-ED. Swanson, H. L. (1995b). Using the cognitive processing test to assess ability: Development of a dynamic assessment measure. School Psychology Review, 24, 672–693. Swanson, H. L. (1996). Individual and age-related differences in children’s working memory. Memory & Cognition, 24, 70–82. Swanson, H. L. (1999). What develops in working memory? A life span perspective. Developmental Psychology, 35, 986–1000. Swanson, H. L. (2003). Age-related differences in learning disabled and skilled readers’ working memory. Journal of Experimental Child Psychology, 85, 1–31. Swanson, H. L. (2008). Working memory and intelligence in children: What develops? Journal of Educational Psychology, 100, 581–602. Swanson, H. L. (2010). Does the dynamic testing of working memory predict growth in nonword fluency and vocabulary in children with reading disabilities. Journal of Cognitive Education and Psychology, 9, 51–77. Swanson, H. L. (in press). Dynamic testing, working memory and reading comprehension growth in children. Journal of Learning Disabilities. Swanson, H. L., Ashbaker, M., & Lee, C. (1996). Working-memory in learning disabled readers as a function of processing demands. Journal of Child Experimental Psychology, 61, 242–275. Swanson, H. L., & Howard, C. B. (2005). Children with reading disabilities: Does dynamic assessment help in the classification? Learning Disability Quarterly, 28, 17–34. Swanson, H. L., & Jerman, O. (2006). Math disabilities: A selective meta-analysis of the literature. Review of Educational Research, 76, 249–274. Swanson, H. L., Jerman, O., & Zheng, X. (2008). Growth in working memory and mathematical problem solving in children at risk and not at risk for serious math difficulties. Journal of Educational Psychology, 100, 343–379.
29
Predictive Validity of Dynamic Testing
Swanson, H. L., & Lussier, C. M. (2001). A selective synthesis of the experimental literature on dynamic assessment. Review of Educational Research, 71, 321–363. Swanson, H. L., Saez, L., Gerber, M., & Leafstedt, J. (2004). Literacy and cognitive functioning in bilingual and nonbilingual children at or not at risk for reading disabilities. Journal of Educational Psychology, 96, 3–18. Swanson, H. L., & Siegel, L. (2001). Learning disabilities as a working memory deficit. Issues in Education: Contributions from Educational Psychology, 7, 1–48. van der Sluis, S., van der Leij, A., & de Jong, P. F. (2005). Working memory in Dutch children with reading- and arithmetic-related LD. Journal of Learning Disabilities, 38(3), 207–221. Wagner, R., & Torgesen, J. (1999). Test of word reading efficiency. Austin, TX: Pro-Ed. Wilkinson, G. S. (1993). The Wide Range Achievement Test-3. Wilmington, DE: Wide Range, Inc. Willcutt, E. G., Pennington, B. F., Olson, R. K., Chhabildas, N., & Hulslander, J. (2005). Neuropsychological analyses of comorbidity between reading disability and attention deficit hyperactivity disorder: In search of the common deficit. Developmental Neuropsychology, 27, 35–78. Woodcock, R. W. (1998). Woodcock reading mastery test – Revised. Minneapolis, MN: American Guidance.
APPENDIX. COMPOSITE SCORES (Z-SCORES) FOR WORKING MEMORY TASKS AS A FUNCTION OF TESTING CONDITIONS AND TESTING WAVES Variable
Wave 1 Vinitial Vgain Vmain Vprobe Visinitial Visgain Vismain Visprobe Wave 2 Vinitial Vgain
RD-Only
RD þ MD
Low VIQ Readers
Skilled Readers
Mean SD Mean SD
Mean
SD
Mean
SD
0.83 0.25 0.72 0.85 1.19 1.2 0.75 0.22 1.05 0.76 0.2 1.16 0.65 0.14 0.86 0.83 0.82 1.28 0.69 0.26 1.01 0.74 0.25 1.03
0.13 1.07 0.29 0.01 0.1 0.64 0.28 0.14
0.75 0.99 0.82 0.76 0.75 0.87 0.74 0.55
0.55 1.66 1.03 0.11 0.39 1.0 0.75 0.02
0.73 0.65 0.7 0.67 0.64 0.88 0.81 0.9
0.11 0.84 0.24 0.61 1.46 0.72 1.47 1.21
0.14 1.64
0.64 0.85
0.59 2.02
0.65 1.13
0.39 1.0 0.36 0.01 0.21 0.69 0.01 0.14
30
H. LEE SWANSON AND MICHAEL OROSCO
Variable
RD-Only
RD þ MD
Mean SD Mean SD 0.65 Vmain 0.11 Vprobe Visinitial 0.21 Visgain 0.95 0.76 Vismain Visprobe 0.05
1.13 0.46 1.28 0.61 0.02 0.84 0.69 0.07 1.14 0.9 0.75 1.47 0.93 0.65 1.34 0.65 0.18 0.72
Wave 3 Vinitial 0.26 0.6 0.34 1.74 0.55 1.62 Vgain Vmain 0.81 0.86 1.07 Vprobe 0.06 0.69 0.39 Visinitial 0.57 0.51 0.83 1.39 0.52 1.92 Visgain Vismain 1.15 0.61 1.62 Visprobe 0.04 0.85 0.23
0.66 0.61 0.62 0.7 0.77 0.63 0.7 0.58
Low VIQ Readers
Skilled Readers
Mean
SD
Mean
SD
0.72 0.22 0.39 1.08 0.73 0.25
0.79 0.75 0.83 1.09 1.11 0.67
1.08 0.28 0.68 1.55 1.23 0.12
0.7 0.8 0.76 0.9 0.89 0.67
0.13 1.53 0.79 0.45 0.53 1.73 1.33 0.08
0.74 0.83 0.92 0.73 0.88 1.08 1.15 0.92
0.78 2.37 1.87 0.37 0.77 1.7 1.44 0.35
0.96 0.95 1.07 0.79 0.53 0.86 0.7 0.68
Note: Vinitial, initial testing on verbal measures; Visinitial, initial testing on visual-spatial measures; Vgain, gain testing on verbal measures; Visgain, gain testing on visual-spatial measures; Vmain, maintenance testing on verbal measures; Vismain, maintenance testing on visual-spatial measures.
APPLICATIONS OF CURRICULUMBASED MEASURES IN MAKING DECISIONS WITH MULTIPLE REFERENCE POINTS Gerald Tindal and Joseph F. T. Nese ABSTRACT We write this chapter using a historical discourse, both in the chronology of research and in the development that has occurred over the years with curriculum-based measurement (CBM). More practically, however, we depict the chronology in terms of the sequence of decisions that educators make as they provide special services to students with disabilities. In the first part of the chapter, we begin with a pair of seminal documents that were written in the late 1970s to begin the story of CBM. In the second part of the chapter, we begin with the first decision an educator needs to make in providing special services and then we continue through the chronology of decisions to affect change in learning for individual students. In the end, we conclude with the need to integrate these decisions with multiple references for interpreting data: normative to allocate resources, criterion to diagnose skill deficits, and individual to evaluate instruction.
Assessment and Intervention Advances in Learning and Behavioral Disabilities, Volume 24, 31–58 Copyright r 2011 by Emerald Group Publishing Limited All rights of reproduction in any form reserved ISSN: 0735-004X/doi:10.1108/S0735-004X(2011)0000024004
31
32
GERALD TINDAL AND JOSEPH F. T. NESE
EARLY VIEWS ON INSTRUCTIONAL EVALUATION In the late 1970s, Deno and Mirkin developed a concept they entitled databased program modification, espousing the need for measures that teachers could use to evaluate instruction over time (Deno & Mirkin, 1977). They emphasized specific features of these measures to allow generating alternate forms so that a time series data display could be used to determine if programs were working or needed to be adjusted. For example, the measures had to be easy to create, quick to administer, usable by all, from the curriculum, and with sufficient technical adequacy. In the end, they laid the groundwork for an experimental view of teaching in which any intervention was considered a hypothesis that needed vindication. At nearly the same time, Engelmann, Granzin, and Severson (1979) wrote about the need for diagnosing instruction. Their thesis was that problems in learning were seldom the fault of students; rather they often resulted from ineffective programs even though we spend an inordinate amount of time directly trying to diagnose the problem as the student’s. A central warrant of their thesis was that, while it is impossible to ascertain the certainty of problems ‘‘inside’’ the student, it is very much possible to determine programs are not working, change them, and only upon continued lack of success, make inferences about the problem being with the student. Both approaches relied upon the scientific method in which a null hypothesis (a notion that is not really believed with much certainty) is posited and countered by an alternative hypothesis. Rather than prove anything in a definitive manner, teachers needed to disprove that their actions are ineffective (posit a null hypothesis and then provide data to counter it). For Deno and Mirkin (1977), experimental teaching provided a time series design in which each student’s performance is compared to previous performance (in level, slope, and variation), with the impetus for instructional change residing in the lack of student change. For Engelmann et al. (1979), the null hypothesis is similarly situated with repeated failure to affect change as the only convincing evidence that supports the problem as being with the student (and not an instructional one). Ironically, they approached the problem from opposite ends of the continuum. For Deno and Mirkin (1977), instruction was tentative and ascertained only through measurement. For Engelmann et al. (1979), diagnosis was entirely instructional with measurement only implied. Yet, in an interesting confluence of history, both have become signatures of effective teaching that has current play in any ‘‘response to intervention’’ (RTI) model. In this chapter, we provide a brief history of this perspective
Applications of CBMs in Making Decisions
33
and then discuss the next generation of curriculum-based measurement (CBM) that features alignment to content standards and rigorous statistical modeling analytics. We review the empirical literature that has resulted from more widespread adoption of CBM, in both the kinds of problems being investigated as well as the variety of applications appearing in educational systems. We conclude with classroom and student case studies that echo these developments. CBM research initially focused on measurement validation, different sampling plans, and measurement designs. A number of studies also focused on highlighting the central feature of CBM as reflecting a ‘‘general outcome’’ measure rather than a curriculum-based one. Although efforts to change the language failed and CBM became the parlance in use, more recent efforts with the National Center on Response to Intervention (NCRTI) (www.rti4success.org) reflect a return to more precise language that references progress monitoring as oriented to either mastery monitoring or general outcomes. Mastery monitoring was initially defined by Deno and Mirkin (1977) in terms of the graphic display: ‘‘A progress (or mastery) graph is constructed to display the time it is taking a student to master a set (usually ordered over time in terms of sequence and/or complexity) of instructional objectives’’ (p. 33) with the y-axis reflecting units traversed and the x-axis reflecting time taken (usually days). The most significant feature is that the domain for measurement comes from a constricted sample of items (problems) being taught during the evaluation period. The classic example of mastery monitoring is the weekly spelling list in which 10–15 words are taught during the week, and then students receive a test on this list of words at the end of the week. Another feature of mastery monitoring is the graphic display, reflecting a series of step increases plotted against some definition of mastery; with mastery, performance moves up one unit for the next unit of time, and without mastery, performance is recorded at the same level for the next unit of time. Without incremental movement up in mastery, the graphic display of performance would simply hover at a steady level over time. General outcome measures were likewise described by Deno and Mirkin (1977) in terms of performance graphs: ‘‘designed to display how a student’s behavior changes on a single taskyover time’’ (p. 34) with the y-axis reflecting how well the student performed on the task and the x-axis reflecting time (in days or weeks). In this system, items (problems) are sampled across an alternate unit of instructional time, from the early through late skills that are being introduced over a year (or possibly over years). The sampling plan effectively includes items that preview what is to
34
GERALD TINDAL AND JOSEPH F. T. NESE
be taught as well as reviews what has been taught. The graphic display allows raw scores to change without converting them to a common scale. The graphic display includes three metrics to evaluate programs: (a) slope of improvement, (b) variation among successive data values, and (c) immediate changes after a program has been introduced. By looking for increases in slope, decreases in variation, and large changes in level (from pre- to immediately postintervention), teachers can determine whether or not their programs are effective.
THREE DECADES OF RESEARCH ON CBM FROM THE 1980s THROUGH 2010 In one of the early definitions of CBM, Deno (1987) stated: the term curriculum-based assessment, generally refers to any approach that uses direct observation and recording of a student’s performance in the local school curriculum as a basis for gathering information to make instructional decisionsyAs noted earlier, the term curriculum-based measurement refers to a specific set of procedures created through a research and development program y and grew out of the Data-Based Program Modification system developed by Deno and Mirkin (1977) (p. 41).
CBMs, which sample skills related to the curriculum material covered in a given year of instruction, provide teachers with a snapshot of their students’ current level of performance as well as a mechanism for tracking the progress students make in gaining desired academic skills. Historically, CBMs typically are very brief individually administered measures (Deno, 2003). CBM is distinct in two important respects: (a) the procedures reflect technically adequate measures with reliability and validity present to a degree that equals or exceeds that of most achievement tests and (b) ‘‘growth is described by an increasing score on a standard, or constant, task. The most common application of CBM requires that a student’s performance in each curriculum area be measured on a single global task repeatedly across time’’ (Deno, 1987, p. 41). After the initial research conducted at the Institute for Research on Learning Disabilities (IRLD) at the University of Minnesota, scores of the technical reports became publications in the refereed literature; eventually, as the practice of CBM became more widespread, scores of chapters were written with two important books published with Shinn as the editor and most of the original researchers contributing chapters (1989, 1998). Much of this
Applications of CBMs in Making Decisions
35
research has been with and on the use of performance measures or general outcome measures and not the mastery measures.
Reading Research The history of research on CBM is rich and rather extensive, from the initial work in the late 1970s and early 1980s on through more recent research in the 2000s. Wallace, Espin, McMaster, and Deno (2007) summarize this research in a recent special series on curriculum-based measurement. Over the initial 20 years of research on reading CBM, a number of technical characteristics were investigated, including: 1. Content-related evidence addressing domain size (Fuchs, Tindal, & Deno, 1984) and difficulty of material (Fuchs & Deno, 1992; Shinn, Gleason, & Tindal, 1989). 2. Criterion-related evidence in the relation of oral reading fluency to reading comprehension (Shinn, Good, Knutson, Tilly, & Colllins, 1992) and predictions of performance (Fewster & Macmillan, 2002). 3. Construct validation with respect to rival hypotheses related to psychological variables (Kranzler, Brownell, & Miller, 1998), as well as other psychometric concerns (Fuchs, Fuchs, & Maxwell, 1988; Mehrens & Clarizio, 1993). 4. The effects of curriculum-based planning on grouping students (Wesson, Vierthaler, & Haubrich, 1989) and making instructional decisions (Allinder, 1996; Fuchs, Fuchs, & Stecker, 1989; Wesson, et al., 1988). As a useful formative assessment method, CBM has been extensively researched, a summary of which is beyond the scope of this chapter; these topics are simply illustrative of the kind of research that has been done over the past 30 years. Much of this research, however, has been conducted with elementary students and has focused on assessment of reading. With the establishment of CBM as a useful means of assessment for decision-making, we can apply its utility to mathematics, as it has been applied to reading.
Mathematics Research The research on mathematics did not begin as early as that done in reading and writing. Rather, work began after the initial research done with the IRLD with the best summary provided by Foegen, Jiban, and Deno (2007).
36
GERALD TINDAL AND JOSEPH F. T. NESE
In this publication, the research reported from 32 studies is summarized in terms of three stages (Fuchs, 2004): (a) stage 1, with an emphasis on initial technical adequacy, particularly reliability and criterion validity evidence; (b) stage 2, with the focus on the technical characteristics of slope (change over time); and (c) stage 3, in which instructional utility of the measures is addressed. 1. The dominance of research is on elementary grade levels in which 17 studies provided evidence on the technical characteristics of the measures (stage 1) with monitoring basic skills progress measures (Fuchs, Hamlett, & Fuchs, 1998). The studies were split between progress measures (reflecting curriculum sampling within grades) and performance measures (reflecting robust measures across grades). The results confirmed many of the measures being developed as both reliable and with adequate relations to other measures of achievement. 2. In the early mathematics measures for students in prekindergarten to first grade, all of the studies were published since 2000 and most of them have relied upon performance measures. The focus has been on number identification, quantity discrimination, and missing numbers. 3. The least well-developed research in mathematics has been with middleschool grades. Calhoon (2008) has most aptly phrased the problem at the middle- and high-school level: ‘‘The needs for CBM assessments at the secondary level are limitless’’ (p. 238). Out of 578 articles, dissertations, and reports related to CBM, there have been only four published studies on math progress monitoring for middle school students (Foegen et al., 2007). Of these four, two investigated measures involving facts and estimation (Foegen, 2000; Foegen, et al., 2007), and two focused on measures that reflected a conceptual understanding of math (Helwig, Anderson, & Tindal, 2002; Helwig & Tindal, 2002). All four studies showed promising validity and/or reliability. The essential features of progress monitoring assessments have changed very little since the 1970s. Such measures are still expected to sample from the year’s worth of curriculum. They are meant to provide teachers with meaningful information about the progress students are making in mastering that material. In addition, to enhance their utility, progress monitoring measures are intended to be easy to administer, score, and interpret. However, whereas four decades ago the researchers deemed CBMs as not requiring any particular expertise to develop, the increasing stakes associated with assessment results as well as advances in psychometrics have significantly altered one perspective: We now recognize that the
Applications of CBMs in Making Decisions
37
creation of reliable and valid progress monitoring measures requires specialized knowledge beyond what most public school teachers possess. This realization spurred our creation of ‘‘next generation’’ elementary school reading CBMs, measures created using rigorous statistical modeling analytics previously the domain of large-scale assessments.
NEXT GENERATION CBM: ALIGNMENT TO STANDARDS, ITEM SCALING, AND TECHNICAL ADEQUACY Next-generation CBM needs to reflect the changing educational landscape with its emphasis on accountability. We developed easyCBM to reflect this high-stakes environment; in the process, we focused on the need to integrate the decisions teachers would make so that the same data system could be used across multiple decisions and at the same time be predictive of performance on a large-scale summative test. Rather than make a referral decision based on a single measure used only once for one decision (to locate students at risk), we developed screening measures that also would be related to formative progress measures. Not only would the measures be appropriate for all students so teachers could understand their differences, but they also would be useful for an individual student so teachers could understand what instructional programs were effective and make an individual difference (Deno, 1990). To accomplish this outcome and relate decisions using common measures, however, next-generation CBMs were needed that avoided three significant problems represented in the research just cited. The first problem was alignment to standards, the second was proper scaling, and the third and last was the documentation of technical adequacy with the most recent efforts being promulgated by the NCRTI.
Alignment to Standards In the most recent reauthorization of the Elementary and Secondary Education Act (ESEA), the federal government announced that a key priority in education is for all states to develop and adopt standards in reading and mathematics that build through high school (U.S. Department of Education Office of Planning Evaluation and Policy Development, 2010).
38
GERALD TINDAL AND JOSEPH F. T. NESE
Academic standards stipulate and define what students must know and be able to do at each grade level. The U.S. Department of Education also pledged support for the development and use of assessments aligned to ‘‘standards in English language arts and mathematics that accurately measure student academic achievement and growth, provide feedback to support and improve teaching, and measure school success and progress’’ (U.S. Department of Education, 2010, p. 11). Thus, the call has been made in education for curriculum standards, and formative assessments aligned to those standards. Progress is meaningful when it is directed, and CBMs that are aligned with content standards may provide more useful information for teachers about student performance and progress. CBM aligned to standards can put the direction and rate of progress into context to best address individual student learning and make informed instructional decisions. Many states and several groups have made concerted efforts to establish curriculum standards, notably the Common Core State Standards Initiative, a state-led effort coordinated by the National Governors Association Center for Best Practices (NGA Center) and the Council of Chief State School Officers (CCSSO). Yet to date, there are no published research studies on the alignment of CBMs to content standards, and easyCBM is the only CBM system that has developed assessments to be aligned to national or state standards (Alonzo, Tindal, Ulmer, & Glasgow, 2006). The easyCBM mathematics system was developed based on the National Council of Teachers of Mathematics (NCTM) Curriculum Focal Points for Kindergarten through Grade 8 Mathematics (National Council of Teachers of Mathematics, 2006). The NCTM Focal Points outline the areas of mathematics instruction focus and objective for each grade level that have been adopted by many states as the basis of their state content standards in mathematics (e.g., Oregon). The easyCBM math items were written directly to the NCTM focal point standards; more specifically, an item was written to a particular objective within a single focal point standard. The results of an alignment study indicated that the easyCBM math items consistently and strongly aligned with the NCTM Focal Points standards (Nese et al., 2010). These results provide content validation of the easyCBM math assessment system, and also lend support to the feasibility and benefits of integrating standards into the process of CBM development. But as content standards become more prevalent and the technical properties of CBMs become more robust, the field needs to better study and understand the importance of alignment of CBM items and systems to content standards.
Applications of CBMs in Making Decisions
39
Alignment with content standards also allows CBMs and high-stakes state tests to be complementary. If CBMs and the state achievement test are both developed to be aligned to the same content standards, the predictive and criterion validity of the CBMs will be enhanced, effectively increasing teachers’ instructional decisions. But the purpose of the two tests are discrete, as high-stakes state tests aim to assess year-end content mastery, and CBMs aim to assess academic status and growth across the year. Because CBMs need to be sensitive to academic growth, their ceilings should not be parallel to the knowledge proficiency cut-point associated with state tests. CBMs need to offer a large sample of items with a range of difficulties in order to accurately measure student growth. Thus, an effective CBM system must develop items that are representative of the classroom or state curriculum standards, and also items that represent a wide range of difficulties.
Use of Item Response Theory for Scaling CBM In easyCBM, the potential exists to assist teachers in moving from individual differences to making an individual difference through analysis of item-level data in making diagnostic decisions about what to teach and when. As we noted in the opening review of Deno and Mirkin (1977) and Engelmann et al. (1979), instruction is the focus of any progress monitoring system. However, rather than simply wait for data to be compiled over time as instructional programs are delivered, we have designed a data system that allows teachers to target specific skills based on the data, using item response theory (IRT). IRT has become the theoretical foundation for most current test development given its theoretical measurement principles and its potential to address measurement difficulties (Embretson & Reise, 2000). Embretson and Reise (2000) outlined several advantageous psychometric assessment properties that IRT offers, including shorter, more reliable tests; equating different test forms; providing an unbiased estimates of model parameters that can be obtained from unrepresentative samples; obtaining scores to meaningfully compare students even when initial status scores differ; and providing a standard error of measurement (SEM) that can differ across scores but generalize across populations. This last property is particularly important because it concerns the quality of the test as well as the interpretation of individual student scores. As a consequence, it allows the SEM to vary across students such that the measurement error is smaller (and
40
GERALD TINDAL AND JOSEPH F. T. NESE
thus scores are more reliable) for a specific group of students of interest (rather than equally broad for the entire distribution) (Embretson & Reise, 2000). Despite the psychometric advantages that IRT offers, it has not been widely adopted in the research on and development of CBM systems. However, easyCBM capitalizes on the properties of IRT to distribute items appropriately across forms so that each form is of equivalent difficulty. Form equivalency is paramount for a CBM system that purports to offer reliable estimates of academic growth, because it eliminates the test make-up as a reason for score change. easyCBM also uses IRT in developing assessments that provide maximum information for low-achieving students. That is, assessments are comparable and accessible for all students, but are particularly useful in progress monitoring for lower performing students with potential for poor learning outcomes, because these students are at the center of the decision-making process. IRT models the probability of endorsing an item correctly that depends upon a student’s ability and the properties of the item. IRT offers a robust diagnostic analysis of an item, including item difficulty, item discrimination (i.e., how the probability of answering an item correctly changes with student ability), and even a parameter to account for student guessing. easyCBM uses a specific IRT model, the Rasch model, which includes a parameter for individual student ability (e.g., math, vocabulary, or comprehension) and item difficulty and models the relation between the two. For example, as a student’s math ability increases, or as a math item’s difficulty decreases, the probability that a student will answer the item correctly increases. If the student’s math ability is equal to the difficulty of a math item, that student will have a 0.50 probability of answering that question correctly. This is an ideal scenario in test development because such an item affords the most information about a particular student. All of the IRT parameters are scaled in the same metric as individual student ability and thus can be used to immediately target (match) specific items with specific students. easyCBM uses IRT to better understand distributions of both individuals and items as a bridge in moving from individual differences to making an individual difference. Using IRT, all items are displayed on a logit scale that reflects the item difficulty. The advantage of this scale is that items from different measures can be placed on the same scale, allowing teachers to understand student performance relative to the different measures. In this scale a positive number represents a difficult item and a negative number represents an easy item. For example in Table 1, VQ20 (vocabulary question #20) is the most difficult item (with a scaled score of 3.00). In contrast, vocabulary question #1 is the easiest item
Applications of CBMs in Making Decisions
Table 1.
41
Item Measure (Difficulty) for Vocabulary and Comprehension Items.
Item
Measure
VQ20 CQ20 VQ7 VQ14 CQ11 VQ9 CQ12 CQ10 VQ24 CQ6 CQ9 CQ3 VQ15 VQ13 CQ14 VQ23 CQ7 VQ5 CQ18 CQ5 VQ18 VQ16 VQ21 CQ4 VQ4 VQ22 CQ19 VQ12 VQ19 CQ16 CQ15 VQ6 VQ8 VQ25 CQ17 VQ10 CQ13 VQ3 VQ11 VQ2 CQ8 VQ17
3.00 1.44 1.26 1.26 1.02 0.84 0.81 0.80 0.71 0.68 0.66 0.56 0.56 0.54 0.52 0.25 0.22 0.19 0.19 0.14 0.14 0.14 0.01 0.06 0.10 0.28 0.35 0.37 0.41 0.42 0.50 0.60 0.67 0.68 0.82 0.91 0.95 0.96 1.04 1.06 1.11 1.15
42
GERALD TINDAL AND JOSEPH F. T. NESE
Table 1. (Continued ). Item
Measure
CQ2 CQ1 VQ1
1.16 1.18 1.64
Note: VQ represents vocabulary question, followed by the item number. CQ stands for comprehension question followed by the item number.
(with a scaled score of 1.64). Both vocabulary and (multiple-choice) comprehension items are distributed in between (CQ# stands for comprehension question item number). In all, 22 items are measured below zero (on the easy end of the scale) and 23 items are measured above zero (on the difficult end of the scale). See Table 1. Another advantage of this scale is that items and persons can be placed on the same scale. As in the table, the most difficult items are positive (at the top) and the easiest items are negative (at the bottom). The same group of items as displayed in Table 1 is now displayed in Fig. 1 (with VQ20 on the top right side and VQ1 on the bottom right side). In this figure, however, students are added on the left with each dot (.) representing a student and every seven students depicted as a ‘‘#’’. In this figure, one student is on the scale at each end with VQ20 and VQ1. In addition, M represents the mean, S represents one standard deviation out, and T represents two standard deviations out. Notice that M, S, and T are present for both the students (on the left) and items (on the right). In this particular figure, therefore, it is possible to determine how the students and items distribute themselves (on the same scale). An important conclusion is that the average for students (M on the left side) is higher than the mean of the items (M on the right side). In other words, students are slightly higher in ability than the items, which means that low students are sufficiently low and there should be few false negatives in any screening for determination of risk. Another way to say this is that plenty of items exist, even for the low students (see Fig. 1). The future of IRT for CBMs is as open as it is promising, with potential for improvement and advancement. IRT can offer much beyond scaling and accessing tests. IRT also paves the way for computer-based as well as computer-adaptive tests, which can shorten test length, testing time, and offer a more reliable estimate of ability for all students. This is particularly important in measuring growth within and across years, using CBM benchmark and progress monitoring assessments. Another promising
43
Applications of CBMs in Making Decisions <more> 6
.
5 .
4
3
2
1
0
–1
.# . . ## T . .##### # .##### . .######### .###### S .####### T .######### .###### .######## .########### .######### M ########### S .######### .######### .########## .####### .####### S M .######### .#### .#### .#### .# S .# .# T . . . T
VQ20
CQ20 VQ14
VQ7
CQ11 CQ10 CQ12 VQ9 CQ6 CQ9 VQ24 CQ14 CQ3 VQ13 VQ15 VQTOT VQ23 CQ18 CQ5 CQ7 VQ16 VQ5 CQ4 VQ21 VQ18 VQ4 CQ19 VQ12 VQ19 VQ22 CQ15 CQ16 VQ25 VQ6 VQ8 CQ17 VQ10 CQ13 VQ11 VQ2 VQ3 CQ1 CQ2 CQ8 VQ17
VQ1
–2
. –3
Fig. 1. Item–Person Map (1,358 Persons with 49 Items Measured and 1,198 Persons with 47 Items Measured). Note: Students are added on the left and items on the right. Each dot (.) represents a student and each ‘#’ represents seven students. M represents the mean, S represents one standard deviation out, and T represents two standard deviations out.
44
GERALD TINDAL AND JOSEPH F. T. NESE
development lies in the use of distracter options. Typically, IRT allows test developers to analyze items and options in the development of a diagnostic assessment that provide more detailed information about a student’s knowledge and skills in a targeted area. An analysis of patterns of incorrect item responses, based on the construct of the distracters, also provides teachers with critical estimations of specific student misunderstandings. This feedback would offer practical implications and recommendations for both instruction and grouping, an important but missing part of most CBM within RTI.
Technical Adequacy and the National Center on Response to Intervention Criteria The documentation of the technical adequacy of CBM has been intermittently published by peer reviewed journals. These articles typically focus on alternate form, test–retest, or internal item reliability; the reliability of growth rates; benchmark standards for performance or growth; and predictive validity with criterion achievement measures. The NCRTI has created a clearing house for CBM systems by developing a review process of these systems, which has centralized and formalized the documentation of CBM technical adequacy. The mission of the NCRTI, funded by the Office of Special Education Programs (OSEP), is to build the capacity of states to help districts implement sound models for RTI. As such, NCRTI reviews CBM systems designed to identify students at risk for poor learning outcomes and monitor students’ academic progress, particularly in reading and math. The goal of the NCRTI review is to provide technical assistance to participating states, districts, and schools for successful implementation of the progress monitoring tools. Specific criteria are applied to the CBM systems, including different measures of reliability and validity and benchmark criteria for acceptable year-end academic performance and within- and across-year growth. The NCRTI review process exemplifies the increased technical and practical standards to which CBM systems are accountable. As an example, we can describe the validation work for items and forms of the easyCBM system. This work aims to satisfy the criteria set by the NCRTI (column headings in Table 2) for each measurement area of the system (row headings in Table 2). Separate analyses must be done by separate minority groups (e.g., racial minority and those receiving special education services) within each grade level K-8, to satisfy the 10 NCRTI
easyCBM
Tool
Math Reading, comprehension Reading, letter names Reading, letter sounds Passage reading fluency Reading, phone segmenting Word reading fluency
Area
Table 2.
Reliability of the Reliability Validity of the Predictive Alternate Sensitive to End-ofRates of Norms Performance-Level of the Performance-Level Validity of Forms Student Year Improvement Disaggregated Improvement Benchmarks Specified for Diverse Score Slope Score the Slope of Populations Improvement
Disaggregated Reliability and Validity Data
National Center for Response to Intervention Progress Monitoring Tools Chart.
Applications of CBMs in Making Decisions 45
46
GERALD TINDAL AND JOSEPH F. T. NESE
criteria for each of the 7 easyCBM measurement areas. Thus, the validation work is expansive and systematic. To demonstrate that the 20 alternate forms of each easyCBM measure were of equal and controlled difficulty, IRT analyses (described earlier in the chapter) were used to present the distribution of item difficulty across forms. In addition, alternate form reliability studies were conducted and indicated, for example, that passage reading fluency alternate form reliability ranged from .87 to .97 (Alonzo, Mariano, & Nese, 2010). The reliability of the performance-level scores were reported through internal reliability analyses of the measures, split-half reliability analyses by full grade-level sample and by ethnic groups, and test–retest reliability. The reliability of the growth slope for each measure was derived from two-level, repeated measures growth analyses conditioned on initial status; results indicated high reliability for most measures (e.g., on average, reliability for the math assessments ranged from .41 to .61, word reading fluency from .62 to .77). Norm-referenced benchmarks for year-end performance and withinyear growth (i.e., the average slope of improvement in measure score) were specified across grade levels and across ethnic groups using both nationally and locally representative samples. These norms have direct and indirect implications for teachers and administrators. For example, the proportion of students correctly classified to meet or not-meet state math proficiency based on these benchmarks ranged from .76 to .88. In other words, these CBM measures can help classify students as at risk of failing the year-end state math test with a high degree of reliability and accuracy. Validity analyses were broad and diverse. The innovative alignment study detailed earlier in the chapter provided robust support for content validity (Nese et al., 2010). In support of construct validity, the easyCBM measures were found to correlate quite highly with each other (generally above .65) and with the state test (generally above .70). In addition, confirmatory factor analyses (CFA) provided support for the theoretical measurement models developed for reading and math assessments by demonstrating excellent fit statistics. Multiple regression of fall, winter, and spring benchmark scores on state achievement test scores demonstrated support for concurrent and predictive validity. Results indicated that the easyCBM spring assessments showed high concurrent validity of the state test scores, and the fall and winter assessments showed high predictive validity of the state test scores. In addition, the separate easyCBM measures each accounted for a moderately high amount of variance in the state reading and math scores. Lastly, predictive validity information for the slope of improvement (i.e., correlation between the slope and achievement outcome) was provided using two-level,
Applications of CBMs in Making Decisions
47
repeated measures growth analyses. The interpretation of this criterion in particular is tenuous, because a zero correlation between the two would indicate that all students demonstrated significant growth, regardless of where they began the year academically. Our analyses, however, conditioned on fall achievement, indicated a range of predictive validity estimates for improvement slopes (e.g., on average, passage reading fluency ranged from .10 to .38, and reading comprehension from .46 to .59). In brief, the technical properties of CBM systems must be robust, if only because implications are critical for districts, schools, teachers, and students. The results of a CBM system often are applied in decisions about K-8 student placement intervention and special education eligibility, classroom instruction, school progress, and school and district accountability. For CBMs to be used effectively in the decision-making process of a problem-solving model like RTI, the measures must be valid so that results are meaningful, and the system must offer teachers’ benchmarks for expected performance and growth so that results are useful. The NCRTI process to CBM systems evaluation and review is based on norm-referenced, risk-rating approach. That is, a student’s performance is defined by a comparison to peers’ performance, or individual differences, and scores are often interpreted as standing within a group, or risk of ‘‘failure.’’ We look now from individual student differences to making an individual difference for each student. Essential Data in Moving from Individual Differences to Making an Individual Difference The most significant issue in moving from individual differences to making an individual difference is the change in metric from noting where students exist in a group to helping them progress in their learning. Currently, three systems are available for organizing assignment of students to: (a) tiers of instruction in an RTI model or (b) specific groups for monitoring their progress, and (c) diagnostic error patterns with items. The dominant use is with the former practice, but all three uses shift the emphasis from what has been learned to what are students learning. Benchmarking and Risk within an RTI Model RTI is generally described as a multi-tiered prevention system, but the most common system used in practice and referenced in research consists of three tiers (Fuchs, Mock, Morgan, & Young, 2003). Tier 1 consists of the regular
48
GERALD TINDAL AND JOSEPH F. T. NESE
curriculum and instruction delivered to all general education students; Tier 2 consists of supplemental instruction in an identified academic domain, delivered to students identified as not successfully responding to Tier 1 instruction; and Tier 3 consists of intensive instructional intervention delivered to individual students identified as not making satisfactory progress in Tier 2. A main principle of RTI, therefore, is the systematic assessment of students’ response to high quality, research-based instruction to identify students who would benefit most from the supplementary or intensive instruction of a higher tier. Benchmarking generally refers to the assessment process used to identify struggling students, often referred to as universal screening because the assessment is given to all students (including students in Tiers 2 and 3 as a safety check so students have the opportunity to move up or down through the support system). As part of a CBM system, these benchmark assessments are typically administered three times a year, during the fall, winter, and spring to ensure students continue to meet expectations throughout the year. Although the assessments can be of any academic domain, the most frequent assessments across grade levels include different measures of reading (e.g., fluency, comprehension, and vocabulary) and math (e.g., operations, algebra, and geometry). Along with the benchmark assessments, teachers need a complementary method to identify or classify students who may be at risk of low achievement and eligible for Tier 2 services. Most frequently, at-risk students are identified using a cut-point on a CBM benchmark assessment (Hintze & Silberglitt, 2005; Wayman, Wallace, Wiley, Ticha´, & Espin, 2007), and those students below the designated cut-point are candidates for Tier 2 instruction. Establishing these cut-points for risk is a critical matter, as they are used to direct service delivery. The methodological ideal would be for the CBM cutpoint to maximize the proportion of students predicted to be both below and above a cut-point on a criterion measure so that no classification errors are made. An inherent problem in the process of risk, however, is identifying the true status of a condition; asking the educational question: at risk of what? In the field of medicine, for example, the true status of a condition such as mumps can be easily confirmed with a simple blood test. But operationally defining ‘‘low achievement’’ or ‘‘poor learning outcomes’’ is a tenuous status. Within the current context of No Child Left Behind (NCLB) in which students are expected to make Adequate Yearly Progress (AYP) and schools and districts are held accountable, risk is often associated with proficiency on high-stakes state tests. Thus, the answer to the educational question has become: at risk of failing the state reading or math test.
Applications of CBMs in Making Decisions
49
Grouping for Monitoring Progress Benchmarking, however, does not need to be attached solely to a specific learning outcome on an external criterion, such as a state test, as the basis for assigning risk potential and assignment to an instructional tier. Districts and schools may also create cut-points to organize resources (financial and human) for monitoring progress. The data produced by CBM benchmarking can be used to create instructional groups based on performance or to assign specific teams of teachers for various instructional programs. For example, a school may target students who score below the 20th percentile on a CBM benchmark reading assessment for an academic intervention. Instructional grouping, whether homogeneous or heterogeneous in skill level, is a controversial topic in education. Proponents claim that it allows teachers to adapt learning outcomes, instructional activities, and pace to better meet the needs of students (Lou et al., 1996; Vaughn et al., 2003). In contrast, opponents claim that students may experience less focused instruction, lowered academic expectations, and less academic engagement (Elbaum, Schumm, & Vaughn, 1997). Although grouping has not been a standard practice, it has been sufficiently common to yield some results. Among a random sample of over 200 primary-grade teachers from across the United States, almost two-thirds of teachers reported using homogenous instruction groups, most reporting that they did so to best meet the needs of their students (Chorzempa & Graham, 2006). Some teachers who did not use homogenous groups reported that it was not allowed by their school or district, or that it was not compatible with their curriculum, but most teachers simply used heterogeneous groupings instead. Groups can be homogeneous or heterogeneous in skill level, depending on need, which may help with allocation of resources within the classroom, including personnel, intervention materials, and time. But homogeneous groupings can help teachers establish appropriate academic goals for each student by providing an appropriate comparative context made up of students of a similar skill level. Silberglitt and Hintze (2007) emphasized the importance of referencing students to the correct group by comparing student rates of within-year reading growth based on initial level of performance. Results indicated that the highest and lowest achieving student groups had significantly lower slopes than did the average student group, and consequently suggested that the academic growth of students may be misinterpreted if viewed in comparison to the growth of a different group.
50
GERALD TINDAL AND JOSEPH F. T. NESE
USING EASYCBM TO ENACT THE ENTIRE DECISION-MAKING PROCESS We use real data from the easyCBM system to articulate the intricacies of benchmarking, risk rating, grouping, and progress monitoring (http:// easyCBM.coms). easyCBM is an online benchmark and progress monitoring assessment system, available for free teacher use, that includes multiple different reading and mathematics measures developed for students from K-8. The measures on the system were developed using IRT and represent the psychometric properties expected of modern technically adequate assessments: All items on all measures have good fit (Alonzo, Liu, & Tindal, 2007; Alonzo & Tindal, 2007) as determined by a one-parameter logistic Rasch model with mean square outfits within the acceptable range of 0.5–1.5. Items represent a range of difficulties. Finally, the forms are equivalent in difficulty within each grade. Screening (for Documenting Individual Differences) Many schools, teachers, and administrators use the easyCBM system as part of their RTI programs, and we present here a specific example of a student in such a school. Within this model, all students are administered math and reading CBM benchmark assessments during the fall, winter, and spring. The reading benchmarks include assessments of oral reading fluency (ORF) or passage reading fluency (PRF), vocabulary (VOC), and multiple-choice reading comprehension (MCRC). ORF/PRF is a one-minute test of reading accuracy and fluency in which the total number of correct words per minute (cwpm) is calculated by subtracting the number of incorrectly read words from the total number of words read. The ORF/PRF assessment is administered to each student individually. VOC is a test of students’ knowledge of words in which a student chooses a word’s synonym from three word options. The VOC assessment is group-administered by classroom teachers using an online computer administration format. The MCRC is test of reading comprehension in which a student reads a narrative fiction text and answers 20 questions about the text (literal, inferential, and evaluative). The MCRC assessment is also online and group-administered. A Class Roster When teachers begin the school year, all students are screened, including students in all tiers of instruction if the school is using an RTI model. The
Applications of CBMs in Making Decisions
51
purpose of the screener is to determine which students are at risk of failure in reading, using normative benchmarks in various grades to stratify students into three groups (a) strategic (dark gray), (b) intensive (medium gray), and (c) on track (light gray). Although districts are free to establish these groups as they want, most set the strategic group as students below the 10th percentile rank (PR), students between the 11th and 30th PR as intensive, and finally students on track as above the 30th PR. The color coding is applied to each measure and a cumulative risk rating applied overall in the following manner. First, performance on each measure is rated, with strategic students receiving 2 points, intensive students receiving 1 point, and students on track receiving 0 points. Second, the points are added and the following point awards are used to assign students to a final risk rating (though districts are also free to establish these cut-offs): (5–6 points are considered strategic, 3–4 points are considered intensive, and 0–2 points are considered on track). Each teacher receives a class roster with their students listed alphabetically and the option to sort by level of risk (which is depicted in Table 3). As can be seen in this table, the risk rating is applied in the fall and in the spring so teachers may see change in risk groups as well as overall status at the end of the year (blank cells represent students who have moved in during the year and having missing data; therefore they are coded according to the values present for the particular season) (see Table 3). A Case Study Student A began third grade with a benchmark PRF score of 58 cwpm, which was just below the 20th percentile (Fig. 2); a benchmark VOC score which was well below the 10th percentile (Fig. 3); and a benchmark MCRC score which was just above the 20th percentile (Fig. 4). Because this student scored near or below the 20th percentile in all three benchmark assessments, she was targeted for a Tier 2 intervention intended to address her specific academic needs, and placed in a group of students of reading with similar performance and skills for supplemental reading instruction. As a result of her easyCBM benchmark reading scores, beginning on September 15, Student A began an intervention labeled by the school, ‘‘Instructional Differentiation with Progress Monitoring.’’ This intervention is demarcated by a vertical intervention line in Figs. 2–4. In a strictly quantitative analyses of the data (based on a ROC analysis), Student A’s spring benchmark scores were above the cut score and she was predicted to pass the year-end state reading test; in fact, she did not pass this test. But her intervention reference group perhaps provided a more
5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5
Grade
F M M M M F F M F F M F M M F F F M M M F F M F F F F M F M M M M
Gender
50 50 90
50
50 90
50
50
Disab. Code
3 1 1 1 1 3 1 1 1 1 1 3 1 1 3 1 1 1 3 1 1 3 1 1 3 3 1 1 1 3 1 1 1
Ethnic
20 18 16 18 18 21 22
155 130 106 142 124 138 155
143 132 149 103 131 126
11 18 12 9 11 14
13 15 15
20 18 20 15 17
147 155 137 179 139
115 149 127
18 22 16 23 20 15 20
Fall Vocab.
159 184 184 210 118 162 168
Fall PRF
9 6 13 12 15 8
5 9 11
13 16 12 15 16 15 13
18 11 6 13 15
18 16 14 17 15 12 14
Fall MCRC
4 4 3 6 3 5
5 4 5
1 2 5 1 2 0 1
0 3 2 3 1
1 0 2 0 1 4 1
Fall Factor
H H S H S H
H H H
L S H L S L L
L S S S L
L L S L L H L
Fall Risk Category
185 192 186 181 168 178 199 199 185 183 160 207 145 215 179 146 130 175 105 168 170 178 155 152 157 141 157 147 195 138 120 160 134
Spring PRF
22 23 22 22 23 24 22 21 20 20 21 22 21 19 24 20 22 19 23 17 20 18 19 18 18 16 12 12 16 19 20 17 18
Spring Vocab.
16 16 16 16 16 16 18 15 17 18 15 14 16 17 15 19 16 14 17 17 14 15 14 15 13 15 10 15 13 13 14 8 13
Spring MCRC
0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 3 3 3 4 4 4 4 4 4 4 4
Spring Factor
Class Roster to Stratify Students into Three Groups based on Normative Reading Benchmarks.
L L L L L L L L L L L L L L L S S S S S S S S S S H H H H H H H H
Spring Risk Category
Note: Dark gray represents the strategic group, students below the 10th percentile rank (PR). Medium gray represents the intensive group, students between the 11th and 30th PR. Light gray represents the on track group, students above the 30th PR.
1 2 7 8 9 16 17 21 22 23 24 25 26 27 28 40 41 42 43 44 45 46 50 51 52 53 54 55 56 57 58 59 60
Last Name
Table 3. 52 GERALD TINDAL AND JOSEPH F. T. NESE
Applications of CBMs in Making Decisions
53
Fig. 2. Student A’s Third Grade easyCBM Passage Reading Fluency Scores. Note: The vertical intervention line represents the implementation of Student A’s Tier 2 intervention. The black diagonal line represents a rough estimate of Student A’s slope, or passage reading fluency growth across the year. easyCBMs graphs also include a dark gray line at the bottom representing students’ scores at the 10th percentile; a medium gray line above the lowest line reflecting scores at the 20th percentile; a line reflecting scores at the 50th percentile; and a light gray line at the top representing scores at the 90th percentile.
appropriate indicator of her yearly progress, and a more promising estimate of her learning for her teachers and parents than the state score might. Student A was correctly identified as needing a reading intervention at the beginning of the year; she was assessed using CBM and monitored to identify specific areas of need, and as a result she made practical gains in reading skills.
Progress Monitoring (for Documenting an Individual Difference) Figs. 2 and 4 show Student A’s academic progress across the school year in reading fluency and comprehension, respectively (no vocabulary progress
54
GERALD TINDAL AND JOSEPH F. T. NESE
Fig. 3. Student A’s Third Grade easyCBM Vocabulary Scores. Note: easyCBMs graphs also include a dark gray line at the bottom representing students’ scores at the 10th percentile; a medium gray line above the lowest line reflecting scores at the 20th percentile; a line reflecting scores at the 50th percentile; and a light gray line at the top representing scores at the 90th percentile.
monitoring assessments were available to Student A during the school year). Her spring easyCBM reading benchmark scores (PRF ¼ 95, VOC ¼ 19, MCRC ¼ 18) suggest that she made progress throughout the year, although it is difficult to qualify this progress. In an ipsative approach, Student A’s academic gains during the year were of practical significance, but because she received a group intervention, the teacher or reading specialist could reference her growth with her peers and make a more meaningful interpretation of the data. This grouping method allows both Student A’s status and growth to be qualified to determine whether her response to the intervention is meeting expectations so that instruction can be continued, discontinued, or revised based on appropriate student references. Student A’s spring PRF was above the 25th percentile, and two progress monitoring tests after the spring benchmark showed fluency close to the 50th percentile (Fig. 2). Her scores on the MCRC progress monitoring assessments appeared to increase consistently across the year (Fig. 2), and a high spring
Applications of CBMs in Making Decisions
55
Fig. 4. Student A’s Third Grade easyCBM Multiple-Choice Reading Comprehension Scores. Note: The vertical intervention line represents the mplementation of Student A’s Tier 2 intervention. The black diagonal line represents a rough estimate of Student A’s slope, or passage reading fluency growth across the year. easyCBMs graphs also include a dark gray line at the bottom representing students’ scores at the 10th percentile; a medium gray line above the lowest line reflecting scores at the 20th percentile; a line reflecting scores at the 50th percentile; and a light gray line at the top representing scores at the 90th percentile.
comprehension score indicated that she was well above the 50th percentile in reading comprehension at year’s end. No vocabulary progress monitoring measures were administered to Student A throughout the year, so the data provided only a gain score and not a growth trajectory. Student A ended the school year with a score on the spring vocabulary benchmark that was close to the 10th percentile, indicating an area that perhaps needs further attention.
REFERENCES Allinder, R. (1996). When some is not better than none: Effects of differential implementation of curriculum-based measurement. Exceptional Children, 62, 525–535.
56
GERALD TINDAL AND JOSEPH F. T. NESE
Alonzo, J., Liu, K., & Tindal, G. (2007). Examining the technical adequacy of reading comprehension measures in a progress monitoring assessment system. Technical Report # 41. Behavioral Research and Teaching, Eugene, OR. Alonzo, J., Mariano, G., & Nese, J. F. (2010). Reliability of the easyCBM reading assessments. Paper presented at the 18th annual meeting of the Pacific Coast Research Conference, Coronado, CA. Alonzo, J., & Tindal, G. (2007). Examining the technical adequacy of word reading and passage reading fluency measures in a progress monitoring assessment system. Technical Report # 40. Behavioral Research and Teaching, Eugene, OR. Alonzo, J., Tindal, G., Ulmer, K., & Glasgow, A. (2006). easyCBM online progress monitoring assessment system. Eugene, OR: Center for Educational Assessment Accountability. Available at http://easycbm.com Calhoon, M. (2008). Curriculum-based measurement for mathmatics at the high school level: What we do not know and what we need to know. Assessment for Effective Intervention, 33, 234–239. doi: 10.1177/15345084073/3488 Chorzempa, B. F., & Graham, S. (2006). Primary-grade teachers’ use of within-class ability grouping in reading. Journal of Educational Psychology, 98, 529–541. doi: 10.1037/ 0022-0663.98.3.529 Deno, S. (1987). Curriculum-based measurement. Teaching Exceptional Children, 20(1), 41–47. Deno, S. (1990). Individual differences and individual difference: The essential difference of special education. The Journal of Special Education, 24(2), 159–173. Deno, S. (2003). Developments in curriculum-based measurement. The Journal of Special Education, 37, 184–192. doi: 10.1177/0022669030370030801 Deno, S., & Mirkin, P. (1977). Data based program modification: A manual. Minneapolis, MN: Leadership Training Institute for Special Education. Elbaum, B. E., Schumm, J. S., & Vaughn, S. (1997). Urban middle elementary students’ perceptions of grouping formats for reading instruction. The Elementary School Journal, 97, 475–500. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum Associates. Engelmann, S., Granzin, A., & Severson, H. (1979). Diagnosing instruction. The Journal of Special Education, 13(4), 355–363. Fewster, S., & Macmillan, P. (2002). School-based evidence for the validity of curriculum-based measurement of reading and writing. Remedial and Special Education, 23, 149–156. doi: 10.1177/07419325020230030301 Foegen, A. (2000). Technical adequacy of general outcome measures for middle-school mathematics. Assessment for Effective Intervention, 25, 175–203. doi: 10.1177/ 073724770002500301 Foegen, A., Jiban, C., & Deno, S. (2007). Progress monitoring measures in mathematics: A review of the literature. Journal of Special Education, 41, 121–139. doi: 10.1177/ 00224669070410020101 Fuchs, D., Mock, D., Morgan, P. L., & Young, C. L. (2003). Responsiveness-to-intervention: Definitions, evidence, and implications for the learning disabilities construct. Learning Disabilities Research & Practice, 18, 157–171. Fuchs, L. (2004). The past, present, and future of curriculum-based measurement research. School Psychology Review, 33, 188–192.
Applications of CBMs in Making Decisions
57
Fuchs, L., & Deno, S. (1992). Effects of curriculum within curriculum-based measurement. Exceptional Children, 58, 232–243. Fuchs, L., Fuchs, D., & Maxwell, L. (1988). The validity of informal reading comprehension measures. Remedial and Special Education, 9(2), 20–28. Fuchs, L., Fuchs, D., & Stecker, P. (1989). Effects of curriculum-based measurement on teachers’ instructional planning. Journal of Learning Disabilities, 22, 51–59. doi: 10.1177/ 002221948902200110 Fuchs, L., Hamlett, C. L., & Fuchs, D. (1998). Monitoring basic skills progress: Basic math contempts and applications [computer program manual]. Austin, TX: Pro Ed. Fuchs, L., Tindal, G., & Deno, S. (1984). Methodological issues in curriculum-based reading assessment. Diagnostique, 9, 191–207. doi: 10.1177/073724778400900401 Helwig, R., Anderson, L., & Tindal, G. (2002). Using a concept-grounded, curriculum-based measure in mathematics to predict statewide test scores for middle school students with LD. Journal of Special Education, 36, 102–112. doi: 10.1177/00224669020360020501 Helwig, R., & Tindal, G. (2002). Using general outcome measures in mathematics to measure adequate yearly progress as mandated by Title I. Assessment for Effective Intervention, 28, 9–18. doi: 10.1177/073724770202800102 Hintze, J., & Silberglitt, B. (2005). A longitudinal examination of the diagnostic accuracy and predictive validity of R-CBM and high stakes testing. School Psychology Review, 34, 372–386. Kranzler, J., Brownell, M., & Miller, M. (1998). The construct validity of curriculum-based measurement of reading: An empirical test of a plausible rival hypothesis. Journal of School Psychology, 36, 399–415. doi: 10.1016/S0022-4405(98)00018-1 Lou, Y., Abrami, P. C., Spence, J. C., Poulsen, C., Chambers, B., & d’Apollonia, S. (1996). Within-class grouping: A meta-analysis. Review of Educational Research, 66, 423–458. Mehrens, W., & Clarizio, H. (1993). Curriculum-based measurement: Conceptual and psychometric considerations. Psychology in the Schools, 30, 214–254. doi: 10.1002/ 1520-6807(199307)30:3o241::AID-PITS2310300307W3.0.CO;2-J National Council of Teachers of Mathematics. (2006). Curriculum focal points for prekindergarten through Grade 8 mathematics: A quest for coherence. Reston, VA: National Council of Teachers of Mathematics. Nese, J. F., Lai, C.F., Anderson, D., Park, B. J., Tindal, G., & Alonso, J., (2010). The alignment of easyCBM math measures to curriculum standards. Technical Report #1002. Behavioral Research and Teaching, Eugene, OR. Shinn, M. (1989). Curriculum-based measurement: Assessing special children. New York, NY: Guilford Press. Shinn, M. (1998). Advanced applications of curriculum-based measurement. New York, NY: Guilford Press. Shinn, M., Gleason, M., & Tindal, G. (1989). Varying the difficulty of testing materials: Implications for curriculum-based measurement. Journal of Special Education, 23, 223–233. doi: 10.1177/002246698902300208 Shinn, M., Good, R., Knutson, N., Tilly, W., & Colllins, V. (1992). Curriculum-based measurement of oral reading fluency: A confirmatory analysis of its relation to reading. School Psychology Review, 21, 459–479. Silberglitt, B., & Hintze, J. (2007). How much growth can we expect? A conditional analysis of R-CBM growth rates by level of performance. Exceptional Children, 74, 71–84.
58
GERALD TINDAL AND JOSEPH F. T. NESE
U.S. Department of Education Office of Planning Evaluation and Policy Development. (2010). ESEA Blueprint for Reform. Washington, DC. Vaughn, S., Linan-Thompson, S., Kouzedanani, D., Bryant, P., Dickson, S., & Blazis, S. (2003). Reading instruction grouping for students with reading difficulties. Remedial and Special Education, 24, 301–315. Wallace, T., Espin, C., McMaster, K., & Deno, S. (2007). CBM progress monitoring within a standards-based system. The Journal of Special Education, 41, 66–67. doi: 10.1177/ 00224669070410020201 Wayman, M., Wallace, T., Wiley, H., Ticha´, R., & Espin, C. (2007). Literature synthesis on curriculum-based measurement in reading. Journal of Special Education, 41, 85–120. doi: 10.1177/00224669070410020401 Wesson, C., Deno, S., Mirkin, P., Sevcik, B., Skiba, R., King, R., et al. (1988). A casual analysis of the relationships among on-going measurement and evaluation, the structure of instruction, and student achievement. The Journal of Special Education, 22, 330–343. doi: 10.1177/002246698802200304 Wesson, C., Vierthaler, J., & Haubrich, P. (1989). The discriminative validity of curriculumbased measures for establishing reading groups. Reading Research and Instruction, 29(1), 23–32.
EVOLVING STANDARDS OF DIAGNOSTIC ACCURACY IN PREDICTING AND AVOIDING ACADEMIC FAILURE Amanda M. VanDerHeyden ABSTRACT Since its first appearance in policy, the diagnostic construct of learning disability (LD) has struggled to demonstrate its validity. Any diagnostic construct requires the operationalization of a ‘‘true positive’’ diagnosis to permit strong analyses of diagnostic accuracy and associated intervention outcomes. Because there is no ‘‘true positive’’ definition for LD, diagnostic accuracy and outcome research are disparate and difficult to translate into meaningful actions for diagnosticians and intervention teams in schools. In this chapter, a new framework of decision making centered on consequential validity, evidence-based education, and shared decision making is proposed for evaluating the relative costs and benefits of alternative actions when making decisions about whether to conduct assessment or intervention, and what assessments or interventions to implement.
Assessment and Intervention Advances in Learning and Behavioral Disabilities, Volume 24, 59–78 Copyright r 2011 by Emerald Group Publishing Limited All rights of reproduction in any form reserved ISSN: 0735-004X/doi:10.1108/S0735-004X(2011)0000024005
59
60
AMANDA M. VANDERHEYDEN
EVOLVING STANDARDS OF DIAGNOSTIC ACCURACY IN ASSESSMENT OF LEARNING DISABILITY Some allege that the learning disabilities (LD) construct was born from a desire to explain poor achievement in the absence of readily understood causes in ways that were not (1) stigmatizing to the student and (2) could lead to remediation if only the child were diagnosed and then provided with the ‘‘right’’ instruction (Sleeter, 1986; Zigmond, 1993). Early and strong popular endorsement of LD as a valid diagnostic construct was grounded in what turned out to be the undue belief that diagnosis of LD would lead to specialized interventions that would produce normalized performance for struggling learners (Gresham & Witt, 1997; Kavale & Forness, 1999). Early research primarily came from an advocacy position that the LD construct was valid and subsequent research was designed to provide rich descriptions of individuals diagnosed with LD rather than questioning the discriminant, predictive, and consequential evidence for the LD diagnosis in the first place. The IQ-achievement discrepancy definition of LD was codified in law and regulatory provisions in 1977 by PL 94-142 (the precursor to Individuals with Disabilities in Education Act) and almost immediately received criticism and questions even as the rate of LD diagnosis proliferated in schools (Algozzine, Ysseldyke, & Christenson, 1983; Ysseldyke, Vanderwood, & Shriner, 1997).
OVERRELIANCE ON CORRELATION: ABSENCE OF CONVERGING AND DIVERGING EVIDENCE The LD construct, operationalized as achievement that differs from achievement that is expected based on measures of cognitive processing, has suffered from a lack of convergent and discriminant validity evidence. Valid diagnosis requires that application of diagnostic criteria reliably result in a homogeneous grouping of individuals who share the diagnosed condition and that these individuals differ from others who do not meet diagnostic criteria and do not share the diagnosed condition (Campbell & Fiske, 1959). Consider for a moment the diagnostic criteria for breast cancer. Individuals with breast cancer will share certain symptoms. Some of these symptoms are useful for diagnostic decision making and some are not. Useful symptoms must meet convergent and discriminant validity
Predicting and Avoiding Academic Failure
61
standards. For example, a symptom like fatigue may be a symptom that is shared by most individuals with breast cancer. Fatigue as a diagnostic marker for breast cancer does contribute to homogeneous grouping of individuals with breast cancer because most individuals with breast cancer experience fatigue. A converging pattern is observed: fatigue is positively correlated with breast cancer. The problem is that absence of fatigue does not discriminate well those individuals who do not have breast cancer. In other words, people who do not have breast cancer also commonly experience fatigue. Hence, a diverging pattern is not present: not having fatigue is not correlated with not having breast cancer. Therefore, fatigue has little value as a diagnostic criterion for breast cancer. Useful diagnostic criteria are those measurable symptoms that are shared by individuals with the diagnostic condition (converging evidence) and not shared by those without the diagnostic condition (diverging evidence). To validate the diagnostic construct of LD, it is necessary to demonstrate that students with LD can be readily discriminated from similarly lowperforming students without LD on the basis of proposed measures of the construct. Lack of discriminant validity evidence has been a notable challenge to reliable and valid diagnosis of LD with compelling data demonstrating that errors of classification are prevalent when trying to distinguish among subgroups of low-performing students (Macmann, Barnett, Lombard, Kocher-Belton, & Sharpe, 1989); that measures of IQ are not meaningful in predicting who will successfully respond to intervention and who will not (Lennon & Slesinski, 1999; Vellutino, Scanlon, & Lyon, 2000; Vellutino et al., 1996; Vellutino, Scanlon, & Tanzman, 1998; Vellutino, Scanlon, & Zhang, 2007); and that IQachievement discrepant readers and IQ-achievement consistent readers have not been found to differ systematically on a continuum of measures including phonological awareness, rapid naming, verbal short-term memory, and vocabulary lexical skills (Stuebing et al., 2002). IQ-achievement difference scores (i.e., discrepancies) are unreliable and do not differ predictably between typically achieving learners and low-achieving learners (Kratochwill, Clements, & Kalymon, 2007). Various discrepancy criteria do not result in the same diagnostic decision for the same student with a given set of test scores and different tests may result in different results for the same student given the same discrepancy criterion (Fuchs, Fuchs, & Compton, 2004; Kavale & Flanagan, 2007). These data have led many researchers to conclude that cognitive processing measures are not capable of discriminating students with LD from so-called garden-variety low achievers, especially in the area of reading. Further, specific instructional
62
AMANDA M. VANDERHEYDEN
techniques have not been demonstrated to convey unique benefit to students with LD versus low-performing students without LD (Vellutino et al., 1996, 1998, 2000, 2007) and in practice, special instructional techniques have not been associated with special education services provided to students with LD (Ysseldyke & Christenson, 1987; Ysseldyke, Thurlow, Christenson, & Weiss, 1987; Ysseldyke, O’Sullivan, Thurlow, & Christenson, 1989). The diagnostic challenges associated with figuring out who really has LD and who is ‘‘just’’ a struggling learner led researchers and frontline diagnosticians to adjust diagnostic procedures across settings and time, creating a moving diagnostic target. These diagnostic realities have contributed to a climate where practitioners wish to find help for struggling students, view special education as a way to obtain help, view diagnosis as a necessary means to obtain help through special education, and given the flexibility in the current diagnostic criteria can make a diagnosis of LD with little accountability if that diagnosis is made in error. In fact, advocates have prioritized overdiagnosis of LD in the name of protecting against false negative identification errors. Given the lack of consequential validity evidence for the diagnosis of LD, overdiagnosis of LD is an error of consequence that should be reconsidered by parents, advocates, and school systems.
CONSEQUENTIAL VALIDITY AND THE LD CONSTRUCT AS A POSSIBLE RED HERRING The idea that educational services provided to students should lead to appreciably positive changes in their everyday lives is consonant with the ideals of education and psychology (Wolf, 1978; Hayes, Nelson, & Jarrett, 1987). In 1995, Messick expanded the notion of construct validity to include not just the reliability and validity of scores obtained on measures but also the intended and unintended social consequences of the use of the scores to reach decisions. Academic failure and diagnosis of LD have been associated with high poverty rates and ethnic and gender disproportionality (Donovan & Cross, 2002). When an individual carries a higher probability of receiving a diagnosis of LD because he is male, of a minority ethnicity, or receives free and reduced lunch, the equity of diagnostic procedures must be questioned. Is the diagnosis presumed to legitimately affect children differently based on ethnic, poverty, or gender status? Are the criteria biased in some way? If the condition is presumed to disproportionately affect individuals, is there
Predicting and Avoiding Academic Failure
63
demonstrated benefit that comes from making the diagnosis and does this benefit outweigh potential for harm? Given that specialized interventions are not available on the basis of diagnosis and outcomes have not been shown to be improved through services in special education (Kavale & Forness, 1999), the consequential validity of the LD diagnosis is empirically unsupported (Barnett, Macmann, & Carey, 1992). If consequential validity is a foundation of our practice, then perhaps focusing on whether or not a child truly does or truly does not have LD is a waste of resources. Perhaps asking whether or not a student really has or really does not have LD is the wrong question and will continue to lead us down fruitless paths of increasingly complicated diagnostic criteria designed to rule out every possible imagined cause of poor performance. Such a path will be counterproductive because increasingly cumbersome rule-out procedures are not likely to be implemented with fidelity in schools and there will always remain the possibility that there is some cause of poor performance that we failed to imagine and assess and rule out when making the diagnosis.
THE DIAGNOSTIC CONUNDRUM OF LD: WHAT IS A TRUE POSITIVE? IS LOW ACHIEVEMENT EVER REALLY UNEXPECTED? Kavale and Flanagan (2007) argued that LD is unexpected underachievement and then described a method for determining when underachievement is indeed unexpected. The problem with their argument is that underachievement is a relative term that is determined on the basis of what is expected, which also must be determined from some measure of student potential for achievement. Kavale and Flanagan state, ‘‘A non-significant discrepancy may be found simply because the cognitive abilities/processes that are responsible for low achievement have attenuated the total test score y If those specific abilities/processes could be removed from the total test score and in so doing a significant discrepancy emerged, then this finding would suggest underachievement. In short, while the finding of a non-significant discrepancy may rule out underachievement in some cases, it does not rule out underachievement in all cases’’ (p. 133). These comments underscore the impossibility of diagnosing LD in a stable, replicable, and meaningful way because the diagnostic criteria as described by Kavale and
64
AMANDA M. VANDERHEYDEN
Flanagan are fluid and are determined in part by the battery of measures to which the child happens to be exposed. At the heart of the problem here is the fact that there has yet to be an operationalized definition of a ‘‘true positive’’ for LD. A ‘‘true positive’’ is a person who truly has LD. Because we have no incontrovertible index, no way of knowing if someone really has or really does not have LD, defining a true positive for LD requires articulating what measures with what scores indicate that a person truly has LD and therein lies the problem. Diagnostic accuracy studies evaluating the utility of various assessment procedures, cutscores, and decision rules require a definition of a true positive and true negative. For example, to conduct diagnostic accuracy studies, researchers will categorize students into one of four cells on a grid. The column headings are criterion positive and criterion negative (these are often referred to as the ‘‘gold standard’’ criteria). The row headings are test positive and test negative. Each child can be sorted into one of the four cells based on his or her scores on test measures and whether or not he or she meets established, agreed-upon criteria for being a true positive (first column) or true negative (second column). Once children are sorted into cells, then classification agreement analyses (e.g., sensitivity, specificity) are possible. Defining a true positive is the first step, otherwise, studies of various symptoms and cutscores will be disparate and chase a moving target. The classification agreement analyses published in the literature over the last 10 years illustrate well the lack of a definition of a true positive for LD. These studies have commonly used outcomes like frequently appearing in the risk range due to low level of performance and slow growth relative to peers (Speece, Case, & Molloy, 2003), failing to respond to short-term intervention (VanDerHeyden, Witt, & Naquin, 2003), or performing below some benchmark on a criterion measure in reading (Hintze, Ryan, & Stoner, 2003). Although these studies represented an important step forward in beginning to quantify the utility of various screening criteria for identifying a need for intervention, from a broader perspective, it seems illogical that frequent debate has centered on the details associated with measuring the presence of LD when we have not yet even defined what LD is in a way that has uniform support. The trouble with LD as a diagnostic construct is that its primary symptom, poor academic achievement, is neither a unique nor specific marker for the presence of LD. So, just like fatigue in the example earlier in this chapter, poor academic achievement, even when it is unexpected, is not uniquely and specifically associated with the presence of LD. Children may demonstrate lower than expected achievement for a
Predicting and Avoiding Academic Failure
65
number of reasons that have little to do with having LD. For example, children may demonstrate low achievement because they do not like a task and are not highly motivated to give their best performances. Instruction may not have adequately met the needs of a given student. A student may have failed to master the prerequisite skills required to be successful on a given task. Therefore, diagnosis of LD necessitates a series of rule-out decisions. Diagnosticians must rule out alternative explanations of potential causes of poor achievement. After attempting to rule out all possible alternative causes of low achievement, the diagnostician concludes that the child has some condition called LD by default (there are no other tenable explanations to explain low performance and so the child must have a within-child problem called LD). One fundamental assumption in the paradigm of a rule-out diagnosis of LD is that a child who does not attain expected benchmarks in the face of environmental supports to assure high motivation and powerful intervention is a child who truly has LD. As a researcher and practitioner, this gives me qualms. In my scientific framework, there is no flawed learner, but rather there are only flawed interventions. I define a failed RtI as a situation in which we have not yet found the right intervention. I am not ready to conclude that the intervention has successfully ruled-in the presence of some condition we call LD. Again, failed response to intervention has to be a unique and specific marker for the presence of LD and until we can define what that is, we cannot make that leap. Second, interventions fail for many reasons, the most common of which involves poor implementation integrity (Wickstrom, Jones, LaFleur, & Witt, 1998; Witt, Noell, LaFleur, & Mortenson, 1997). Much of the foregoing argument is simply not relevant to the political context in which we operate where diagnosis is required to provide special education services. However, the field could make a paradigm shift as has happened over the years, providing intervention to those students and evaluating the effects of those interventions. Whether we call these students LD or not matters little to me. What we do with those students is what matters and certainly we can do better than what we have done. LD is a diagnosis of exclusion and diagnoses of exclusion are particularly susceptible to error (Hunt & Walker, 1971). Where there is no ‘‘true positive’’ definition and instead diagnosis depends on ruling out hypothesized alternative causes, there is always the possibility that the diagnostician has failed to identify a potential causal variable and/or failed to correctly measure one of the rule-out criteria. Because a ‘‘true positive’’ for LD is
66
AMANDA M. VANDERHEYDEN
unmeasurable, tenable diagnostic accuracy studies are impossible. As a result of discordant definitions of LD, the research base for LD rightfully could be described as disparate. This state of affairs is not just a problem for our science, it is a problem of major consequence ethically and socially. Understanding and operationalizing the construct of LD is fraught with competing contingencies among advocates, policymakers, researchers, practitioners, parents, and students. Fuchs, Fuchs, and Stecker (2010) contextualize RtI as a framework at the center of a host of competing agendas held by various stakeholders. Fuchs and colleagues seem to characterize groups of researchers and organizations as being ‘‘for’’ or ‘‘against’’ special education and then conclude by suggesting that everyone is missing the point. Indeed, it is possible that for all the improvements in using data to filter students into levels of intervention, that we have no tenable data suggesting that those students who have always struggled have shown much improvement as a result of these efforts.
WHAT IS A TRUE POSITIVE IN RTI? Fuchs et al. (2010) assert that there are students who are poorly served in education (general and special) and that these are largely the same students who have always been poorly served (Fuchs, Fuchs, Mathes, & Lipsey, 2000). As for how RtI intersects with this group of low-achieving students, I believe that these children are disproportionately unresponsive to intervention relative to their at-risk peers. On the other hand, their lack of responsiveness is not diagnostic for the presence of LD. Lack of responsiveness might function as a specific marker for ruling in the presence of LD, but only so long as the gold standard in that scenario is lack of responsiveness to X or Y intervention. Reaching broader conclusions about the presence or absence of disability based on failed RtI requires operationalization of a true positive and dealing with the substantial threat that the ‘‘right’’ intervention was not implemented or implemented well. Otherwise, diagnosing LD is akin to proving the negative, which simply cannot be done. With an unreliable or unstable set of diagnostic criteria and a resulting disparate research base, it is no wonder to me that the most vulnerable of students have not been well served by our efforts to reform the systems and rules governing support for struggling students. Selecting measurable outcomes like reduced risk status, improved academic performance, improved equity, accelerated rates of learning, give us, in effect, a set of ‘‘true positive’’ conditions that we can define, aim for,
Predicting and Avoiding Academic Failure
67
and evaluate. Defining true positive outcomes allows us to focus on attaining the outcomes that matter, and avoid the vulnerabilities associated with making error-prone discriminations between students who fundamentally exhibit the same symptoms, that is, trouble learning. Thus far, efforts to diagnose and serve students with LD have not demonstrated appreciably positive effects (Kavale & Forness, 1999) and so to move toward evidencebased practice in education, we must begin to define the outcomes we care about, align our efforts with those outcomes, and measure progress toward those outcomes, at both the system and the individual student level.
WHAT IS EVIDENCE-BASED PRACTICE AND WHY DOES IT MATTER IN EDUCATION? The integration of science into frontline clinical decision making first occurred in medicine (Sackett, Rosenberg, Gray, Haynes, & Richardson, 1996). Evidence-based medicine gave diagnosticians a framework for quantifying the utility of certain symptoms and diagnostic tests to rule-in or rule-out diagnostic conditions and identifying treatments that were likely to yield the greatest benefit to the patient with the least risk of aversive, painful, or dangerous side effects. Evidence-based medicine was important because it established science as the ‘‘best’’ basis for action and implied a decision system that seeks to consider the cost relative to the benefits at each stage of decision making (initial screening, follow-up assessment, treatment evaluation). It is not a surprise that the framework of evidence-based practice resonated with educators because it shared many similarities to data-based decision-making models that were emerging in schools (Deno & Mirkin, 1977; Marston, 1989). At the core of evidence-based practice models is an emphasis on the consequential or treatment validity of assessments and diagnoses. The validity of an assessment is irrevocably tied to the yield it brings in measurable improvements for a person exposed to that assessment. This measurable yield also considers the cost associated with giving the assessment or making the diagnosis that includes tangible and intangible costs of decision errors and the cost of the assessment itself. Evidence-based practice offers a framework for evaluating not whether an assessment is useful, but rather how much more useful that assessment is compared to available alternatives which include not doing the assessment at all. Importantly, evidence-based practice gives implementers a way to avoid
68
AMANDA M. VANDERHEYDEN
the errors associated with mistaking correlation alone for strong evidence or simple accuracy for correct decision making (Meehl & Rosen, 1955). The question is not whether performance on a given measure correlates with presence of a condition we wish to diagnose, but rather the measurable extent to which use of the assessment leads to a diagnostic conclusion that increases the odds of positive outcomes for the person exposed to the assessment (VanDerHeyden, 2010). This question allows us to focus on the ingredients that matter in a diagnostic framework. Educational decision makers have neglected to consider how prevalence interacts with assessment accuracy in mathematically predictable ways (VanDerHeyden, 2010). Educators must begin to examine whether each additional assessment provides an incremental benefit in decision-making accuracy that makes a measurable difference in the outcome attained by the student. Implicit in such a framework is the notion of efficiency. It is possible that strong universal prevention strategies will outweigh any benefit conveyed by additional assessment for the vast majority of students. Obviously, student performance data collected at routine intervals will be the fuel that drives the car of data-based decision making and multitiered intervention in schools. Bushell and Baer (1994 [2008]) described two worlds in education: One is the world in which we choose everything we want to teach from the universe of measurable goals, and then find the teaching techniques to maximize those measures. In the other world, we choose only how we will teach, as if there were an a priori correct way; whatever we get by teaching that way then must be what we want or should want. It is especially easy to believe that, when what we want is not measurable – or at least not measured. Perhaps the second world is so consistently preferred over the first because the first is doubly embarrassing: It requires us to decide what we want to teach, and it is always embarrassing to discover that in some small part we don’t know and in some small part we don’t agree. And it exposes us to a series of surprises as we discover what techniques teach the goals better than others. (p. 266)
Bushell and Bear recognized the folly of instructing without regard to the learning that either happens or does not happen. A theme in educational reform over the last three decades has been that instructional design cannot be separated from its result (i.e., learning). It is the learning, the quality of the learning, the rate of learning, and the robustness of the learning that must guide the instruction that follows. Consequently, frequent student assessment has become a cornerstone of effective teaching practice (National Council of Teachers of Mathematics, 2000; National Mathematics Advisory Panel, 2008; National Reading Panel, 2000). Yeh (2007) conducted a very compelling analysis of the return for various investments
Predicting and Avoiding Academic Failure
69
that could be made to enhance learning in schools. In his analysis he computed effect sizes on student achievement for the following conditions: frequent student assessment (2–5 times per week), 10% increase in per pupil spending, voucher programs, charter schools, and increased accountability. He then computed the cost for each approach. Frequent student assessment was 4 times as effective as increased spending per pupil, 6 times as effective as vouchers, 64 times more effective than charter schools, and 6 times more effective than increased accountability even after accounting for the increased costs associated with conducting frequent student assessment. These findings are consonant with familiar data demonstrating the effects of frequent progress monitoring and decision making on student achievement (Fuchs, Fuchs, Hamlett, & Stecker, 1991; Fuchs, Fuchs, Mathes, & Simmons, 1997).
IMPLEMENTATION IS COMPLICATED AND FAILS MORE THAN IT SUCCEEDS Perhaps the most enduring implications of the evidence-based practice movement in education will be the challenges associated with implementation. Implementation failures are not new phenomena. Implementations have failed as long as there have been new recommended programs to try (Fixsen & Blase´, 1993). In education, the framework in science has been to develop and evaluate programs separate from the contexts in which they will be installed or implemented. This framework has resulted in data that are well-controlled but all too often demonstrate no significant effect or have little chance of producing similar effects in the real world (Cook, Tankersley, & Landrum, 2009). In a prophetic article in 1978, Montrose Wolf reflected on implementation failures related to his work developing effective treatment programs for youth exhibiting disruptive behaviors. Wolf commented, ‘‘y the first time we tried to replicate the Achievement Place program in another community, that community gave us feedback in a most drastic manner. Before we really knew they had complaints about our program, they had ‘fired’ us’’ (p. 206). And later in the article, y we were told by many communities that one of the most important characteristics of teaching-parents was ‘‘warmth.’’ When quizzed about ‘‘warmth,’’ the community members indicated that they wanted teaching-parents who ‘‘know how to relate to youths.’’ For some time, our response to this request was to disagree with them. We argued, ‘‘What you really need is someone who knows how to give and take away points at the right time.’’ (p. 207)
70
AMANDA M. VANDERHEYDEN
Wolf concluded, ‘‘A new consensus seems to be developing. It seems that if we aspire to social importance, then we must develop systems that allow our consumers to provide us feedback about how our applications relate to their values, to their reinforcers’’ (p. 213). Wolf recognized then that implementation failures were events that were worthy of analysis, that causes of implementation failures could be identified. His suggestion was to expand the outcomes of behavior-change efforts to include measurement of consumer satisfaction, and this recommendation is pertinent in today’s climate of educational accountability and school system reform. Implementation is a behavior-change process that requires matching of environmental demands with scientific know-how in ways that implementers will have the skill and the motivation to implement (Neef, 1995). Implementation requires, therefore, close and continual contact with the relevant outcome data that should specifically link to the outcomes that are important to or are prized by the system. That is, implementation requires initial equipment, training, and installment, but these conditions are not sufficient for full implementation. Full implementation is incremental and requires constant tending of actual implementation and the effects to adjust the methods used, build environmental capacity to implement, and ensure that those responsible for the implementation experience positive results from their perspective so that implementation will be reinforced and will sustain (Fixsen & Blase´, 1993). There is a long legacy in education of failed innovations. One of the most salient examples of a failed innovation involves direct instruction (Carnine, 2000). In a compelling commentary on what it takes to get evidence-based practices implemented in classrooms, Carnine reiterates the need for analysis and implementation to maintain direct and frequent contact with the student data outcomes that matter. Effective practices will not be adopted if dissemination decisions are politicized and researchers, policy makers, and frontline implementers have not agreed on which outcomes matter most and should be used to guide implementation. Carnine’s commentary highlights the damage of philosophy-driven educational practices at a direct and detrimental cost to datadriven practices where effects on student learning outcomes are the prioritized effects. Carnine writes, ‘‘The best way for a profession to ensure its continued autonomy is to adopt methods that ensure the safety and efficacy of its practices’’ (p. 10). So the key question will remain, once the field has agreed on outcomes related to optimal student learning and satisfaction as the driving force for specialized interventions in schools, how to get those interventions effectively implemented in classrooms. There has been an ever-present
Predicting and Avoiding Academic Failure
71
tendency in education to focus on intervention selection and to neglect intervention management. This tendency is directly responsible for the majority of intervention failures. Finding an intervention that should work to improve a particular learning problem is not terribly challenging. Getting that intervention implemented in the classroom is substantially more challenging. Research on intervention integrity allows us to reach the following conclusions: (1) intervention failure should be a rare event, but in many settings it is not, (2) most interventions fail because direct measurement of implementation accuracy is not collected or implementation failures are not responded to by adults in the intervention setting. Interventions fail because there is no direct and frequent measurement of both implementation and implementation effects with systematic adjustments to improve effectiveness (Fixsen & Blase´, 1993). There is not enough attention paid to what outcomes merit priority and consensus for how these outcomes will be measured.
WHAT ABOUT SHARED DECISION MAKING IN EDUCATION? GAMBLING AND PROSTRATE SCREENING MAY PAVE THE WAY In medicine, a great deal of attention has been paid to involving patients in medical decision making about what assessments to order and what treatments to provide. Shared decision making has been framed as a way to increase patient buy in with the idea that if patients have a shared role in the decisions, compliance with prescribed treatments will be greater and patients (and systems) will experience superior outcomes. Like many promising initiatives, shared decision making has been more difficult to accomplish in practice. Each decision about an assessment to order carries with it implications related to the accuracy of the test (sensitivity and specificity) in the context in which it is ordered (higher- or lower-risk setting for the condition) and given certain measured symptoms of the patient (symptoms may increase likelihood of a true positive or true negative) and the costs associated with ordering the test (what happens to the patient if there is a false positive decision error? what happens to the patient if there is a false negative decision error?). For highly trained diagnosticians, the systematic influence on decision error that arises from predictable sources like prevalence of a condition in a given setting (e.g., a high-risk setting implies a higher rate of true positive cases in that setting) is frequently, even
72
AMANDA M. VANDERHEYDEN
typically, underestimated and poorly understood (Meehl & Rosen, 1955). Hoffman, Wilkes, Day, Bell, and Higa (2006) developed a decision-making aid that could be used by decision makers to determine relative risk when reaching decisions about what assessments to conduct and what interventions to begin. Their decision aid begins with two side-by-side pie charts (see Fig. 1) that illustrate the probability of a patient (a) doing well (i.e., having no symptoms), (b) experiencing negative side effects (e.g., incontinence), or (c) dying if the 65-year-old patient elects not to have the prostrate screening (chart on left) and if the patient chooses to have the prostrate screening (chart on right). Visually, one can see right away, that the chances of dying are greater if one elects not to have the screening, but only slightly so. Because the test is associated with a possible side effect of incontinence, one can also see that having the test increases chances of becoming incontinent. To make the concept of probability more accessible to decision makers, the same probabilities were used to build two dartboards as in Fig. 2 where a decision maker could choose which dartboard he or she would choose to throw a dart at (i.e., not having the screening on the left or having the screening on the right). Finally, because dartboard games do include more than just random chance (i.e., there is arguably some skill involved in hitting a mark with a dart), the aid was adjusted to represent a roulette wheel as in Fig. 3. Here a decision maker can look at the graph and decide based on probabilities which test to order or not order. This kind of probabilistic thinking is absent in decisions about screening and intervention in schools.
No PSA Screening Asymptomatic
Annual PSA Screening
Risk of incontinence and/or impotence
Death from Prostate Cancer
Fig. 1. Probability of Positive Outcomes (Light Gray), Negative Side Effects (Dark Gray), and Death (White) for a 65-Year Old Patient Who Chooses not to Undergo Screening (On Left) and Who Chooses to Undergo Screening (On Right). Reprinted with Permission from Hoffman et al. (2006).
73
Predicting and Avoiding Academic Failure
No PSA Screening Asymptomatic
Fig. 2.
Risk of incontinence and/or impotence
Death from Prostate Cancer
Same Probabilities as in Figure 2, Displayed On a Dart Board. Reprinted with Permission from Hoffman et al. (2006).
No PSA Screening Asymptomatic
Outcome:
Fig. 3.
Annual PSA Screening
Annual PSA Screening
Risk of incontinence and/or impotence
Death from Prostate Cancer
Outcome:
Same Probabilities as in Figure 2, Displayed On a Roulette Wheel. Reprinted with Permission from Hoffman et al. (2006).
In education, there has been an a priori position that more intervention is better even when the data tell us otherwise (Kavale & Forness, 1999). The aid becomes even more useful when one considers the relative risks and benefits of potential treatments. In the example provided by Hoffman and colleagues, knowing that a disease has 40% mortality that can be cut in half with treatment but the treatment carries with it minor side effects in 10% of
74
AMANDA M. VANDERHEYDEN
No Treatment Asymptomatic
With Treatment
Risk of Side Effects
Death from Disease
Fig. 4. Hypothetical Probability of Positive Outcomes (Light Gray), Negative Side Effects (Dark Gray), and Death (White) Given No Treatment (On Left) and Treatment (On Right) for Prostate Cancer. Odds Favor Choosing to Have Treatment. Reprinted with Permission from Hoffman et al. (2006).
No Treatment Asymptomatic
With Treatment
Risk of Side Effects
Death from Disease
Fig. 5. Hypothetical Probabilities for Various Outcomes When Choosing Not to Have Treatment (On Left) and Choosing to Have Treatment (On Right). In This Example, Treatment is Not The Obviously Favored Option. Decision Makers Will Have to Evaluate the Relative Costs of Negative Side Effects Versus the Odds of Death in Electing to Have the Treatment. Reprinted with Permission from Hoffman et al. (2006).
patients, decision makers can readily see that treatment ought to be the favored option based on probability (see Fig. 4). However, doubling the side effects of treatment as in Fig. 5 makes the decision more complicated and
Predicting and Avoiding Academic Failure
75
requires a judgment about how ‘‘bad’’ the negative side effects are before determining that treatment is worthwhile (and better than dying) because the chances favor that with treatment or without the combined risk of death and/or side effects is the same. In education, we are rarely talking about death. Education’s version of death might be academic failure and dropout from school. Similarly, assessment decisions do carry risks. For example, early reading screening carries substantial false positive error risks (Hintze et al., 2003). Savvy decision makers should question what are the consequences of not screening for reading compared to the consequences of screening for reading? My guess is that empirical evidence would support the decision to screen because of the reduced risk of failing to learn to read if intervention is provided. Choosing one treatment over another, however, complicates the decision. Now decision makers need to understand what are the costs associated with participating in treatment (i.e., loss of instructional time in the classroom, segregation from classroom peers) and the probability of treatment success or failure (child learns to read successfully, child does not learn to read successfully) and make judgments accordingly. For screening, data are available to guide decision makers in knowing whether the benefits outweigh the risks, although they are not typically explained with great precision in this way. In determining whether or not to begin treatment and which treatment to begin, however, the data are much less well developed. Researchers and practitioners should measure student outcomes, measure implementation of interventions designed to improve outcomes, and quantify the effects and choices for consumers in ways that convey the relative risks and benefits of each assessment and treatment option.
REFERENCES Algozzine, B., Ysseldyke, J. E., & Christenson, S. (1983). An analysis of the incidence of special class placement: The masses are burgeoning. The Journal of Special Education, 17, 141–147. Barnett, D. W., Macmann, G. M., & Carey, K. T. (1992). Early intervention and the assessment of developmental skills: Challenges and directions. Topics in Early Childhood Special Education, 12, 21–43. Bushell, D., Jr., & Baer, D. M. (1994 [2008]). Measurably superior instruction means close, continual contact with the relevant outcome data. Revolutionary! In: R. Detrich, R. Keyworth & J. States (Eds), Advances in evidence-based education: A roadmap to evidence-based education (pp. 265–275). Oakland, CA: The Wing Institute. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105. Carnine, D. (2000). Why education experts resist effective practices (and what it would take to make education more like medicine). Available at http://www.edexcellence.net/foundation/ publication/publication.cfm?id ¼ 46. Retrieved on March 16, 2004.
76
AMANDA M. VANDERHEYDEN
Cook, B. G., Tankersley, M., & Landrum, T. J. (2009). Determining evidence-based practices in special education. Exceptional Children, 75, 365–383. Deno, S. L., & Mirkin, P. K. (1977). Data-based program modification: A manual. Reston, VA: Council for Exceptional Children. Donovan, M. S., & Cross, C. T. (2002). Minority students in special and gifted education. Washington, DC: National Academy Press. Fixsen, D. L., & Blase´, K. A. (1993). Creating new realities: Program development and dissemination. Journal of Applied Behavior Analysis, 26, 597–615. Fuchs, D., Fuchs, L. S., & Compton, D. L. (2004). Identifying reading disabilities by responsiveness to instruction: Specifying measures and criteria. Learning Disability Quarterly, 27, 216–228. Fuchs, D., Fuchs, L. S., Mathes, P. G., & Lipsey, M. W. (2000). Reading differences between low-achieving students with and without learning disabilities: A meta-analysis. In: R. Gersten, E. P. Schiller & S. Vaughn (Eds), Contemporary special education research (pp. 81–104). Mahwah, NJ: Erlbaum. Fuchs, D., Fuchs, L. S., Mathes, P. G., & Simmons, D. C. (1997). Peer-assisted learning strategies: Making classrooms more responsive to diversity. American Educational Research Journal, 34, 174–206. Fuchs, D., Fuchs, L. S., & Stecker, P. M. (2010). The ‘‘blurring’’ of special education in a new continuum of general education placements and services. Exceptional Children, 76, 301–323. Fuchs, L. S., Fuchs, D., Hamlett, C. L., & Stecker, P. M. (1991). Effects of curriculum-based measurement and consultation on teacher planning and student achievement in mathematics operations. American Educational Research Journal, 28, 617–641. Gresham, F. M., & Witt, J. C. (1997). Utility of intelligence tests for treatment planning, classification, and placement decisions: Recent empirical findings and future directions. School Psychology Quarterly, 12, 249–267. Hayes, S. C., Nelson, R. O., & Jarrett, R. B. (1987). The treatment utility of assessment: A functional approach to evaluating assessment quality. American Psychologist, 42, 963–974. Hintze, J. M., Ryan, A. L., & Stoner, G. (2003). Concurrent validity and diagnostic accuracy of the dynamic indicators of basic early literacy skills and the comprehensive test of phonological processing. School Psychology Review, 32, 541–556. Hoffman, J. R., Wilkes, M. S., Day, F. C., Bell, D. S., & Higa, J. K. (2006). The roulette wheel: An aid to informed decision making. PLoS Med, 3(6), e137. Hunt, W. A., & Walker, R. E. (1971). Cue utilization in diagnostic judgment. Journal of Clinical Psychology, 62–64. Kavale, K. A., & Flanagan, D. P. (2007). Ability-achievement discrepancy, response to intervention, and assessment of cognitive abilities/processes in specific learning disability identification: Toward a contemporary operational definition. In: S. Jimerson, M. Burns & A. VanDerHeyden (Eds), Handbook of response to intervention: The science and practice of assessment and intervention (pp. 130–147). New York: Springer. Kavale, K. A., & Forness, S. R. (1999). Effectiveness of special education. In: C. R. Reynolds & T. B. Gutkin (Eds), The handbook of school psychology (3rd ed., pp. 984–1024). New York: Wiley. Kratochwill, T. R., Clements, M. A., & Kalymon, K. M. (2007). Response to intervention: Conceptual and methodological issues in implementation. In: S. Jimerson, M. Burns & A. VanDerHeyden (Eds), Handbook of response to intervention: The science and practice of assessment and intervention (pp. 25–52). New York: Springer.
Predicting and Avoiding Academic Failure
77
Lennon, J. E., & Slesinski, C. (1999). Early intervention in reading: Results of a screening and intervention program for kindergarten students. School Psychology Review, 28, 353–364. Macmann, G. M., Barnett, D. W., Lombard, T. J., Kocher-Belton, E., & Sharpe, M. N. (1989). On the actuarial classification of children: Fundamental studies of classification agreement. The Journal of Special Education, 23, 127–149. Marston, D. B. (1989). A curriculum-based measurement approach to assessing academic performance: What it is and why do it. In: M. R. Shinn (Ed.), Curriculum-based measurement: Assessing special children (pp. 18–78). New York: Guilford Press. Meehl, P. E., & Rosen, A. (1955). Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores. Psychological Bulletin, 52, 194–215. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749. National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics. Reston, VA: Author. National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the national mathematics advisory panel. Washington, DC: U.S. Department of Education. National Reading Panel. (2000). Report of the national reading panel. Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. NIH Publication no. 00-4769. Washington, DC: U.S. Government Printing Office. Neef, N. A. (1995). Research on training trainers in program implementation: An introduction and future directions. Journal of Applied Behavior Analysis, 28, 297–299. Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (Eds). (1996). Editorial: Evidence based medicine: What it is and what it isn’t. British Medical Journal, 312, 71–72. PMid:3527718. Sleeter, C. E. (1986). Learning disabilities: The social construction of a special education category. Exceptional Children, 53, 46–54. Speece, D. L., Case, L. P., & Molloy, D. E. (2003). Responsiveness to general education instruction as the first gate to learning disabilities identification. Learning Disabilities Research and Practice, 18, 147–156. doi:10.1111/1540-5826.00071 Stuebing, K., Fletcher, J., LeDoux, J., Lyon, G. R., Shaywitz, S., & Shaywitz, B. (2002). Validity of IQ-discrepancy classifications of reading disabilities: A meta-analysis. American Educational Research Journal, 39, 469–518. VanDerHeyden, A. M. (2010). Determining early mathematical risk: Ideas for extending the research. School Psychology Review, 39, 196–202. VanDerHeyden, A. M. (2010). Technical adequacy of response to intervention decisions. Exceptional Children, 77, 335–350. VanDerHeyden, A. M., Witt, J. C., & Naquin, G. (2003). Development and validation of a process for screening referrals to special education. School Psychology Review, 32, 204– 227PMid:15505962. Vellutino, F., Scanlon, D., Sipay, E., Small, S., Pratt, A., Chen, R., & Denckla, M. (1996). Cognitive profiles of difficult-to-remediate and readily remediated poor readers: Early intervention as a vehicle for distinguishing between cognitive and experiential deficits as basic causes of specific reading disability. Journal of Educational Psychology, 88, 601–638.
78
AMANDA M. VANDERHEYDEN
Vellutino, F. R., Scanlon, D. M., & Lyon, G. R. (2000). Differentiating between difficult-toremediate and readily remediated poor readers: More evidence against the IQachievement discrepancy definition of reading disability. Journal of Learning Disabilities, 33, 223–238. Vellutino, F. R., Scanlon, D. M., & Tanzman, M. S. (1998). The case for early intervention in diagnosing specific reading disability. Journal of School Psychology, 36, 367–397. Vellutino, F. R., Scanlon, D. M., & Zhang, H. (2007). Identifying reading disability based on response to intervention: Evidence from early intervention research. In: S. R. Jimerson, M. K. Burns & A. M. VanDerHeyden (Eds), Handbook of response to intervention: The science and practice of assessment and intervention (pp. 185–211). New York: Springer. Wickstrom, K. F., Jones, K. M., LaFleur, L. H., & Witt, J. C. (1998). An analysis of treatment integrity in school-based behavioral consultation. School Psychology Quarterly, 13, 141–154. Witt, J. C., Noell, G. H., LaFleur, L. H., & Mortenson, B. P. (1997). Teacher use of interventions in general education settings: Measurement and analysis of the independent variable. Journal of Applied Behavior Analysis, 30, 693–696. Wolf, M. M. (1978). Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11, 203–214. Yeh, S. S. (2007). The cost effectiveness of five policies for improving student achievement. Journal of Evaluation, 28, 416–436. Ysseldyke, J. E., & Christenson, S. L. (1987). Evaluating students’ instructional environments. Remedial and Special Education, 8, 17–24. Ysseldyke, J. E., O’Sullivan, P. J., Thurlow, M. L., & Christenson, S. L. (1989). Qualitative differences in reading and math instruction received by handicapped students. Remedial and Special Education, 10, 14–20. Ysseldyke, J. E., Thurlow, M. L., Christenson, S. L., & Weiss, J. (1987). Time allocated to instruction of mentally retarded, learning disabled, emotionally disturbed, and nonhandicapped elementary students. The Journal of Special Education, 21, 43–55. Ysseldyke, J. E., Vanderwood, M. L., & Shriner, J. (1997). Changes over the past decade in special education referral to placement probability: An incredibly reliable practice. Diagnostique, 23, 193–201. Zigmond, N. (1993). Learning disabilities from an educational perspective. In: G. R. Lyon, D. B. Gray, J. F. Kavanaugh & N. A. Krasnegor (Eds), Better understanding learning disabilities: New views from research and their implications for education and public policies (pp. 27–56). Baltimore: Brookes.
HOW TO USE PSYCHOLOGICAL TESTS FOR FUNCTIONAL DIAGNOSIS: THE CASE OF ASSESSMENT OF LEARNING DISABILITIES Giulia Balboni and Roberto Cubelli ABSTRACT Very often the diagnostic process provides only a descriptive label of the disorder. A useful diagnosis must reach the precise description and the functional interpretation of the deficits shown by a given individual. To this aim, four separate but intertwined steps are required: (1) clinical interview to collect personal and clinical anamnesis and to figure out the sociocultural context; (2) screening tests to describe the general cognitive picture and to detect the impaired functions; (3) standardized batteries to categorize the specific cognitive disorders according to the accredited taxonomies; (4) ad hoc investigation to identify the impaired cognitive components in the individual patient according to a sound theoretical model of the functional architecture of the cognitive processes. Only doing in this way, personalized educational and rehabilitative interventions may be planned and specific goals can be achieved.
Assessment and Intervention Advances in Learning and Behavioral Disabilities, Volume 24, 79–92 Copyright r 2011 by Emerald Group Publishing Limited All rights of reproduction in any form reserved ISSN: 0735-004X/doi:10.1108/S0735-004X(2011)0000024006
79
80
GIULIA BALBONI AND ROBERTO CUBELLI
The term ‘‘diagnosis’’ means ‘‘discerning, distinguishing’’ and derives from the Greek word diagignoskein, which literally signifies ‘‘to know thoroughly’’ (from dia- ‘‘apart’’ and gignoskein ‘‘to learn’’). It follows that the diagnostic process is aimed to the recognition of a disease or condition and the interpretation of the underlying causes by its outward signs and symptoms. A useful diagnosis allows achieving the precise description and the functional interpretation of the behavioral or cognitive disorders of a given individual, i.e., it is intended to identify the impaired processes and mechanisms that may explain the observed pathological pattern of performance (e.g., Ellis & Young, 1988). In this way, when possible, personalized educational and rehabilitative interventions may be planned and specific goals for ameliorating everyday life can be achieved. Very often the diagnostic process consists of only the assignment of a descriptive label of the disorder, thus confirming what is already known without providing any useful information on the damaged components of the cognitive system. For example, in the case of a student with reading difficulties, it may happen that the diagnosis is limited to categorize the apparent disorder according to the available classification systems, like DSMIV-TR or ICD-10, based on the reduced levels of speed and accuracy. In this way, the existence of a problem in reading a written text is ascertained but no useful information is provided to teachers or parents who usually are aware of the reading problem and decide to consult professionals and experts to deal with this critical disorder. In clinical settings, the emphasis given to the objective property of psychological tests and the availability of normative scores has promoted the idea that tests are self-sufficient tools. In other words, arbitrary psychometric measures, like the number of the standard deviations below the mean obtained by the normative sample or the score assumed as pathological cut-off, are often considered sufficient to formulate a cognitive diagnosis and conclude a neuropsychological assessment. In some cases, in adjunct of the DSM classification, more sophisticated diagnosis allows identifying subtypes of a specific disorder. In the case of dyslexia, e.g., the pattern of performance in different tasks and the qualitative analysis of errors, based upon the dual route model (Coltheart & Rastle, 1994; Coltheart, Rastle, Perry, Langdon, & Ziegler, 2001) can lead to distinguishing between phonological and surface dyslexia. In the former, novel words are read less accurately than known words with mainly segmental errors; in the latter, regular words are read better than irregular words, with regularization errors and difficulties in segmenting letter strings. However, categorizing the behavioral pattern is not sufficient since the same clinical label can be applied to individuals showing impairments at different
Psychological Tests for Functional Diagnosis
81
cognitive levels (for different forms of developmental surface dyslexia see Friedmann & Lukov, 2008). The diagnostic process must go beyond any very broad clinical category and must lead, for each individual patient, to the identification of the specifically impaired cognitive processes. Actually, to obtain a useful diagnosis, and not just a label, a deeper assessment of the individual history and the overall pattern of abilities and disabilities is needed. Using tests without investigating the personal and clinical history impedes to evaluate all factors associated to the disorder (e.g., the social-cultural level or the medical interventions) that can have a role in the genesis of the disorder or can interact with its manifestation. Categorizing without attempting any interpretation of the origin of the difficulties does not allow the identification of the cognitive components specifically impaired and therefore prevents the possibility to plan an effective and personalized intervention. In order to achieve a functional interpretation of the disorder for all individual patients, four separate but intertwined steps are needed, each of them requiring different instruments (see Cubelli & Della Sala, in press): (1) clinical interview to collect personal and clinical anamnesis and to grasp the sociocultural context; (2) screening tests to describe the general cognitive picture and to detect the impaired functions; (3) standardized batteries to categorize the specific disorders according to the shared taxonomies; (4) ad hoc investigation to identify the impaired cognitive components according to a sound theoretical model of the processes investigated. Each stage is crucial to obtain a correct and useful functional diagnosis; the entire process must be always completed.
STEPS OF THE FUNCTIONAL DIAGNOSIS Clinical Interview The first step should provide information concerning the personal and clinical history of the person to be evaluated. The clinical interview is a qualitative method of evaluation based on oral questions regarding specific contents and aimed to isolate and contextualize the complained difficulties (e.g., Lang, 2003; Lis, Venuti, & De Zordo, 1995). It allows taking into consideration a given cognitive problem (e.g., reading difficulties), estimating how it interferes with daily living activities and how it is subjectively experienced. The clinical interview is useful to investigate the context within which the complaint arises and to ascertain whether other issues, either
82
GIULIA BALBONI AND ROBERTO CUBELLI
personal (e.g., emotional and motivational factors), familiar (e.g., sociocultural level), or educational (e.g., poor teaching), may play a causal role. At the end of this first step, the professional should realize whether the difficulties are secondary to these factors or, alternatively, reflect a specific, primary cognitive disorder which is independent of the context. In the case of students with learning disabilities, the parents should be interviewed first, then the teachers, finally the students. Parents should be met both simultaneously and separately. When met individually, the information provided by the parents, either consistent or conflicting, is useful to draw a complete picture of the difficulties to be evaluated and their impact on the everyday activities. In contrast, interviewing both parents contribute to establish a collaborative relationship which is necessary to plan and conduct an efficient intervention. The clinical interview is aimed to describe: (1) the alleged learning deficits; (2) the medical history, with special attention paid to any physical or mental disorders which could be associated to the learning deficits, such as visual and auditory defects, language disorder, hyperactivity and attention deficits, emotional disorders like depression or anxiety; (3) all social and personal factors which can influence the learning attitude, i.e., the interaction with peers, classmates, and teachers, the motivation and personal interests; the school activities and methods of education; (4) the household environment, e.g., sociocultural level of parents and relatives, emotional interaction and conflicts, relevant facts and events. The interview with the teachers is aimed to address the same issues; in particular, it has to provide information on the educational experience and learning difficulties. As well as for the parents, also for the teachers, the clinical interview is critical to establish a cooperative relationship (Bigozzi, 2000). The interview with the student (e.g., Greenspan, 2003; Morrison & Anders, 2001) is crucial to assess whether they are aware of the difficulties and available to collaborate. The professional must develop a ‘‘diagnosing alliance’’ to achieve an adequate diagnosis and, if necessary, an effective intervention (Del Corno & Lang, 1996). Different areas should be investigated: cognitive level, inclination and preferences, social interaction, copying strategies, meta-cognitive judgments.
Screening Tests The second step should allow confirming the presence of the alleged difficulties. For this purpose, screening tests for assessing the different cognitive abilities must be used. Screening tests are brief and easy instruments
Psychological Tests for Functional Diagnosis
83
developed to be administered to large groups of people in order to detect signs and defective behavior (e.g., Crocker & Algina, 1986; Streiner, 2003). To be useful, the screening tests must have specific psychometric properties, known as diagnostic efficiency statistics (e.g., Hsu, 2002; Streiner, 2003). First, screening tests must be sensitive and specific (e.g., Glaros & Kline, 1988; Hasselblad & Hedges, 1995). The sensitivity of a test is defined as the accuracy in identifying persons who have the disorder and corresponds to the proportion of persons having the disorder who are detected by the given test. Given the score on a screening test obtained by two groups of people with and without a specific disorder, the sensitivity is equal to the number of impaired individuals having a pathological score (true positives) ratio the number of persons belonging to the clinical group (the true positives plus those who do not fail the test, i.e., the false negatives). The specificity of a test is the accuracy in detecting people who do not have a given disorder and corresponds to the proportion of persons without the disorder who are correctly classified by the test. The specificity is equal to the number of persons with a negative score on the test, indicating the absence of the disorder (true negatives), ratio the number of persons without the disorder (the true negatives plus those erroneously classified as showing the disorder, i.e., the false positives). Sensitivity and specificity can be combined into a single statistic called likelihood ratio (e.g., Kraemer et al., 1999). This statistics tells what the odds are that a positive result has come from a person with the disorder and is equal to: Likelihood ratio ¼
Sensitivity True Positive Rate ¼ 1 Specificity False Positive Rate
For any given test score, the likelihood ratio may be computed via the receiver operating characteristic analysis (ROC; e.g., Kraemer et al., 1999; McFall & Treat, 1999). The higher is the likelihood ratio the lower is the likelihood of false positives and false negatives that may be detected by means of the test. The number of false negatives must be low to avoid failures in identifying individuals showing the target disorder. However, information gained by means of the clinical interview is useful to prevent a person, who obtained a score within the normal range on a single screening test but shows difficulties in everyday activities and suffered with some illness, from being wrongly classified as unimpaired. Also the false positives should be few to prevent individuals to be further evaluated, thus inducing stress and lack of confidence in the examiner and the examinee, and ambiguous findings which are impossible to interpret (Streiner, 2003).
84
GIULIA BALBONI AND ROBERTO CUBELLI
A screening test is an effective diagnostic tool when among people with a pathological score on a given test, the percentage of those who really show the deficit is higher than the incidence of that disorder in the general population (base rate; e.g., Hsu, 2002; Streiner, 2003). To have a measure of the increase in the predictive values over what would be expected by going with the base rate, the incremental positive predictive power (IPPP) must be computed (also called incremental validity; Gibertini, Brandenburg, & Retzlaff, 1986; Kraemer et al., 1999). IPPP ¼
PPP PðDxÞ 1 PðDxÞ
Positive predictive power (PPP) is equal to the number of persons with a disorder ratio the number of persons detected with the disorder by the test, and P(Dx) is the probability of having the disorder (base rate). The ratio is equal to zero when the test adds nothing to the base rate and is equal to 1 when there are no diagnostic errors. On screening tests, people of the control groups make few mistakes, most of them perform flawlessly. Therefore, the typical distribution of raw scores on screening tests does not approximate to the normal distribution curve, but is leptokurtic with some degrees of skewed (respectively positively or negatively based on the score being equal to the number of wrongly or correctly answered items). Given this kind of nonnormal distribution, the percentile rank (PR) is the most useful normative score (e.g., Allen & Yen, 2002; Crocker & Algina, 1986). An individual is classified as being affected by a given disorder if the score on the screening test is higher or lower (based on the raw score being equal to the number of wrongly or correctly answered items, respectively) than a conventional threshold. To identify this threshold, a criterion or a normative approach can be used. Based on the criterion approach, the threshold should be the PR corresponding to the raw score which allows discriminating two groups of persons with and without the disorder, the raw score for which the likelihood ratio is highest (e.g., Scott, Fletcher, & Martell, 2000). Based on the normative approach, given a test administered to a sample representative of the general population showing that the rate of the disorder is not different to the base rate in the general population, the threshold should be the PR that corresponds to the base rate. For example, if the base rate is 15%, the PR threshold should be equal to 15 (and people diagnosed as defective are those obtaining lower scores) when the raw score is the number of the correct answers, or equal to 85 (people diagnosed as defective are those obtaining
Psychological Tests for Functional Diagnosis
85
higher scores) when the raw score is the number of the incorrect answers. The final threshold should be close to both the thresholds identified with the criterion and normative approaches. The PR scores are less reliable for raw scores in the central part of the distribution than for those in the extremes; therefore, to identify the threshold as well as the normative score obtained by an individual a percentile band equal to the PR71 standard error of measurement should be considered (e.g., Crocker & Algina, 1986; see, for more details, Crawford, Garthwaite, & Slick, 2009). To detect learning disabilities, several screening tests are available. Professionals have to evaluate the likelihood ratio, IPPP, and all methods to compute the threshold of the different test and to identify the most valid tests for assessing each cognitive process. One of the most valid screening tests for reading difficulties available in Italy is the MT test, which measures performance speed and accuracy in children from one to ten grades (Cornoldi & Colpo, 1998, 1995; Cornoldi, Baldi, & Friso, 2010). The MT test includes short texts to be read aloud; the total time of reading and of the number and type of errors (e.g., omission or addition of syllables and words; pause longer than 5 seconds) are the dependent variables. Two different raw scores are computed: the number of seconds used for each one syllable and the numbers of errors. The normative scores are similar to the PRs (i.e., cumulative percentage); for each text four thresholds are available indicating four levels of performance: (a) very low with immediate intervention needed; (b) low with attention required; (c) just above the critical threshold; (d) well within the normal range suggesting a learned ability to read. For the same texts, the confidence interval computed using the standard error of measurement is available. The test can be administered at the beginning, at the middle, and at the end of the school year. A series of screening tests provide psychometric measures on reading and writing as well as on other cognitive functions, e.g., perception and memory, thus allowing the identification of the specific domains which need attention and further investigation.
Standardized Batteries The third step is aimed to classify an individual performance by assigning a clinical labeling according to standard diagnostic criteria, e.g., DSM-IV-TR. Using information gleaned from the preliminary steps, the cognitive processes alleged to be impaired should be addressed. The diagnosis is not based on a single test, but on the pattern of dissociation and association between tasks (see Ellis & Young, 1988).
86
GIULIA BALBONI AND ROBERTO CUBELLI
To diagnose specific disorders, the assessment must be based on theoretical models describing the functional architecture of the mental processes and predicting the specific consequences of a selective impairment. For example, the Italian Battery for the Assessment of Developmental Dyslexia and Dysorthography-2 (Sartori, Job, & Tressoldi, 2007) is based on the dual route model of reading (Coltheart & Rastle, 1994) and aims to investigate the integrity of the lexical and sublexical systems. Diagnostic batteries must be reliable and valid. Based on the European Federation of Psychologists’ Associations Standing Committee on Tests and Testing (Mun˜iz, 2009), the internal consistency is adequate if higher than .7, good if between .8 and .9, or excellent if higher than .9. Several methods are available to investigate the validity of a diagnostic test. The differentiation between groups with and without the disorder to be detected is one of the most useful (diagnosis must be realized in agreement of a manual of diagnostic criteria and independently from the performance of the two samples). Moreover, also the incremental validity of a test should be investigated; it represents the amount of variance in correctly detecting the disorder that the test could explain over another similar test (e.g., Hunsley & Mayer, 2003; for an example, see Balboni, Naglieri, & Cubelli, 2010). This information is helpful when determining the relative value of each test available for the measurement of a specific disorder. The typical distribution of raw scores on diagnostic tests approximates to the normal distribution curve; also the normative scores like, e.g., the T score and the deviation IQ score have a normal distribution. With this kind of scores, a performance on a test may be labeled as indicating a disorder when the normative score of an individual is below the mean of the general population, i.e., proximately one or two SD below. Moreover, to avoid the effects of the error of measurement, the single normative score must be converted in an interval obtained by subtracting and adding to the score the standard error of measurement of the test multiplied for 1.96 (5% of errors) or 2.58 (1%; e.g., Crocker & Algina, 1986). Actually, to make a diagnosis, to take into account the score on a single test is not enough; the performances on two different tests (or groups of tests), selected in agreement with the theoretical model of the investigated cognitive process, must be compared (e.g., Coltheart, 2001; Sartori, 1988). Typically, a dissociation is defined as occurring when a person shows a ‘‘deficit’’ on test A but is ‘‘normal’’ or ‘‘within normal limits’’ on test B. To reveal whether a dissociation occurs, a fully operational definition of ‘‘deficit’’ and ‘‘normal’’ is necessary, and methods to make comparisons
Psychological Tests for Functional Diagnosis
87
based on inferential statistics and not just descriptive statistics must be used (Crawford, Garthwaite, & Gray, 2003). If the normative group of a test is representative of the general population, the most common method of inferring a deficit is to use the normal deviate z. For this purpose, the raw score on a test of the person being evaluated is converted to the normative score (e.g., T score) based on the normative tables of the test. Then, the corresponding z-score is computed using the following formula: z¼
NS NS Mean NSSD
where NS is the normative score obtained by the individual, e.g., 45 T; NS Mean and NS SD are the mean and the SD of the normative score used, respectively, e.g., 50 T and 10 T. Then, z refers to the table of areas under the normal curve. If the probability of a z-score at least as large as the one obtained is less than .05 (or some other selected alpha value), the performance is considered to be significantly lower than the normative sample and the person labeled with a deficit in that test. Otherwise, if the probability of a z-score at least as large as the one obtained is more than .05, it is inferred the absence of the presumed deficit. If a representative normative sample is not available and therefore the normative score are computed with a sample of modest size, the method proposed by Crawford and Howell (1998) based on a modified t-test (Sokal & Rohlf, 1995) can be used. Given a defective performance on test A and a good performance on test B, to ascertain whether the difference between the z-scores obtained in the two tests is large enough for inferring a dissociation, the difference in z-scores should be compared with the distribution of differences in z-scores obtained by the normative sample. For this purpose, test A and test B must be standardized with the same normative group and the correlation between the scores in the two tests has to be available. If the normative group is representative, the standard deviation of the difference AB (SDAB; Payne & Jones, 1957) must be computed using the following formula: SDAB ¼
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 2 rAB
where rAB is the correlation between the two tests in the normative sample. Then, a z-score for the difference (zD) may be calculated dividing the difference in z-scores by SDAB . Using the table of areas under the normal curve, it is possible to test if zD exceeds the required critical value (i.e., 1.64
88
GIULIA BALBONI AND ROBERTO CUBELLI
for one-tailed significance or 1.96 for a two-tailed test at the .05 level). If the normative group is not representative, to calculate zD Crawford, Howell, and Garthwaite (1998) proposed to divide the difference in z-scores by the standard error of the difference in the normative group and not by the standard deviation of the difference. In this way, a quantity is yielding that is distributed as t with n1 degrees of freedom (see Crawford et al., 1998, for more details). Also methods for obtaining confidence limits on the estimates of test score extreme as well as of test score differences had been developed (Crawford & Garthwaite, 2002). All these methods can be used to compare the scores obtained on different tasks with different scales of the test battery or to compare scores obtained in different tests (provided that the correlation between the scores in the two tests obtained by the control group is available). When assessing learning disabilities, critical dissociation has to be considered. For instance, the label ‘‘Surface Dyslexia’’ requires a normal ability to read aloud regular words and novel words coupled to an impaired ability to read aloud irregular words. The diagnosis should be confirmed by the analysis of errors. In languages with opaque orthography, e.g., English or French, the phonological information corresponding to the graphemes can be ambiguous: e.g., in English the letter sequence oEAW is pronounced differently in the words head, beat, heart, and great; similarly, in French the letter sequence oENW is read differently in the words chre´tiente´ [Christianity] and impatiente [impatient]. For a number of words, the correct pronunciation cannot be derived using the grapheme-to-phoneme conversion rules; therefore, to be read flawlessly ‘‘exception words’’ have to be learned by heart and stored in the lexicon with their particular orthography to phonology mappings. Typically, individual with Surface Dyslexia produce regularization errors, with irregular words pronounced with the typical phoneme to grapheme correspondences: e.g., disease - decease (see Folegatti, Pia, Berti, & Cubelli, submitted). Overall, the different error types depend on the specific properties of the languages: for instance, in Hebrew most errors are incorrect vowel pronunciations in reading unvoweled homographs (Birnboim & Share, 1995), whereas in Italian most errors are word stress mis-assignment (e.g., Job, Sartori, Masterton, & Coltheart, 1983; Toraldo, Cattani, Zonca, Saletta, & Luzzatti, 2006).
Ad Hoc Investigation The last step of the assessment is aimed to identify in each individual the functional locus of impairment underlying the assigned clinical label. For
Psychological Tests for Functional Diagnosis
89
this purpose, additional tests are needed. In the case of surface dyslexia, for instance, the damage can be located at several different loci in the lexical route. As suggested by Coltheart and Funnell (1987), it can be due to (1) an impairment at the level of the orthographic input lexicon; (2) an impaired connection between the orthographic input lexicon and both the phonological output lexicon and the semantic system; and (3) an impairment at the level of the phonological output lexicon. Different patterns of performances can be predicted. For example, the performance on the Orthographic Decision test (Pizzoli, Lami, Pignatti, D’Alessandro, & Cubelli, 2008), whereby patients are asked to discriminate between a word and a pseudohomophone distracter (e.g., Brain and Brane), is expected to be defective (at chance) if the deficit occurs at the level of the orthographic input lexicon, but within the normal range if the deficit is due to impaired postorthographic mechanisms. At this stage of assessment the tests are used to go beyond the clinical label. The investigation should be adapted for the single patient and in some cases requires the adoption of ad hoc prepared tests. The choice of the testing material should be theoretically founded, i.e., based on explicit models, and not behaviorally oriented, i.e., based on visible signs. Using this approach, it has been possible to identify different subtypes of developmental surface dyslexia (Friedmann & Lukov, 2008) and developmental surface dysgraphia (Gvion, Friedmann, Rubin-Zaxenberg, & Mor, 2010), thus getting a functional diagnosis for each individual patient.
CONCLUSION A useful diagnosis consists in the precise description of the defective behavior and error types and in the functional interpretation of the underlying impaired cognitive mechanisms. Comparing the overall pattern of performances (described in terms of dissociation between tasks and qualitative analysis of errors) with a theoretical model describing the functional architecture of the assessed cognitive process allows deriving the identification of the specific loci of impairment in each single patient. A descriptive label of the disorder only confirms what is already known without providing any explanation regarding the damaged functional components of the cognitive system. To obtain a functional diagnosis, four separate steps with different diagnostic instruments are required: (1) clinical interview; (2) screening tests; (3) standardized battery; (4) ad hoc
90
GIULIA BALBONI AND ROBERTO CUBELLI
investigation. Only in this way it would be possible to gain all information needed to plan tailored educational and rehabilitative interventions.
ACKNOWLEDGMENT The authors are grateful to Margherita Lang for her comments and suggestions.
REFERENCES Allen, M. J., & Yen, W. M. (2002). Introduction to measurement theory. Long Grove, IL: Waveland Press, Inc. Balboni, G., Naglieri, J. A., & Cubelli, R. (2010). Concurrent and predictive validity of the Raven Progressive Matrices and the Naglieri Nonverbal Ability Test. Journal of Psychoeducational Assessment, 28, 222–235. Bigozzi, L. (2000). Apprendimento e riabilitazione a scuola. Aspetti psicologici. Roma, Italy: Carocci. Birnboim, S. L., & Share, D. L. (1995). Surface dyslexia in Hebrew: A case study. Cognitive Neuropsychology, 12, 825–846. Coltheart, M. (2001). Assumptions and methods in cognitive neuropsychology. In: B. Rapp (Ed.), The handbook of cognitive neuropsychology (pp. 3–21). Philadelphia: Psychology Press. Coltheart, M., & Funnell, E. (1987). Reading and writing: One lexicon or two?. In: A. Allport, D. Mackay, W. Prinz & E. Sheerer (Eds), Language perception and production (pp. 313–339). London: Academic Press. Coltheart, M., & Rastle, K. (1994). Serial processing in reading aloud: Evidence for dual-route models of reading. Journal of Experimental Psychology: Human Perception and Performance, 20, 1197–1211. Coltheart, M., Rastle, K., Perry, C., Langdon, R., & Ziegler, J. (2001). DRC: A dual route cascaded model of visual word recognition and reading aloud. Psychological Review, 108, 204–256. Cornoldi, C., Baldi, A. P., & Friso, G. (2010). Prove MT Avanzate di lettura e matematica-2 per il biennio della scuola superiore di II grado. Firenze, Italy: Giunti OS. Cornoldi, C., & Colpo, G. (1995). Nuove prove di lettura MT per la scuola media inferiore. Firenze: Giunti OS. Cornoldi, C., & Colpo, G. (1998). Prove di lettura MT per la scuola elementare-2. Firenze: Giunti OS. Crawford, J. R., & Garthwaite, P. H. (2002). Investigation of the single case in neuropsychology: Confidence limits on the abnormality of test scores and test score differences. Neuropsychologia, 40, 1196–1208. Crawford, J. R., Garthwaite, P. H., & Gray, C. D. (2003). Wanted: Full operational definitions of dissociations in single-case studies. Cortex, 39, 357–370.
Psychological Tests for Functional Diagnosis
91
Crawford, J. R., Garthwaite, P. H., & Slick, D. J. (2009). On percentile norms in neuropsychology: Proposed reporting standards and methods for quantifying the uncertainty over the percentile ranks of test scores. The Clinical Neuropsychologist, 23, 1173–1195. Crawford, J. R., & Howell, D. C. (1998). Comparing an individual’s test score against norms derived from small samples. The Clinical Neuropsychologist, 12, 482–486. Crawford, J. R., Howell, D. C., & Garthwaite, P. H. (1998). Payne and Jones revisited: Estimating the abnormality of test score differences using a modified paired samples t-test. Journal of Clinical and Experimental Neuropsychology, 20, 898–905. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Belmont, CA: Wadsworth, Thomson Learning. Cubelli, R., & Della Sala, S. (in press). The purposes of neuropsychological assessment and how to achieve them. Advances in Clinical Neuroscience and Rehabilitation. Del Corno, F., & Lang, M. (1996). La relazione con il paziente. Milano: Libreria Cortina. Ellis, A., & Young, A. (1988). Human cognitive neuropsychology. Hillsdale, NJ: Lawrence Erlbaum. Folegatti, A., Pia, L., Berti. A., & Cubelli, R. (submitted). Stress assignment errors in surface dyslexia: Evidence from two Italian patients with a selective deficit of the orthographic input lexicon. Friedmann, N., & Lukov, L. (2008). Developmental surface dyslexias. Cortex, 44, 1146–1160. Gibertini, M., Brandenburg, N., & Retzlaff, P. (1986). The operating characteristics of the Million Clinical Multiaxial Inventory. Journal of Personality Assessment, 50, 554–567. Glaros, A. G., & Kline, R. B. (1988). Understanding the accuracy of tests with cutting scores: The sensitivity, specificity, and predictive value model. Journal of Clinical Psychology, 44, 1013–1023. Greenspan, S. I. (2003). The clinical interview of the child (3rd ed.). Arlington, VA: American Psychiatric Publishing. Gvion, A., Friedmann, N., Rubin-Zaxenberg, I., & Mor, R. (2010). Subtypes of developmental surface dysgraphia. Procedia Social and Behavioral Sciences, 6, 145–147. Hasselblad, V., & Hedges, L. V. (1995). Meta-analysis of screening and diagnostic tests. Quantitative Methods in Psychology, 111, 167–178. Hsu, L. M. (2002). Diagnostic validity statistics and the MCMI-III. Psychological Assessment, 14, 410–422. Hunsley, J., & Mayer, G. J. (2003). The incremental validity of psychological testing and assessment: Conceptual, methodological, and statistical issue. Psychological Assessment, 15, 446–455. Job, R., Sartori, G., Masterton, J., & Coltheart, M. (1983). Developmental surface dyslexia in Italian. In: R. N. Malatesha & H. Whitaker (Eds), Dyslexia: A global issue (pp. 133–141). The Hague: Martinus Njihoff. Kraemer, H. C., Kazdin, A. E., Offord, D. R., Kessler, R. C., Jensen, P. S., & Kupfer, D. J. (1999). Measuring the potency of risk factors for clinical or policy significance. Psychological Methods, 4, 257–271. Lang, M. (2003). Il colloquio clinico. Milano, Italy: Libreria Cortina. Lis, A., Venuti, P., & De Zordo, M. R. (1995). Il colloquio come strumento psicologico (2nd ed.). Firenze, Italy: Giunti. McFall, R. M., & Treat, T. A. (1999). Quantifying the information value of clinical assessments with signal detection theory. Annual Review of Psychology, 50, 215–241.
92
GIULIA BALBONI AND ROBERTO CUBELLI
Morrison, J., & Anders, T. F. (2001). Interviewing children and adolescents. New York: Guilford Press. Mun˜iz, J. (2009, July). The role of EFPA in setting standards for tests and test use. Paper presented at the 11th European Congress of Psychology, Oslo, Norway. Payne, R. W., & Jones, G. (1957). Statistics for the investigation of individual cases. Journal of Clinical Psychology, 13, 115–121. Pizzoli, C., Lami, L., Pignatti, B., D’Alessandro, D., & Cubelli, R. (2008). Il compito di decisione ortografica per la valutazione dell’acquisizione lessicale in eta` evolutiva. Psicologia Clinica dello Sviluppo, 12, 25–40. Sartori, G. (1988). From models to neuropsychological data and vice versa. In: G. Denes, P. Bisiacchi, C. Semenza & E. Andreewsky (Eds), Perspectives in cognitive neuropsychology (pp. 59–73). Hove, UK: Erlbaum. Sartori, G., Job, R., & Tressoldi, P. E. (2007). DDE-2 Batteria per la valutazione della dislessia e della disortografia evolutiva-2. Firenze, Italy: Giunti OS. Scott, M. S., Fletcher, K. L., & Martell, B. M. (2000). Selecting components for a screening test to identify three-year-olds at risk for mild learning problems. Education and Training in Mental Retardation and Developmental Disabilities, 35, 208–221. Sokal, R. R., & Rohlf, J. F. (1995). Biometry. San Francisco: W.H. Freeman. Streiner, D. L. (2003). Diagnostics tests: Using and misusing diagnostic and screening tests. Journal of Personality Assessment, 81, 209–219. Toraldo, A., Cattani, B., Zonca, G., Saletta, P., & Luzzatti, C. (2006). Reading disorders in a language with shallow orthography: A multiple single case study in Italian. Aphasiology, 20(9–11), 823–850.
A COMPARISON OF OBSERVATIONAL TECHNIQUES FOR ASSESSING STUDENTS’ SOCIAL BEHAVIOR Jugnu Agrawal, Dannette Allen-Bronaugh and Margo A. Mastropieri ABSTRACT This study compares two methods of data collection for students’ social behaviors. One method employed time sampling procedures, while the other method used handheld computerized devices and the Multi-Option Observation System for Experimental Studies (MOOSES) system. Both coding systems were used to assess social behaviors of students with emotional disabilities during writing instruction. The middle-school-aged students, all classified as having EBD, were enrolled in classes to improve their written expression. Students were assessed for on-task, off-task, and multitask behaviors. Results revealed some surprising differences. When students were relatively consistent with attendance and on-task behaviors, the methods yielded comparable results; however, when students were more disruptive and demonstrated more inconsistent behaviors, different patterns emerged. Implications and recommendations for future research and practice are discussed. Assessment and Intervention Advances in Learning and Behavioral Disabilities, Volume 24, 93–110 Copyright r 2011 by Emerald Group Publishing Limited All rights of reproduction in any form reserved ISSN: 0735-004X/doi:10.1108/S0735-004X(2011)0000024007
93
94
JUGNU AGRAWAL ET AL.
Assessing students’ behaviors is a critical component of evaluating both instruction and behavioral intervention for students. A variety of methods and procedures are available to researchers. These methods vary not only in the way the data are collected, but also how data may be analyzed. Time sampling and Multi-Option Observation System for Experimental Studies (MOOSES) are two assessment methods that have been extensively used to collect observational data in many studies. Time sampling is a procedure during which a time segment is divided into intervals and behavior is recorded at each prespecified time interval (Brulle & Repp, 1984). Observers record whether an occurrence of the behavior is present at the end of a predetermined interval, such as 30 seconds. This method is useful when an observer cannot watch the participant continuously in a particular setting or during a particular task. This method permits observations of a group of participants simultaneously, for example, a teacher can use this method to collect data on behaviors, such as student on-task engagement, for a larger group of individual students in her classroom. Moreover, this procedure is relatively cost effective. A stopwatch, a beep tape (a device with recorded signals such as clicks or beeps as prompts to observe and record behaviors), headphones, a clipboard, paper, and pencil are the only tools required. Time sampling has been used extensively to observe on-task behaviors of students with emotional and behavioral disorders (EBD) (e.g., Blood, 2010; Mastropieri et al., 2009). Time sampling can be completed in a relatively cost-effective way since the required materials are inexpensive. With advancing technologies, additional assessment and observational approaches are now available. A variety of systems use handheld computerized devices, which have the convenience of collecting and exporting data into files that are ready for data analysis. MOOSES is one program that allows for real-time collection and analysis of data obtained from observational processes and participants (Tapp & Wehby, 1993). MOOSES provides opportunities for more flexible data collection using researcherdesigned codes, including mutually exclusive events, interval coding, coding actual minutes and seconds of behavioral occurrences, and toggling between behaviors. Data collapsing and analytical procedures are also embedded within the program. MOOSES has been employed in multiple studies to code a variety of observational data for students of all ages and for assessing teacher behaviors (Reinke, Lewis-Palmer, & Merrell, 2008; WebsterStratton, Reid, & Stoolmiller, 2008). Several studies have used MOOSES to study student engagement, on-task behavior, teacher and student
Comparison of Observational Techniques
95
interactions, and their responses during instruction (e.g., Kamps & Greenwood, 2005; Tankersley, Kamps, Mancina, & Weidinger, 1996). Miller, Lane, and Wehby (2005) studied the effects of social skills training on the inappropriate behaviors of elementary school students with highincidence disabilities and used MOOSES to record the occurrence of inappropriate classroom behaviors and attending during baseline, intervention, and postintervention phases. MOOSES provides researchers with additional observational and recording functions that are not possible with time sampling procedures. However, MOOSES is more costly, requires the purchase of handheld computerized devices, and the software. This study was designed to determine whether differences existed between the two methods of assessment when used to measure students’ social behaviors during academic instruction.
METHOD Students Participants were seven middle school students, six males and one female, who were all classified as having emotional or behavioral disabilities. All students were participants in projects designed to improve their written expression as part of a larger project, but none of the presently analyzed data were included in those reports (Mastropieri et al., 2009, 2010, 2011). Four students attended a specialized middle school for students with EBD and three attended a traditional middle school. The racial and ethnic composition was diverse: three were African American, one Hispanic, two Caucasian, and one Asian. The average age was 13.3 years (range from 12.3 to 14.5 years). Table 1 presents the student characteristics. Students’ emotional and behavioral issues included social problems, academic engagement, self-regulation, following directions, and attendance. Students exhibited challenging behaviors, which is why they were selected for the current study. Setting The settings for this investigation were two middle schools in a large public school district on the east coast. One school with a population of approximately 100 was for students with EBD. There were more males
Male
Male
Male
Male
Male
Male
Female
Wilton
Liam
Edgar
Evan
George
Malcolm
Sally
African American
African American African American
Hispanic
Caucasian
Asian
Caucasian
Ethnicity
13.10
13.20
13.60
14.50
13.00
12.30
13.80
Age (Years, Months)
8
8
8
8
7
7
8
Grade Level
ED, OHI
ED, OHI
ED, LD, ESOL ED, LD
OHI, AUT, ED Serv AUT, ED Serv OHI, ED Serve
Special Education Categories
Work independently; decrease attention-seeking behaviors, following directions Self-awareness, coping skills
Coping strategies, attendance
Stress reduction strategies
Social problem solving, social awareness Verbal interpersonal interactions, on-task behavior Self-regulation
Behavioral Goals
Student Characteristics.
WISC IV; VCI 87; PRI 82; WMI 97; PSI 80; Full Scale 82; WJ BWL 80
WISC IV; VCI 100; PRI 106; WMI 80; PSI 75; Full Scale GAI 103; WJ BWL 80; TOWL III 96 WISC IV; VCI 83; PRI 94; WMI 62; PSI 73; Full Scale 75; WJ BWL 78 RIAS; VI 65; NI 98; CI 81; CMI 54: WJ BWL 86 WASI; VA 98; PA 104; Full Scale 101; WJ BWL 80
SBIS; VR 109; QR 108; AVR 92; STM 93; Composite 101; VABS 93/80 WJ BWL 92; UNIT 112
Test Scores
ED, emotional disabilities; ESOL, English for speakers of other languages; LD, learning disabilities; OHI, other health impairments; AUT, autism spectrum disorder; ED Serv, receiving emotional disabilities services; WISC IV, Wechsler Intelligence Scale for Children–fourth edition (Wechsler, 2003); VCI, Verbal Comprehension Index; PRI, Perceptual Reasoning Index; WMI, Working Memory Index; PSI, Processing Speed Index; Full Scale, Full Scale IQ; Full Scale GAI, Full Scale IQ for General Ability Index; WJ BWL, Woodcock Johnson Broad Written Language; WRM, Woodcock Reading Mastery Tests; TOWL-3, Test of Oral Written Language (Hammill & Larsen, 1996); UNIT, Universal Nonverbal Intelligence Test (Bracken & McCallum, 1998); SBIS, Stanford–Binet Intelligence Scale – fourth edition (Thorndike, Hagen, & Sattler, 1986); VR, verbal reasoning; QR, quantitative reasoning; STM, short-term memory; Composite, composite score; VABS, Vineland Adaptive Behavior Scale (Sparrow, Balla, Cicchetti, & Doll, 1984); RIAS, Reynolds Intellectual Assessment Scales Full Scale (Reynolds & Kamphaus, 2003); VI, Verbal Intelligence Index; NI, Nonverbal Intelligence Index; CI, Composite Intelligence Index; CMI, Composite Memory Index; WASI, Wechsler Abbreviated Scale of Intelligence (The Psychological Corporation, 1999); VA, verbal reasoning; PA, performance.
Gender
Name
Table 1.
96 JUGNU AGRAWAL ET AL.
Comparison of Observational Techniques
97
(82%) than females (17%) enrolled, and 53% received free and reduced lunch. The sample was ethnically and racially diverse: 37% were Caucasian, 25% were African American, 23% were Hispanic, and 6% were Asian. The second school was a traditional middle school with a population of approximately 1,000 seventh and eighth grade students of whom 52% were males and 48% were females. Fifteen percent of the students received special education services and 11% of the students received free and reduced lunch. The racial and ethnic composition included 62% Caucasian, 19% Asian, 7% Hispanic, and 4% African Americans.
Project Staff The research team included two advanced doctoral students and a faculty member who had been participating in observing and recording behavior on similar projects. All had received extensive training in observing and recording social behaviors using videotapes, beep tapes, and MOOSES. Additionally, all had previous experience assessing, observing, and recording student behaviors.
MATERIALS Time Sampling A data-recording sheet was created to collect observational data using the time sampling method (see Fig. 1) with Microsoft Excel. The landscape orientation sheet was divided into 30-second segments to equal 30 scoring units for a 15-minute recording period. Qualitative notes about the classrooms, lessons, students, and behaviors could be written on a designated space on the sheet. Numeric codes were devised to record specific student behaviors. For example, one was for on-task, two for offtask, three for out of classroom, four for when students return to classroom. Table 2 provides all behavioral codes for both time sampling and MOOSES. A digitally recorded audio file containing clicks every 30 seconds for 15 minutes was created and included on a handheld computerized device; however, this file could also have been installed on an ipod or a comparable more inexpensive device. Earbud headphones were used with the handheld device so that only observers could hear the clicks.
Fig. 1.
Coding Sheet Used for Time Sampling.
98 JUGNU AGRAWAL ET AL.
99
Comparison of Observational Techniques
Table 2.
Codes for Student Behavior.
Time Sampling
MOOSES
1 ¼ On-task behavior 2 ¼ Off-task behavior 3 ¼ Out of classroom 4 ¼ Student returns to classroom 5 ¼ Multitask
On-task (ot) behavior Off-task (ft) behavior Multitasking (mt) behaviors Amount of time in class (ic) Amount of time out of class (oc)
MOOSES An HP handheld computerized device was installed with the Minimooses program. Minimoose is the program designed to accompany the MOOSES program that is installed on PocketPC handheld devices to collect data. A PC with MOOSES software installed was used to record and analyze data. For the purpose of this study, the frequency and duration and sequential analysis tools of the MOOSES software were used. The Mininmooses and MOOSES were set up with the codes for the behavioral categories (see Table 2 for specific codes). Minimooses was set up on mixed mode for a total of 300-second (5 minutes) duration for observation, with 25 seconds for observation and 5 seconds for recording. It was decided to use 5-minute segments of coding for MOOSES and complete three MOOSES 5-minute segments during the identical 15-minute time sampling interval to have a direct comparison across observational techniques. Fiveminute samples per student have been used to provide samples of social behavior repeatedly in the literature. Frequently, 5-minute samples are added together for individuals across sessions. In addition, when observing larger numbers of students in a session, observers collect data on one student for 5 minutes, move to the next student for 5 minutes, and to subsequent students using MOOSES for 5 minutes each time. This procedure permits observers to collect continuous data on individual students, but also multiple students within a setting during a single class period. For example, Tankersley et al. (1996) reported using the 5-minute observational data collection sample successfully. Both Conditions Videotaped samples of student behaviors during instruction were selected for use in the current study. All selected segments contained the same
100
JUGNU AGRAWAL ET AL.
writing instructional sequences, and were edited to obtain video clips of the 7 targeted students over 11 instructional sessions. The videotapes were obtained as part of a larger project designed to teach written expression and had been used to assess fidelity of treatment. However, these tapes were never used to code on-task behaviors. Target students and segments were selected if segments included (a) unobstructed views of the target students for 15 minutes of a lesson; (b) sufficient sound clarity to hear teacher and student comments throughout the segment; and (c) sufficient sequential instructional days with specific students present. Microsoft Excel and Statistical Package for the Social Sciences (SPSS) were used for analyzing and coding the data obtained from both data collection procedures. All student social behaviors were operationally defined as described next. On-task Behaviors During whole group instruction, looking at the teacher, staying in the chair, listening to the teacher, and answering teacher’s questions were defined on-task behaviors. During individual work time, working actively on the writing process, including using the planning and organizing sheets, writing essays, and general compliant behavior such as keeping hands and feet to self and talking only when appropriate, were noted as on-task behaviors. Off-task Behaviors During whole group instruction and individual work time, putting heads down on desks, talking to peers, doodling, fidgeting with inappropriate materials, getting up and walking around in the classroom, not responding to teacher’s questions, and general noncompliant behaviors such as refusing to participate or speaking out inappropriately were regarded as off-task behaviors. Multitask Behaviors The code of multitask was included because it was noted that many students appeared to exhibit both on- and off-tasks behaviors simultaneously. For example, it was frequently noted that students would be participating actively during class discussions, but also be involved with fidgeting behaviors with inappropriate materials. It was decided that observers could reliably code multitask when the dominant behavior was actually on-task, but simultaneously the student was involved in some type of off-task behavior.
Comparison of Observational Techniques
101
Procedures Researcher Training All researchers had prior experience in coding using the time sampling method. The researchers were trained by a faculty member to code and analyze the data using Minimooses and MOOSES. Multiple videotaped practice sessions were used to train the researchers in coding using both methods of data collection. Ten videos were used for training reliability. Once 100% reliability and comfort level using the handheld devices with Minimooses codes was obtained, coding began. Coding Procedures Once segments of student behavior over time were identified, 15-minute segments were coded using time sampling procedures. The same 15-minute segments were divided into three 5-minute segments and coded using the MOOSES procedure. This procedure was repeated across instructional days for all seven students to obtain a sufficient sample of observational data using each method. Reliability of Observation Interobserver agreement (IOA) was calculated for 30% of the observations for time sampling and MOOSES. The overall average IOA for time sampling was 92% (range 90.0–96.6%). The overall average IOA for MOOSES was 97% (range 89.6–100%). MOOSES was used to calculate IOA for frequency and duration measures. Percentages of agreement scores were calculated using the following formula: agreements/(agreements þ disagreements) 100 (MacLean, Tapp, & Johnson, 1985).
RESULTS Observational data from time sampling for each student was entered into Microsoft Excel. Those individual data points for on-task behavior were summed and added with the total multitask behaviors to obtain total ontask scores for each student for each video segment. It was decided to combine multitask with on-task behaviors because the findings separately paralleled one another. Scores were averaged and converted into mean percent of time on-task for each segment and for overall mean percentages. The data collected using Minimooses on the handheld devices was downloaded to a computer and analyzed using MOOSES software.
102
JUGNU AGRAWAL ET AL.
MOOSES calculated the percentage of on-task, off-task, and multitask behaviors. Mean percent of time on-task for each 5-minute segment was added to the mean percent of multitask for each 5-minute segment to yield a total time on-task for each 5-minute segment per student per observation. Descriptive data for each of the four segments (one 15-minute time sampling segment and three 5-minute MOOSES data segments) are displayed in Table 3 by student across all observation periods. Considerable variability was obtained in total on-task behaviors ranging from 10% to 100%. It was observed that overall the total on-task behavior by student appeared somewhat less than desirable for most students with the exception of Wilton. For example, for Wilton, time sampling obtained an overall 88% time on-task, while MOOSES-coded segments revealed 90%, 91%, and 92%, respectively, for segments 1, 2, and 3. In this case it appears that either observational technique provided percentages of time on-task that could be considered highly similar. This is also seen in Fig. 2, which displays Wilton’s on-task behavior by coding technique across instructional days. As can be seen, when examining daily data, some segments of MOOSES coding revealed more variability with substantially less on-task behavior. But overall Wilton’s on-task behavior was quite similar across instructional days and he had fairly consistent on-task behavior, indicating that either coding method revealed comparable results. Conversely, Liam’s results revealed more variability in on-task behavior across methods of observation. He obtained a mean of 81% on-task behavior with time sampling, but his MOOSES mean on-task scores revealed more variability with ranges from 75% (segment 1), to 84% (segment 2), to 92% (segment 3). Liam’s behavior ranged from 70% to 100% for on-task behavior with time sampling; however, his range was 29–100% for on-task using MOOSES. A different picture of Liam’s on-task behavior would be obtained depending on the observational technique selected and the amount of time observed. For example, if Liam were only observed for the first 5-minute segment, he was on-task only 75% of the time. In reviewing Liam’s daily mean data on Fig. 3, a different picture of his on-task behavior emerges because his behavior ranges from a low of 30% to 100% on-task during the first 5-minutes coding segment. However, when only examining the third 5-minute segment of his behavior, Liam was on-task 92% of the time. Liam’s on-task behavior ranges from a student who is mostly on-task to one who has challenges staying on-task when using time sampling versus the MOOSES. But perhaps even more interesting is how the 5-minute segments selected to code using MOOSES resulted in extreme variability for Liam’s on-task behaviors.
First 5 minutes MOOSES Second 5 minutes MOOSES Third 5 minutes MOOSES
Time Sampling
88 (0.11) 67–100 90 (0.11) 69–100 91 (0.10) 70–100 82 (0.16) 59–100
Wilton Mean (SD) Range
81(0.16) 70–100 75 (0.23) 29–100 84 (0.16) 60–100 92 (0.09) 60–100
85 (0.17) 52–100 75 (0.19) 50–100 74 (0.20) 50–100 90 (0.08) 40–100
Edgar Mean (SD) Range 67 (0.20) 37–93 68 (0.26) 39–100 78 (0.20) 50–100 73 (0.19) 28–100
Evan Mean (SD) Range 50 (0.27) 10–90 51 (0.26) 9–100 72 (0.15) 50–90 60 (0.30) 29–100
George Mean (SD) Range
Mean Percent Data for On-task Behavior.
Liam Mean (SD) Range
Table 3.
63(0.17) 27–87 65 (0.33) 30–100 81 (0.13) 49–100 58 (0.16) 23–80
Malcolm Mean (SD) Range
63 (0.17) 17–80 69 (0.32) 9–100 65 (0.23) 20–90 64 (0.25) 49–100
Sally Mean (SD) Range
Comparison of Observational Techniques 103
104
JUGNU AGRAWAL ET AL. 100
Mean Percentages
90
80 Time Sampling 1st 5 min MOOSES 2nd 5 min MOOSES 3rd 5min MOOSES
70
60
50
40
Fig. 2.
1
2
3
4
5
6 7 Days
8
9
10
11
Wilton’s Mean On-task Behavior by Day by Time Sampling versus 5-Minute MOOSES Samples.
100 90
Mean Percentages
80 70 Time Sampling 1st 5min MOOSES 2nd 5min MOOSES 3rd 5min MOOSES
60 50 40 30 20
Fig. 3.
1
2
3
4
5
6 7 Days
8
9
10
11
Liam’s Mean On-task Behavior by Day by Time Sampling versus 5-Minute MOOSES Samples.
105
Comparison of Observational Techniques
Edgar’s patterns of on-task behavior also revealed significant variability. His mean 85% on-task with time sampling is somewhat the average of his three MOOSES segments of 75%, 74%, and 90%. However, this variability indicates that the selection of a method and timing sampling may obfuscate his actual on-task behavior. George and Malcolm’s data revealed generally low levels of on-task behavior, but the large ranges observed in on-task behavior observed indicate enormous variability in their on-task behavior over days. George obtained a mean of 50% on-task with time sampling (range 10–90%). When observing the same segments, but coding using the MOOSES his first 5-minute segment was 51 (range 9–100%) and highly similar to time sampling, but the second segment (M ¼ 72) and the third segment (M ¼ 60) represent more variability. Malcolm’s pattern of data is highly similar to George’s in that he obtained generally low levels of on-task behavior, but also demonstrated significant variability between the time sampling (M ¼ 63) and some MOOSES segments with means of 81–58. Fig. 4 displays graphically the extreme variability obtained in on-task behavior by days across the two methods and by the 5-minute segments. Evan and Sally both presented generally low levels of on-task behavior. But even with those low levels there were differences noted across the
100 90
Mean Percentages
80 70 60 Time Sampling 1st 5min MOOSES 2nd 5min MOOSES 3rd 5min MOOSES
50 40 30 20 10 0
Fig. 4.
1
2
3
4 Days
5
6
7
Malcolm’s Mean On-task Behavior by Day by Time Sampling versus 5-Minute MOOSES Samples.
106
JUGNU AGRAWAL ET AL. 100 90
Mean Percentages
80 70 60 Time Sampling 1st 5 min3 MOOSES 2nd 5 min MOOSES 3rd 5 min MOOSES
50 40 30 20 10 0
1
2
3
4
5
6
7
Days
Fig. 5.
Sally’s Mean On-task Behavior by Day by Time Sampling versus 5-Minute MOOSES Samples.
techniques employed. Sally’s time sampling mean of 63% ranged from 17% to 80%. Given such variability it is interesting to note that her means with each MOOSES segment appear more comparable to time sampling with means of 69, 65, and 64. However, Sally’s MOOSES ranges from 9% to 100%. Fig. 5 graphically shows Sally’s variability across days and highlights how the selection of a particular 5-minute segment might provide different on-task behaviors for her. Across these students with EBD, each observational technique does in fact present reliable and accurate indicators of time on-task. However, there is also considerable variability within student outcomes across measures when comparing the two techniques. Perhaps even more interesting, different 5-minute segments can reveal significant variability in on-task behavior.
DISCUSSION In the present study, different patterns of on-task behavior were obtained when different observational techniques were selected. Moreover, variability
Comparison of Observational Techniques
107
in on-task behavior was also obtained when selecting different 5-minute samples of students’ behavior for some students. It must be recalled that this entire sample was identified as having EBD, so their behaviors might be considered more erratic than most. However, even within this present sample, students who were fairly consistent with their behavior during observation periods looked highly similar no matter which observation technique was employed. This was not only in the case of Wilton, who actually had relatively higher on-task behaviors, but also in the case of Sally, who had consistently lower on-task behavior. However, students who were behaviorally inconsistent more frequently appeared differently with respect to mean on-task behavior with the different observation techniques. These students also exhibited more variability across the three 5-minute segments in on-task when using the MOOSES. In the case of students with inconsistent on-task behaviors, MOOSES, if completed for substantially longer time periods on each student, may provide a more accurate picture of on-task behaviors, but the selection of a 5-minute sample may not yield a clear picture on on-task behavior. Many studies that employed MOOSES coded single students for 5 minutes and then moved to a different student for the next 5 minutes. When using such a procedure, researchers should be cautioned to look more closely at larger samples of individual behavior to improve accuracy of findings. In situation of coding multiple student behaviors in a class, time sampling may provide more information on larger groups of students simultaneously. However, time sampling will not provide the detailed duration and frequency data that MOOSES can provide on a single student. The selection of an observation technique may be based on multiple factors, but may also produce somewhat different findings. Both observational techniques have strengths and limitations. Time sampling is less expensive than MOOSES. In the present study, time sampling record sheets were created using Microsoft Excel, which is a commonly available program. Multiple students can be observed and coded simultaneously using time sampling, but it is more challenging to code multiple behaviors. The present time sampling record sheet contained space to document qualitative field notes on student behavior and setting. However, since time sampling captures a snapshot of the student behavior only at the end of the time segment, it is possible that students’ behaviors may be inaccurately reflected for the time prior to recording. Moreover, the device used to listen to the beeps can fail to start or cease operating during observations. MOOSES has several major advantages over time sampling alone. Multiple variables related to one student can be coded simultaneously.
108
JUGNU AGRAWAL ET AL.
In the current study, additional variables like praise academic, praise social, reprimand academic, and reprimand social were coded but not analyzed. MOOSES software calculates the total duration of the time on-task, offtask, and multitask for each student in percentages as well as total seconds. MOOSES software calculates the percentages and duration of a behavior in a simple way. Conversely, MOOSES software is expensive because it also requires the purchase of handheld computerized devices to house the Minimooses. Usually a single student’s behavior is coded using MOOSES. In this case, the duration codes on the Minimooses were adjusted to reflect the 5-minute observation duration with 25 seconds of observation interval and 5 seconds of recording interval. The codes in MOOSES are case sensitive and codes on the handheld devices must match the codes on the computer. As with any technology, users have to gain familiarity when using technology to make any required changes and adjustments to the MOOSES. In the present study, sometimes the Minimooses failed to start on time for the observation. In those instances the observation had to be conducted again, which was time consuming and cumbersome. In the present study, all that was required was to restart the movie; however, had this been live coding some behavioral coding would have been missed. Finally, the Minimooses screen does not provide the option of noting qualitative details. Both observational techniques have promise for providing an assessment of social behavior, such as time on-task. A careful consideration of the costs and benefits associated with each technique need to be clearly evaluated prior to selecting a technique. More importantly, however, this study provided initial evidence that different results are obtained with the different observational techniques. This was especially apparent when only 5-minute time segments were used to collect samples of student behavior. Careful consideration about the goals and objectives for assessing student behavior should be completed prior to selecting a technique, and those decisions should be weighed carefully with the costs and benefits associated with each potential observational technique. Future research replicating the study with a larger sample may provide additional evidence and guidelines for observers.
ACKNOWLEDGMENT Partial support for this research was provided by Grant No. H325D070008 from the U.S. Department of Education, Office of Special Education Programs, and Grant No. R324A070199-07 from the U.S. Department of
Comparison of Observational Techniques
109
Education, Institute of Educational Sciences, awarded to Pennsylvania State University with a subcontract to George Mason University.
REFERENCES Blood, E. (2010). Effects of student response systems on participation and learning of students with emotional and behavioral disorders. Behavioral Disorders, 35, 214–228. Bracken, B. A., & McCallum, R. S. (1998). Universal nonverbal intelligence test. Itsaka, IL: Riverside. Brulle, A. R., & Repp, A. C. (1984). An investigation of momentary time sampling procedures with time series data. British Journal of Psychology, 75, 481–485. Hammill, D., & Larsen, S. (1996). Test of written language-3. Austin, TX: Pro-ED. Kamps, D. M., & Greenwood, C. R. (2005). Formulating secondary-level reading interventions. Journal of Learning Disabilities, 38, 500–509. doi: 10.1177/00222194050380060501 MacLean, W. E., Jr., Tapp, J. T., Sr., & Johnson, W. L. (1985). Alternate methods and software for calculating interobserver agreement for continuous observation data. Journal of Psychopathology and Behavioral Assessment, 7, 65–73. doi: 10.1007/BF00961847 Mastropieri, M. A., Scruggs, T. E., Cuenca-Sanchez, Y., Irby, N., Mills, S., & Mason, L. H. (2010). Persuading students with emotional disabilities to write: Findings from a design experiment. In: T. E. Scruggs & M. A. Mastropieri (Eds), Literacy and learning: Advances in learning and behavioral disabilities (Vol. 23, pp. 237–268). Bingley, UK: Emerald. doi: 10.1108/S0735-004X(2010)0000023011 Mastropieri, M. A., Scruggs, T. E., Irby, N., Allen-Bronaugh, D., Thompson, C., Guckert, M., et al. (2011). Fluent persuasive writing with counter reasons: Success with students with emotional disabilities. Paper to be presented at the annual meeting of the American Educational Research Association, New Orleans. Mastropieri, M. A., Scruggs, T. E., Mills, S., Irby, N., Cuenca-Sanchez, Y., & Bronaugh, D. A. (2009). Persuading students with emotional disabilities to write fluently. Behavioral Disorders, 35, 19–40. Miller, M., Lane, K. L., & Wehby, J. (2005). Social skills instruction for students with highincidence disabilities: A school-based intervention to address acquisition deficits. Preventing School Failure, 49, 27–40. doi: 10.3200/PSFL.49.2.27-39 Reynolds, C. R., & Kamphaus, R. W. (2003). Reynolds intellectual assessment scales. Lutz, FL: Psychological Assessment Resources Inc. Reinke, W. M., Lewis-Palmer, T., & Merrell, K. (2008). The classroom check-up: A classwide consultation model for increasing praise and decreasing disruptive behavior. School Psychology Review, 37, 315–332. Sparrow, S. S., Balla, D. A., Cicchetti, D. V., & Doll, E. A. (1984). Vineland adaptive behavior scale: Interview edition survey form manual. Circle Pines, MN: American Guidance Service. Tankersley, M., Kamps, D., Mancina, C., & Weidinger, D. (1996). Social interventions for head start children with behavioral risks: Implementation and outcomes. Journal of Emotional and Behavioral Disabilities, 4, 171–181. doi: 10.1177/106342669600400304 Tapp, J., & Wehby, J. (1993). MOOSES: Multiple option observation system for experimental studies. Available at http://www.eric.ed.gov.mutex.gmu.edu/PDFS/ED361372.pdf
110
JUGNU AGRAWAL ET AL.
The Psychological Corporation. (1999). Wechsler abbreviated scale of intelligence. San Antonio, TX: The Psychological Corporation. Thorndike, R. L., Hagen, E. P., & Sattler, J. M. (1986). Stanford–Binet intelligence scale (4th ed.). Rolling Meadows, IL: Riverside Publishing. Webster-Stratton, C., Reid, M. J., & Stoolmiller, M. (2008). Preventing conduct problems and improving school readiness: Evaluation of the incredible years teacher and child training programs in high-risk schools. The Journal of Child Psychology and Psychiatry, 49, 471–488. doi: 10.1111/j.1469-7610.2007.01861.x Wechsler, D. (2003). Wechsler intelligence scale for children (4th ed.). San Antonio, TX: The Psychological Corporation.
CURRICULUM-BASED MEASUREMENT FOR BEGINNING WRITERS: RECENT DEVELOPMENTS AND FUTURE DIRECTIONS Kristen L. McMaster, Kristen D. Ritchey and Erica Lembke ABSTRACT Many students with learning disabilities (LD) experience significant difficulties in developing writing proficiency. Early identification and intervention can prevent long-term writing problems. Early identification and intervention require reliable and valid writing assessments that can be used to identify students at risk and monitor their progress in response to intervention. One promising approach to assessing students’ performance and progress in writing is Curriculum-Based Measurement (CBM). In this chapter, we provide an overview of CBM. Next, we describe a theoretical framework for writing development, and discuss implications of this framework for developing writing assessments. We then describe current efforts to develop a seamless and flexible approach to monitoring student progress in writing in the early elementary grades, and highlight Assessment and Intervention Advances in Learning and Behavioral Disabilities, Volume 24, 111–148 Copyright r 2011 by Emerald Group Publishing Limited All rights of reproduction in any form reserved ISSN: 0735-004X/doi:10.1108/S0735-004X(2011)0000024008
111
112
KRISTEN L. MCMASTER ET AL.
important directions for future research. We end with a discussion of how teachers might eventually use CBM to make data-based decisions to provide effective individualized interventions for students who experience writing difficulties.
Writing is a critical skill for students to develop during their school years. Learning to write plays an important role in overall literacy development (Shanahan, 2006), and is an essential tool for students to communicate what they know (Graham & Perin, 2007) via written assignments, assessments, and essays or reports. Further, because writing is often used to synthesize and evaluate information, it may have an important impact on students’ development of content knowledge, including their ability to integrate knowledge and think critically (Bangert-Drowns, Hurley, & Wilkinson, 2004; Shanahan, 2004). In other words, not only must students learn to write, but writing also plays an important role in learning. Despite its importance, writing has not received a great deal of attention in educational research or practice (Berninger et al., 2006b), especially compared to reading and mathematics (Shanahan, 2006). National statistics appear to reflect this lack of attention: The latest Nation’s Report Card revealed that, in 2002, 72% of fourth graders, 69% of eighth graders, and 77% of twelfth graders were below proficient in writing (Persky, Daane, & Jin, 2003); and in 2007, 65% of 8th graders and 75% of 12th graders were performing below proficient (Salahu-Din, Persky, & Miller, 2008). Although these data suggest a small amount of improvement, proportions of students performing below proficient in writing remain unacceptable. Among those students not reaching proficient levels in writing are many students with learning disabilities (LD; Graham, Harris, & Larsen, 2001). For example, Salahu-Din et al. (2008) found that only 6% of eighth graders with LD reached the proficient level in writing; 46% were below the basic level. The writing of students with LD is often characterized by difficulties with handwriting, spelling, and mechanics, as well as by fewer ideas; poorer organization, planning, and revising; and lower quality than that of their peers without LD (Graham & Harris, 2002; Troia, 2006). These writing problems often go undetected until the late-elementary or middle-school grades, when they become increasingly difficult to remediate (Graham & Harris, 2005). Persistent writing difficulties can have long-term, negative consequences. They can affect students’ overall academic performance, diminish their chances of attending college, and limit employment opportunities (Graham & Perin, 2007). Early identification and intervention
113
CBM for Beginning Writers
with students at risk for serious writing difficulties or disabilities could prevent such consequences for many students (Berninger, Nielsen, Abbott, Wijsman, & Raskind, 2008; Berninger et al., 2006b). Early identification and intervention require reliable and valid assessment tools that can be used to (a) identify students in need of early writing intervention, (b) monitor students’ performance and progress during intervention, and (c) evaluate the effectiveness of interventions and make data-based decisions regarding the appropriateness of instruction. One assessment approach that shows promise as a tool for such data-based instructional decision making is Curriculum-Based Measurement (CBM; Deno, 1985). Over 30 years of research has been conducted on CBM in core academic areas (reading, mathematics, spelling, written expression, and content areas; see Foegen, Jiban, & Deno, 2007; McMaster & Espin, 2007; Wayman, Wallace, Wiley, Ticha, & Espin, 2007 for reviews). In recent years, more work has been conducted to develop and examine CBM for beginning writers, including research by the coauthors of this chapter. The purpose of this chapter is to describe recent advances in CBM for beginning writers, as well as to identify future directions for this work. First, we provide an overview of CBM. Then, we describe recent developments in CBM for beginning writers, starting with the theoretical framework in which these measures are grounded, followed by current efforts to develop tasks that yield reliable, valid, and instructionally useful progress-monitoring data. Throughout the chapter, we indicate areas in need of further research. We end with a case example of how teachers might use CBM to make data-based decisions to provide effective interventions for students who experience writing difficulties.
CBM OVERVIEW CBM was developed by Deno and colleagues (e.g., Deno, 1985; Deno, Mirkin, & Marston, 1982) as part of the Institute for Research on Learning Disabilities (IRLD) at the University of Minnesota to be a set of simple, efficient procedures that would provide general outcome indicators of student performance and progress. CBM was designed to reflect vital signs of academic health; that is, to indicate whether a student was making sufficient progress toward important academic goals, or whether progress was insufficient, indicating an underlying problem requiring further diagnosis and intervention (Deno, 1985). To provide general outcome
114
KRISTEN L. MCMASTER ET AL.
indicators of students’ performance and progress in a given academic area, Deno and colleagues sought measurement approaches that involved direct observation of academic behaviors. To be useful for ongoing instructional decision making, they sought to develop measures drawn directly from the curriculum in which a student’s progress was to be monitored, and that were reliable and valid, simple and efficient, easy to understand, and inexpensive (Deno). For example, in reading, the most common CBM approach involves listening to a student read aloud from grade-level text while the number of words read correctly in 1 min is recorded. Repeated (e.g., weekly) assessments using alternate passages of equivalent difficulty are administered to provide a picture of the student’s progress within the reading curriculum over time. Elementary students’ scores on CBM read-aloud measures have correlated strongly with standardized reading measures (see Wayman et al., 2007), suggesting that the CBM read-aloud approach provides the type of general outcome indicator that Deno and his colleagues were seeking. Further, the measures have been found to distinguish among students of different skill levels, and to be sensitive to growth made in brief time periods (Wayman et al., 2007). Similar approaches are used in mathematics (e.g., students complete basic math facts or math problems; see Foegen et al., 2007), spelling (students spell words that represent the corpus of words in the spelling curriculum; Fuchs, Fuchs, Hamlett, & Allinder, 1991a, 1991b), and written expression (students write in response to a story or picture prompt; see McMaster & Espin, 2007). CBM for writing (CBM-W) research initially involved story or picture prompts (e.g., Deno, Mirkin, & Marston, 1982); students provided a written response to the prompts for 3–5 min, and the prompts were then scored for number of words written (WW), words spelled correctly (WSC), and correct word sequences (CWS, which accounts for both spelling and grammar; Videen, Deno, & Marston, 1982). These measures served as reliable and valid indicators of student writing proficiency for upper-elementary students, and were relatively simple and efficient to administer and score; however, the measures yielded somewhat weak reliability (rs ¼ .20–.47; Deno et al., 1982), and weak to moderate criterion validity (rs ¼ .23–.63; Gansle et al., 2004; Jewell & Malecki, 2005; Parker, Tindal, & Hasbrouck, 1991) for early elementary students. Thus, our collective research teams have begun to explore CBM approaches that might serve as better indicators of young children’s developing writing skills. A search for general outcome indicators of early writing proficiency begs the question: what constitutes early writing? Theoretical models of writing development have provided guidance in
115
CBM for Beginning Writers
determining how we might gauge the vital signs of students’ early writing proficiency.
THEORETICAL FRAMEWORK FOR DEVELOPING CBM FOR BEGINNING WRITERS Skilled writers engage in multiple processes while writing, and for young writers, these processes typically begin to develop in early childhood, and continue to develop during the early elementary grades. Theoretical models of early writing developed by Bereiter and Scardamalia (1987) and by Berninger and colleagues (Berninger, 2000; Berninger, Fuller, & Whitaker, 1996; Berninger & Swanson, 1994) account for children’s growing writing proficiency. Berninger et al. (2002) organized these factors into a ‘‘simple view’’ of writing, in which transcription (handwriting or keyboarding, spelling) provides a foundation for early writing development, allowing the writer to translate ideas into text; and executive functions (conscious attention, planning, reviewing, revising, and self-regulation) serve to manage the writing process, all within the constraints of short-term, working, and long-term memory. The design of early writing assessments requires sensitivity and attention to the developmental challenges faced by young writers. For young writers, transcription and ideation are the most relevant factors influencing the text produced by children. The process our respective research teams have used in the development of CBM tasks and scoring procedures for beginning writers are informed by these aspects of writing development.
Transcription Transcription is the process of getting words onto paper, and handwriting (or keyboarding) and spelling are components of transcription. Young writers tend to focus on producing smaller units, such as forming individual letters and spelling individual words. As students’ transcription skills develop and become automatic, working memory capacity can then be devoted to higher-level processes such as composition and self-regulation (McCutchen, 2000), allowing students to engage more fully in planning and revising (Berninger & Swanson, 1994; Berninger et al., 2002). Students are then able to produce text that is longer and more complex and begin
116
KRISTEN L. MCMASTER ET AL.
transition from writing individual words to clauses, sentences, and longer texts. Transcription skills are important to writing quality. For example, Graham, Berninger, Abbott, Abbott, and Whitaker (1997) found that handwriting fluency and spelling explained a large proportion of the variance in both the length and quality of children’s writing; and Graham, Berninger, Weintraub, and Schafer (1998) found that copying tasks discriminate across a wide age range (grades 1–9) in terms of handwriting speed and legibility. Abbott, Berninger, and Fayol (2010) found that spelling is a strong predictor of paragraph writing in grades 1–7. Further, experimental and quasi-experimental studies have shown improvements in students’ writing quality in response to interventions targeting handwriting and spelling (Berninger et al., 1997; Berninger et al., 1998; Graham, Harris, & Fink, 2000; Graham, Harris, & Fink-Chorzempa, 2002; Jones & Christensen, 1999), suggesting a causal relation between development of transcription skills and overall writing proficiency. As described in more detail below, our respective research teams have developed CBM tasks for beginning writers that are intended to tap transcription skills, with the hypothesis that such measures would serve as general outcome indicators of children’s developing writing proficiency. Such measures have included dictated letter- and sound-writing tasks for very young children (Ritchey, 2004, 2006), dictated spelling tasks (e.g., Ritchey, Coker, & McCraw, 2010; Lembke, Deno, & Hall, 2003) and copying tasks (e.g., Lembke et al., 2003; McMaster, Du, & Petursdottir, 2009) for children as they progress through the early elementary grades.
Ideation Ideation is the process of generating ideas for written content, and draws on oral language sources such as vocabulary and domain knowledge. Juel, Griffith, and Gough (1986) reported that the number of generated ideas was a unique predictor of first and second graders’ writing quality. Coker (2006) found that receptive vocabulary proficiency predicted end-of-the-year writing quality for first-grade students. Written vocabulary also predicted writing quality for second- and fourth-grade students (Olinghouse & Leaird, 2009). Although there is limited research on the relation between ideation and early writing, assessments that do not attempt to capture ideation may miss a critical feature of written development.
CBM for Beginning Writers
117
Thus, our research teams have developed CBM tasks that require students to generate their own ideas. Given that young writers’ executive functions (e.g., conscious attention, planning, reviewing, revising, and self-regulation) are in development and are likely to be dependent on outside guidance and assistance (Berninger & Amtmann, 2003), we have added varying levels of scaffolding to these tasks, by creating more simple tasks (e.g., writing words or sentences rather than entire passages) and prompts (including letters, words, and pictures) intended to help young writers generate words and ideas to include in their writing. These measures have included Letter prompts to elicit word-level writing (McMaster et al., 2009), Picture-Word prompts (McMaster et al.) and Sentence-Writing prompts (Coker & Ritchey, 2010) to elicit sentence-level writing, and Picture prompts presented in a theme (McMaster et al., 2009) or a story sequence (Ritchey & Coker, in press) to elicit discourse-level (story) writing.
RECENT DEVELOPMENTS IN CBM FOR BEGINNING WRITERS Researchers (Fuchs, 2004; Wallace, Espin, McMaster, & Deno, 2007) have recently articulated important directions for programmatic CBM research. First, in a summary of the past, present, and future of CBM research, Fuchs outlined three stages needed to establish the utility of CBM for progress monitoring: Stage 1 involves examining technical features of the static score (i.e., a score obtained at a single time point), Stage 2 involves examining technical features of slope (i.e., rate of growth across time), and Stage 3 involves examining instructional utility. Fuchs indicated that, over the past 30 years, CBM researchers have addressed each of these three stages to varying degrees in reading, writing, spelling, and math, but that a disproportionate amount of this research has focused on Stage 1. Whereas Stage 1 is a critical first step in validating CBM, and is useful for identifying reliable and valid measures for screening purposes, Fuchs encouraged CBM researchers to ‘‘capitalize on CBM’s unique contribution for modeling progress and enhancing instruction’’ (p. 191) by focusing further attention on Stages 2 and 3. Second, in an introduction to a special issue of The Journal of Special Education featuring reviews of CBM research in reading, mathematics, and writing, Wallace et al. (2007) presented their vision of a ‘‘seamless and flexible system of progress monitoring’’ (p. 66) that could be used with
118
KRISTEN L. MCMASTER ET AL.
students across varying ages and skill levels, as well as across settings and curricula. Such a system of progress monitoring would be reliable and valid for indexing student performance, progress, and response to instruction both within and across grades. We use Fuchs’s (2004) three stages to summarize the work conducted thus far in CBM for beginning writers. We use Wallace et al.’s (2007) vision of a seamless and flexible system of progress monitoring to reflect upon what we currently know about CBM for beginning writers and to discuss future directions for a coherent program of research in this area. A summary of CBM tasks and scoring procedures, along with the stages and grade levels at which this research has been conducted, are presented in Table 1. For Stage 1 research, an ‘‘R’’ denotes measures with sufficient reliability (rW.70) and a ‘‘V’’ denotes measures with sufficient criterion-related validity (rW.50). Because there is not a consensus on criteria by which to judge reliability of measures, we based our judgment of sufficient reliability and validity on other standardized writing measures, for which reliability estimates have typically ranged from r ¼ .70 to above .90 and criterion-related validity coefficients have rarely exceeded r ¼ .60 (Taylor, 2003).
Stage 1 Research Much of our collective work thus far has focused on technical features of the static score, an important first step to establishing approaches that show promise as general outcome indicators of early writing proficiency (Fuchs, 2004). Ritchey and Coker have systematically examined measures developed at the subword, word, sentence, and discourse levels, in line with a perspective of how young children’s writing skills develop beginning in kindergarten and proceeding through early elementary grades (through grade 3). Lembke and McMaster and their colleagues have taken a more comparative approach, by administering tasks at word, sentence, and discourse levels and comparing their technical characteristics, primarily for students in grades 1 and 2. Subword-Level Measure Development Ritchey (2004, 2006) designed two CBM tasks to target writing skills at the earliest stages of writing development. The first task, Letter Writing, assesses students’ ability to write individual upper- and lower-case letters from dictation. Sixty kindergarten children were assessed in March
T&I
T&I
Picture-worde,f
T
T
Sentence writingg
Sentence level Sentence dictationc,d Sentence copyingc,d,e,f
T&I
3 min; WW, WSC, CWS; qualitative 3–5 min; CLS, WW, WSC, CWS
3 min; WW, WSC, CWS, C-IWS 3–5 min; CLS, WW, WSC, CWS
T
Word copying c,d,e
Letter prompt
N.T.; CLS 3 min; CLS, C-ILS, WW, WSC 2–3 min; CLS, C-ILS, WW, WSC 3–5 min; CLS, WW, WSC
T T
e
N.T.; CLS
N.T.; CL N.T.; CL
Duration and Scoring
T
T T
Domain
R
R
R&V
R&V R&V
K
R&V
R&V
R&V
R&V
R
R&V
R&V
Grade 1
V
V
V
V
Grade 2 Grade 3
Stage 1: Technical Features of the Static Score
Bi-monthly growthW0
K
Weekly growthW0 Reliable slopes; Weekly growthW0 Bi-monthly growthW0 Reliable slopes; Weekly growthW0
Weekly growthW0 Weekly growthW0
Grade 1
Grade 2
Stage 2: Technical Features of the Slope
Grade 3
Summary of CBM Tasks and Scoring Procedures for Beginning Writers by Stage of Research and Grade Level.
Word level Real word spellingb Nonsense wordb Word dictationc,d
Subword level Letter writinga Sound spellinga
CBM Task
Table 1.
CBM for Beginning Writers 119
T&I
3 min; WW, WSC, CWS 3–5 min; CLS, WW, WSC, CWS 5 min; WW, WSC, CWS, qualitative
3–5 min; WW, WSC, CWS
Duration and Scoring K
(Insufficient R & V)
R&V
R&V
Grade 1
R
R&V
R&V
R&V
Grade 2 Grade 3
Stage 1: Technical Features of the Static Score K
Reliable slopes; weekly growthW0
Grade 1
Bimonthly growthW0 for CWS and qualitative scoring only
Grade 2
Grade 3
Bimonthly growthW0 for all scores
Stage 2: Technical Features of the Slope
Note. T ¼ transcription, I ¼ ideation, CL ¼ correct letters, CLS ¼ correct letter sequences, C-ILS ¼ correct minus incorrect letter sequences, WW ¼ words written, WSC ¼ words spelled correctly, CWS ¼ correct word sequences, C-IWS ¼ correct minus incorrect word sequences, R ¼ sufficient reliability (rW0.70), V ¼ sufficient validity (rW.50). a Ritchey (2004, 2006). b Ritchey (2006), Ritchey et al. (2010). c Lembke et al. (2003). d Hampton et al. (2011). e McMaster et al. (2009). f McMaster et al. (2011). g Coker and Ritchey (2010). h McMaster and Campbell (2008). i Ritchey and Coker (in press).
Picture storyi
T&I
T&I
Photo prompte,h
Picture-theme
T&I
Discourse level Story prompte,f,h
e
Domain
CBM Task
Table 1. (Continued ) 120 KRISTEN L. MCMASTER ET AL.
CBM for Beginning Writers
121
and May. Reliability coefficients were r ¼ .90 for both test–retest and alternate-form reliability (at an eight-week interval), split-half reliability coefficients were r ¼ .90 and .92 (for March and May assessments, respectively), and internal consistency reliability was .94 (coefficient alpha for both March and May assessment points). Scores on Letter Writing were positively related to early reading and writing skills, including performance on norm-referenced and CBM-Reading assessments. Criterion-related validity coefficients with reading ranged from r ¼ .50 to .69. Criterionrelated validity coefficients with a second sample of kindergarten children (N ¼ 86) were investigated using the Test of Early Written Language (TEWL; Hresko, Herron, & Peak, 1996). The validity coefficient was r ¼ .67 with the Basic Writing subtest. The second subword-level task is Sound Spelling. This task requires children to spell individual letters after being presented with a sound (e.g., write the letter that makes the /t/ sound). In this way, basic encoding can be assessed at a stage when students have emerging, but perhaps incomplete, word-spelling skills. The technical features of Sound Spelling were investigated in the same two samples of children as Letter Writing. For Sound Spelling, the reliability coefficients were r ¼ .90 for test–retest and alternate-form reliability (at eight weeks), split-half reliability coefficients were r ¼ .92 and .91 (for March and May assessments, respectively), and internal consistency was .91 and .90 (for March and May assessments, respectively). Criterion-related validity coefficients with reading ranged from r ¼ .58 to .77. The criterion-related validity estimate with the TEWL Basic Writing subtest (Hresko et al., 1996) was r ¼ .67. The letter-writing tasks described above can be used to assess early writing development; however, the seamlessness of these measures within an ideal assessment framework may be limited due to ceiling effects once children learn to write all of their letters. Initial study of these measures did not include a timed component; yet, a timed component may help to eliminate ceiling effects and to be a more sensitive indicator of writing growth over time. Word-Level Measure Development Given the empirical relation between transcription skills and writing composition described above, it seems reasonable to expect that spelling measures might provide an indicator of children’s early writing development, particularly as they reach the ceiling on letter- and sound-writing tasks. Specifically, word-level spelling tasks may be used to assess the extent to which students are beginning to use letter-sound associations, combined
122
KRISTEN L. MCMASTER ET AL.
with phonological segmentation skills, to phonetically recode words. Word-level tasks may be helpful to use when assessing students during the developmental transition from writing individual letters to writing phrases and sentences. Traditional CBM spelling measures have assessed global outcomes of spelling, primarily across upper-elementary grade levels (e.g., Fuchs et al., 1991a, 1991b) with little work at first grade or below. A question remains regarding which particular words or word types might be used as indicators of writing skill. Two CBM spelling tasks have been developed for students in kindergarten to investigate early spelling skills limited to basic words following consonant–vowel–consonant patterns (c–v–c). One task, Real Word Spelling, requires spelling of real words (e.g., mad) while Nonsense Word Spelling requires spelling of phonetically regular pseudo words (e.g., sud). The spelling tasks have five items for children in kindergarten and include one word with each short vowel sound. Initial study focused on the relation between these tasks and students’ literacy development during the second half of kindergarten (Ritchey, 2006). Students’ responses were scored using correct letter sequences (CLS), an index of the number of correct adjacent letters (White & Haring, 1980). For Real Word Spelling, the reliability coefficients ranged from .84 to .90 (alternate forms and internal consistency). Nonsense Word Spelling yielded similar estimates with a range of .86–.89 for reliability and 92% for interrater agreement. The tasks were related to measures of reading with coefficients ranging from r ¼ .40 to. 50. The correlation between Real Word Spelling and Nonsense Word Spelling was r ¼ .92, suggesting that the tasks are capturing similar aspects of early writing skill and that one task, rather than both, may be sufficient to assess students’ spelling knowledge development. Ritchey et al. (2010) compared scoring procedures for the Real Word Spelling task with a different sample of kindergarten children. CLS was compared to a scoring procedure that involves counting the number of correct sounds, a phonological scoring system developed by Tangel and Blachman (1992), and a correct/incorrect scoring system. All scores were related to reading and writing outcomes with criterion-related validity coefficients around r ¼ .50 with the TEWL Basic Writing subtest (Hresko et al., 1996). All scores were also sensitive to bimonthly growth, with the largest slope estimates for the phonological scoring metric. These findings are tempered by applicability to