The burden of proof: the validity as improvement of instructional practice.

The burden of proof: the validity as improvement of instructional practice. In a quiet but profound way, the work of Samuel Messick Samuel J. Messick III (3 April 1931 – 6 October 1998) was an American psychologist professor whose work at the Educational Testing Service examined construct validity. (1980, 1988,1994) has revolutionized how educational researchers establish claimsfor the validity of assessment procedures. Messick's framework forunderstanding validity was initially applied to the use of intelligenceor achievement tests for placement of students into special education orremedial programs. We believe that his framework has equally exciting,challenging, and powerful implications for improving instructionalpractice for children with disabilities. The purpose of this article isto explore these implications in mathematics assessment and instructionin special education.Recently, three state-of-the-art innovations in special educationmathematics assessment were described in Exceptional Children (Fuchs,Fuchs, & Hamlett, 1994; Gerber, Semmel, & Semmel, 1994; Woodward& Howard, 1994) and the Journal of Special Education Technology(Woodward & Carnine, 1993).All three innovations view assessment as a means to improvemathematics instruction for students with learning disabilities. Yet allthree take different paths to achieving this end. In one case, expertsystems identify errors and develop hypotheses concerning underlyingmisconceptions in subtraction subtraction,fundamental operation of arithmetic; the inverse of addition. If a and b are real numbers (see number), then the number a−b is that number (called the difference) which when added to b (the subtractor) equals (Woodward & Howard, 1994). In asecond, student responses are analyzed and problem types generated whereadditional instruction is required (Fuchs et al., 1994).In the third case, instantaneous prompting and graphical elaborationis provided to prevent errors and to help a student solve a multidigitmultiplication problem (Gerber et al., 1994). In this article, wediscuss these three innovative projects using Messick's (1988)framework as a tool for understanding what really is entailed in claimsmade about "the appropriateness, meaningfulness, andusefulness" (p. 35) of inferences drawn from assessment data. Inanalyzing these three approaches, we discuss not only current thinkingon assessment, but also changing views on how to improve specialeducation math instruction.Messick argued convincingly that validity comprises not onlytechnical and statistical concerns, but also social and instructionalones. It is incumbent on researchers involved in assessment to do thefollowing:* Explain instructional theory Instructional theory is a discipline that focuses on how to structure material for promoting the education of humans, particularly youth. Originating in the United States in the late 1970s, instructional theory underlying assessment.* Explain in detail expected uses of assessment data by teachers.* Present data on how teachers actually use the assessment procedure.* Demonstrate the consequences--both positive and negative, bothintended and unintended--of use on student learning.Whereas commonly used validity indexes (concurrent, predictive,construct) remain interesting and important, they are not sufficientjustification for use. They are what Messick calls the first face ofvalidity, or evidential ev��i��den��tial?adj. LawOf, providing, or constituting evidence: evidential material.ev interpretation of the meaning(s) of assessmentdata; they indicate that the underlying constructs have some theoreticalbasis (see Figure 1, left column). Validity also entails bothpersuasive, logical arguments supporting use, (Figure 1, right column),and empirical data demonstrating the consequences of that assessment,i.e., that it does in fact help guide instruction to enhance studentlearning.A HISTORICAL INTERLUDEFor many years, the key to successful reading instruction forstudents with learning disabilities was believed to lie in anunderstanding of auditory and visual processing Visual processing is the sequence of steps that information takes as it flows from visual sensors to cognitive processing. The sensors may be zoological eyes or they may be cameras or sensor arrays that sense various portions of the electromagnetic spectrum. deficits. Kavale (1981)conducted a comprehensive meta-analysis of extant research and concludedthat a significant correlation existed between auditory processingskills and reading abilities. In Messick's terminology, he providedevidence of construct validity construct validity,n the degree to which an experimentally-determined definition matches the theoretical definition. to support test interpretation.However, Kavale and Mattson (1983) subsequently found that trainingin auditory skills did not improve reading ability. In Messick'sterminology, no evidence existed to support use.More recently, Adams (I 990) noted that a set of variables, labeledphonemic pho��ne��mic?adj.1. Of or relating to phonemes.2. Of or relating to phonemics.3. Serving to distinguish phonemes or distinctive features. or phonological awareness Phonological awareness is the conscious sensitivity to the sound structure of language. It includes the ability to auditorily distinguish parts of speech, such as syllables and phonemes. , were consistently predictive ofsubsequent reading performance. In Messick's framework, there wasan evidential basis for test interpretation. Further, unlike theIllinois Test of Psycholinguistic psy��cho��lin��guis��tics?n. (used with a sing. verb)The study of the influence of psychological factors on the development, use, and interpretation of language. Abilities (ITPA ITPA International Truck Parts AssociationITPA International Tax Planning AssociationITPA Inosine TriphosphataseITPA International Tokamak Physics ActivityITPA Independent Telecommunications Pioneer AssociationITPA Ibm Tivoli Performance Analyzer ) constructs,subsequent research has found that explicit instruction in phonologicalawareness skills such as sound segmentation and blending can lead togrowth in reading. Stanovich (1994) reported, "Instructionalinterventions for preschool children at risk for learning difficultiesthat involve the conscious and explicit teaching of sound segmentation(and spelling sound correspondences) have been found to lead to fasterrates of reading and spelling acquisition" (p. 268). Thus, therewas an evidential basis for test use as well as for test interpretation.And the tests were demonstrated to have consequential validity (i.e.,when the constructs guide practice, they help students learn to readmore rapidly).Fully determining the validity of an assessment process transcendswhat any one researcher can accomplish. It is a task for a community ofresearchers and practitioners (L. Fuchs, personal communication, July19, 1994) to consider meanings and utility of assessment procedures inrelation to current thinking about how to improve instructional practiceand issues raised by studies of implementation.BEYOND SURFACE PERFORMANCE: ASSESSMENTS TO DESCRIBE UNDERLYINGMISCONCEPTIONSTheories and conceptions of how best to teach mathematics to studentswith disabilities are essential foundations for validation ofinstructional assessments. Special education currently is responding toseveral important trends that have emerged from the mathematics reformmovement as exemplified in the National Council of Teachers ofMathematics (NCTM NCTM National Council of Teachers of MathematicsNCTM Nationally Certified Teacher of MusicNCTM North Carolina Transportation MuseumNCTM National Capital Trolley MuseumNCTM Nationally Certified in Therapeutic Massage ) Standards (1989) and in research by Ball (1993);Hiebert et al. (1994); Kaplan, Yamamoto, and Ginsburg (1989) and Lampert(1986). The aim is to teach mathematics for understanding (Prawat,Remillard, Putnam, and Heaton (1992) and to reform mathematicsinstruction so that students learn how to think mathematically--"toexplore, conjecture, and reason logically" (NCTM, 1989, p. 5 [citedin Ball, 1993]). According to according toprep.1. As stated or indicated by; on the authority of: according to historians.2. In keeping with: according to instructions.3. Hiebert et al. (1994), "This means... asking appropriate questions ... resisting the temptation to provideclues that lead students too smoothly to a solution" (p. 20), andconceptualizing all computational problems as problem-solvingexperiences (Hiebert et al., 1994), rather than as a series of tasks tobe performed following explicitly taught rules.Clearly, this thinking is quite divergent from most conventionalthinking in special education. Many students with learning disabilitieshave been shown to have limited verbal expressive abilities, and it isunclear how successful discourse/discussion may be as an instructionalstrategy (Hofmeister, 1993). Nevertheless, T. P. Carpenter (personalcommunication, April 14, 1994) noted that development of reasoningabilities about numerical concepts is a critical focus of instructionfor students with learning disabilities. Increasingly, mathematicalreasoning is viewed as essential for survival in the workplace (Reich,1991), and the validity of our efforts to prepare special educationstudents for the workplace may well depend on the extent to which theypossess in-depth understanding of mathematical principles and are ableto reason mathematically (Woodward & Gersten, 1991).Perhaps the primary impact of the mathematics reform movement hasbeen to emphasize, as the most crucial instructional goal, helpingstudents link their intuitive knowledge of the world of numbers withformal mathematical principles. The most stunning application of thisphilosophy to date remains Lampert's (1986) demonstration thatfourth graders could grasp the associative, distributive, andcommutative com��mu��ta��tive?adj.1. Relating to, involving, or characterized by substitution, interchange, or exchange.2. Independent of order. laws when multidigit multiplication was taught asproblem-solving and when continual attempts were made to link conceptualunderstanding to computation through the systematic use of a range ofalternative representational systems representational systems,n.pl a neurolinguistic programming term for the senses (visual, auditory, olfactory, kinesthetic, and gustatory). .Because of the interest in how each individual student thinksmathematically, some of those who approach mathematics from aconstructivist con��struc��tiv��ism?n.A movement in modern art originating in Moscow in 1920 and characterized by the use of industrial materials such as glass, sheet metal, and plastic to create nonrepresentational, often geometric objects. (Cobb, 1994) vantage point have been interested inassessing and understanding the thinking behind students' errorpatterns. These errant strategies (often referred to as"bugs") "can be considered as clues to thinking processesrather than indicators of lack of learning" (Kaplan et al., 1989,p. 62).The focus on a detailed understanding of the thinking underlyingindividual students' error patterns is reflected in two of theassessment approaches reviewed here--Dynamath (Gerber et al., 1994) andTorus (Woodward & Carnine, 1993; Woodward & Howard, 1994). Bothare in the constructivist tradition, as described by Cobb (1994), whonotes, "Constructivists are typically concerned with the quality ofindividual interpretive activity, with the development of ways ofknowing at a more micro-level" (p. 15, emphasis added). Theconceptual frameworks underlying Dynamath and Torus are the assumptionthat individualized instruction Individualized instruction is a method of instruction in which content, instructional materials, instructional media, and pace of learning are based upon the abilities and interests of each individual learner. should be based on an understanding ofeach student's misconceptions, although they take different tackson how to use this information to enhance instruction.Woodward and Howard (1994) argued that, in many areas of mathematicsperformance, it is important to look beyond "surface"performance to begin to understand directly the source of eachindividual student's misconceptions, in order to improve and reformspecial education math instruction. Gerber et al. (1994) also relied onan analysis of errors, although the information was intended less foruse by teachers than by a computerized "tutor" that wouldstructure subsequent instruction.The results of these approaches for improved assessment can beevaluated and given meaning only in terms of coherent constructs andtheories about how children learn and about the relationship betweenteaching and learning. We present here an analysis of how theseapproaches accomplish this, and we identify fruitful directions forcontinued validation research.VALUE IMPLICATIONS OF ASSESSMENT PROCEDURES THAT FOCUS ON STUDENTMISCONCEPTIONSFor Gerber et al. (1994) and Woodward and Howard (1994), the hope isthat assessment can help teachers transcend limitations of currentpractice. Both have offered detailed expositions of what they view asexpanded alternatives to current theory and practice. They articulatedcomplex, elegant, theoretical models of instruction.Both Gerber et al. (1994) and Woodward and Howard (1994) grappledspecifically with the issue of moving thinking about instruction inspecial education beyond the behavioral/direct instruction traditiontoward a more constructivist approach based, in part, oninformation-processing theories. From somewhat different perspectives,both Torus and Dynamath posit detailed understanding of student errorpatterns and bugs" (faulty algorithms) as a key to understandingwhat to provide in the way of refined, individualized instruction tostudents with learning disabilities. Both incorporate aspects of theinformation-processing paradigm that has been persistent in learningdisabilities research for at least 3 decades.The Dynamath system of dynamic assessment assumes that highlyspecific, ongoing, and instantaneous feedback on the exact instructionalstep or steps is the optimal way to teach. The goal of Dynamath is tomake explicit the precise nature of students' procedural errors andto provide precise prompts or remediation, with a level of detailvirtually impossible for a teacher to emulate. Dynamath is an attempt tomathematically capture the "zone of proximal development Lev Vygotsky's notion of zone of proximal development (зона ближайшего развития), often abbreviated ZPD "(Vygotsky, 1978), through providing an optimal instructional range foreach student that offers the right mix of challenge and success.According to Gerber et al. (1994),Dynamath does not attempt to simulate ahuman teacher's specific or "expert"repertoire of assisting examples, prompts,explanations or behaviors. Rather, it consistsof its own repertoire related to specificcapabilities possessed by computerhardware-orchestration of graphical display,movement, color, sequence, and auditorycues. (p. 116)As errors occur, Dynamath prevents a student from even seeing theincorrect answer, and immediately initiates increasing levels ofprompts, drawing on the student's database over time to dynamicallyreorganize its degree of instructional challenge so as to maximizesuccessful performance.Unlike Gerber et al. (1994), Woodward and Howard (1994) asserted thatefforless learning is not optimal. They made explicit their primaryassumption: "Systematic errors or misconceptions ... are groundedin an active, conscious process of attempting to determine `what to donext'" (p. 127, emphasis added). They saw value in trying tounderstand what goes on in students' minds as they make errors. TheTorus system represents an attempt to use students' performancedata to make the underlying source of errors explicit to teachers.Students' errors are compared to a "library of bugs," acatalogue of common computational errors and likely associatedmisconceptions.EXAMINING TEACHERS' USE OF ASSESSMENT DATA: POTENTIAL RESEARCHQUESTIONSWoodward and Howard (1994) noted that for activities requiringprocedural ("how to") knowledge, such as arithmeticcomputation:Errors in procedural knowledge Procedural knowledge is the knowledge exercised in the performance of some task. See below for the specific meaning of this term in cognitive psychology and intellectual property law. , particularly ifthey are systematic, may not be easilycontained or corrected by immediatefeedback, discrimination practice and othertechniques that behaviorists haverecommended. On the contrary, instructionthat accentuates a mastery of procedures ...may do little to reduce a student's errorpatterns. Misconceptions appear to be resilientto the assumed benefits of "more practice."(pp. 126-127, emphasis added)A particularly appealing aspect of this hypothesis is that it is opento empirical investigation, and the developers have begun some researchin this area.Although Torus makes it possible for teachers to attempt to directlyremediate individual misconceptions, this is not recommended by thedevelopers. On the contrary, they asserted that "the solution forcontrolling misconceptions is in increasing conceptualunderstanding" (p. 133). They envisioned teachers using Torusassessment "infrequently, as part of a more global means ofexamining student understanding" (p. 134). This approach allows forlarge degrees of teacher latitude in using assessment data, consistentwith the developers' belief that an understanding of underlyingprocesses can enhance the quality of instruction.Procedural errors, in their view, should always be considered"more than just incorrect responses" (p. 127). Theirhypothesis is that special education teachers will use an awareness ofmisconceptions underlying error patterns from Torus diagnoses to helpthem provide more informed mathematics instruction. But to establishevidential validity for use of assessment data, they must still addressthe following two questions:* In reality, how do teachers use diagnostic information aboutstudents' misconceptions in their daily teaching of math?* Do these uses have a significant effect on students'understanding of and ability to use mathematics?Concerns have been raised about teachers' abilities to applysubtle assessment data to improve the quality of instruction (Ball,1990; Bereiter & Kurland, 1981-1982; Carnine & Woodward, 1988).In Messick's scheme, it is crucial to study empirically howteachers use Torus data and to examine the impact on both theirday-to-day teaching and their conceptions of mathematics instruction forstudents with learning disabilities (Richardson, 1994). Shifts in eitheror both domains are potentially valuable. And they are researchable(Ball, 1990, 1993; Richardson, 1994).Gerber et al. defined their approach as related to, but distinctfrom, three traditions of assessment that have greatly influencedspecial education research: Vygotskian, applied behavioral, andcognitive-behavioral. Their goal was to incorporate the best features ofeach, "to dynamically create and relate explicit models of studentusers with mild disabilities, the domain of knowledge being assessed ...and forms of instructional assistance" (p. 115), and to explorerelationships among these.DynaMath is less ambiguous about its analysis of errors and the waysin which the resulting data are utilized to improve instruction. Basedon an analysis of errors made in previous performance, problems tailoredto individual learners are generated, and then students are promptedthrough steps in the procedure that have proven problematic for them.The role of the teacher in this instance shifts toward providing aconceptual context within which the computer-directed tutoring fits.Evidential questions of use include the following:1. To what extent does Dynamath free up teachers to work on moreconceptual aspects of mathematics instruction?2. What are students' attitudes and reactions to the detailedlevel of feedback provided?3. What is the impact on students' growth in mathematics?Despite their computational emphasis and early stage of developmentand testing, both Dynamath and Toirus are intended to furtherstudents' conceptual understanding of mathematics. Approaches likeTorus or Dynamath may provide useful research tools for understandingthe extent to which misconceptions truly are resistant to remedialinstruction, and for analyzing how students respond to various types ofinstructional strategies. The hope is that, if implemented beyond theprototype phase, such tools will prove both practical and beneficial.PRAGMATICS pragmaticsIn linguistics and philosophy, the study of the use of natural language in communication; more generally, the study of the relations between languages and their users. : IMPLICATIONS OF A LESS THEORY--DRIVEN APPROACH TOWARDMATH ASSESSMENT AND INSTRUCTIONFuchs et al. (1994) would concur that the major issue is not whetherinstruction should be more conceptually based, but how to accomplishthis objective. However, they have largely sidestepped theoreticalissues:Because we relied on the advice of "experts,"rather than our own solutions to the presentinginstructional problems, we chose not tosuperimpose su��per��im��pose?tr.v. su��per��im��posed, su��per��im��pos��ing, su��per��im��pos��es1. To lay or place (something) on or over something else.2. a theoretical orientation, such ascognitive or behavioral, that would requireaccepting or rejecting potential solutionsbased on the theoretical perspective fromwhich they were derived. In fact, each of ourexperts relied on strategies representingmultiple theoretical orientations. Rather, weelected to adopt a pragmatic philosophicalposition in which there are no ultimateprinciples or self-evident values, but rather inwhich the test of an idea is its capacity tosolve the particular problems it addresses. (p.140)Their stated goals in improving mathematics assessment are (a) toanalyze performance and indicate to the teacher the types of problemswhere additional work is needed and (b) to provide suggestions regardingpossible instructional approaches gathered from expert teachers.The clarity and common sense underlying their straightforwardapproach is not to be underestimated. The impetus behind provision of amenu of alternate instructional strategies for each problem type wastheir earlier finding that when faced with student performance dataindicating that a current instructional approach was not effective,teachers often did not generate workable alternatives on their own.Fuchs et al. subsequently built a database of such alternatives.It is noteworthy that, despite its lack of a theoretical orientationtoward math instruction, use of the Fuchs et al. system led to enhancedstudent outcomes. The use of more varied instructional procedures byteachers, prompted by the expert system, led to significant achievementgains for students. As Fuchs et al. noted, the teachers actuallyimplemented many of the recommendations offered and shifted how theytaught. Thus, there is some evidential basis supporting use of thisassessment.Two other important research questions suggested by application ofMessick's framework are as follows:* What is the relative effectiveness among the suggested alternativeinstructional strategies?* Why do certain approaches work better than others for students withlearning disabilities?Secondary analyses of the data generated by Fuchs et al. (1994) mightwell shed some light on which instructional strategies are trulyeffective in helping students with learning disabilities betterunderstand mathematical concepts. By suggesting alternative algorithms,the expert system may have pushed some teachers toward explanationsabout underlying concepts that are increasingly viewed as important fora students' understanding of mathematics.Because the recommended alternate teaching strategies emanated fromexpert mathematics teachers, it is likely that elements ofcognitive/information processing approaches were present (Leinhardt& Smith, 1985). For example, if alternate representational systemsoffer clearer, richer explanations of concepts such as equality orcommutativity com��mu��ta��tive?adj.1. Relating to, involving, or characterized by substitution, interchange, or exchange.2. Independent of order. , or simply if more time is taken to discuss a strategy,understanding and proficiency in math may increase.TRANSCENDING THE COGNITIVE AND BEHAVIORAL TRADITIONS OF INSTRUCTIONThe apparent polarity between behavioral and cognitive/informationprocessing approaches in thinking about assessment--whether our mainconcerns as teachers should be on carefully utilizing what we seeovertly, or whether assessment can help us see more clearly underlyingproblems that can help us more intelligently present concepts tostudents--has been a recurrent issue as tests have been designed, used,and critiqued in the field of learning disabilities for the past 30years (Salvia salvia:see sage. salviaAny of about 700 species of herbaceous and woody plants that make up the genus Salvia, in the mint family. Some members (e.g., sage) are important as sources of flavouring. & Ysseldyke, 1981). An emerging body of literature hasargued, however, that seemingly strong differences betweenphilosophically incongruent in��con��gru��ent?adj.1. Not congruent.2. Incongruous.in��congru��ence n. positions often dissipate when one focuseson the specific details of instructional design Instructional design is the practice of arranging media (communication technology) and content to help learners and teachers transfer knowledge most effectively. The process consists broadly of determining the current state of learner understanding, defining the end goal of or instructionalinteractions, and directly measures their impact on learning (Dixon& Carnine, 1994; Gersten & Jimenez, 1994; Stanovich, 1993). Forexample, the work of Leinhardt (1990) and Leinhardt and Smith (1985) onexpert mathematics teaching indicates that distinctions betweenconstructivist and behavioral conceptions of math instruction dissolvewhen one details precisely how expert teachers explain concepts, providefeedback, and remediate problems.Lampert's (1986) previously cited research illustrates thispoint. A careful reading of her research reveals that she accomplishedher goal of imparting young students with deep working knowledge ofabstract mathematical principles by following a key principle ofinstructional design validated by behavioral research--restricting therange of examples (Carnine, 1976). She included only multiplication andaddition problems, eliminating those involving subtraction or division,so students could really get the feel of the mathematical principles.Research from divergent traditions seems to support thatmisconceptions often develop because students are taught with weak orinsufficient conceptual frameworks and because teachers explain toolittle, fail to provide conceptual explanations, and fail to build ashared language with the student by providing adequate opportunities topractice, discuss responses, and receive meaningful feedback (Ball,1990; Kelly, Gersten, & Carnine, 1990; Leinhardt & Smith, 1985;Woodward & Gersten, 1991). Ball (1993) noted that true understandingof mathematical concepts entails "building bridges between theexperiences of the child and the knowledge of the expert" (p. 375),and research has only begun to address how teachers can build thesebridges for students with learning disabilities.UNRESOLVED ISSUES AND DIRECTIONS FOR CONTINUED RESEARCHAll three approaches examined here were attempts to--sometimessubtly, sometimes dramatically--reshape the nature of mathematicsinstruction for students with learning disabilities. All three force usto think more about how to teach mathematics to special educationstudents, what types of assessment data to collect, and how to use thosedata.An implicit assumption in all three approaches is that ongoingcollection of certain types of assessment data will serve to improveboth the quality of teaching and student outcomes. This assumption hasbeen the Holy Grail of a line of special education research dating atleast as far back as Lindsley (1964), that recently has been rejuvenated(Hanley, 1994).Second, all three approaches placed at least implicit value onachieving maximal individualization individualization,n the process of tailoring remedies or treatments to cure a set of symptoms in an indiv-idual instead of basing treatment on the common features of the disease. in instruction through improvedassessment procedures. A decade's worth, of observational research(Haynes & Jenkins, 1986; McIntosh, Vaughn, Schumm, Haager & Lee,1993; Woodward & Gersten, 1992) has demonstrated that most specialeducation instruction is not individualized in any but the most cursoryfashion. Since individualization of instruction continues to be acornerstone of educational practice for children with disabilities, aslegally mandated and operationalized within the individualized educationprogram In the United States an Individualized Education Program, commonly referred to as an IEP, is mandated by the Individuals with Disabilities Education Act (IDEA). In Canada an equivalent document is called an Individual Education Plan. (IEP IEPIn currencies, this is the abbreviation for the Irish Punt.Notes:The currency market, also known as the Foreign Exchange market, is the largest financial market in the world, with a daily average volume of over US $1 trillion. ), these applications of assessment are well anchored withrespect to Messick's (1988) consequential validity.Mathematics assessment and instruction are currently undergoing majorconceptual challenges and shifts. Thus evidential and consequentialvalidity questions remain (and beg!) to be asked and answered withrespect to the meanings and uses of "improved" assessments inmath. Do these elegant and exhaustive data collection strategiesactually result in significant improvements in the use of more varied,more effective instructional strategies by teachers? And how much and inwhich ways does student performance improve accordingly?DynaMath sidesteps the question of use of assessment data by having acomputer assume the instructor role, freeing the teacher to work onconceptual contexts. Yet it is ironic that, in many ways, this complex,ambitious attempt to cognitively analyze mathematical understandingappears to resemble programmed instruction programmed instruction,method of presenting new subject matter to students in a graded sequence of controlled steps. Students work through the programmed material by themselves at their own speed and after each step test their comprehension by answering an of 20 years ago. At the sametime, several validity questions endure: Is the neo-Vygotskianremediation model of Dynamath too complex? Does it truly fit howstudents learn? Are certain steps unnecessary?The research of Meyer (1982), for instance, indicated that whenfeedback on one aspect of reading (decoding) is extremely detailed andspecific, the student can be diverted from the actual task athand--reading. When even minor slips due to temporary loss of attentionare provided with detailed prompts, students may become agitated ag��i��tate?v. ag��i��tat��ed, ag��i��tat��ing, ag��i��tatesv.tr.1. To cause to move with violence or sudden force.2. , bored,or embarrassed. Similarly, the extremely detailed feedback DynaMathprovides may be too much for students; it may diagnose and prescribe ata level that is far more microscopic than human processing of the task.The assertion that teacher knowledge of bugs and misconceptions willlead to more informed and effective instruction, in many ways parallelsthe interest in miscues that has often been linked with the wholelanguage movement (Goodman, 1989). Insofar in��so��far?adv.To such an extent.Adv. 1. insofar - to the degree or extent that; "insofar as it can be ascertained, the horse lung is comparable to that of man"; "so far as it is reasonably practical he should practice as the extant researchliterature in reading instruction has any relevance for mathematics,there may be good reason for caution.Despite two decades of often fascinating research on miscues inreading (K. Goodman, 1973; Y. M. Goodman, 1989, 1994; Leu Leuleucine. Leuabbr.leucineLeuleucine. , 1982), littlein the way of valid instructional practice has emerged. The research ofStanovich (1993) explains why. Empirical studies Empirical studies in social sciences are when the research ends are based on evidence and not just theory. This is done to comply with the scientific method that asserts the objective discovery of knowledge based on verifiable facts of evidence. have consistently shownthat proficient readers invariably in��var��i��a��ble?adj.Not changing or subject to change; constant.in��vari��a��bil use phonological pho��nol��o��gy?n. pl. pho��nol��o��gies1. The study of speech sounds in language or a language with reference to their distribution and patterning and to tacit rules governing pronunciation.2. cues, not contextcues. Thus prematurely labeling a beginning reader or a less proficientreader by her miscue mis��cue?n.1. Games A stroke in billiards that misses or just brushes the ball because of a slip of the cue.2. A mistake.intr.v. mis��cued, mis��cu��ing, mis��cues1. pattern as a "context reader" is unlikelyto be helpful. Once the student becomes a proficient reader, she willinvariably lose her pattern of making errors. At least in this case,according to Stanovich, attempts to "enter the student'smind" through miscue analysis Miscue analysis was originally developed by Ken Goodman for the purpose of understanding the reading process. It is a diagnostic tool that helps researchers/teachers gain insight into the reading process. will not be productive.In other words Adv. 1. in other words - otherwise stated; "in other words, we are broke"put differently , it may not be important to understand the exactnature of the "bug" or misconception, as Dynamath and Torusattempt to do, in order to provide more responsive, more effectiveinstruction. It may be that certain--or many--of these misconceptionsare developmental. As students become more proficient with a procedureor grasp the underlying logic, some of the "bugs" maydisappear with sensitive instruction and practice.Are normal developmental characteristics taken too seriously withinDynamath or Torus? (i.e., Are they reified into permanence well beyondtheir developmental nature?) Because both Dynamath and Torus are at muchearlier stages of development than the Fuchs et al. system, they haveyet to broach broach(broch) a fine barbed instrument for dressing a tooth canal or extracting the pulp. broachn.A dental instrument for removing the pulp of a tooth or exploring its canal. this question. Research on interpretation of assessmentdata and the relationship between assessment and changes in studentperformance, paralleling that of Fuchs et al., may be anticipated at alater date. As Messick (1988) makes clear: "Appropriateness,meaningfulness, and usefulness ... (of assessments) ... are inseparable,and the unifying force is empirically grounded constructinterpretation" (p. 35).Increasingly, all discussions of assessment validity will need toinclude a discussion of consequential validity. For example, indiscussing criteria for determining whether assessments linked tocurricula reform movements are valid, Linn linn?n. Scots1. A waterfall.2. A steep ravine.[Scottish Gaelic linne, pool, waterfall.] (1994) recently concludedthat, for innovative assessments, "the evaluation of consequences,both intended and unintended, takes on greater priority. Evidence isalso needed that the uses and interpretations are contributing toenhanced student achievement" (p. 80).As Messick (1988) noted, the burden of proof, ultimately, is not onlyon the developers, but the entire research/practice community. The threeapproaches examined here offer great potential as research tools fordelineating, articulating, and understanding the means for improving thequality of special education math instruction.REFERENCESAdams, M. J. (1990). Beginning to read: Thinking and learning aboutprint. Cambridge, MA: Massachusetts Institute of Technology Press. Ball,D. L. (1990). Reflections and deflections of policy: The case of CarolTurner. Educational Evaluation Educational evaluation is the evaluation process of characterizing and appraising some aspect/s of an educational process.There are two common purposes in educational evaluation which are, at times, in conflict with one another. and Policy Analysis, 12, 247-249. Ball,D. L. (1993). With an eye on the mathematical horizon: Dilemmas ofteaching elementary school elementary school:see school. mathematics. Elementary School Journal Published by the University of Chicago Press, The Elementary School Journal is an academic journal which has served researchers, teacher educators, and practitioners in elementary and middle school education for over one hundred years. ,93(4), 373-397. Bereiter, C., & Kurland, M. (1981-1982). Aconstructive look at Follow Through results. Interchange, 12(1), 1-22.Carnine, D. (1976). Similar sound separation and cumulative introductionin learning letter-sound correspondences. Journal of EducationalResearch, 69(10), 368-372. Carnine, D., & Woodward, J. (1988). Areply to Swanson's "Paradigms Lost": Learningdisabilities and the new ghost in the old machine. Journal of LearningDisabilities, 21(4), 233-243. Cobb, P. (1994). Constructivism constructivism,Russian art movement founded c.1913 by Vladimir Tatlin, related to the movement known as suprematism. After 1916 the brothers Naum Gabo and Antoine Pevsner gave new impetus to Tatlin's art of purely abstract (although politically intended) inmathematics and science education. Educational Researcher, 23(7), 4.Dixon, R., & Carnine, D. (1994). Ideologies, practices and theirimplications for special education. Journal of Special Education, 28(3),356-367. Fuchs, L. S., Fuchs, D., & Hamlett, C. L. (1994).Strengthening the connection between assessment and instructionalplanning with expert systems. Exceptional Children, 61, 138-146. Gerber,M. M., Semmel, D. S., & Semmel, M. I. (1994). Computer-based dynamicassessment of multidigit multiplication. Exceptional Children, 61,114-125. Gersten. R., & Jimenez, R. (1994). A delicate balance:Enhancing literacy instruction for students of English as a secondlanguage. The Reading Teacher, 47(6), 438-449. Goodman, K. (1973).Theoretically based studies of patterns of miscues in oral readingperformance. Detroit: Wayne State University Wayne State University,at Detroit, Mich.; state supported; coeducational; established 1956 as a successor to Wayne Univ. (formed 1934 by a merger of five city colleges). . Educational ResourcesInformation Center, ED 079 708. Goodman, Y. M. (1994, December). Thirtyyears of miscue research. Paper presented at the annual meeting of theNational Reading Conference, San Diego San Diego(săn dēā`gō), city (1990 pop. 1,110,549), seat of San Diego co., S Calif., on San Diego Bay; inc. 1850. San Diego includes the unincorporated communities of La Jolla and Spring Valley. Coronado is across the bay. . Goodman, Y. M. (1989). Roots ofthe whole-language movement. Elementary School Journal, 90(9). 113-127.Hanley. T. V. (1994). The need for technological advances in assessmentrelated to national educational reform. Exceptional Children, 61,222-229. Haynes. M. C., & Jenkins, J. R. (1986). Reading instructionin special education resource rooms. American Educational ResearchJournal, 23(2), 161-190. Hiebert, J., Wearne, D., Carpenter, T. P.,Fennema, E., Fuson, K., Human, P., Olivier, A., & Murray, H. (1994,April). Authentic problem solving problem solvingProcess involved in finding a solution to a problem. Many animals routinely solve problems of locomotion, food finding, and shelter through trial and error. in mathematics. Paper presented at theannual meeting of the American Educational Research Association, NewOrleans New Orleans(ôr`lēənz –lənz, ôrlēnz`), city (2006 pop. 187,525), coextensive with Orleans parish, SE La., between the Mississippi River and Lake Pontchartrain, 107 mi (172 km) by water from the river mouth; founded . Hofmeister, A. M. (1993). Elitism e��lit��ismor ����lit��ism ?n.1. The belief that certain persons or members of certain classes or groups deserve favored treatment by virtue of their perceived superiority, as in intellect, social status, or financial resources. and reform in schoolmathematics. Remedial and Special Education. 14(6), 8-13. Kaplan. R. G.,Yamamoto, T., & Ginsburg, H. P. (1989). Teaching mathematicsconcepts. In L.B. Resnick & L.E. Klopfer (Eds.), Toward the thinkingcurriculum: Current cognitive research (pp. 59-82). Alexandria, VA:Association for Supervision and Curriculum Development The Association for Supervision and Curriculum Development, or ASCD, is a membership-based nonprofit organization founded in 1943. It has more than 175,000 members in 135 countries, including superintendents, supervisors, principals, teachers, professors of education, and . Kavale. K. A.(1981). The relationship between auditory perceptual skills and readingability: A meta-analysis. Journal of Learning Disabilities, 14, 539-546.Kavale, K. A., & Mattson, P. D. (I 983). One jumped off the balancebeam: A meta-analysis of perceptual-motor training. Journal of LearningDisabilities, 16(2). 165-173. Kelly, B., Gersten, R., & Carnine, D.(1990). Student error patterns as a function of curriculum design.Journal of Learning Disabilities, 23(l), 23-32. Lampert, M. (1986).Knowing, doing, and teaching multiplication. Cognition and Instruction,3(4), 305-342, Leinhardt, G. (1990). Towards understanding instructionalexplanations (Tech. Rep. No. CLIP 90-03). Pittsburgh, PA: University ofPittsburgh, Learning Research and Development Center. Leinhardt, G.,& Smith, D. (1985). Expertise in mathematics instruction: Subjectmatter knowledge. Journal of Educational Psychology, 77, 247-271. (ERICDocument Reproduction Service No. ED 334 150) Leu, D. J., Jr. (1982).Oral reading error analysis: A critical review of research andapplication. Reading Research Quarterly, 17, 420-437. Lindsley, O. R.(1964). Direct measurement and prosthesis prosthesis(prŏs`thĭsĭs): see artificial limb. prosthesisArtificial substitute for a missing part of the body, usually an arm or leg. of retarded behavior. Journalof Education, 147, 62-81. Linn, R. L. (1994). Performance assessment:Policy promises and technical measurement standards. EducationalResearcher, 23(9), 4-14. McIntosh, R., Vaughn, S., Schumm, J. S.,Haager, D., & Lee, O. (1993). Observations of students with learningdisabilities in general education classrooms. Exceptional Children, 60,249-26 1. Messick, S. (1980). Test validity and the ethics ofassessment. American Psychologist, 35, 1012-1027. Messick, S. (1988).The once and future issues of validity: Assessing the meaning andconsequences of measurement. In H. Wainer & H. I. Braun (Eds.), Testvalidity (pp. 33-46). Hillsdale, NJ: Lawrence Erlbaum. Messick, S.(1994). The interplay of evidence and consequences in the validation ofperformance assessments. Educational Researcher, 23(2), 13-23. Meyer, L.A. (1982). An experimental study of the treatment effects ofword-analysis and word-supply correction procedures during word attack.Reading Research Quarterly, 4, 544-555. National Council of Teachers ofMathematics. (I 989). Curriculum and evaluation standards for schoolmathematics. Reston, VA: Author. Prawat, R. S., Remillard, J., Putnam,R. T., & Heaton, R. M. (1992). Teaching mathematics forunderstanding: Case studies of four fifth-grade teachers. ElementarySchool Journal, 93(2), 145-152. Reich, R. (1991). The work of nations.New York New York, state, United StatesNew York,Middle Atlantic state of the United States. It is bordered by Vermont, Massachusetts, Connecticut, and the Atlantic Ocean (E), New Jersey and Pennsylvania (S), Lakes Erie and Ontario and the Canadian province of : Alfred A. Knopf. Richardson, V. (1994). Conducting research onpractice. Educational Researcher, 23(5), 5-10. Salvia, J., &Ysseldyke, J. E. (1981). Assessment in special and remedial education.Boston: Houghton-Mifflin. Stanovich, K. E. (1993). Romance and reality.The Reading Teacher, 47(4), 280-291. Stanovich, K. E. (1904).Constructivism in reading education. The Journal of Special Education,28, 259-274. Vygotsky, L. S. (1978). Mind in society: The development ofhigher psychological processes (M. Cole, V. John-Steiner, S. Scribner,& E. Souberman, Eds. and Trans.). Cambridge, MA: Harvard UniversityPress The Harvard University Press is a publishing house, a division of Harvard University, that is highly respected in academic publishing. It was established on January 13, 1913. In 2005, it published 220 new titles. . Woodward, J., & Carnine, D. (1993). Uses of technology formathematics assessment and instruction: Reflection on a decade ofinnovations. Journal of Special Education Technology, 12(1), 38-48.Woodward, J., & Gersten, R. (1991). A conceptual approach tomathematical knowledge: Implications for special education (Tech. Rep.91-3). Eugene, OR: Eugene Research Institute. Woodward, J., &Gersten, R. (1992). Innovative technology for secondary learningdisabled students: A multi-faceted study of implementation. ExceptionalChildren, 58,407-421. Woodward, J., & Howard, L. (1994).Misconceptions of youth. Exceptional Children, 61, 126-137.ABOUT THE AUTHORSRUSSELL GERSTEN (CEC (Central Electronic Complex) The set of hardware that defines a mainframe, which includes the CPU(s), memory, channels, controllers and power supplies included in the box. Some CECs, such as IBM's Multiprise 2000 and 3000, include data storage devices as well. OR Federation), Professor of Special Education,University of Oregon The University of Oregon is a public university located in Eugene, Oregon. The university was founded in 1876, graduating its first class two years later. The University of Oregon is one of 60 members of the Association of American Universities. , Eugene, and Senior Researcher, Eugene ResearchInstitute. THOMAS KEATING (CEC #216), Researcher and Project Director,Eugene Research Institute. LARRY K. IRVIN (CEC #216), Professor ofSpecial Education, University of Oregon, Eugene, Researcher, EugeneResearch Institute and Research Professor, Teaching Research Divisio,Oregon State System of Higher Education.Address all correspondence to Russell Gersten, Eugene ResearchInstitute, 1400 High Street, Suite C, Eugene, OR 97401.We wish to thank Douglas Carnine, Sharon Vaughn, Susan Brengelman,and Scott Baker for their thoughtful feedback on earlier versions ofthis article; Lynn Fuchs and Charlie Greenwood for their support of thiseffort; John Woodward for his suggestions; and Matthew Cranor for hisexpert editorial assistance.Manuscript received March 1994; revision accepted January 1995.